In Amazon CTO Werner Vogels’ technology predictions for 2024, he envisions a near future where generative AI becomes more culturally aware. This involves training large language models (LLMs), a type of AI algorithm, on a diverse range of data, leading to more nuanced and accurate results. The goal is to make generative AI more accessible and useful to users worldwide.
However, LLMs rely on data from the internet, which is mostly associated with high-resource languages. This term typically refers to languages, such as English, that have an abundance of linguistic resources available for natural language processing (NLP) tasks.
It is critical that organisations gain the ability to easily customise their LLMs with local data in their native languages to foster social inclusion, stimulate economic growth by opening up new markets, and create improved citizen experiences.
Culture influences everything. Recognising this, AI Singapore (AISG), a national program launched by Singapore’s National Research Foundation to enhance the country’s AI capabilities, is making its LLMs more culturally-accurate, localised, and tailored to Southeast Asia.
SEA-LION Family—a first for the region
Building on AWS’s scalable compute infrastructure, AISG developed SEA-LION, a family of LLMs that is specifically pre-trained and instruct-tuned (a powerful fine-tuning method that allows for greater control over LLM behavior) for Southeast Asian languages and cultures.
A first for the region, SEA-LION also serves as the foundation for Singapore’s National Multimodal LLM Program, contributing to the island’s capabilities in AI research and innovation. This initiative aligns with the National AI Strategy 2.0 that outlines plans to deepen the use of AI in Singapore.
The model will focus on more commonly used languages in Southeast Asia, including Bahasa Indonesia, Bahasa Melayu, Thai, and Vietnamese, and will eventually be extended to include other Southeast Asian languages like Burmese and Lao.
Building regionally represented LLMs also requires rich, hyper-local data in relevant languages. An example of local language nuances is the term “LOL” (an abbreviation of “laughing out loud” in English). In Thailand, people commonly use “55555”, while Indonesians often use “wkwkwk”. LLMs trained on culturally diverse training data, like SEA-LION, enhances the ability of generative AI applications to grasp nuanced aspects of human experiences and navigate complex societal challenges.
Accelerating hyper-local Generative AI in Southeast Asia
SEA-LION will be available on Amazon SageMaker JumpStart by February 2024. The platform provides pre-trained, publicly available models to help customers around the world get started with machine learning (ML).
The relatively small 3-billion and 7-billion parameter variants of SEA-LION that have been released so far were trained using Amazon EC2, a service that provides scalable compute capacity in the cloud.
These smaller variants are more flexible and accessible than commonly used LLMs in the market today, which typically have hundreds of billions of parameters.
"Language and culture-specific LLMs like AISG’s SEA-LION will enable smoother cross-cultural communication and understanding, preserve cultural nuances, and help governments and businesses better serve citizens and customers in Southeast Asia."
By the end of January 2024, AISG will launch a commercial version of instruction tuning parameters for SEA-LION, which will have greater capacity to capture nuances in Southeast Asian languages, improve contextual understanding, enhance multilingual reasoning, and generate context-rich outputs.
"AISG's focus on underrepresented languages throughout Southeast Asia is significant. As the majority of our customers at Tokopedia interact with us in Bahasa Indonesia, developing models with that local fluency will enhance our ability to connect with customers and improve their experiences," said Paul Condylis, associate vice president of Data Science at Tokopedia. "We're in the early stages of experimenting with SEA-LION and the progress is promising."
“As technology rapidly evolves in Asia-Pacific, we see an exciting opportunity to tailor language models to reflect the region’s rich linguistic diversity and revolutionise the way information is processed and shared. We are pleased to work closely with AISG on the SEA-LION LLM project that leverages AWS resources. This project helps to bridge the digital divide by improving cross-cultural communication and providing enhanced localised experiences. Furthermore, businesses and governments can tap into NCS’ deep experience in addressing business needs through the use of customised LLM-powered applications and harness the full potential of the SEA-LION family of LLMs,” said Ying Shaowei, chief scientist, NCS.
Cost-effective ML and compute
Building, training, and deploying an LLM requires time, significant compute resources, and expertise. AISG is working with AWS to address these challenges by harnessing the cloud’s powerful performance for tasks like ML training and high-performance computing using NVIDIA A100 Tensor Core GPUs, which deliver top-notch throughput and fast, responsive networking.
The compact size of SEA-LION also makes it relatively more cost-effective and efficient than larger LLMs, which can have hundreds of times more parameters. Smaller LLMs allow developers to deploy faster, are cheaper to fine-tune, and perform more quickly during inference. The ease of deploying smaller LLMs on mobile devices, or at the edge, also helps businesses more easily adopt and create applications.
“Building an LLM requires reliable cloud infrastructure that is readily available exactly when needed, and AWS is instrumental in helping us scale cost-effectively. We built a 3 billion parameter LLM in just three months with AWS, and we have since scaled the model to 7 billion parameters, extending its reach to more audiences,” said Dr Leslie Teo, senior director of AI Products at AISG.
“By working with AWS, we can focus solely on training our models instead of managing infrastructure. This accelerates the development of unique LLMs that reflect our region’s diversity,” added Teo.
Elsie Tan, country manager of Worldwide Public Sector in Singapore at AWS, said: “Language and culture-specific LLMs like AISG’s SEA-LION will enable smoother cross-cultural communication and understanding, preserve cultural nuances, and help governments and businesses better serve citizens and customers in Southeast Asia.
“We’re excited to help AISG unlock the potential of generative AI to deliver significant benefits to ASEAN’s diverse economies. We look forward to helping AISG extend the reach of its LLMs with our extensive network of customers and partners in the region.”
Start building generative AI applications for your industry with AWS today.
