Kuwain: Arabic Language Injection Boosts English LLM Without Retraining

This is a Plain English Papers summary of a research paper called Kuwain: Arabic Language Injection Boosts English LLM Without Retraining. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Breaking New Ground: Adding Arabic to English-Only Models

Most large language models focus primarily on English, making them less effective for languages with distinct characteristics like Arabic. This English-centric nature creates a significant barrier for users of non-English languages, particularly those with different linguistic structures and writing systems.

Kuwain 1.5B offers a novel solution for adding Arabic capabilities to an existing English language model without compromising its original abilities. Rather than retraining the entire model from scratch—a computationally expensive approach—the researchers developed a method that strategically adds new layers to a pre-trained model while expanding its vocabulary.

This approach stands out because it:

• Preserves the model's original English knowledge
• Adds robust Arabic language capabilities
• Requires significantly fewer resources than training a new model
• Achieves competitive performance despite a small model size

Model performance vs size visualization: Points represent models on the Arabic NLP benchmark, where point size indicates efficiency (average score divided by model size). Larger points show better performance-to-size ratio.

The visualization above demonstrates how Kuwain compares to other models despite its small size, showing remarkable efficiency in the performance-to-size ratio. This innovative approach addresses a critical gap in language technology, providing a pathway for efficient multilingual model development that could extend to other underrepresented languages.

The State of Multilingual and Arabic-Focused LLMs

Approaches to Creating Multilingual Models

Training multilingual models from scratch requires enormous computational resources and data. The complexity of optimizing randomly initialized parameters demands extensive training data to guide effective performance. Several approaches have emerged to make this process more efficient:

1. Continuous pre-training extends an existing model's pre-training on new language data, reducing costs compared to starting from scratch. This approach has been used for German (LeoLM-7B), Japanese (Swallow-MS-7b), and Greek (Meltemi-7B).

2. Vocabulary expansion adapts a model's tokenizer to accommodate new languages, often critical for languages with different writing systems.

3. Mixture-of-Experts (MoE) techniques train separate language components on distinct multilingual subsets, reducing interference between languages.

However, these approaches face common challenges. Models trained with continuous pre-training often perform worse on original language tasks—LeoLM-7B showed a -2.6% decline on English benchmarks, while Swallow-MS-7b saw a -5.3% drop. This phenomenon, known as catastrophic forgetting, remains a significant obstacle in multilingual model development.

The Landscape of Arabic Language Models

Recent years have seen several notable Arabic-focused language models, each with distinct approaches and limitations:

Jais is an open-source bilingual model trained from scratch for Arabic and English tasks. While innovative, it suffers from significant imbalance—it was exposed to substantially more English data than Arabic, with much of the Arabic content translated from English sources. This creates outputs that often reflect English cultural contexts rather than authentic Arabic cultural nuances.

AceGPT adapts the English-focused LLaMA2 model through additional pre-training and tuning. However, it retains LLaMA2's original tokenizer, which processes Arabic at the character level, leading to inefficient representation of Arabic morphology and vocabulary.

ArabianGPT, built on the GPT-2-small architecture, features an advanced tokenizer optimized for Arabic linguistics. Its exclusive focus on Arabic limits its ability to handle multilingual tasks or leverage knowledge transfer from English.

ALLaM enhances Arabic language processing by leveraging a mixture of Arabic and English data, incorporating vocabulary expansion and continued pre-training. While effective, it requires extensive computational resources.

These models share common limitations: high training costs, degradation in original language performance, and inadequate tokenization strategies—particularly problematic for morphologically rich languages like Arabic.

Kuwain addresses these limitations with a lightweight, cost-efficient alternative that extends an existing English-centric model while preserving its original capabilities. This approach makes Kuwain uniquely positioned as a practical solution for multilingual expansion in resource-constrained environments.

Building a Balanced Bilingual Dataset

Sourcing High-Quality Arabic and English Data

The Kuwain training dataset comprises 110 billion tokens, with a strategic split of 90 billion tokens in Arabic and 20 billion in English. This composition reflects the model's focus on Arabic capabilities while maintaining sufficient English representation to preserve original knowledge.

The data comes from publicly available open-source collections:

• CulturaX
• C4
• ArabicText 2022 (including ArabicWeb22-A, ArabicWeb16, OSCAR, ArabicWeb22-B, CC100-AR, and Arabic Tweets)

Beyond Modern Standard Arabic, the dataset incorporates dialectal Arabic from Hugging Face repositories, preserving the linguistic richness and diversity of regional Arabic varieties. This inclusion is crucial for developing a model that can handle the full spectrum of Arabic usage.

To maintain evaluation integrity, the researchers ensured none of the evaluation datasets appeared in the training data, preventing data leakage that could artificially inflate performance metrics.

Rigorous Arabic Text Cleaning Process

Arabic text presents unique challenges that require specialized cleaning techniques. The researchers implemented a comprehensive set of processing steps:

• Removing corrupted characters and repetitions
• Stripping markup and elongation characters
• Normalizing white space
• Preserving Quranic symbols and special characters
• Normalizing encoding inconsistencies
• Unifying orthographic variants (different forms of Arabic letters)
• Filtering malformed examples and short text

This cleaning process was released as a customizable open-source script to support reproducibility and future Arabic text pre-processing applications. For English data, the researchers adopted the filtering pipeline from the BLOOM project, which promotes clean, diverse content.

The meticulous attention to data quality, particularly for Arabic, creates a foundation for effective model training that respects the language's unique characteristics while maintaining consistency and clarity.

A Two-Pronged Approach: Layer Extension and Vocabulary Expansion

Strategic Layer Addition: Building Upon Existing Knowledge

Inspired by Llama-Pro, the researchers developed a layer extension approach that adds new capabilities while preserving the original model's knowledge. This technique involves inserting new layers into an existing model architecture rather than retraining the entire model.

Models decoder architecture showing the typical structure of decoder layers in foundation models.

The technical foundation of this approach builds on the standard decoder block structure used in popular models like Llama, Mistral, Phi-3, and Gemma-2. These models share similar decoder architectures with minor variations, consisting of multi-head self-attention (MHSA), feed-forward networks (FFN), and normalization layers.

To extend the model, the researchers introduced additional identity blocks that initially preserve the model's original behavior. These identity blocks are achieved by initializing the additional weight matrices to zero, ensuring they don't disrupt existing functionalities. During training, only these newly added layers and the final layer of the original model are updated, keeping most of the original parameters frozen.

This strategic approach:

• Preserves the base model's knowledge
• Reduces training time and computational requirements
• Allows for targeted learning of new language features
• Minimizes the risk of catastrophic forgetting

Arabic-Optimized Tokenization: Speaking a New Language

Extending a tokenizer's vocabulary is critical when adapting a model to a new language, especially when the original tokenizer isn't well-suited for that language. Many open-source models like LLaMA-2, Phi-3, and Mistral include only 28 Arabic tokens—just enough to cover the Arabic alphabet. This limited representation fails to capture Arabic's morphological richness and lexical diversity.

Vocabulary expansion by adding new Arabic tokens to the tokenizer.

Extension of model layers, where newly added layers were trainable while the original layers remained frozen.

The researchers trained a specialized tokenizer using SentencePiece with their Arabic training data, resulting in 26,000 additional tokens optimized for Arabic. This expanded vocabulary was integrated with the base model's tokenizer, creating a final vocabulary of 54,000 tokens.

This vocabulary expansion improves tokenization efficiency, enabling the model to represent Arabic more effectively. The benefits extend beyond performance improvements to include:

• Shorter training time due to more efficient token representation
• Increased effective context length for better handling of Arabic text
• Better capture of Arabic morphological patterns and word formations
• Reduced computational resources needed for training

The combination of strategic layer extension and vocabulary expansion forms the technical core of Kuwain's approach, enabling efficient adaptation to Arabic while preserving English capabilities.

Experimental Design: Finding the Optimal Configuration

Finding the Perfect Layer Arrangement

The researchers conducted extensive experiments to determine both the ideal number and optimal placement of additional layers. They explored configurations ranging from 6 to 10 layers, running each experiment for 20,000 training steps on a subset of the dataset.

Training loss chart showing instability when freezing the last layer

Training loss chart showing stable training when enabling the last layer and avoiding consecutive layers

Training loss chart showing degraded performance when stacking new layers consecutively

Key findings from these experiments revealed several architectural principles:

1. Avoid consecutive layer insertion - Placing new layers back-to-back led to training instability
2. Keep the final encoder layer trainable - This proved essential for stable training outcomes
3. Distribute new layers strategically - The optimal placement didn't follow a regular pattern or equal intervals
4. Eight new layers proved optimal - While configurations with 8-10 layers performed similarly, 8 layers (a 30% increase in model size) delivered the best results

These experiments highlight the importance of architectural choices in model adaptation. As shown in the loss charts, freezing the last layer led to unstable training (chart a), while enabling the last layer and avoiding consecutive layer placement produced stable training dynamics (chart b). Stacking multiple new layers consecutively significantly worsened performance (chart c).

The distributed 8-layer configuration emerged as the most effective architecture, balancing added capacity with training stability and efficient knowledge transfer.

Tokenizer Optimization: Balancing Vocabulary Size and Expansion Ratio

The researchers evaluated their proposed tokenizer against two prominent Arabic language models: AraBERT and Jais. This analysis focused on two critical metrics:

1. Vocabulary Size: The number of unique tokens in the tokenizer's vocabulary, affecting its ability to represent linguistic features
2. Expansion Ratio: The ratio of tokenized text length to original text length, indicating how efficiently the tokenizer processes text

Comparison of tokenization metrics across different Arabic language models. The vocab size used presents the Arabic vocab for each model, where the full vocab has a different number.

An ideal tokenizer balances a reasonable expansion ratio with a manageable vocabulary size. While Jais achieves the best expansion ratio at 2.19, the Kuwain tokenizer demonstrates superior efficiency when considering the tradeoff between this metric and vocabulary size. With just 26K tokens, Kuwain achieves an expansion ratio of 2.30—very close to Jais's performance despite using 41% fewer tokens.

This analysis, based on tokenizing over one million examples, suggests that Kuwain's tokenizer achieves effective text representation without an expansive vocabulary, leading to more efficient model training and deployment.

The Minimum English Data Requirement

A crucial question in adapting models to new languages is how much original language data is needed to maintain performance. The researchers discovered that Kuwain requires only 20% of the original English data to maintain performance on English tasks—significantly less than the 50% ratio used in comparable approaches.

Performance comparison on English benchmarks between TinyLlama and two variants of Kuwain: trained with 20% English data (Kuwain) and with less than 20% English data (Kuwain-lt-$\phi$). Results show how English performance varies with data proportion.

The results show that reducing English data below 20% (as in Kuwain-lt-φ) caused a notable decline in English benchmark performance, with the average score dropping from 52.99 to 49.56. In contrast, training with 20% English data maintained performance comparable to—and in some cases slightly better than—the original TinyLlama model.

This finding has significant implications for efficient multilingual model development: a minimal amount of original language data serves as an "anchor" that helps the model retain existing knowledge while efficiently learning new skills.

Traditional Methods vs. Layer Extension: A Clear Comparison

To evaluate the impact of their layer extension approach, the researchers created a baseline model (Kuwain-Naive) trained using conventional continued pre-training. This model used the same Arabic dataset and vocabulary expansion but without layer extension—the entire model was trained rather than just additional layers.

English evaluation of TinyLlama vs. Kuwain-Naive (standard pre-training).

The results revealed a stark contrast: while Kuwain-Naive effectively learned Arabic, it suffered catastrophic forgetting of English knowledge. Its average performance on English benchmarks dropped significantly from 52.99 to 46.85.

In contrast, the layer extension approach (Kuwain) maintained strong English performance while achieving Arabic capabilities comparable to Kuwain-Naive: