As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models are limited in their ability to train for English, which is a problem for many other languages. |
| Approach: | They propose to adapt existing generative language models to new languages by retraining lexical embeddings without tuning the Transformer layers. |
| Outcome: | The proposed method achieves lexical embeddings for Italian and Dutch that are aligned with the original English lexicals. |
Similar Papers
On the Multilingual Capabilities of Very Large-Scale English Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Generative Pre-trained Transformers (GPTs) have been scaled to unprecedented sizes in the history of machine learning. |
| Approach: | They investigate the potential and limits of Generative Pre-trained Transformers in three tasks . they find it can be almost as useful for many languages as it is for English . |
| Outcome: | The proposed model can perform tasks in five different languages, and its potential is explored . it can learn from a few examples "via text interaction" and is scalable to many languages . |
GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages (2024.lrec-main)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, Magnus Sahlgren
| Challenge: | a growing interest in building and applying large language models for languages other than English is fueling interest in developing LLMs for smaller languages. |
| Approach: | They describe the development process for the first native large generative language model for the North Germanic languages, GPT-SW3. |
| Outcome: | The proposed model is based on the generative language model for the North Germanic languages . it is a first-generation model with a high-quality data set and a low cost of implementation . |
A Simple Recipe for Multilingual Grammatical Error Correction (2021.acl-short)
Copied to clipboard
| Challenge: | Modern approaches view the task of Grammatical Error Correction (GEC) as monolingual text-to-text rewriting and employ encoderdecoder neural architectures. |
| Approach: | They propose a language-agnostic method to generate a large number of synthetic examples and use large-scale multilingual language models to train state-of-the-art GEC models. |
| Outcome: | The proposed method surpasses state-of-the-art results on GEC benchmarks in English, Czech, German and Russian. |
Tokenizer-Aware Cross-Lingual Adaptation of Decoder-Only LLMs through Embedding Relearning and Swapping (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been primarily focused on English, leaving the multilingual ability unexplored. |
| Approach: | They propose a technique that creates new tokenizers and tunes embeddings on fixed model weights for target language adaptation. |
| Outcome: | The proposed method is light-weight and performant but has limitations for older models and high resource languages. |
LexFit: Lexical Fine-Tuning of Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based language models implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. |
| Approach: | They propose to expose and enrich lexical knowledge from transformer-based language models to serve as effective decontextualized word encoders even when fed input words "in isolation" |
| Outcome: | The proposed model outperforms standard static WEs and vanilla LMs in lexical tasks over four established tasks in 8 languages. |
Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing multilingual machine translation models need to be upgraded as data becomes available in more languages. |
| Approach: | They propose three techniques that speed up the effective learning of new languages and alleviate catastrophic forgetting . |
| Outcome: | The proposed techniques exceed the performance of a same-sized baseline model with 30% computation and recover the performance a larger model trained from scratch with over 50% reduction in computation. |
Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish (2022.lrec-1)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren
| Challenge: | a prerequisite for building large-scale generative models for other languages is access to large amounts of high-quality text data and powerful computational resources. |
| Approach: | They present a 3.5 billion parameter autoregressive language model, trained on a 100 GB Swedish corpus. |
| Outcome: | The proposed model performs well on a 100 GB Swedish corpus and is competent in comparison with existing models of similar size. |
Synthetic Data in the Era of Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | 'synthetic data' is a data generated with the assistance of large language models to make dataset construction faster and cheaper. |
| Approach: | This tutorial seeks to build a shared understanding of recent progress in synthetic data generation from NLP and related fields by grouping and describing major methods, applications, and open problems. |
| Outcome: | This tutorial will describe methods, applications, and open problems that have been developed and are being used to improve the quality and efficiency of synthetic data generation. |
Unsupervised Lexical Simplification with Context Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing unsupervised lexical simplification methods only use monolingual data and pre-trained models. |
| Approach: | They propose an unsupervised method that generates substitutes based on monolingual data and pre-trained language models. |
| Outcome: | The proposed method outperforms existing models on the TSAR-2022 task in English, Portuguese, and Spanish. |
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. |
| Approach: | They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples. |
| Outcome: | The proposed method outperforms existing methods on diverse classification tasks. |