| Challenge: | Currently, open-source large language models are limited to tasks involving the English language. |
| Approach: | They propose to use QLoRA to train a Romanian-adapted LLM with 7 billion parameters and quantized to 4 bits to improve model's performance. |
| Outcome: | The proposed model outperforms the other LLMs on four out of the seven tasks investigated using zero-shot prompting. |
Similar Papers
“Vorbești Românește?” A Recipe to Train Powerful Romanian LLMs with English Instructions (2024.findings-emnlp)
Copied to clipboard
Mihai Masala, Denis Ilie-Ablachim, Alexandru Dima, Dragos Georgian Corlatescu, Miruna-Andreea Zavelca, Ovio Olaru, Simina-Maria Terian, Andrei Terian, Marius Leordeanu, Horia Velicu, Marius Popescu, Mihai Dascalu, Traian Rebedea
| Challenge: | Large Language Models (LLMs) have achieved almost human-like performance on various tasks. |
| Approach: | They are the first to collect and translate a large collection of texts, instructions, and benchmarks and train, evaluate and release open-source LLMs tailored for Romanian. |
| Outcome: | The proposed model trains, evaluates and releases open-source models tailored for Romanian. |
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with capturing long-distance dependencies within sequences to deeply understand semantics. |
| Approach: | They propose a system that captures relevant information within a fixed window size and provides precise answers to queries. |
| Outcome: | The proposed system can read Harry Potter within 30s and accurately answer the questions. |
TableLlama: Towards Open Large Generalist Models for Tables (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for interpreting, augmenting, and querying semi-structured tables require pretraining on tables or special model architecture design. |
| Approach: | They construct a dataset with a variety of tables and tasks for instruction tuning and evaluating LLMs. |
| Outcome: | The proposed model achieves comparable or better performance on 7 out of 8 in-domain tasks compared with the base model on 6 out-of-domain datasets. |
Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Multilingual large language models are expensive to pretrain and suffer from imbalances across languages and datasets. |
| Approach: | They propose a family of Indian language-only autoregressive language models trained on open-source language-specific data for the five most spoken Indian languages. |
| Outcome: | The proposed model outperforms most larger models up to 8B across all five languages. |
LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Recent advances in AI can be attributed to the remarkable performance of Large Language Models (LLMs) success of LLMs depends on specific training techniques, such as instruction tuning and prompting . |
| Approach: | They explore the capabilities of Large Language Models (LLMs) in various tasks and languages . they also examine their performance, fine-tuning, instructions tuning, and close vs. open models . |
| Outcome: | The proposed model can be used for speech and multimodal tasks across modalities, languages, and dialects. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a low-bit training solution that requires substantial training resources. |
| Approach: | They propose an algorithm that reduces memory consumption by low-bit representations with minimal accuracy loss. |
| Outcome: | EfficientQAT achieves 2-bit Llama-2-70B model on single GPU in 41 hours . compared to previous methods, it obtains model with less than 3 points accuracy degradation . |
Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan (2021.findings-acl)
Copied to clipboard
Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, Marta Villegas
| Challenge: | Multilingual language models have been a crucial breakthrough for under-resourced languages . however, the superiority of language-specific models has already been proven for underresourced ones . |
| Approach: | They propose to build a monolingual monolingual model that is comparable to state-of-the-art large multilingual models. |
| Outcome: | The proposed model consistently outperforms state-of-the-art models across tasks and settings. |
Can Post-Training Quantization Benefit from an Additional QLoRA Integration? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large language models require considerable computing resources, which can be costly and often unavailable. |
| Approach: | They propose to integrate 4-bit Post-training Quantization with QLoRA to address these issues . they demonstrate that the integration outperforms standard quantization and fine-tuning . |
| Outcome: | The proposed integration outperforms standard PTQ and 16-bit full-parameter fine-tuning on LLMs. |
Language Adaptation of Large Language Models: An Empirical Study on LLaMA2 (2025.coling-main)
Copied to clipboard
| Challenge: | Popularity of Large Language Models (LLMs) has seen a skyrocketing increase in recent years. |
| Approach: | They present a systematic review of the language adaptation process for Large Language Models including vocabulary expansion, continued pre-training, and instruction fine-tuning. |
| Outcome: | The proposed model is based on empirical studies conducted on LLaMA2 and discussions on various settings affecting the model's capabilities. |