Papers with BLOOM
Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024.eacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are statistical models trained to assign probability to human-generated text. |
| Approach: | They evaluate language models' ability to reproduce variability that humans exhibit in the ‘next word prediction’ task. |
| Outcome: | The language models are trained to assign probability to human-generated text . they exhibit low calibration to human uncertainty, and advise against it . |
The ROOTS Search Tool: Data Transparency for LLMs (2023.acl-demo)
Copied to clipboard
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, Anna Rogers
| Challenge: | a 1.6TB multilingual text corpus is currently the largest language model . large language models are ubiquitous in modern NLP, used directly to generate text and as building blocks in downstream applications. |
| Approach: | They propose a search engine for the 1.6TB multilingual ROOTS corpus offering both fuzzy and exact search capabilities. |
| Outcome: | The ROOTS Search Tool is an open-source search engine for the 1.6TB multilingual ROOTs corpus. |
What Language Model to Train if You Have One Million GPU Hours? (2022.findings-emnlp)
Copied to clipboard
Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy
| Challenge: | Recent years have seen the advent of large language models characterized by emergent capabilities arising from sheer scale alone. |
| Approach: | They propose to use a multilingual model to compare performance to the English-only model by ablation at the billion-parameter scale. |
| Outcome: | The proposed model is based on a multilingual model and its performance against the English-only model. |
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024.naacl-long)
Copied to clipboard
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi
| Challenge: | Recent advances in few-shot generalization in natural language processing focus on English. |
| Approach: | They propose a benchmark that unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
| Outcome: | The proposed framework unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023.emnlp-main)
Copied to clipboard
| Challenge: | asymmetric outliers in transformer language models are a challenge for post-training quantization . we propose a framework for outlier suppression that can be seamlessly migrated into subsequent modules . |
| Approach: | They propose a framework for post-training quantization that includes the channel-wise shifting and scaling for concentration. |
| Outcome: | The proposed framework can be migrated into subsequent modules while maintaining equivalence. |
How Well Can Large Language Models Reflect? A Human Evaluation of LLM-generated Reflections for Motivational Interviewing Dialogues (2025.coling-main)
Copied to clipboard
Erkan Basar, Xin Sun, Iris Hendrickx, Jan de Wit, Tibor Bosse, Gert-Jan De Bruijn, Jos A. Bosch, Emiel Krahmer
| Challenge: | Motivational Interviewing (MI) is a counseling technique that promotes behavioral change through reflective responses to mirror or refine client statements. |
| Approach: | They assess the potential of Large Language Models (LLMs) to generate MI reflections via three LLMs: GPT-4, Llama-2, and BLOOM. |
| Outcome: | The proposed models generate meaningful reflections comparable to human therapists, but significant challenges remain. |
Word-level Cross-lingual Structure in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional performance across a broad spectrum of cross-lingual Natural Language Processing (NLP) tasks. |
| Approach: | They propose to use Word-level Cross-lingual Structure to prove that the word-level embedding on the hidden layers isomorphic between languages. |
| Outcome: | The proposed method significantly improves on two representative LLM foundations, LLaMA2 and BLOOM. |
Cross-lingual Editing in Multilingual Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models editing techniques (METs) can efficiently update outdated LLMs without retraining. |
| Approach: | They propose a cross-lingual model editing paradigm where a fact is edited in one language and the subsequent update propagation is observed across other languages. |
| Outcome: | The proposed techniques perform well in multilingual models with knowledge stored in multiple languages. |
Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing and potential applications of open-ended text generation are farreaching, spanning domains such as QA, story generation, open-end dialogue, and ChatGPT 1 . |
| Approach: | They propose a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models by a set of structural and stylistic prompts. |
| Outcome: | The proposed method can be generalized to other large models like BLOOM and OPT. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual generative language models (LMs) are fluent in a large variety of languages. |
| Approach: | They analyze formality distributions of XGLM and BLOOM’s predictions in 5 languages and classify 1,200 generations per language as formal, informal, or incohesive. |
| Outcome: | The proposed models generate a significant amount of informal predictions even when prompted with formal text. |
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM (2025.coling-main)
Copied to clipboard
| Challenge: | Many large language models (LLMs) support many languages, while others only support a few, e.g. the Llama series. |
| Approach: | They present a case study on BLOOM to understand three pertinent factors affecting performance: the number of languages, language exposure, and similarity between training and test languages. |
| Outcome: | The proposed model can be used to perform multilingual tasks on 1 to 52 languages. |
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work aimed to improve task performance of large language models by rewriting or tuning them manually, but manual rewrite is time-consuming and requires subjective interpretation. |
| Approach: | They propose a gradient-free, edit-based search approach for improving task instructions for large language models. |
| Outcome: | The proposed approach outperforms manual rewriting and purely example-based prompts while allowing for API-based tuning. |
DecoMT: Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows the power of few-shot prompting with large language models for tasks like machine translation, summarization, and question answering. |
| Approach: | They propose a few-shot prompting approach that decomposes the translation process into word chunks. |
| Outcome: | The proposed approach outperforms established few-shot prompting models with 8 chrF++ scores across languages. |
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024.acl-long)
Copied to clipboard
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
| Challenge: | Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent. |
| Approach: | They propose a novel detection method to pinpoint language-specific neurons within LLMs by selectively activating or deactivating these neurons. |
| Outcome: | The proposed method can “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
Evaluating the Factual Consistency of Large Language Models Through News Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs generally assign a higher score to factually consistent summaries than to factualally inconsistent summary. |
| Approach: | They propose a benchmark to measure whether large language models prefer factually consistent continuations of inputs. |
| Outcome: | The proposed benchmark compares the scores an LLM assigns to a factually consistent versus a inconsistent summary for an input news article. |
On the Reliability of Large Language Models for Causal Discovery (2025.acl-long)
Copied to clipboard
| Challenge: | Existing statistical methods to identify causal relationships from observational data remain elusive. |
| Approach: | They examine the impact of memorization for accurate causal relation prediction, the influence of incorrect causal relations in pre-training data and the contextual nuances that influence LLMs’ understanding of causal relations. |
| Outcome: | The proposed models are effective in recognizing causal relations that occur frequently in pre-training data, but their ability to generalize to new or rare causal relations is limited. |
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent multilingual models support limited number of human languages due to lack of training data for low resource languages. |
| Approach: | They propose a multilingual multilingual LLM that scales to 100 languages . they use a human feedback dataset and a data set to perform multilingual instruction tuning . |
| Outcome: | The proposed model outperforms its peers on five multilingual benchmarks. |
FLOR: On the Effectiveness of Language Adaptation (2024.lrec-main)
Copied to clipboard
Severino Da Dalt, Joan Llop, Irene Baucells, Marc Pamies, Yishi Xu, Aitor Gonzalez-Agirre, Marta Villegas
| Challenge: | Large language models have amply proven their capabilities, but low- and mid-resource languages do not have access to the necessary means to train such models from scratch. |
| Approach: | They use a 26B tokens corpus to further pre-train BLOOM, giving rise to FLOR models. |
| Outcome: | The proposed model achieves consistent gains across Catalan and Spanish tasks. |
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)
Copied to clipboard
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina
| Challenge: | Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance . |
| Approach: | They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. |
| Outcome: | The proposed model can be extended to other languages without incurring prohibitively large costs. |
Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies suggest that large language models can transfer skills learned in one language to others, but internal mechanisms behind this ability remain unclear. |
| Approach: | They find that LLMs map semantically identical inputs from different languages into a common semantic latent space that allows for consistent processing across languages. |
| Outcome: | The findings highlight the structural evolution of multilingual models during training and scaling up. |
Probing the Emergence of Cross-lingual Alignment during LLM Training (2024.findings-acl)
Copied to clipboard
| Challenge: | Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. |
| Approach: | They propose that LLMs can align languages without explicit supervision from parallel sentences without a single linguistic feature. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer even when the vocabularies of two languages have a null intersection, i.e., no tokens are shared. |
Preference Tuning For Toxicity Mitigation Generalizes Across Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their increasing global use. |
| Approach: | They propose to use English preference tuning to study cross-lingual detoxification of LLMs. |
| Outcome: | The proposed method reduces toxicity in multilingual LLMs by reducing the probability of mGPT-1.3B generating toxic continuations across 17 languages. |
Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved great success in question answering, but their robustness is insufficient to support their practical applications. |
| Approach: | They propose a method which regularizes the model's output and an efficient side block to reduce its inference time. |
| Outcome: | The proposed method achieves comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8 to 4.4 speedup compared to previous methods. |
TUBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning (2025.findings-acl)
Copied to clipboard
Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin I. P. Rubinstein, Trevor Cohn
| Challenge: | Despite the increasing support for multilingual capabilities, the impact of backdoor attacks on LLMs remains under-explored. |
| Approach: | They propose to use poisoned instructiontuning data to attack multilingual LLMs . their results show that more powerful models show increased susceptibility to transferable cross-lingual backdoor attacks . |
| Outcome: | The proposed attack is effective in models like BLOOM and GPT-4o with high success rates in more than 7 out of 12 languages. |
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for cross-lingual semantic parsing are not able to perform tasks on a wide range of datasets. |
| Approach: | They propose a benchmark for cross-lingual semantic parsing that uses 22 natural languages and 8 meaning representations to translate queries into MRs. |
| Outcome: | The proposed benchmarks cover 22 natural languages and 8 meaning representations on 164 domains and 5 tasks covering a wide range of multilingual language models. |
Zero-Shot Multi-Label Topic Inference with Sentence Encoders and LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we focus on Zero-shot approaches for inferring topics from documents where both the document and topics were never seen by a model previously. |
| Approach: | They propose to use Sentence Encoders and Large Language Models to perform a "definition-wild zero-shot topic inference" where users define or provide topics of interest in real-time. |
| Outcome: | The proposed methods outperform ChatGPT-3.5 and PaLM and Sentence-BERT on the definition-wild zero-shot topic inference task on seven datasets. |
PIVOINE: Instruction Tuning for Open-world Entity Profiling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for information extraction focus on a closed-world setting, but PIVOINE is a promising solution to tackle the open-world problem of entity profiling. |
| Approach: | They propose to develop an LLM that performs Open-world Entity Profiling with instruction tuning to extract desirable entity profiles . they construct INSTRUCTOPENWIKI, a substantial instruction-tuning dataset for Open-World Entity Profiles . |
| Outcome: | The proposed model outperforms existing methods and ChatGPT-based baselines on unseen and out-of-ontology cases. |
Pre-Trained Language Models Represent Some Geographic Populations Better than Others (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on measuring the degree to which pre-trained language models capture purely linguistic knowledge and reasoning abilities and world knowledge. |
| Approach: | They use geography to demarcate different populations around the world and comparable corpora to measure how well two families of LLMs perform across these different populations. |
| Outcome: | The results show that pre-trained models perform better for some populations than others. |
Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large language models have been poorly studied . a recent study showed that non-expert users can jailbreak LLMs by manipulating their prompts . |
| Approach: | They propose a formalism and a taxonomy of known (and possible) jailbreaks . they propose generating a dataset of model outputs across 3700 jailbreak prompts a 'prompt' attack is a new attack popularly categorized as "prompting injection attacks" |
| Outcome: | The proposed model exploits 3700 jailbreak prompts over 4 tasks to analyze their effectiveness . authors show that the model can learn to perform a new task on unseen examples . |