Papers by Noah Smith
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for translation of ambiguous text use context to disambiguate meaning . current models for MTs consistently translate English idioms literally, whereas LMs are context-aware . |
| Approach: | They use a dataset of 512 pairs of English sentences to study semantic ambiguities . they use literal and figurative idioms to disambiguate intended meaning . |
| Outcome: | The results show that current models translate English idioms literally, even when the context suggests a figurative interpretation. |
Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
Challenges in Automated Debiasing for Toxic Language Detection (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing toxic language data are limited in their ability to prevent biased behavior in toxic language detection systems. |
| Approach: | They propose to debiase toxic language detection models using lexical and dialectal markers using synthetic labels instead of traditional methods. |
| Outcome: | The proposed method reduces dialectal associations with toxicity despite the use of synthetic labels . |
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)
Copied to clipboard
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith
| Challenge: | Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance. |
| Approach: | They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example. |
| Outcome: | The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. |
Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how novel language models generate training text . large LMs and constrained decoding strategies both decrease novelty . |
| Approach: | They develop a novel search tool inspired by genomic data to find n-grams in training data. |
| Outcome: | The proposed tool can search for n-grams over a corpus in constant time w.r.t. large LMs and more constrained decoding strategies both decrease novelty. |
Set the Clock: Temporal Alignment of Pretrained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. |
| Approach: | They construct a time-sensitive question dataset and use it to examine temporal alignment methods to align their internal knowledge to a target time. |
| Outcome: | The proposed methods improve LLaMa2's performance by 62% if they are fine tuned to the year 2022 . |
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024.emnlp-main)
Copied to clipboard
Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah Smith, Luke Zettlemoyer
| Challenge: | Multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. |
| Approach: | They propose Cross-lingual Expert Language Models (X-ELM) which mitigates inter-language competition by independently training language models on subsets of the multilingual corpus. |
| Outcome: | The proposed model outperforms jointly trained multilingual models across all 16 considered languages and transfer the gains to downstream tasks. |
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov
| Challenge: | Language models have evolved from being research prototypes to commercialized products offered as web APIs. |
| Approach: | They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages. |
| Outcome: | The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results. |
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
Time is Encoded in the Weights of Finetuned Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Time vectors are used to customize language models to new time periods. |
| Approach: | They propose a tool to customize language models to new time periods by using time vectors . they show that time is encoded in the weight space of finetuned models . |
| Outcome: | The proposed tool improves performance on text from a time period without training. |
Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects (2024.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Adelani, Daud Abolade, Noah Smith, Yulia Tsvetkov
| Challenge: | Recent efforts to develop NLP tools for low-resource languages focus on their standard dialects. |
| Approach: | They propose a high-quality parallel text and speech corpus for Yoruba . they use native speakers to collect data from four regional yoruba dialects . |
| Outcome: | The proposed dataset shows that dialect-adaptive finetuning can narrow performance disparities . the dataset will be released publicly under an open license . |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the advances of language models, they still produce text that contains trivial commonsense errors. |
| Approach: | They propose a general-purpose commonsense statement verification model that learns to estimate the plausibility of declarative statements based on commonsensical knowledge. |
| Outcome: | The proposed model outperforms existing models that can be repurposed for commonsense verification, even including GPT-3.5/ChatGPT/GPT-4. |
Demystifying Prompts in Language Models via Perplexity Estimation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models can be prompted to perform a wide variety of tasks with zero- and few-shot learning. |
| Approach: | They propose a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
| Outcome: | The proposed method extends a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |