Papers by Ben Bergen
Emergent Inabilities? Inverse Scaling Over the Course of Pretraining (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has found that increased number of model parameters and increased size of the training dataset positively influence model performance. |
| Approach: | They investigate whether language models' performance on specific tasks can decrease over the course of training. |
| Outcome: | The proposed model size-based scaling is found on 8 tasks on which Pythia 12B shows decreased performance over the course of training. |
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract grammatical knowledge is key to linguistic generalization in humans . strong evidence for grammatikal abstraction in humans comes from structural priming . |
| Approach: | They compare human models of crosslingual structural priming to human models . they find evidence for abstract monolingual and crosslingual grammatical representations . |
| Outcome: | The results show that grammatical representations in multilingual models are similar to humans . the strongest evidence for grammatikal abstraction in humans comes from structural priming . |
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Crosslingual transfer is crucial to contemporary language models’ multilingual capabilities, but how it occurs is not well understood. |
| Approach: | They use structural priming to study grammatical representations in humans by controlling for training data quantity and language exposure. |
| Outcome: | The proposed model is able to learn a language in two languages and has a higher likelihood of learning a prepositional object (PO) dative sentence than a double object (DO) . |
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual language models are widely used to extend NLP systems to low-resource languages. |
| Approach: | They pre-train over 10,000 monolingual and multilingual language models for over 250 languages including multiple language families that are under-studied in NLP. |
| Outcome: | The results show that adding multilingual data improves low-resource language modeling performance, similar to increasing low-source dataset sizes by up to 33%. |
Are explicit belief representations necessary? A comparison between Large Language Models and Bayesian probabilistic models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have indirect pragmatic capabilities, but their performance on Theory of Mind tasks is mixed. |
| Approach: | They propose to use Bayesian probabilistic model to make inferences about others' beliefs to predict human belief inference. |
| Outcome: | The proposed model outperforms the Rational Speech Act (RSA) framework in predicting human belief inferences, even though it does not explicitly encode belief representations. |
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shown that language models can select the most likely or plausible of a set of possible events, but they are far from robust. |
| Approach: | They focus on whether language models can select the most likely or plausible of a set of possibilities and compare them to a broader behavior that humans exhibit largely unconsciously. |
| Outcome: | The proposed models perform worse than expected under certain conditions, compared with Llama 3, Gemma 2, and Mistral NeMo, and they are significantly more sensible than leaves. |
Does GPT-4 pass the Turing test? (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) such as GPT-4 seem well designed for Turing's game. |
| Approach: | They evaluated GPT-4 in a public online Turing test and found it passed in 49.7% of games. |
| Outcome: | The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%) participants’ decisions were based mainly on linguistic style (35%) and socioemotional traits (27%). |