Papers by Carsten Eickhoff
Parameter-efficient Modularised Bias Mitigation via AdapterFusion (2023.eacl-main)
Copied to clipboard
Deepak Kumar, Oleg Lesota, George Zerveas, Daniel Cohen, Carsten Eickhoff, Markus Schedl, Navid Rekabsaz
| Challenge: | Large pre-trained language models contain societal biases and carry along these biase . Current approaches to mitigate these bias impose debiasing by updating model parameters, effectively transferring model to irreversible debiased state. |
| Approach: | They propose to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand while keeping the core model untouched. |
| Outcome: | The proposed approach improves or maintains effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance while granting parameter-efficiency and easy switching between the original and debiased models. |
CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Contextual document embedding reranking is an efficient and efficient retrieval framework. |
| Approach: | They propose a highly efficient retrieval framework that uses contextual document embedding reranking to incorporate ranking context into training. |
| Outcome: | The proposed framework reduces the computational overhead of a first-stage method and can be used as stand-alone retrieval models. |
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models (LLMs) exhibit factual inconsistencies across languages . authors identify two primary sources of error: insufficient engagement of reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer. |
| Approach: | They propose two vector interventions to redirect the model toward better internal paths for higher factual consistency. |
| Outcome: | The proposed interventions increase the recall accuracy by over 35 percent for the lowest-performing language. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Enhancing Retrieval-Augmented Generation: A Study of Best Practices (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented generation systems have shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses. |
| Approach: | They propose to integrate query expansion, various novel retrieval strategies, and a Contrastive In-Context Learning RAG to improve response quality. |
| Outcome: | The proposed RAGs incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. |
UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages. (2026.findings-acl)
Copied to clipboard
Tassallah Abdullahi, Macton Mgonzo, Mardiyyah Oduwole, Paul Okewunmi, Abraham Toluwase Owodunni, Ritambhara Singh, Carsten Eickhoff
| Challenge: | Current guardian models are predominantly Western-centric and optimized for high-resource languages . low-resourced African languages are vulnerable to evolving harms, cross-lingual failures, cultural misalignment . |
| Approach: | They propose a policy-based safety benchmark for African languages built from adversarial queries authored by 155 domain experts across sensitive fields. |
| Outcome: | The proposed model overestimates multilingual safety, cross-lingual transfer provides partial but insufficient coverage, and dynamic models struggle to localize African-language contexts. |
Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts (2025.emnlp-main)
Copied to clipboard
Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh, Carsten Eickhoff
| Challenge: | Multimodal Large Language Models perform well on visual question answering tasks, but it remains unclear whether their reasoning relies more on memorized world knowledge or on visual information present in the input image. |
| Approach: | They propose a dataset of visual-realistic counterfactuals that put world knowledge priors into conflict with visual input. |
| Outcome: | The proposed dataset puts world knowledge priors into conflict with visual input . it shows that model predictions shift toward visual evidence in mid-to-late layers . |
SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for state tracking are limited and state changes are less densely distributed over utterances. |
| Approach: | They propose to turn to simplified, fully observable systems that show some of these properties. |
| Outcome: | The proposed system shows that state changes occur infrequently while messages are "chatter" it allows for rich descriptions of state while avoiding the complexities of other settings. |
Interpretability Analysis of Arithmetic In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) solve arithmetic with only a few in-context examples, yet the computations that connect those examples to the answer remain opaque. |
| Approach: | They propose to use in-context examples to illustrate how large language models process ICEs to isolate partial-sum representations in three-operand tasks and investigate their influence on final logits. |
| Outcome: | The proposed model performs better than previous models on three-operand tasks. |
CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Cross-lingual summarization (CLS) has attracted increasing interest due to the availability of large-scale web-mined datasets and the advancements of multilingual language models. |
| Approach: | They propose a dataset of cross-lingual code-switched summaries in Chinese and English . they show that leveraging existing CLS resources does not improve performance . |
| Outcome: | The proposed method does not improve on CroCoSum, indicating the limited generalizability of existing approaches. |
Self-Supervised Neural Topic Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Topic models are useful tools for analyzing and interpreting the main underlying themes of large corpora of text. |
| Approach: | They propose a self-supervised neural topic model that learns a topic representation jointly from three co-occurring words and a document that the triple originates from. |
| Outcome: | The proposed model outperforms existing topic models in coherence metrics and document clustering accuracy. |
Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to training with binary relevance labels uses synthetic data . contrastive learning with binary correlations leaves out subtle nuances useful for ranking . |
| Approach: | They propose to use waterstein distance as a loss function for training transformer-based retrievers with graduated relevance labels instead of real documents. |
| Outcome: | The proposed method outperforms conventional training with InfoNCE by a large margin on MARCO and BEIR benchmarks without using real documents. |
NEWTS: A Corpus for News Topic-Focused Summarization (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarking corpora provide concordant pairs of full and abridged versions of Web, news or professional content. |
| Approach: | They propose a topical summarization corpus called NEWTS that is annotated via crowd-sourcing. |
| Outcome: | The proposed model can condition summaries on a desired range of themes . the proposed model outperforms Lead-3 baselines on most benchmark datasets . |
Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (2026.acl-long)
Copied to clipboard
William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Carsten Eickhoff, Ritambhara Singh, Kyle Mahowald
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
MATCHA: Matching Text via Contrastive Semantic Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | MATCHA is an automatic metric that rewards semantic agreement with a reference and penalizes contradictions. |
| Approach: | They introduce a metric that jointly rewards semantic agreement with a reference and penalizes contradictions. |
| Outcome: | The proposed metric outperforms popular metrics on eight public benchmarks compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. |
SIMSUM: Document-level Text Simplification via Simultaneous Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Document-level text simplification is a specific type of simplification which involves simplifying documents consisting of several sentences by rewriting them into fewer or more sentences. |
| Approach: | They propose a new two-stage framework SIMSUM for automated document-level text simplification which uses explicit summarization and simplification models and guides the generation using the main keywords of a source text. |
| Outcome: | The proposed model outperforms baseline models on two document-level simplification datasets, namely D-Wikipedia and Wiki-Doc. |
Outlier Dimensions Encode Task Specific Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that fine-tuning outlier dimensions is detrimental to the representational quality of embeddings. |
| Approach: | They investigate how fine-tuning impacts outlier dimensions by testing their hypothesis that a single outlier dimension can complete downstream tasks with a minimal error rate. |
| Outcome: | The proposed model can encode crucial task-specific knowledge and the value of a representation in a single outlier dimension drives downstream model decisions. |
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind (2025.findings-acl)
Copied to clipboard
William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, Ritambhara Singh
| Challenge: | Multimodal Large Language Models struggle with visual reasoning, despite strong performance on vision-language tasks. |
| Approach: | They propose a visually cued chain-of-thought prompting that enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams. |
| Outcome: | The proposed model improves GPT-4o's accuracy on an irregular polygon side-counting task from 7% to 93%. |
Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces (2020.emnlp-main)
Copied to clipboard
| Challenge: | popular pretrained models encode both denotation and connotation as one entangled representation . a researcher using a pretrained representation can confuse words with connotations . |
| Approach: | They propose a nerual netowrk that decomposes a pretrained representation as independent denotation and connotation representations. |
| Outcome: | The proposed model improves document rankings by comparing denotation and connotation representations with extrinsic representations. |
Text Simplification via Adaptive Teaching (2024.findings-acl)
Copied to clipboard
| Challenge: | Text simplification is the process of rewriting a text using simpler vocabulary and grammatical structure in order to make it more accessible and understandable for a larger audience. |
| Approach: | They propose a model for text simplification based on adaptive teaching using a teacher network and a text generation network. |
| Outcome: | The proposed model outperforms the current state-of-the-art model on the Wiki-Doc and D-Wikipedia datasets and performs well on human evaluations in terms of text simplicity, correctness, and fluency. |
Enhancing the Ranking Context of Dense Retrieval through Reciprocal Nearest Neighbors (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sparse annotation poses persistent challenges to training dense retrieval models . despite potential future endeavors to extend annotation, issue of false negatives persists . |
| Approach: | They propose a method that smooths out the annotation of unlabeled relevant documents . they use reciprocal nearest neighbors to estimate relevance and rerank candidates . |
| Outcome: | The proposed method reduces the issue of false negatives in contrastive learning by reducing sparsity. |
Pretraining on Interactions for Learning Grounded Affordance Representations (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing studies of affordances have not integrated into formal semantics. |
| Approach: | They propose to integrate 3D objects' trajectories into a neural network to predict their traversories. |
| Outcome: | The proposed model outperforms 2D computer vision models and is more accurate than expected. |
Beyond Multiple Choice: Evaluating Steering Vectors for Summarization (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent methods for controlling language models can often be classified into three main strategies: prompt engineering, trainable decoding mechanisms, fine-tuning according to specific objectives. |
| Approach: | They evaluate steering vectors for controlling topical focus, sentiment, toxicity, and readability in abstractive summaries across the SAMSum, NEWTS, and arXiv datasets. |
| Outcome: | The proposed method is effective in free-form generation, but high steering strengths induce degenerate repetition and factual hallucinations. |
Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25 (2025.emnlp-main)
Copied to clipboard
| Challenge: | Interpretability in information retrieval (IR) models is coarse-grained and poorly understood . a cross-encoder model extracts traditional relevance signals, such as term frequency and inverse document frequency . |
| Approach: | They analyze how a common IR model extracts traditional relevance signals . this is similar to the probabilistic ranking function BM25 . |
| Outcome: | The proposed model extracts traditional relevance signals in early-to-middle layers, similar to BM25 . the model then combine these concepts in later layers, laying the groundwork for future interventions . |
Language Models Implement Simple Word2Vec-style Vector Arithmetic (2024.naacl-long)
Copied to clipboard
| Challenge: | a primary criticism of language models is their inscrutability. |
| Approach: | They propose to use a vector arithmetic style mechanism to solve relational tasks . they find that this mechanism is specific to tasks that require retrieval from pretraining memory . |
| Outcome: | The proposed model reduces to a simple additive update for a variety of tasks . the findings contribute to proving that the models are interpretable and reliable . |
IsoScore: Measuring the Uniformity of Embedding Space Utilization (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies suggest that contextualized word embedding models do not isotropically project tokens into vector space. |
| Approach: | They propose to use a tool to measure isotropy to quantify the degree to which a point cloud uniformly utilizes the ambient vector space. |
| Outcome: | The proposed tool is the only available tool that accurately measures how uniformly distributed variance is across dimensions in vector space. |