Papers by Benjamin Heinzerling
BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages (L18-1)
Copied to clipboard
| Challenge: | In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively . pre-trained subword embeddings for BPE units are commonly available . |
| Approach: | They present a collection of pre-trained subword embeddings in 275 languages . they use fine-grained entity typing as testbed to evaluate BPEmb . |
| Outcome: | The proposed method performs better than other methods, but requires less resources and no tokenization. |
Can LMs Store and Retrieve 1-to-N Relational Knowledge? (2023.acl-srw)
Copied to clipboard
| Challenge: | Pretraining language models on large amounts of text has made it difficult to store and retrieve world knowledge. |
| Approach: | They propose to view pretrained language models as knowledge bases by examining their ability to store and retrieve world knowledge. |
| Outcome: | The proposed model can store and retrieve world knowledge with high accuracy, but it is not clear how accurately it can handle 1-to-N relational knowledge. |
Monotonic Representation of Numeric Attributes in Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902, but how this information is encoded in the model’s internal representations is not understood well. |
| Approach: | They propose a method for finding and editing representations of numeric properties such as Karl Popper’s birth year. |
| Outcome: | The proposed method can express an increasingly late birthyear by patching activations along a “birthyear” direction. |
When Choosing Plausible Alternatives, Clever Hans can be Clever (D19-60)
Copied to clipboard
| Challenge: | Pretrained language models have shown large improvements in the commonsense reasoning benchmark COPA, but recent work has identified superficial cues in benchmark datasets which are predictive of the correct answer. |
| Approach: | They propose an extension of COPA that does not suffer from easy-to-exploit single token cues and exploits them. |
| Outcome: | The proposed extension of COPA does not suffer from easy-to-exploit single token cues. |
Prompting for explanations improves Adversarial NLI. Is this true? {Yes} it is {true} because {it weakens superficial cues} (2023.findings-eacl)
Copied to clipboard
| Challenge: | Explanation prompts are used to generate an explanation for a given input . they are also used to improve model performance on adversarial benchmarks . |
| Approach: | They propose to use explanation prompts to generate an explanation that supports a label . they argue that prompting for explanations weakens superficial cues . |
| Outcome: | The proposed explanation prompts outperform label-only prompts on adversarial benchmarks. |
COPA-SSE: Semi-structured Explanations for Commonsense Reasoning (2022.lrec-1)
Copied to clipboard
| Challenge: | Semi-structured explanations for Choice of Plausible Alternatives (COPA-SSE) are a crowdsourced dataset of 9,747 common sense explanations . |
| Approach: | They propose a semi-structured approach to explain Choice of Plausible Alternatives questions using a crowdsourced dataset of 9,747 common sense explanations with ConceptNet relations but freely written concepts. |
| Outcome: | The proposed explanations are geared towards commonsense reasoners operating on knowledge graphs and serve as a starting point for improving such systems. |
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing evidence for the stages-of-inference hypothesis is that early layers of language models map their subword tokenized input to more meaningful representations that form the model’s “inner vocabulary”. |
| Approach: | They propose an analytical decomposition of first-layer attention in language models that quantifies the relative contributions of position-related, token-related and mixed effects. |
| Outcome: | The proposed analysis yields interpretable terms that quantify the relative contributions of position-related, token-related and mixed effects. |
Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing bi-encoder architectures do not allow any sharing between text and knowledge graphs . john sutter: experimental results show that enabling full interaction yields strong improvements. |
| Approach: | They propose cross-stitch bi-encoders that allow full interaction between text and KG . they say the amount of sharing is dynamically controlled via cross-attention-based gates . |
| Outcome: | Experimental results show that bi-encoder architectures yield strong improvements . cross-stitch mechanism allows sharing and updating representations between two encoders . |
Test-time Augmentation for Factual Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve factual probing are relation-specific and do not generalize to unseen relation types. |
| Approach: | They propose to use test-time augmentation to augment and ensemble prompts at test time to reduce sensitivity to prompt variations. |
| Outcome: | The proposed method improves model confidence, but for other models, it leads to degradation. |
Representational Analysis of Binding in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has shown that LMs use a concept called Binding ID (BI) to mark entity-attribute pairs, but have not captured the information from entity activations. |
| Approach: | They propose to localize the Binding ID mechanism by localizing BI information in LMs by encoding it in a low-rank subspace. |
| Outcome: | The proposed model can infer attributes for a given entity from a container . |
Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles (2025.acl-long)
Copied to clipboard
Munachiso S Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka, Benjamin Heinzerling, Velibor Bojkovic, Hilal AlQuabeh, Martin Takáč, Kentaro Inui
| Challenge: | Existing models that use sequential data can bypass the limitations of unidirectional causality. |
| Approach: | They propose a mechanism that enables large language models to bypass unidirectional causality . they propose 'cycle tokens' that enable recall of preceding tokens from succeeding ones . |
| Outcome: | The proposed model bypasses the limitations of unidirectional causality by enabling recall of preceding contexts. |
The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have focused on simple factual recall, but we have not explored how this is used in more complex queries. |
| Approach: | They propose to identify low-dimensional subspaces which encode numerical attributes associated with entities in comparison prompts. |
| Outcome: | The proposed model can answer numeric comparison questions using a low-dimensional subspace of theembedding space. |
How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders (2025.findings-emnlp)
Copied to clipboard
Tatsuro Inaba, Go Kamoda, Kentaro Inui, Masaru Isonuma, Yusuke Miyao, Yohei Oseki, Yu Takagi, Benjamin Heinzerling
| Challenge: | Using sparse autoencoders, we explore how bilingual language models develop complex internal representations. |
| Approach: | They employ sparse autoencoders to analyze bilingual language models' internal representations. |
| Outcome: | The proposed method integrates decomposed representations from a fully trained model into a mid-training model. |
Cell-Based Representation of Relational Binding in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work has found evidence that Large Language Models (LLMs) are able to track entities across discourse . however, the mechanism by which they bind entities, relations, and attributes remains unclear . |
| Approach: | They propose a low-dimensional cell-based binding representation for relational binding . they also show that context-specific CBR representations are related by translation vectors . |
| Outcome: | The proposed model encodes a low-dimensional cell-based binding representation (CBR) a translation vector in activation space enables cross-context transfer, the study shows . |
Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation (P19-1)
Copied to clipboard
| Challenge: | Pretrained contextual and non-contextual subword embeddings are available in over 250 languages, allowing massively multilingual NLP. |
| Approach: | They compare pretrained contextual and non-contextual subword embeddings with a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. |
| Outcome: | The proposed method outperforms non-contextual embeddings on multilingual named entity recognition and part-of-speech tagging. |
On Entity Identification in Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shed light on the internal mechanisms of language models that can recall factual knowledge composed of entities and relations. |
| Approach: | They propose a framework analogous to clustering quality metrics to analyze the correspondence between entities and their mentions. |
| Outcome: | The proposed framework is analogous to clustering quality metrics. |
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2021.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have been suggested as an alternative or complement to structured knowledge bases . however, this paradigm has only been considered in a very limited setting . |
| Approach: | They propose a paradigm that allows LMs to store a large number of entities . they propose LM-as-KB paradigm which allows querying stored facts . |
| Outcome: | The proposed paradigm allows handling 21k entities whose name is found in common LM vocabularies . the proposed paradigm has only been considered in a very limited setting . |
Learning to Learn to be Right for the Right Reasons (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work shows that models trained on held-out data perform poorly on hard instances . previous methods have resorted to manual methods of encouraging models not to overfit to superficial cues . |
| Approach: | They propose to explicitly learn a model that does well on both easy and hard tests . they use Choice of Plausible Alternatives and Commonsense Explanation to evaluate the model . |
| Outcome: | The proposed model performs well on easy and hard tests with superficial cues but performs poorly on hard ones without superficial cuings. |