Papers by Yonatan Belinkov
Copied to clipboard
| Challenge: | Recent work shows that transformer-based deep NLP models are over-parameterized and do not require all the representational power lent by the rich architectural choices during inference. |
| Approach: | They define a notion of Redundancy and propose a feature-based transfer learning procedure which maintains 97% performance while using at-most 10% of the original neurons. |
| Outcome: | The proposed model maintains 97% performance while using 10% of the original neurons. |
Copied to clipboard
| Challenge: | Recent research suggests large language models encode meta-information about their own outputs. |
| Approach: | They investigate whether large language models possess similar privileged knowledge about answer correctness . they train correctness classifiers on question representations from a model’s hidden states and external models . |
| Outcome: | The proposed model outperforms peer-model models in factual knowledge tasks, but shows no advantage in math reasoning. |
Copied to clipboard
| Challenge: | Existing approaches to mitigate social biases require explicit annotation of demographic information for each sample. |
| Approach: | They propose a method that leverages predefined demographic texts and incorporates a regularization term during the fine-tuning process to mitigate bias in language models. |
| Outcome: | The proposed method outperforms debiasing methods with limited demographic-annotated data. |
Copied to clipboard
| Challenge: | Recent work has explored sparse autoencoders (SAEs) to perform precise interventions on monosemantic features, but most SAE-based methods operate at inference time, which does not create persistent changes in the model’s parameters. |
| Approach: | They propose a parameter-efficient method for persistent concept unlearning using SAEs that automatically identifies salient SAE features across multiple layers and suppresses their activations. |
| Outcome: | The proposed method outperforms previous methods on safety-critical unlearning tasks from the WMDP benchmark, successfully removing harmful knowledge while preserving general and in-domain capabilities. |
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Copied to clipboard
| Challenge: | Language models (LMs) produce a chain of thought (CoT) when prompted to think step-by-step, but it is unclear whether the reasoning encoded in the CoT is faithful to the models’ parametric beliefs. |
| Approach: | They propose a framework for measuring parametric faithfulness of generated reasoning by unlearning reasoning steps (FUR) they propose to erase information contained in reasoning steps from model parameters and measure faithfulness as the resulting effect on the model’s prediction. |
| Outcome: | The proposed framework erases information contained in reasoning steps from model parameters and measures faithfulness as the resulting effect on the model’s prediction. |
Copied to clipboard
| Challenge: | Current approaches to address this issue involve costly dataset scrubbing or model filtering through unlearning and model editing. |
| Approach: | They propose a method for unlearning sensitive information from language models . they curate email and URL datasets and a social security number dataset . |
| Outcome: | The proposed method shows superior performance and robustness to extraction attacks on real-world datasets. |
Copied to clipboard
| Challenge: | Recent work has shown that contextualized word representations are a viable alternative to simple word prediction tasks. |
| Approach: | They propose to use subword units and characters to model morphology, syntax, and semantics instead of word embeddings. |
| Outcome: | The proposed representations are better for modeling syntax and more robust to noisy input. |
Copied to clipboard
| Challenge: | Modern optimizers provide a spectacular array of tweaks to stabilize training trajectories and accelerate Stochastic Gradient Descent (SGD). |
| Approach: | They propose a fast-forward approach to accelerate large segments of SGD training . they alternate between Adam SGD for burn-in and accelerating by line search . |
| Outcome: | The proposed approach speeds up training without compromising model performance. |
Copied to clipboard
| Challenge: | a tutorial aims to introduce the nascent field of interpretability and analysis of neural networks in NLP . |
| Approach: | This tutorial will introduce the nascent field of interpretability and analysis of neural networks in NLP. |
| Outcome: | This tutorial will introduce the nascent field of interpretability and analysis of neural networks in NLP. |
Copied to clipboard
| Challenge: | Increasing the ability of large language models to perform latent multihop reasoning is crucial for reducing the cost and deployment challenges. |
| Approach: | They propose an interpretability method that traces how logits propagate across layers and positions toward the final prediction. |
| Outcome: | The proposed method improves accuracy on five reasoning datasets. |
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Approach: | They propose an unsupervised approach to learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Outcome: | The proposed approach matches the performance of existing supervised methods by identifying features with low output scores and identifying them with input and output scores. |
Copied to clipboard
| Challenge: | Text-to-image diffusion models use a latent text prompt to guide image generation . however, the process by which the encoder produces the text representation is unknown . |
| Approach: | They propose a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. |
| Outcome: | The proposed method provides valuable insights into the text encoder component in T2I pipelines. |
Copied to clipboard
| Challenge: | Dense retrieval models based on text representations have proven very effective, but when applied off-the-shelf they often experience a severe drop in performance. |
| Approach: | They propose to interpret the vector representations produced by dual encoders by projecting them into the model’s vocabulary space. |
| Outcome: | The proposed model significantly improves on the BEIR benchmark and in zero-shot settings. |
Copied to clipboard
| Challenge: | Existing studies of gender bias in NLP focus on extrinsic or intrinsic bias, but the relationship between extrindic and intrinsic bias is relatively unknown. |
| Approach: | They propose a framework to measure extrinsic and intrinsic bias together and propose metric to measure debiasing and intrinsic debiases. |
| Outcome: | The proposed framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner. |
Copied to clipboard
| Challenge: | Existing evaluation methods do not expose compositionality of emergent communication . compositionality is a trait that enables the construction of complex meanings from the meaning of parts. |
| Approach: | They propose to find best-match between emergent words and natural language concepts to assess compositionality of emergentic communication. |
| Outcome: | The proposed algorithm provides a global score and translation-map between emergent words and natural language concepts. |
Copied to clipboard
| Challenge: | Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. |
| Approach: | They apply causal mediation analysis to pre-trained neural language models to investigate their models' preferences for grammatical inflections and whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. |
| Outcome: | The proposed model can predict correct token from grammatically minimally different continuations with high accuracy even in difficult contexts. |
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
Copied to clipboard
| Challenge: | Model merging has emerged as a promising technique for combining fine-tuned models into a single expert model without retraining. |
| Approach: | They propose a model merging technique that preserves weak model knowledge . they define mergeability as a property of model updates that captures how well they retain trained knowledge when merged with other model updates. |
| Outcome: | The proposed method preserves weak knowledge in the base model. |
Copied to clipboard
| Challenge: | Recent work shows that deep NLP models capture linguistic knowledge but little attention is paid to individual neurons. |
| Approach: | They conduct a neuron-level analysis of pre-trained neural language models to determine linguistic properties. |
| Outcome: | The proposed model is more localized and disjoint when predicting properties than BERT and others. |
Copied to clipboard
| Challenge: | Numerous methods to mitigate social biases require prior knowledge of the demographics in the dataset, such as gender or race. |
| Approach: | They propose a method for bias removal without prior knowledge of demographics in the dataset. |
| Outcome: | Experiments with racial and gender biases in sentiment classification and occupation classification tasks show that BLIND mitigates biase . BLINT is competitive with methods that require demographic information and sometimes surpasses them. |
Copied to clipboard
| Challenge: | Existing methods to investigate whether sentence representations from NMT systems capture distinct semantic phenomena are limited. |
| Approach: | They propose a process to investigate the extent to which sentence representations arising from neural machine translation systems encode distinct semantic phenomena. |
| Outcome: | The proposed model is suited to supporting inferences at the syntax-semantics interface, compared to anaphora resolution requiring world knowledge. |
Copied to clipboard
| Challenge: | a corpus of late antique and medieval Hebrew poetry is rich in metaphors and similes . scholars in the humanities need to distinguish between figurative and literal language . |
| Approach: | They present a corpus of late antique and medieval Hebrew poetry with expert annotations of metaphor . they hope to facilitate further research in this area . |
| Outcome: | The proposed dataset includes a corpus of late antique and medieval Hebrew poetry with expert annotations of metaphor. |
Copied to clipboard
| Challenge: | Recent debiasing methods in natural language understanding improve performance on out-of-distribution datasets by pressuring models into making unbiased predictions. |
| Approach: | They propose a general probing-based framework that allows for post-hoc interpretation of biases in language models and use an information-theoretic approach to measure the extractability of certain biase . |
| Outcome: | The proposed framework allows for post-hoc interpretation of biases in language models and measures the extractability of certain biase . |
Copied to clipboard
| Challenge: | Recent work in interpretability suggests we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary space, a transformation that makes them more human interpretable. |
| Approach: | They propose a tool to visualize a forward pass of Generative Pre-trained Transformers as an interactive flow graph with nodes representing neurons or hidden states and edges representing interactions between them. |
| Outcome: | The proposed visualization simplifies huge amounts of data into easy-to-read graphs that can reflect the models’ internal processing, uncovering the contribution of each component to the models' final prediction. |
Copied to clipboard
| Challenge: | Natural language inference datasets contain artifacts and biases that allow models to perform poorly by using a biased subset of the input without considering the remainder features. |
| Approach: | They reformulate a natural language inference task as a generative task . they find that this approach is highly robust to large amounts of bias . |
| Outcome: | The proposed model is highly robust to large amounts of bias. |
Copied to clipboard
| Challenge: | Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself and might under-represent domain specific or rare facts. |
| Approach: | They propose a method that transforms a factual corpus into a benchmark evaluating an LM's propensity to generate true facts from the corpus . |
| Outcome: | The proposed framework transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. |
Copied to clipboard
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
Copied to clipboard
| Challenge: | Existing circuit analysis methods treat component positions as equally relevant across input positions, limiting their ability to capture cross-positional interactions or mechanisms that vary across positions. |
| Approach: | They propose to extend edge attribution patching to differentiate between token positions and introduce a dataset schema to enable position-aware circuit discovery in datasets with variable length examples. |
| Outcome: | The proposed approach enables position-aware circuit discovery even on tasks with variable length examples. |
Copied to clipboard
| Challenge: | Recent interpretability methods project weights and hidden states obtained from the forward pass to the models’ vocabularies, helping to uncover how information flows within LMs. |
| Approach: | They propose to cast a gradient matrix as a low-rank linear combination of forward and backward passes’ inputs and then to project these gradients into vocabulary items. |
| Outcome: | The proposed method can be cast as a low-rank linear combination of forward and backward passes’ inputs and project these gradients into vocabulary items. |
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Copied to clipboard
| Challenge: | Existing evaluations of attribution methods focus on the English language . plausibility and faithfulness are two main criteria for plausible and faithful attributions . |
| Approach: | They propose a cross-lingual strategy to measure faithfulness based on word alignments. |
| Outcome: | The proposed approach eliminates drawbacks of erasure-based evaluations and provides a multilingual dataset with highlights to support future studies. |
Copied to clipboard
| Challenge: | Text-to-image generative models encode factual associations that can quickly become outdated, diminishing their utility for end-users. |
| Approach: | They propose a method for editing factual associations in text-to-image models without retraining or explicit input from end-users. |
| Outcome: | The proposed method improves generalization and preservation of unrelated concepts on an existing dataset and compares with other methods. |
Copied to clipboard
| Challenge: | Natural language processing models tend to learn and encode social biases present in the data. |
| Approach: | They propose a method for removing non-linear encoded concepts from neural representations by iteratively training neural classifiers to predict a particular attribute, followed by a projection of the representation on a hypersurface. |
| Outcome: | The proposed method removes non-linear encoded concepts from neural representations. |
Copied to clipboard
| Challenge: | Prior work focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. |
| Approach: | They investigate how semantic information is distributed across token representations in text-to-image prompts by patching techniques to uncover encoding patterns. |
| Outcome: | The proposed model can improve alignment and generation quality by modifying the diffusion stage and the cross-attention mechanism. |
Copied to clipboard
| Challenge: | Existing and novel similarity measures are used to analyze contextual word representations . different architectures have rather similar representations, but different individual neurons. |
| Approach: | They propose a method to analyze contextual word representation models using similarity analysis. |
| Outcome: | The proposed approach can be used to analyze model similarity without external annotations. |
Copied to clipboard
| Challenge: | Prior work on large language model (LLM) hallucinations associated with model uncertainty or inaccurate knowledge. |
| Approach: | They define and investigate a type of hallucination where a model can answer a question correctly but a perturbation causes it to produce a hallucinous response with high certainty. |
| Outcome: | The proposed mitigations outperform existing methods on CHOKE hallucinations . the findings highlight the need to understand their origins and improve mitigation strategies . |
Copied to clipboard
| Challenge: | Animacy is not always expressed directly in language, but it manifests indirectly in English . atypically animate entities are easier to remember and prioritized in visual processing . |
| Approach: | They find that LMs behave much like humans when presented with entities whose animacy is typical. |
| Outcome: | The proposed model can learn about animacy even when presented with atypically animate entities. |
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows . |
| Approach: | They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase . |
| Outcome: | The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods . |
Copied to clipboard
| Challenge: | Existing similarity measures perform mediocrely on standard benchmarks . |
| Approach: | They develop a similarity measure based on contrastive learning that learns a parameterized measure by using both similar and dissimilar examples. |
| Outcome: | The proposed measure achieves much higher accuracy than previous similarity measures . it is more suitable for the analysis of neural networks, revealing new insights . |
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Copied to clipboard
| Challenge: | Recent studies have shown that strong natural language understanding models are prone to relying on unwanted dataset biases without learning the underlying task. |
| Approach: | They propose two learning strategies to train neural models that are more robust to dataset biases and transfer better to out-of-domain datasets. |
| Outcome: | The proposed methods improve robustness in all settings and transfer better to out-of-domain datasets. |
Copied to clipboard
| Challenge: | Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. |
| Approach: | They conduct the first in-depth analysis of the role padding tokens play in T2I diffusion models by using two causal techniques to analyze how information is encoded in the representation of tokens across different components of the pipeline. |
| Outcome: | The proposed techniques reveal that padding tokens may affect the model’s output during text encoding, during the diffusion process, or be effectively ignored. |
Copied to clipboard
| Challenge: | Common language models typically predict the next word given a past context. |
| Approach: | They propose a method that aligns the given context and the following phrase . they define syntactic heights and phrase segmentation rules to enable it to learn . |
| Outcome: | The proposed model outperforms strong baseline models on Wikitext-103 dataset. |
Copied to clipboard
| Challenge: | Existing efforts to address context window limitation for off-the-shelf LLMs involve training specialized architectures. |
| Approach: | They propose a method that carves a long context into chunks and restricts attention to apply only within each window. |
| Outcome: | The proposed method shows significant improvements on in-context learning tasks with diverse input and output spaces. |