Papers by Elena Voita
Information Flow Routes: Automatically Interpreting Language Models at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current state-of-the-art language models (LMs) are built on top of the Transformer architecture. |
| Approach: | They propose to build graphs where nodes correspond to token representations and edges to computations . they show that attention heads and subword merging heads are important . |
| Outcome: | The proposed model can analyze behavior for specific types of predictions, or different domains. |
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (P19-1)
Copied to clipboard
| Challenge: | et al., 2017) show that multi-head attention is important for neural machine translation. |
| Approach: | They evaluate the contribution made by individual attention heads to the overall performance of the Transformer model and analyze the roles played by them in the encoder. |
| Outcome: | The proposed pruning method removes the vast majority of heads without affecting performance. |
Context-Aware Monolingual Repair for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | et al., 2018) show that human raters prefer corrected translations over the baseline ones. |
| Approach: | They propose a monolingual model to correct inconsistencies between sentences . they use monolingual document-level data to train the model . |
| Outcome: | The proposed model improves translations of contextual phenomena in English-Russian translation task. |
Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Statistical MT decomposes the translation task into distinct components that are learned separately. |
| Approach: | They show that neural machine translation models acquire different competences over the course of training . previous work shows how to improve some of the competences in NMT by using lexical translation probabilities, phrase memories, alignment information. |
| Outcome: | The proposed model improves translation quality and word-by-word translation, while learning complex reordering patterns. |
Information-Theoretic Probing with Minimum Description Length (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. |
| Approach: | They propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). |
| Outcome: | The proposed method agrees in results and is more informative and stable than the standard probes. |
LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing tools focus on isolated parts of the decision-making process, but LM-TT makes the entire prediction process transparent. |
| Approach: | They present an open-source toolkit for analyzing the internal workings of Transformer-based language models. |
| Outcome: | The LM Transparency Tool makes the entire prediction process transparent . it shows the importance of specific component at each step . |
When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion (P19-1)
Copied to clipboard
| Challenge: | et al., 2018: translation errors due to the lack of extra-sentential context are becoming more and more noticeable among otherwise adequate translations. |
| Approach: | They propose a context-aware translation model that uses sentence-level data to identify inconsistencies . standard metrics are not sensitive to improvements in consistency in document-level translations . |
| Outcome: | The proposed model shows major gains over baseline without sacrificing performance . standard metrics are not sensitive to improvements in document-level translations . |
Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models are influenced by two types of context, source and target, but none explicitly evaluates relative contribution to generation decision. |
| Approach: | They propose to adopt a variant of Layerwise Relevance Propagation which evaluates relative contributions to the generation decision by a proportion of token influence. |
| Outcome: | The proposed model can evaluate the relative contribution of source and target to the generation decision by using a variant of Layerwise Relevance Propagation (LRP) |
Context-Aware Neural Machine Translation Learns Anaphora Resolution (P18-1)
Copied to clipboard
| Challenge: | Standard machine translation systems process sentences in isolation and ignore extra-sentential information. |
| Approach: | They propose a context-aware neural machine translation model that controls flow of information from extended context to the translation model. |
| Outcome: | The proposed model improves on an English-Russian subtitles dataset over its context-agnostic version (+0.7) and over simple concatenation of context and source sentences (+0.6). |
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is becoming more accurate, but hallucinations are extremely pathological . previous work focused on artificial settings where the problem is amplified, disregarding some common types of hallucines . |
| Approach: | They propose a method for alleviating hallucinations at test time that significantly reduces the hallucinic rate. |
| Outcome: | The proposed method significantly reduces the hallucinatory rate in a natural setting. |
Embedding Words in Non-Vector Space with Unsupervised Graph Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | GraphGlove is an unsupervised graph word representations that are learned end-to-end. |
| Approach: | They propose a method to learn weighted graph word representations end-to-end using a weighteable weighte . they adopt a hierarchical graph representation method and modify the GloVe training algorithm to learn graph representations. |
| Outcome: | The proposed method outperforms vector-based methods on word similarity and analogy tasks. |
Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that without artificially encouraging models to hallucinate, existing methods fall short . hallucinations are cases when the model generates output that is partially or fully unrelated to the source sentence. |
| Approach: | They propose a method that evaluates the percentage of the source contribution to a generated translation. |
| Outcome: | The proposed method improves detection accuracy for the most severe hallucinations by a factor of 2. |
Neurons in Large Language Models: Dead, N-gram, Positional (2024.findings-acl)
Copied to clipboard
| Challenge: | We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. |
| Approach: | They analyze a family of large language models that rely on whether an FFN neuron is activated or not . they use a single GPU to analyze the OPT network and use it to analyze data . |
| Outcome: | The proposed model is lightweight and can be run on a single GPU. |
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (D19-1)
Copied to clipboard
| Challenge: | a recent study has shown that deep neural networks are effective with various tasks . a new study examines how representations of tokens evolve between layers under different learning objectives . |
| Approach: | They use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers. |
| Outcome: | The proposed model outperforms untrained models on word identity prediction tasks . the model outpersforms models trained on other linguistic tasks based on the model's objective . |
BPE-Dropout: Simple and Effective Subword Regularization (2020.acl-main)
Copied to clipboard
| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation (2023.emnlp-main)
Copied to clipboard
David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loic Barrault, Marta Costa-jussà
| Challenge: | Previously available quality assessments do not distinguish between hallucinations and omissions. |
| Approach: | They propose to annotate hallucinations and omissions in machine translation using a single language pair. |
| Outcome: | The proposed dataset covers 18 translation directions with varying resource levels and scripts. |