Papers by Kristina Toutanova
Sparse, Dense, and Attentional Representations for Text Retrieval (2021.tacl-1)
Copied to clipboard
| Challenge: | Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. |
| Approach: | They propose a dual-encoder-based neural model that combines the efficiency of dual encoders with expressiveness of more costly attentional architectures. |
| Outcome: | The proposed model outperforms strong alternatives in large-scale retrieval. |
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (N19-1)
Copied to clipboard
| Challenge: | In this paper we build a reading comprehension dataset of yes/no questions that are naturally occurring . they often query for complex, non-factoid information, and require difficult entailment-like inference to solve. |
| Approach: | They build a reading comprehension dataset of yes/no questions that are naturally occurring . they find they are unexpectedly challenging and require difficult inferences to solve . |
| Outcome: | The proposed method achieves 80.4% accuracy compared to 90% accuracy of human annotators and 62% majority-baseline. |
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities, particularly in English, but for less prevalent languages, performance can be significantly lower, making additional adaptation paramount. |
| Approach: | They propose a new adaptation method based on iteratively merging multiple models fine-tuned on a subset of available training data that reduces forgetting while maintaining learning on the target domain. |
| Outcome: | The proposed method outperforms LLAMA-3-8B-based models in German and German while maintaining learning on the target domain. |
Representations for Question Answering from Documents with Tables and Text (2021.eacl-main)
Copied to clipboard
| Challenge: | a study aims to improve question answering on tables by refining table representations based on textual context. |
| Approach: | They aim to improve question answering from tables by refining table representations based on textual context. |
| Outcome: | The proposed method improves on the Natural Questions dataset using text and table representations. |
Zero-Shot Entity Linking by Reading Entity Descriptions (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to link entities to unseen entities require in-domain labeled data. |
| Approach: | They propose a zero-shot entity linking task where mentions must be linked to unseen entities without in-domain labeled data. |
| Outcome: | The proposed task can generalize to unseen entities without metadata or alias tables . the proposed system improves over baselines, including BERT, on a new dataset . |
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations (2023.acl-long)
Copied to clipboard
| Challenge: | People express information needs with multiple preferences or constraints . modern retrieval systems struggle on such queries, a study finds . |
| Approach: | They construct a dataset of 3357 queries that map to a set of Wikipedia entities . they use crowd-sourced data to match constraints with evidence in documents . |
| Outcome: | The proposed dataset challenges models to match constraints mentioned in queries with evidence in documents and correctly perform various set operations. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
Latent Retrieval for Weakly Supervised Open Domain Question Answering (P19-1)
Copied to clipboard
| Challenge: | Recent work on open domain question answering (QA) assumes strong supervision of the supporting evidence and/or assumes a blackbox information retrieval (IR) system to retrieve evidence candidates. |
| Approach: | They propose to jointly learn the retriever and reader from question-answer string pairs and without any IR system. |
| Outcome: | The proposed approach outperforms BM25 on open datasets with a learner and reader by 19 points in exact match. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to semantic parsing only evaluated on synthetic datasets that are not representative of natural language variation. |
| Approach: | They propose a semantic parsing approach that handles both natural language variation and compositional generalization. |
| Outcome: | The proposed model outperforms existing models across compositional generalization challenges on non-synthetic datasets while being competitive with the state-of-the-art on standard evaluations. |
Efficient End-to-End Visual Document Understanding with Rationale Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-processing tools such as optical character recognition (OCR) can map document image inputs to textual tokens, then large language models (LLMs) can reason over text. |
| Approach: | They propose a method that integrates outputs of OCR tools and larger multimodal models as intermediate "rationales" a student model is trained to predict rationales and answers based on visual documents . |
| Outcome: | The proposed model outperforms the base model on three visual document understanding benchmarks with only 1% higher computational cost. |
Joint Passage Ranking for Diverse Multi-Answer Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multi-answer retrieval cannot reason about the set of passages jointly. |
| Approach: | They propose a joint passage retrieval model focusing on reranking to solve multi-answer retrieval problem. |
| Outcome: | The proposed model outperforms baseline models on three multi-answer datasets. |
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to optimize language model pre-training data mixtures are difficult due to the complexity of the data mixture. |
| Approach: | They propose a method to optimize language model pre-training data mixtures by approximating cross-entropy loss via a Mixture of Data Experts (MDE). |
| Outcome: | The proposed method improves performance on a slimPajama dataset with a mixture of data experts. |
Improving Compositional Generalization with Latent Structure and Data Augmentation (2022.naacl-main)
Copied to clipboard
| Challenge: | Generic unstructured neural networks struggle on out-of-distribution compositional generalization. |
| Approach: | They propose a method to recombinate examples from a model called Compositional Structure Learner and add them to a pre-trained sequence-to-sequence model. |
| Outcome: | The proposed model is even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. |
Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Distant supervision assumptions have enabled the creation of large-scale extractive short answer question answering systems. |
| Approach: | They propose to use document-level distant supervision assumptions to pair questions and relevant documents with answer strings. |
| Outcome: | The proposed model outperforms state-of-the-art models by 4.3 points on TriviaQA-Wiki and 1.7 points on NarrativeQA summaries. |