Papers by William Cohen
Neural Models for Reasoning over Multiple Mentions Using Coreference (N18-2)
Copied to clipboard
| Challenge: | Existing Recurrent Neural Network (RNN) layers are biased towards short-term dependencies and hence not suited to such tasks. |
| Approach: | They propose a recurrent layer which is instead biased towards coreferent dependencies and uses coreference annotations extracted from an external system to connect entity mentions belonging to the same cluster. |
| Outcome: | The proposed layer improves performance on Wikihop, LAMBADA and the bAbi AI datasets with large gains when training data is scarce. |
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models store a massive amount of world knowledge implicitly in their parameters, but large models often fail to encode information about rare entities and events. |
| Approach: | They propose a retrieval-augmented model which accesses an external non-parametric memory to augment language generation. |
| Outcome: | The proposed model outperforms existing models by 10-20% absolute on two datasets and under distractor and full-wiki settings. |
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (P19-1)
Copied to clipboard
| Challenge: | Existing text generation metrics rely on reference texts, such as BLEU and ROUGE, but they are too expensive to apply repeatedly. |
| Approach: | They propose a metric which aligns n-grams from the generated texts to the semi-structured data before computing their precision and recall. |
| Outcome: | The proposed metric correlates with human judgments better than existing text generation metrics while being easier to use. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |
Differentiable Open-Ended Commonsense Reasoning (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing commonsense reasoning models work by scoring a question-candidate pair, but new approaches are needed to answer multiple-choice questions. |
| Approach: | They propose to use a corpus of commonsense facts to answer a commonsensical question without any pre-defined choices as a resource. |
| Outcome: | The proposed model outperforms baseline methods by a large margin in the open-ended commonsense reasoning task. |
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2023.findings-acl)
Copied to clipboard
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
| Challenge: | Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model . however, the architecture used for FiD was not designed for retrieval augmented models . |
| Approach: | They propose to make FiD a modified retrieval-augmented language model with a large decoder and memory bandwidth constraints to alleviate memory bandwidth limitations. |
| Outcome: | The proposed architecture outperforms existing models on knowledge-intensive tasks even on large models on many knowledge-based tasks. |
MATE: Multi-view Attention for Table Transformer Efficiency (2021.emnlp-main)
Copied to clipboard
| Challenge: | Tables are ubiquitous on the web, and are rich in information. |
| Approach: | They propose a sparse-attention Transformer architecture for modeling documents that contain large tables. |
| Outcome: | The proposed architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. |
ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers (2022.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for reading comprehension have deterministic answers, but questions in the real world do not always have definite answers. |
| Approach: | They propose a Question Answering (QA) dataset that contains complex questions with conditional answers. |
| Outcome: | The proposed dataset will motivate further research in answering complex questions over long documents. |
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval (2023.acl-long)
Copied to clipboard
| Challenge: | Contrastive learning is the dominant paradigm for learning text representations from parallel text, but finding negative examples can be expensive in terms of compute or manual effort. |
| Approach: | They propose a generative model for learning multilingual text embeddings which encourages source separation in multilingual contexts by an approximation. |
| Outcome: | The proposed model outperforms both a strong contrastive and generative baseline on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval. |
SEMQA: Semi-Extractive Multi-Source Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent proposed long-form question answering systems have shown promising capabilities, but attributing and verifying their generated abstractive answers can be difficult. |
| Approach: | They propose a task that summarises multiple sources in a semi-extractive fashion . they create a dataset with human-written semi-extractive answers to natural and generated questions . |
| Outcome: | The proposed task summarizes multiple sources in a semi-extractive fashion and produces fine in-line attributions by-design that are easy to verify, interpret, and evaluate. |
PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text (D19-1)
Copied to clipboard
| Challenge: | Experimentally PullNet improves over the prior state-of-the-art open domain question answering systems. |
| Approach: | They propose a framework for learning what to retrieve and reasoning with heterogeneous information to find the best answer. |
| Outcome: | The proposed framework improves over the prior state-of-the-art in open domain question answering . it is weakly supervised, requiring question-answer pairs but not gold inference paths . |
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often generate inconsistent summaries containing factual errors or fabricated content. |
| Approach: | They propose to generate representative examples of non-factual summaries through infilling language models and train a robust fact-correction model to post-edit them to improve factual consistency. |
| Outcome: | The proposed model outperforms previous methods in correcting factual errors on two popular summarization datasets. |
MEMORY-VQ: Compression for Tractable Internet-Scale Memory (2024.naacl-short)
Copied to clipboard
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontanon, William Cohen, Sumit Sanghai, Joshua Ainslie
| Challenge: | Memory-based methods like LUMEN pre-compute token representations for retrieved passages to speed up inference. |
| Approach: | They propose a method to reduce storage requirements of memory-augmented models . they use a vector quantization variational autoencoder to compress token representations . |
| Outcome: | The proposed method achieves 16x compression rate with comparable performance on KILT benchmark. |
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (D18-1)
Copied to clipboard
| Challenge: | Specialized neural models have been developed for extracting answers from text alone or Knowledge Bases (KBs) alone. |
| Approach: | They propose a novel model for extracting answers from a question-specific subgraph containing text and KB entities and relations. |
| Outcome: | The proposed model outperforms existing methods in a combination of a KB and entity-linked text in QA over a large text corpus. |
Adaptable and Interpretable Neural MemoryOver Symbolic Knowledge (2021.naacl-main)
Copied to clipboard
| Challenge: | Past research has shown that large neural language models encode surprising amounts of factual information, but augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. |
| Approach: | They propose a neural LM that includes an interpretable neuro-symbolic KB in the form of a "fact memory" their LM improves performance on knowledge-intensive question-answering tasks, sometimes dramatically . |
| Outcome: | The proposed model improves on knowledge-intensive question-answering tasks, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art open-book model. |
PubMedQA: A Dataset for Biomedical Research Question Answering (D19-1)
Copied to clipboard
| Challenge: | PubMedQA is a biomedical question answering dataset based on PubMed abstracts . 68.1% accuracy is achieved, compared to single human performance of 78.0% . |
| Approach: | They propose a biomedical question answering dataset from PubMed abstracts . the dataset is annotated by experts and has 1k instances of QA . |
| Outcome: | The proposed model achieves 68.1% accuracy compared to human performance of 78.0% and majority-baseline of 55.2%. |