Papers by Michael Collins
Sparse, Dense, and Attentional Representations for Text Retrieval (2021.tacl-1)
Copied to clipboard
| Challenge: | Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. |
| Approach: | They propose a dual-encoder-based neural model that combines the efficiency of dual encoders with expressiveness of more costly attentional architectures. |
| Outcome: | The proposed model outperforms strong alternatives in large-scale retrieval. |
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (N19-1)
Copied to clipboard
| Challenge: | In this paper we build a reading comprehension dataset of yes/no questions that are naturally occurring . they often query for complex, non-factoid information, and require difficult entailment-like inference to solve. |
| Approach: | They build a reading comprehension dataset of yes/no questions that are naturally occurring . they find they are unexpectedly challenging and require difficult inferences to solve . |
| Outcome: | The proposed method achieves 80.4% accuracy compared to 90% accuracy of human annotators and 62% majority-baseline. |
Coreference Resolution through a seq2seq Transition-Based System (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent coreference resolution systems use search algorithms to identify mentions and resolve coreference. |
| Approach: | They propose a text-to-text coreference resolution system that uses a semantic paradigm to predict mentions and links jointly. |
| Outcome: | The proposed system achieves state-of-the-art accuracy on CoNLL-2012 datasets with 83.3 F1-score for English, 68.5 F1 score for Arabic, and 74.3 F1 scores for Chinese. |
Synthetic QA Corpora Generation with Roundtrip Consistency (P19-1)
Copied to clipboard
| Challenge: | Existing methods for generating synthetic question answering corpora are not suitable for QA, but can be constructed from widely available natural text. |
| Approach: | They propose a method for generating synthetic question answering corpora by combining question generation and answer extraction models and filtering the results to ensure roundtrip consistency. |
| Outcome: | The proposed model achieves exact match and F1 at less than 0.1% and 0.4% from human performance on SQuAD2 and NQ. |
Decontextualization: Making Sentences Stand-Alone (2021.tacl-1)
Copied to clipboard
| Challenge: | Taking excerpts of text can be problematic, as key pieces may not be explicit in a local window. |
| Approach: | They define a problem of sentence decontextualization by rewriting a sentence to be interpretable out of context while preserving its meaning. |
| Outcome: | The proposed method can be used in question answering and document understanding tasks. |
Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for cross-lingual syntactic analysis have been shown to be effective for low-resource languages. |
| Approach: | They propose to use low-order statistical functions to shape model distributions for semi-supervised learning on low-resource datasets. |
| Outcome: | The proposed method improves POS and LAS on 5 target languages and provides significant gains over strong cross-lingual-transfer-plus-fine-tuning baselines for modest amounts of label data. |
QED: A Framework and Dataset for Explanations in Question Answering (2021.tacl-1)
Copied to clipboard
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, Michael Collins
| Challenge: | Existing question answering systems provide no explanation of reasoning that leads to answer . linguistically informed, extensible framework provides explanations in question answering . |
| Approach: | They propose a linguistically informed, extensible framework for explanations in question answering . they propose an expert-annotated dataset of QED explanations built upon a subset of the Natural Questions dataset . |
| Outcome: | The proposed framework improves the ability of untrained raters to spot errors in QA datasets. |
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)
Copied to clipboard
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki
| Challenge: | Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English. |
| Approach: | They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs. |
| Outcome: | The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs. |
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)
Copied to clipboard
Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva
| Challenge: | Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods. |
| Approach: | They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step. |
| Outcome: | The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions. |
Unsupervised Cross-Lingual Part-of-Speech Tagging for Truly Low-Resource Scenarios (2020.emnlp-main)
Copied to clipboard
| Challenge: | a limited set of translations into one or more high-resource languages are available for POS tagging . a bi-LSTM architecture that uses contextualized word embeddings improves performance . |
| Approach: | They propose an unsupervised cross-lingual transfer approach for part-of-speech tagging . they use the Bible as parallel data to learn POS taggers for target languages . |
| Outcome: | The proposed approach improves accuracy on 12 diverse languages . the Bible is used as a parallel corpus for the study . |
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (D18-1)
Copied to clipboard
| Challenge: | Conditional models are frequently encountered in practice, but there has not been a rigorous theoretical analysis of NCE in this setting. |
| Approach: | They propose to use a ranking-based and ranking-only method for conditional models to estimate parameter estimates. |
| Outcome: | The proposed method avoids calculation of partition function or derivatives at each training step . it is closely related to negative sampling methods, now widely used in NLP . |
Fusion of Detected Objects in Text for Visual Question Answering (D19-1)
Copied to clipboard
| Challenge: | Recent neural architectures such as Transformer and BERT allow for multimodal context, which may help model the meaning of words in general and also sharpen its understanding of instances of words. |
| Approach: | They propose a neural architecture that combines vision and natural language to advance models of multimodal context. |
| Outcome: | The proposed architecture achieves the highest performance on the Visual Commonsense Reasoning benchmark and the best performance to date on the public leaderboard. |
Low-Resource Syntactic Transfer with Unsupervised Source Reordering (N19-1)
Copied to clipboard
| Challenge: | Existing methods for dependency parsing use word order differences between source and target languages. |
| Approach: | They propose a cross-lingual transfer method that takes into account word order differences between source and target languages. |
| Outcome: | The proposed method improves on 68 treebanks (38 languages) on a target language. |
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation (2022.acl-long)
Copied to clipboard
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, Mirella Lapata
| Challenge: | Composition Sampling is a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. |
| Approach: | They propose a method to generate diverse outputs for conditional generation . they use a plan-based neural generation model that is trained to create a composition of the output and then generate by conditioning on it and the input. |
| Outcome: | The proposed method avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. |
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)
Copied to clipboard
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
| Challenge: | Existing methods to explain predictions by highlighting salient features are often unstated. |
| Approach: | They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. |
| Outcome: | The proposed framework allows principled, automatic, model-agnostic evaluation of attributions. |
Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss (2021.tacl-1)
Copied to clipboard
| Challenge: | Named entity recognition is a critical subtask of many domain-specific natural language understanding tasks. |
| Approach: | They propose a novel loss to learn named entity recognizers in the presence of missing entity annotations. |
| Outcome: | The proposed approach outperforms state-of-the-art methods in a challenging setting with only 1,000 biased annotations, averaged across 7 datasets. |