Papers by Matthew Peters
Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Among recent NLP research, multi-document processing is gaining increasing attention due to the need to handle and process an increasing amount of textual data and available documents online. |
| Approach: | They propose to pre-train a generic multi-document model from a cross-document question answering pre-training objective by generating salient sentences from one document and challenging it to recover the sentence from which it was generated. |
| Outcome: | The proposed model outperforms zero-shot GPT-3.5 and GPT-4 in multiple document tasks and generates the correct answer and the salient sentence from a salient document. |
CDLM: Cross-Document Language Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |
Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples (P18-1)
Copied to clipboard
| Challenge: | Statistical parsers are often criticized for their performance outside of the domain they were trained on . we show that word representations reduce the need for domain adaptation when the target domain is syntactically similar to the source domain. |
| Approach: | They propose a way to adapt a parser to a syntactically similar target domain using partial annotations. |
| Outcome: | The proposed model increases the accuracy of a parser on the Wall Street Journal by 1.7% over the previous state-of-the-art model. |
TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan
| Challenge: | Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text. |
| Approach: | They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space. |
| Outcome: | The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
Few-Shot Self-Rationalization with Natural Language Prompts (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for tasks are limited by human-written explanations. |
| Approach: | They propose to use a standardized collection of natural language prompts to create a model that generates free-text explanations for tasks. |
| Outcome: | The proposed model can predict task labels and generate free-text explanations for predictions . plausibility of human explanations is 76%, while human explanation is 51% . |
HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent NLP models have shown the remarkable ability to generalise ‘zero-shot’ to new tasks using only natural language instructions as guidance. |
| Approach: | They introduce Hypernetworks for INstruction Tuning (HINT) which converts task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder. |
| Outcome: | The proposed models outperform strong state-of-the-art models by over 10% when controlling for compute. |
Extracting Latent Steering Vectors from Pretrained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. |
| Approach: | They propose to extract latent vectors directly from pretrained language model decoders without fine-tuning. |
| Outcome: | The proposed approach generates a target sentence nearly perfectly for English sentences . it outperforms pooled hidden states of models on a textual similarity benchmark . |
Construction of the Literature Graph in Semantic Scholar (N18-3)
Copied to clipboard
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, Oren Etzioni
| Challenge: | Fig. 1 summarizes a scalable system for organizing published scientific literature into a heterogeneous graph . authors describe methods used to enable semantic features in www.semanticscholar.org . |
| Approach: | They describe a scalable system for organizing published scientific literature into a heterogeneous graph to facilitate algorithmic manipulation and discovery. |
| Outcome: | The proposed system can be deployed on a scalable platform and report empirical results for each task. |
Does Self-Rationalization Improve Robustness to Spurious Correlations? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Rationalization is fundamental to human reasoning and learning. |
| Approach: | They evaluate robustness to spurious correlations in encoder-decoder and decoder-only models . authors say explanations can come at the cost of robustness . |
| Outcome: | The proposed model outputs are more interpretable and easier to interact with for end-users than nonrationalizing models. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World (2021.acl-long)
Copied to clipboard
Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, Yejin Choi
| Challenge: | PIGLeT model learns physical commonsense knowledge through interaction, then uses this knowledge to ground language. |
| Approach: | They propose a model that learns physical commonsense knowledge through interaction . they factorize PIGLeT into a physical dynamics model and a separate language model . |
| Outcome: | The proposed model outperforms a 100x larger, text-to-text approach in forecasting language . it can read a sentence, simulate neurally what might happen next, and communicate that result through a literal symbolic representation, or natural language. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pretrained language models often need to specialize to specific domains. |
| Approach: | They propose an approach that performs weight-space averaging of adapters trained on different domains. |
| Outcome: | The proposed approach improves performance to new domains without extra training. |
Competency Problems: On Finding and Removing Artifacts in Language Data (2021.emnlp-main)
Copied to clipboard
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, Noah A. Smith
| Challenge: | Recent work in NLP has documented dataset artifacts, bias, and spurious correlations . how to tell which features have spurious instead of legitimate correlations is typically left unspecified . |
| Approach: | They propose a class of competency problems to formalize this notion into a classification . they show that realistic datasets will increasingly deviate from competency problems . |
| Outcome: | The proposed model can be used to show that models are inappropriately affected by these less extreme biases. |
Tailor: Generating and Perturbing Text with Semantic Controls (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies train task-specific generators, relying on training a model for every perturbation. |
| Approach: | They propose a semantically-controlled text generation system that modifies sentences to match target attributes. |
| Outcome: | The proposed system produces textual outputs conditioned on control codes derived from semantic representations. |
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2021.findings-acl)
Copied to clipboard
| Challenge: | Cognitive science and philosophy research has shown that human explanations are contrastive . a contrast case plays a key role in modulating what explanations can be given . |
| Approach: | They propose a method for producing contrastive explanations of model predictions . they edit models' outputs to change model outputs, and then edit them to the contrast case . |
| Outcome: | a new method produces contrastive explanations of model predictions in the form of edits . the edits are minimal and fluent, consistent with human contrastive edits. |
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts. |
| Approach: | They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks. |
| Outcome: | The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters. |