Papers by Alexander Rush

Hop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision (2023.emnlp-main)

Challenge:	PromptSource is a system for creating, sharing, and using natural language prompts . prompts are used to train and query language models in zero-shot learning settings .
Approach:	PromptSource is a system for creating, sharing, and using natural language prompts . et al.: using prompts to train and query language models is emerging area in NLP . they propose a templating language for defining data-linked prompts, a user interface that iterates on prompt development .
Outcome:	PromptSource is a system for creating, sharing, and using natural language prompts . it has a templating language for defining data-linked prompts and a community-driven set of guidelines .

Copied to clipboard

Wenting Zhao, Justin Chiu, Claire Cardie, Alexander Rush

Compound Probabilistic Context-Free Grammars for Grammar Induction (P19-1)

Challenge:	Existing methods rely on supervision for both answers and rationales, but they have limited capacities in modeling interactions between sentences, let alone reasoning across multiple documents.
Approach:	They propose a principled, probabilistic approach for training explainable multi-hop question answering systems without rationale supervision.
Outcome:	The proposed method is more accurate at selecting rationales than previous methods while maintaining similar accuracy in predicting answers.

Copied to clipboard

Yoon Kim, Chris Dyer, Alexander Rush

Syntactic Text Processing

Challenge:	Existing approaches to grammar induction have resorted to manually-engineered features and auxiliary objectives to induce the desired structures.
Approach:	They propose a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar.
Outcome:	Experiments on English and Chinese show that the proposed approach is more efficient than other methods.

Sequence-Level Mixed Sample Data Augmentation (2020.emnlp-main)

Copied to clipboard

Demi Guo, Yoon Kim, Alexander Rush

Unsupervised Text Deidentification (2022.findings-emnlp)

Challenge:	Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language.
Approach:	They propose a data augmentation approach to encourage compositional behavior in neural networks . they propose to softly combine input/output sequences from the training set .
Outcome:	The proposed approach yields 1.0 BLEU improvement on translation datasets over baselines.

Copied to clipboard

John Morris, Justin Chiu, Ramin Zabih, Alexander Rush

Datasets: A Community Library for Natural Language Processing (2021.emnlp-demo)

Challenge:	Existing methods to deidentify textual data are supervised named entity recognition based on 18 identifiers like age, phone number, zip code . prior work ignores this nebulous 18th category, allowing for a supervised method to be used .
Approach:	They propose an unsupervised method that masks words that leak personally-identifying information.
Outcome:	The proposed method masks words that leak personally-identifying information . it eliminates many identifying aspects that would fall outside of the common named entity based approach .

Copied to clipboard

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, Thomas Wolf

Challenge:	Contemporary NLP systems use many different datasets at significantly varying scale and level of annotation.
Approach:	a community library for contemporary NLP is available at https://github.com/datasets . the library includes more than 650 unique datasets and has more than 250 contributors a year after its initial development .
Outcome:	the library includes more than 650 unique datasets and has more than 250 contributors . it supports a variety of cross-dataset research projects and shared tasks .

Low-Complexity Probing via Finding Subnetworks (2021.naacl-main)

Copied to clipboard

Steven Cao, Victor Sanh, Alexander Rush

Commonsense Knowledge Mining from Pretrained Models (D19-1)

Challenge:	Existing approaches to probing neural networks for linguistic properties are to train a shallow multi-layer perceptron (MLP) on top of the model's internal representations.
Approach:	They propose a subtractive pruning-based probe where they find an existing subnetwork that performs the linguistic task of interest.
Outcome:	The proposed probe achieves higher accuracy on pre-trained models and lower accuracy on random models, and better learning on its own.

Copied to clipboard

Joe Davison, Joshua Feldman, Alexander Rush

Predicting Text Preference Via Structured Comparative Reasoning (2024.acl-long)

Challenge:	Several approaches have been proposed for training models for commonsense knowledge base completion (CKBC) due to the sparsity of training data.
Approach:	They propose a method for generating commonsense knowledge using a large, pre-trained bidirectional language model by transforming relational triples into masked sentences.
Outcome:	The proposed method outperforms models trained on held-out test sets on a held-up set, suggesting that it generalizes better than current supervised methods.

Copied to clipboard

Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurzion, Alexander Rush, Jialu Liu, Michael Bendersky

Reasoning

Challenge:	Existing approaches to comparative reasoning rely on pretraining or fine-tuning models at the cost of massive human annotation and computation.
Approach:	They propose a model that prompts LLMs to generate structured intermediate comparisons by proposing aspects for comparison, followed by generating textual comparisons under each aspect.
Outcome:	The proposed model significantly reduces hallucination and improves consistency across various NLP tasks.

Learning Neural Templates for Text Generation (D18-1)

Copied to clipboard

Sam Wiseman, Stuart Shieber, Alexander Rush

Block Pruning For Faster Transformers (2021.emnlp-main)

Challenge:	Encoder-decoder models are uninterpretable and difficult to control in terms of content.
Approach:	They propose a neural generation system using a hidden semi-markov model which learns latent templates jointly with learning to generate.
Outcome:	The proposed model learns useful templates and achieves strong performance nearing that of encoder-decoder models.

Copied to clipboard

François Lagunas, Ella Charlaix, Victor Sanh, Alexander Rush

Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)

Challenge:	Pruning methods have proven to be effective at reducing model size, while distillation methods are proven for speeding up inference.
Approach:	They propose a block pruning approach that integrates structured pruning methods with the movement pruning paradigm for fine-tuning.
Outcome:	The proposed model is 2.4x faster, 74% smaller and faster than distilled models on classification and generation tasks.

Copied to clipboard

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush

Sequence-to-Lattice Models for Fast Translation (2021.findings-emnlp)

Challenge:	Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community.
Approach:	they propose an open-source library that aims to open up advances in machine learning to the wider community.
Outcome:	Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community.

Copied to clipboard

Yuntian Deng, Alexander Rush

I Could’ve Asked That: Reformulating Unanswerable Questions (2024.emnlp-main)

Challenge:	Existing non-autoregressive machine translation models have decoders that are difficult to port to NAT models.
Approach:	They propose a sequence-to-lattice model that replaces the decoder with a search lattice.
Outcome:	The proposed model is faster than past non-autoregressive generation approaches and more accurate than reducing the number of decoder layers.

Copied to clipboard

Wenting Zhao, Ge Gao, Claire Cardie, Alexander Rush

Challenge:	Existing large language models do not assist users in reformulating unanswerable questions . a recent study found that the models failed to reformulate questions based on assumptions that conflict with or cannot be verified with the information available in documents.
Approach:	They evaluate open-source and proprietary LLMs on couldAsk to evaluate their performance . they found that GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time .
Outcome:	The proposed model successfully reformulates questions only 26% and 12% of the time . the proposed model is not able to reformulate questions, but it can be improved .

Parameter-Efficient Transfer Learning with Diff Pruning (2021.acl-long)

Copied to clipboard

Demi Guo, Alexander Rush, Yoon Kim

Rationales for Sequential Predictions (2021.emnlp-main)

Challenge:	Pretrained networks are difficult to deploy for multiple tasks in storage-constrained settings.
Approach:	Diff pruning enables parameter-efficient transfer learning that scales well with new tasks.
Outcome:	Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task.

Copied to clipboard

Keyon Vafa, Yuntian Deng, David Blei, Alexander Rush

Explainability & Interpretability in NLP

Challenge:	Sequence models produce accurate predictions, but their decision making processes are hard to explain.
Approach:	They propose an efficient algorithm to approximate sequential objective by identifying the most faithful rationales.
Outcome:	The proposed algorithm is best at optimizing the sequential objective and provides the most faithful rationales.

Training for Diversity in Image Paragraph Captioning (D18-1)

Copied to clipboard

Luke Melas-Kyriazi, Alexander Rush, George Han

Multimodality

Challenge:	Existing image captioning models have a lack of diversity between sentences . current models have limited their effectiveness due to repetitive paragraphs .
Approach:	They propose to apply sequence-level training to image paragraph captioning models . they find that standard self-critical training produces poor results .
Outcome:	The proposed training improves on the Visual Genome dataset with no architectural changes.

GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction (2021.eacl-main)

Copied to clipboard

Xinya Du, Alexander Rush, Claire Cardie

Information Extraction & Text Mining

Challenge:	Document-level role-filler entity extraction (REE) is a classic problem in information extraction (IE) and NLP.
Approach:	They propose a document-level role-filler entity extraction framework that can make extraction decisions across sentence boundaries and implicitly recognizes noun phrase coreference structure.
Outcome:	The proposed framework performs better than previous models on a MUC-4 dataset.

Simple Unsupervised Summarization by Contextual Matching (P19-1)

Copied to clipboard

Jiawei Zhou, Alexander Rush

Text Generation

Challenge:	Existing methods for sentence summarization require a large amount of parallel data for supervision to work.
Approach:	They propose an unsupervised method for sentence summarization using only language modeling.
Outcome:	The proposed method maintains continuous contextual matching while maintaining output fluency without any paired examples.

MiniChain: A Small Library for Coding with Large Language Models (2023.emnlp-demo)

Copied to clipboard

Alexander Rush

Neural Linguistic Steganography (D19-1)

Challenge:	Programming augmented by large language models (LLMs) opens up many new application areas, but also requires care.
Approach:	They introduce a tool for augmented programming that provides basic primitives for coding LLM calls.
Outcome:	The proposed tool provides core primitives for coding LLM calls and separating out prompt templates.

Copied to clipboard

Zachary Ziegler, Yuntian Deng, Alexander Rush

Tree Prompting: Efficient Task Adaptation without Fine-Tuning (2023.emnlp-main)

Challenge:	linguistic steganography encrypts a secret message into a cover signal . language is a pragmatic cover signal due to its benign occurrence and independence from any one medium.
Approach:	They propose a technique that encrypts a secret message into a cover signal . language is a particularly pragmatic cover signal due to its benign occurrence .
Outcome:	The proposed technique generates realistic looking cover sentences as evaluated by humans while preserving security by matching the cover message distribution with the language model distribution.

Copied to clipboard

Chandan Singh, John Morris, Alexander Rush, Jianfeng Gao, Yuntian Deng

Template Filling with Generative Transformers (2021.naacl-main)

Challenge:	Pretrained language models (LMs) are the main interface for applying them to new tasks, but their large size makes them difficult to fine-tune with gradients for specific downstream tasks.
Approach:	They propose to use training data to form a decision tree based on prompt-LM calls, with each prompt determined by the outcomes of previous calls.
Outcome:	The proposed method improves accuracy over competing methods and is competitive with fine-tuning.

Copied to clipboard

Xinya Du, Alexander Rush, Claire Cardie

Improving Event Duration Prediction via Time-aware Pre-training (2020.findings-emnlp)

Challenge:	Template filling tasks are usually tackled by a pipeline of two separate systems, one for role-filler extraction and another for template/event recognition.
Approach:	They propose a framework that naturally models the dependence between entities within a single event and across multiple events described in a document.
Outcome:	The proposed framework outperforms pipeline-based approaches and other neural baselines that do not model between-event dependencies on documents containing multiple events.

Copied to clipboard

Zonglin Yang, Xinya Du, Alexander Rush, Claire Cardie

Adversarial Semantic Collisions (2020.emnlp-main)

Challenge:	Understanding duration of event expressed in text is crucial task in NLP . current methods focus on developing features and cannot utilize external textual knowledge.
Approach:	They propose two models that incorporate external knowledge by reading temporal-related news sentences.
Outcome:	The proposed models outperform baseline models and capture duration information more accurately.

Copied to clipboard

Congzheng Song, Alexander Rush, Vitaly Shmatikov

Robustness in NLP

Challenge:	Existing approaches to generate semantic collisions for NLP tasks are vulnerable to adversarial examples.
Approach:	They propose gradient-based approaches for generating semantic collisions given white-box access to a model and deploy them against several NLP tasks.
Outcome:	The proposed approaches evade perplexity-based filtering and discuss other potential mitigations.

Deep Latent Variable Models of Natural Language (D18-3)

Copied to clipboard

Alexander Rush, Yoon Kim, Sam Wiseman

Challenge:	In this tutorial, we will discuss the challenges of applying neural variational inference to NLP problems.
Approach:	The tutorial will cover deep latent variable models in the case where exact inference over the latent variables is tractable.
Outcome:	The proposed tutorial will cover deep latent variable models in the case where inference cannot be performed tractably and when it is not .

Entity Tracking Improves Cloze-style Reading Comprehension (D18-1)

Copied to clipboard

Luong Hoang, Sam Wiseman, Alexander Rush

Machine Reading Comprehension

Challenge:	Recent work on reading comprehension tasks has improved with simple approaches, but still trail human performance.
Approach:	They propose to add additional entity features and a multi-task tracking objective to improve model performance . they compare the model's predictions with those of more complicated models .
Outcome:	The proposed model outperforms the current state of the art on the LAMBADA dataset by 8 pts.

Text Embeddings Reveal (Almost) As Much As Text (2023.emnlp-main)

Copied to clipboard

John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander Rush

Representation Learning

Challenge:	a vector database of dense text embeddings stores only the text data, not the original text . a multi-step method that iteratively corrects and re-embeds text can recover 92% of 32-token text inputs exactly.
Approach:	They propose a method that iteratively corrects and re-embeds text to recover 92% of 32-token text inputs exactly.
Outcome:	The proposed method recovers 92% of 32-token text inputs exactly.

Pretraining Without Attention (2023.findings-emnlp)

Copied to clipboard

Junxiong Wang, Jing Yan, Albert Gu, Alexander Rush

Torch-Struct: Deep Structured Prediction Library (2020.acl-demos)

Challenge:	Recent studies show that state-space models (SSMs) outperform standard and deep learning for long-range sequence modeling.
Approach:	They propose a model that combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures.
Outcome:	The proposed model outperforms standard and standard sequence modeling architectures on speech generation and the long range arena benchmarks.

Copied to clipboard

Alexander Rush

Challenge:	Structured prediction is a key area of machine learning and is difficult to utilize in deep learning frameworks.
Approach:	They propose a library for structured prediction that integrates with vectorized, auto-differentiation based frameworks.
Outcome:	The library exploits auto-differentiation to produce readable, fast, and testable code.

Scaling Hidden Markov Language Models (2020.emnlp-main)

Copied to clipboard

Justin Chiu, Alexander Rush

Entity Disambiguation via Fusion Entity Decoding (2024.naacl-long)

Challenge:	Hidden Markov models are a fundamental tool for sequence modeling that separates the hidden state from the emission structure.
Approach:	They propose methods for scaling hidden Markov models to massive state spaces while maintaining efficient exact inference and effective regularization.
Outcome:	The proposed methods are much more accurate than previous HMMs and n-gram-based methods, making progress towards the performance of state-of-the-art NN models.

Copied to clipboard

Junxiong Wang, Ali Mousavi, Omar Attia, Ronak Pradeep, Saloni Potdar, Alexander Rush, Umar Farooq Minhas, Yunyao Li

Posterior Control of Blackbox Generation (2020.acl-main)

Challenge:	Existing generative approaches demonstrate improved accuracy compared to classification approaches under the standardized ZELDA benchmark.
Approach:	They propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions.
Outcome:	The proposed model outperforms existing classification models on the ZELDA benchmark and on retrieval/reader frameworks.

Copied to clipboard

Xiang Lisa Li, Alexander Rush

Text Generation

Challenge:	Existing methods for conditional natural language generation are limited in their ability to produce controlled output.
Approach:	They propose to augment neural generation models with discrete control states learned through a structured latent-variable approach.
Outcome:	The proposed approach improves over benchmarks while providing fine-grained control.

Bottom-Up Abstractive Summarization (D18-1)

Copied to clipboard

Sebastian Gehrmann, Yuntian Deng, Alexander Rush

Text Generation

Challenge:	Existing approaches to summarize text using end-to-end content selectors have had mixed success in content selection, for example copying full sentences from the source document.
Approach:	They propose to use content selectors to over-determine phrases in a source document that should be part of the summary.
Outcome:	The proposed model over-determines phrases in a source document that should be part of the summary while generating fluent summaries.

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference (P19-1)

Copied to clipboard

Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, Alexander Rush

What is Learned in Visually Grounded Neural Syntax Acquisition (2020.acl-main)

Challenge:	Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows .
Approach:	They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase .
Outcome:	The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods .

Copied to clipboard

Noriyuki Kojima, Hadar Averbuch-Elor, Alexander Rush, Yoav Artzi

Multimodality

Challenge:	Visual features are promising for learning bootstrap textual models, but blackbox learning models make it difficult to isolate the specific contribution of visual components.
Approach:	They propose to use alignments between phrases and images as a learning signal for syntax acquisition.
Outcome:	The proposed model performs better than the previous model, but it is significantly less expressive.

Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations (2023.acl-long)

Copied to clipboard

Wenting Zhao, Justin Chiu, Claire Cardie, Alexander Rush

Model Criticism for Long-Form Text Generation (2022.emnlp-main)

Challenge:	Existing approaches for abductive reasoning in natural language processing rely on manual supervision.
Approach:	They propose an approach for abductive commonsense reasoning that exploits the fact that only a subset of explanations is correct for a given context.
Outcome:	The proposed approach outperforms or is comparable to knowledge-augmented zero-shot methods on a diverse set of abductive reasoning datasets.

Copied to clipboard

Yuntian Deng, Volodymyr Kuleshov, Alexander Rush

Symbolic Planning and Code Generation for Grounded Dialogue (2023.emnlp-main)

Challenge:	Language models generate fluent text, but it remains unclear whether output retains coherent high-level structure.
Approach:	They propose to use a statistical tool to evaluate high-level structure of text . they compare distributions between real and generated data in latent space .
Outcome:	The proposed model criticism compares distributions between real and generated data in a latent space . different generative processes identify specific failure modes of the underlying model .

Copied to clipboard

Justin Chiu, Wenting Zhao, Derek Chen, Saujas Vaduguru, Alexander Rush, Daniel Fried

Programming Languages in NLP

Challenge:	Large language models excel at processing and generating text and code, but lack a grounded task-oriented dialogue system that can handle grounding.
Approach:	They propose a modular and interpretable grounded dialogue system that integrates a reader and planner to convert partner utterances into executable code and a symbolic planner to determine the next appropriate response.
Outcome:	The proposed system outperforms the existing state-of-the-art on a one-common dialogue task and improves task success in human evaluations from 56% to 69% in the most challenging setting.

ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (2024.emnlp-main)

Copied to clipboard

Yash Akhauri, Ahmed AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander Rush, Safeen Huda, Mohamed Abdelfattah

Unsupervised Recurrent Neural Network Grammars (N19-1)

Challenge:	Prior work has focused on contextual sparsity, but it has not been successful.
Approach:	They propose a novel pruning predictor that can shadow the LLM behavior and enforce better sparsity patterns.
Outcome:	The proposed model can shadow the LLM behavior and enforce better sparsity patterns, resulting in 15% improvement in end-to-end accuracy compared to prior methods.

Copied to clipboard

Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, Gábor Melis