Papers by Omer Levy
A Few More Examples May Be Worth Billions of Parameters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on few-shot learning for natural language tasks explores the dynamics of scaling up either the number of model parameters or labeled examples while controlling for the other variable by setting it to a constant. |
| Approach: | They explore the dynamics of scaling up the number of model parameters versus the number labeled examples across a wide variety of tasks. |
| Outcome: | The results show that scaling parameters yields performance improvements, while adding examples does not. |
Improving Transformer Models by Reordering their Sublayers (2020.acl-main)
Copied to clipboard
| Challenge: | a sandwich transformer pattern is a new approach to multilayer transformers that can be used for different tasks. |
| Approach: | They propose a transformer ordering pattern that reorders sublayers in a sandwich transformer pattern . they generate random transformer models and train them with the language modeling objective . |
| Outcome: | The proposed pattern improves perplexity on multiple word-level and character-level language modeling benchmarks at no cost in parameters, memory, or training time. |
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions without human supervision. |
| Approach: | They propose to use a dataset of natural language instructions to generate large datasets with no human supervision. |
| Outcome: | The proposed dataset outperforms open-source models on various benchmarks, and is cost-effective. |
Few-Shot Question Answering by Pretraining Span Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining models with recurring span selection are effective, but perform poorly in a few-shot setting. |
| Approach: | They propose recurring span selection scheme that asks model to select correct span in passage with multiple sets of recurring recurrings. |
| Outcome: | The proposed model achieves 72.7 F1 on multiple benchmarks while maintaining competitive performance in the high-resource setting. |
Ultra-Fine Entity Typing (P18-1)
Copied to clipboard
| Challenge: | Experimental results show that a model that can predict ultra-fine types can be crowd-sourced . head words indicate the type of the noun phrases they appear in, and are important for context-sensitive tasks . |
| Approach: | They propose a task where sentences are given with an entity mention . they introduce a new type of distant supervision: head words, which indicate the type of noun phrases they appear in. |
| Outcome: | The proposed model can predict ultra-fine types at varying granularity and performs well on a fine-grained entity typing benchmark. |
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)
Copied to clipboard
| Challenge: | Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text. |
| Approach: | They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span. |
| Outcome: | The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks. |
Transformer Feed-Forward Layers Are Key-Value Memories (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using feed-forward layers, we show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. |
| Approach: | They propose that feed-forward layers in transformer-based language models operate as key-value memories where each key correlates with textual patterns in the training examples and each value induces a distribution over the output vocabulary. |
| Outcome: | The proposed model is based on key-value memories with a key-level correlation with the training examples and a distribution over the output vocabulary. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
Blockwise Self-Attention for Long Document Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in pre-training and fine-tuning methods have drastically reshaped the landscape of natural language processing research. |
| Approach: | They propose a lightweight BERT model that introduces sparse block structures into the attention matrix to reduce memory consumption and training/inference time. |
| Outcome: | The proposed model uses 18.7-36.1% less memory and 12.0-25.1% more time to learn compared to an advanced BERT-based model, RoBERTa. |
Causes and Cures for Interference in Multilingual Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce interference in multilingual machine translation are often computationally intensive and do not always work. |
| Approach: | They propose to reduce interference in multilingual machine translation models by enlarging the model and tuning the sampling temperature to control the proportion of each language pair in the data. |
| Outcome: | The proposed model size, data size, and proportion of each language pair within the dataset determine interference (or synergy) . |
LMentry: A Language Model Benchmark of Elementary Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models are evaluated via perplexity or performance on downstream tasks, but these benchmarks are too complex and difficult to inspect. |
| Approach: | They propose a benchmark that focuses on 25 tasks that humans are expected to perform perfectly, such as writing a sentence containing a specific word or identifying which words in a list belong to a certain category. |
| Outcome: | The proposed benchmarks show that large language models are performing better than previous benchmarks. |
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current datasets targeting ambiguity can be solved by a native speaker with relative ease. |
| Approach: | They present a large-scale dataset based on cryptic crosswords with a cryptical clue. |
| Outcome: | The proposed dataset is based on cryptic crosswords with 523K examples. |
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (N19-1)
Copied to clipboard
| Challenge: | Existing inference models that rely heavily on unsupervised single-word embeddings struggle to learn implied relationships between pairs of words. |
| Approach: | They propose to use word embeddings to learn and use background knowledge about implied relationships between words that are crucial for cross-sentence inference problems. |
| Outcome: | The proposed models gain 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI, and 8.8% on the adversarial SQu AD datasets. |
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens (2022.naacl-main)
Copied to clipboard
| Challenge: | Standard pre-trained language models do not see the characters that compose each token's string representation. |
| Approach: | They probe the embedding layer of pretrained language models and show that models learn the internal character composition of whole word and subword tokens without seeing the characters coupled with the tokens. |
| Outcome: | The embedding layers of RoBERTa and GPT2 hold enough information to accurately spell up to a third of the vocabulary and reach high character ngram overlap across all token types. |
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using positional embeddings, Causal transformer language models learn an implicit notion of absolute positions. |
| Approach: | They propose to use positional embeddings to encode positional information in transformer language models. |
| Outcome: | The proposed model learns an implicit notion of absolute positions across datasets, model sizes, and sequence lengths. |
Deep RNNs Encode Soft Hierarchical Syntax (P18-2)
Copied to clipboard
| Challenge: | Existing studies show that syntactic information is useful for a wide variety of NLP tasks. |
| Approach: | They propose to use word-level representations to learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision. |
| Outcome: | The proposed model encodes significant amounts of syntax even without explicit supervision. |
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models can perform unseen tasks by conditioning on a few input-output demonstrations, but task inference is implicit and the ability of models to explicitly reason about it remains unexplored. |
| Approach: | They propose an instruction induction challenge in which a model is asked to generate a natural language instruction that fits a set of labeled examples. |
| Outcome: | The proposed model achieves 65.7% of human performance while the original model only reaches 9.8% of human performances. |
Neural Machine Translation without Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models operate over subword tokens, but byte-based models employ a different approach . a one-hot representation of each byte does not hurt performance, but it improves BLEU scores . |
| Approach: | They propose to represent every computerized text as a sequence of bytes via UTF-8 . this eliminates the need for an embedding layer and improves performance . |
| Outcome: | The proposed model improves BLEU scores on byte-to-byte translation models compared to character-level models . the proposed model does not require an embedding layer and does not drop out of the decoder . |
Coreference Resolution without Span Representations (2021.acl-short)
Copied to clipboard
| Challenge: | Pretraining has reduced many complex task-specific NLP models to simple lightweight layers. |
| Approach: | They propose a lightweight end-to-end coreference model that removes the dependency on span representations, handcrafted features, pruning heuristics, and more. |
| Outcome: | The proposed model performs competitively with the current standard model, while being simpler and more efficient. |
Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing probing tasks are designed to evaluate the information existing in representations by training a simple classification model. |
| Approach: | They propose to use indicators to query embedding spaces for the existence of certain properties to determine whether a property exists in an embeddable space. |
| Outcome: | The proposed indicators provide a more accurate picture of the information captured and removed compared to probes. |
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)
Copied to clipboard
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
| Challenge: | Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses. |
| Approach: | They propose a new benchmark that places models in context of long texts that require reasoning over long texts. |
| Outcome: | The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods. |
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often used for multi-faceted language generation and evaluation tasks that require complex user constraints or taking into account multiple aspects and criteria. |
| Approach: | They propose a Large Language Model program that consists of branch, solve, and merge modules that are parameterized with specific prompts to the base LLM. |
| Outcome: | The proposed program improves the evaluation correctness and consistency for each LLM by up to 26%, reducing length and pairwise position biases by up 50%, and allowing LLaMA-2-chat to match or outperform GPT-4 on most domains. |
Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation (D19-55)
Copied to clipboard
| Challenge: | Recent machine translation methods are highly sensitive to orthographical variations such as spelling errors. |
| Approach: | They propose to train machine translation models with random synthetic noise at training time . they focus on translation performance on natural typos, and show robustness to such noise . |
| Outcome: | The proposed method significantly improves translation models on natural typos without accessing natural noise data or distribution. |
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM. |
| Approach: | They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets . |
| Outcome: | The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks. |
Learning to Retrieve Passages without Supervision (2022.naacl-main)
Copied to clipboard
| Challenge: | Dense retrievers for open domain question answering have been shown to achieve impressive performance by training on large datasets of question-passage pairs. |
| Approach: | They propose to use recurring spans to create pseudo examples for contrastive learning. |
| Outcome: | The proposed model outperforms all pretrained baselines on a wide range of ODQA datasets and is competitive with BM25, a strong sparse baseline. |
Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)
Copied to clipboard
Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
| Challenge: | Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown. |
| Approach: | They propose an approach to edit and re-align alt-texts associated with images using human annotation. |
| Outcome: | The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks. |
Keep Guessing? When Considering Inference Scaling, Mind the Baselines (2025.findings-naacl)
Copied to clipboard
| Challenge: | Experiments spanning mathematical reasoning and factual knowledge reveal that repeated sampling improves coverage in large language models. |
| Approach: | They define a baseline that enumerates answers according to their prevalence in the training set and then scale it to measure coverage. |
| Outcome: | The proposed model outperforms repeated model sampling for some LLMs while the coverage for others is on par with that of a mixture strategy that obtains k answers by using only 10 model samples and guessing the remaining k-10 attempts via enumeration. |
How to Train BERT with an Academic Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models such as BERT are used in many NLP tasks, but their pretraining phase can be prohibitively expensive for startups and academic research groups. |
| Approach: | They propose a recipe for pretraining a large language model in 24 hours using a low-end deep learning server. |
| Outcome: | The proposed model can be trained on GLUE tasks at fraction of the cost of pretraining. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (D19-1)
Copied to clipboard
| Challenge: | a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations. |
| Approach: | They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective . |
| Outcome: | The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average. |
Simple Local Attentions Remain Competitive for Long-Context Tasks (2022.naacl-main)
Copied to clipboard
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad
| Challenge: | Existing models for NLP tasks require long text sequences beyond the length limit of pretrained models. |
| Approach: | They propose to pretrain large-size NLP models using the same long-doc corpus and fine tune them for real-world long-context tasks. |
| Outcome: | The proposed models can perform better under standard pretraining paradigms than longformer and Longformer. |
BERT for Coreference Resolution: Baselines and Analysis (D19-1)
Copied to clipboard
| Challenge: | Recent BERT-based models have reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition. |
| Approach: | They apply BERT to coreference resolution, achieving a new state of the art on the GAP and OntoNotes benchmarks. |
| Outcome: | A qualitative analysis of model predictions shows that BERT-large is better at distinguishing between related but distinct entities, but there is room for improvement in modeling document-level context, conversations, and mention paraphrasing. |
Can Latent Alignments Improve Autoregressive Machine Translation? (2021.naacl-main)
Copied to clipboard
| Challenge: | Latent alignment objectives improve non-autoregressive models, but can they improve autoregressive ones? e.g., we show that latent alignments are incompatible with teacher forcing. |
| Approach: | They propose latent alignment objectives that use a dynamic program to comb the space of monotonic alignments between the "gold" target sequence and token probabilities the model predicts. |
| Outcome: | The proposed models are incompatible with teacher forcing, the authors show . they show that latent alignment objectives reduce misalignments and focus on original error . |
Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)
Copied to clipboard
| Challenge: | LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. |
| Approach: | They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input. |
| Outcome: | The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients. |
Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling (P18-2)
Copied to clipboard
| Challenge: | Recent models that use gold predicates only use a single predicate at a time. |
| Approach: | They propose an end-to-end approach for jointly predicting all predicates, arguments spans, and the relations between them. |
| Outcome: | The proposed model can model overlapping spans across different predicates in the same output structure without gold predicate predications. |