Papers by Uri Shaham
Small Data, Big Noise: Adversarial Training for Robust ParameterEfficient Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting foundation models to downstream tasks, but current methods struggle with robustness to noise and performance degradation on limited training data. |
| Approach: | They propose a framework that brings adversarial training to PEFT to enhance model robustness and generalization, outperforming alternative approaches. |
| Outcome: | Experiments with two variants of the proposed framework show that it outperforms existing methods in low-resource settings and under word-level and character-level corruptions. |
Causes and Cures for Interference in Multilingual Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce interference in multilingual machine translation are often computationally intensive and do not always work. |
| Approach: | They propose to reduce interference in multilingual machine translation models by enlarging the model and tuning the sampling temperature to control the proportion of each language pair in the data. |
| Outcome: | The proposed model size, data size, and proportion of each language pair within the dataset determine interference (or synergy) . |
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current datasets targeting ambiguity can be solved by a native speaker with relative ease. |
| Approach: | They present a large-scale dataset based on cryptic crosswords with a cryptical clue. |
| Outcome: | The proposed dataset is based on cryptic crosswords with 523K examples. |
Efficient Long-Text Understanding with Short-Text Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing transformer-based pretrained language models cannot be applied to long sequences due to their quadratic complexity. |
| Approach: | They propose a simple approach to long sequences that re-uses battle-tested short-text pretrained LMs. |
| Outcome: | The proposed approach is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. |
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models can perform unseen tasks by conditioning on a few input-output demonstrations, but task inference is implicit and the ability of models to explicitly reason about it remains unexplored. |
| Approach: | They propose an instruction induction challenge in which a model is asked to generate a natural language instruction that fits a set of labeled examples. |
| Outcome: | The proposed model achieves 65.7% of human performance while the original model only reaches 9.8% of human performances. |
Neural Machine Translation without Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models operate over subword tokens, but byte-based models employ a different approach . a one-hot representation of each byte does not hurt performance, but it improves BLEU scores . |
| Approach: | They propose to represent every computerized text as a sequence of bytes via UTF-8 . this eliminates the need for an embedding layer and improves performance . |
| Outcome: | The proposed model improves BLEU scores on byte-to-byte translation models compared to character-level models . the proposed model does not require an embedding layer and does not drop out of the decoder . |
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)
Copied to clipboard
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
| Challenge: | Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses. |
| Approach: | They propose a new benchmark that places models in context of long texts that require reasoning over long texts. |
| Outcome: | The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods. |
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM. |
| Approach: | They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets . |
| Outcome: | The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks. |
Multilingual Instruction Tuning With Just a Pinch of Multilinguality (2024.findings-acl)
Copied to clipboard
| Challenge: | Using multilingual instruction tuning, large language models can be used to follow instructions in multiple languages . a multilingual model can be tuned on a wide range of languages, yet most datasets are limited to English . |
| Approach: | They investigate how multilinguality during instruction tuning affects instruction-following across languages . they find that only 40 multilingual examples improve multilingual instruction- follow . |
| Outcome: | The results show that multilingual models perform better on multilingual mixtures compared to monolingual models . the results suggest that building multilingual instruction-tuned models can be done with only 2-4 languages . |