Papers by Avia Efrat
LMentry: A Language Model Benchmark of Elementary Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models are evaluated via perplexity or performance on downstream tasks, but these benchmarks are too complex and difficult to inspect. |
| Approach: | They propose a benchmark that focuses on 25 tasks that humans are expected to perform perfectly, such as writing a sentence containing a specific word or identifying which words in a list belong to a certain category. |
| Outcome: | The proposed benchmarks show that large language models are performing better than previous benchmarks. |
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current datasets targeting ambiguity can be solved by a native speaker with relative ease. |
| Approach: | They present a large-scale dataset based on cryptic crosswords with a cryptical clue. |
| Outcome: | The proposed dataset is based on cryptic crosswords with 523K examples. |
A Simple and Effective Model for Answering Multi-span Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for reading comprehension restrict output space to a set of single contiguous spans . multi-span questions are problematic because they require multiple inputs - a task that requires a sequence tagging problem . |
| Approach: | They propose a simple architecture for answering multi-span questions by casting the task as a sequence tagging problem. |
| Outcome: | The proposed model significantly improves performance on span extraction questions from DROP and Quoref by 9.9 and 5.5 EM points respectively. |
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)
Copied to clipboard
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
| Challenge: | Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses. |
| Approach: | They propose a new benchmark that places models in context of long texts that require reasoning over long texts. |
| Outcome: | The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods. |
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM. |
| Approach: | They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets . |
| Outcome: | The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks. |