Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, Chris Alberti
| Challenge: | Experts in various fields perform methodical writing tasks to plan, organize, and report their work. |
| Approach: | They propose a benchmark with specifications for 519 methodical writing tasks . they use expert revisions of up to 10 model-generated examples to evaluate contemporary language models. |
| Outcome: | The proposed benchmark includes specifications for 519 methodical writing tasks . it includes examples with input and output examples, and is available at https://dolomites-benchmark.github.io/ . |
Similar Papers
LMentry: A Language Model Benchmark of Elementary Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models are evaluated via perplexity or performance on downstream tasks, but these benchmarks are too complex and difficult to inspect. |
| Approach: | They propose a benchmark that focuses on 25 tasks that humans are expected to perform perfectly, such as writing a sentence containing a specific word or identifying which words in a list belong to a certain category. |
| Outcome: | The proposed benchmarks show that large language models are performing better than previous benchmarks. |
CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation (2025.acl-industry)
Copied to clipboard
| Challenge: | CodeIF assesses the ability of large language models to adhere to task-oriented instructions in code generation tasks. |
| Approach: | They introduce a benchmark designed to assess LLMs' ability to adhere to task-oriented instructions within diverse code generation scenarios. |
| Outcome: | The proposed benchmark assesses LLMs' ability to adhere to task-oriented instructions in code generation tasks across a wide range of complexity levels and programming domains. |
Academics Can Contribute to Domain-Specialized Language Models (2024.emnlp-main)
Copied to clipboard
Mark Dredze, Genta Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David Rosenberg, Sebastian Gehrmann
| Challenge: | Commercially available models dominate academic leaderboards, focusing on creating and adapting general-purpose models . however, general- purpose models often underperform in specialized domains, and domain-specific models yield superior results. |
| Approach: | They advocate for a renewed focus on developing and evaluating domain- and task-specific models . they advocate for an adapted or adapted model that can be used to improve academic leaderboard standings . |
| Outcome: | The proposed model can do well on professional and linguistic examinations, college-level knowledge questions, and collections of reasoning tasks. |
LLMCrit: Teaching Large Language Models to Use Criteria (2024.findings-acl)
Copied to clipboard
| Challenge: | Current research on using criteria to provide feedback on tasks is limited . a general framework that can be used to teach large language models to use criteria is lacking . |
| Approach: | They propose a framework that enables large language models to use criteria for feedback . criteria are extracted from guidelines and construct in-context demonstrations for each criterion . |
| Outcome: | The proposed framework can be used to provide natural language feedback on tasks. |
LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability (2025.findings-emnlp)
Copied to clipboard
Zikai Xiao, Fei Huang, Jianhong Tu, Jianhui Wei, Wen Ma, Yuxuan Zhou, Jian Wu, Bowen Yu, Zuozhu Liu, Junyang Lin
| Challenge: | Existing benchmarks for long-form generation assess real-world queries with hard-to-verify metrics or use synthetic setups that overlook real-life intricacies. |
| Approach: | They propose a new approach that balances verifiable and real-world assessment with Target-Anchored Evaluation. |
| Outcome: | The proposed model balances real-world and verifiable assessment with Target-Anchored Evaluation (TAE) it generates queries, textual materials, and anchors based on verifier targets within real-life scenarios . |
ListOps: A Diagnostic Dataset for Latent Tree Learning (N18-4)
Copied to clipboard
| Challenge: | Existing work on latent tree learning models shows they do not learn plausible grammars . a dataset is created to study the parsing ability of such models in natural language . |
| Approach: | They propose a toy dataset to study the parsing ability of latent tree learning models . they propose 'listops' toy that has a single correct parse strategy that a system needs to learn . |
| Outcome: | The proposed model outperforms existing models on sentence understanding tasks . it can learn grammars that conform to plausible semantics and syntactic formalisms . |
Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for data-to-text generation are saturated, and there is no way to test them. |
| Approach: | They propose a tool for collecting structured data from public APIs to analyze the behavior of open large language models on the task of data-to-text generation. |
| Outcome: | The proposed model can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. |
Systematic Task Exploration with LLMs: A Study in Citation Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) provide unprecedented flexibility in defining and executing complex, creative natural language generation tasks. |
| Approach: | They propose a framework that consists of input manipulation, reference data, and output measurement to explore citation text generation. |
| Outcome: | The proposed framework explores citation text generation, a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm. |
tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Several initiatives release harmonized datasets or provide harmonization codes to preprocess datasets into a consistent format. |
| Approach: | They propose an annotation framework that enables concise, readable, and reusable annotations. |
| Outcome: | The proposed framework outperforms all publicly available text encoders on all tasks. |
LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist . |
| Approach: | This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision. |
| Outcome: | This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario . |