SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
Similar Papers
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020.acl-main)
Copied to clipboard
| Challenge: | In a recent study, we show that holding-out data can overestimate performance of NLP models. |
| Approach: | They propose a task-agnostic methodology for testing NLP models using a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. |
| Outcome: | The proposed method identifies critical failures in commercial and state-of-the-art models. |
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate extremely long-context LLMs or analyze their limitations. |
| Approach: | They propose a Synthetic, Scalable, Systematic evaluation suite for LLMs using SQL execution. |
| Outcome: | The proposed evaluation suite is able to scale text length and difficulty across scenarios and provides strong correlations with real-world benchmarks. |
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features (2023.acl-long)
Copied to clipboard
| Challenge: | Existing datasets do not allow for a fine-grained cross-lingual evaluation and mainly permit comparisons on a language level. |
| Approach: | They propose a morphologically-aware framework for behavioral testing of NLP models that generates tests in light of specific linguistic features in 12 typologically diverse languages. |
| Outcome: | The proposed framework evaluates state-of-the-art language models on the generated tests. |
SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains (2025.emnlp-demos)
Copied to clipboard
Krithika Ramesh, Daniel Smolyak, Zihao Zhao, Nupoor Gandhi, Ritu Agarwal, Margrét V. Bjarnadóttir, Anjalie Field
| Challenge: | SynthTextEval is a toolkit for conducting comprehensive evaluations of synthetic text. |
| Approach: | They propose a toolkit for conducting comprehensive evaluations of synthetic text using large language models. |
| Outcome: | The proposed toolkit can be run over any dataset, but it is aimed at two high-stakes domains: healthcare and law. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
TestAug: A Framework for Augmenting Capability-based NLP Tests (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on capability-based testing requires the developer to compose each individual test template from scratch. |
| Approach: | They propose a capability-based NLP testing framework that requires the developer to only annotate a few test templates while leveraging the GPT-3 engine to generate the majority of test cases. |
| Outcome: | The proposed framework saves the developer's manual efforts and guarantees the correctness of the generated suites with a validity checker. |
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann
| Challenge: | Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios. |
| Approach: | They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks. |
| Outcome: | The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |