| Challenge: | a toolkit for evaluating the quality of universal sentence representations is available for download and preprocessing . word embeddings are not trained to perform well on one specific task, but their value lies in their transferability . evaluation of general-purpose word and sentence embeddables has been problematic . |
| Approach: | They propose a toolkit to evaluate the quality of universal sentence representations. |
| Outcome: | The proposed toolkit includes scripts to download and preprocess datasets and an easy interface to evaluate sentence encoders. |
Similar Papers
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (D19-1)
Copied to clipboard
| Challenge: | Prior work on pretrained sentence embeddings and benchmarks focused on the capabilities of stand-alone sentences. |
| Approach: | They propose a test suite of tasks to evaluate whether sentence representations include broader context information. |
| Outcome: | The proposed training objectives help to encode different aspects of information in document structures. |
Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations (N18-4)
Copied to clipboard
| Challenge: | Sentence representations can capture information that cannot be captured by local features based on character or word Ngrams. |
| Approach: | They propose a supervised regression model using universal sentence representations capable of capturing information that cannot be captured by local features based on character or word Ngrams. |
| Outcome: | The proposed model achieves state-of-the-art performance with only sentence representation features . |
DocAMR: Multi-Sentence AMR Representation and Evaluation (2022.naacl-main)
Copied to clipboard
Tahira Naseem, Austin Blodgett, Sadhana Kumaravel, Tim O’Gorman, Young-Suk Lee, Jeffrey Flanigan, Ramón Astudillo, Radu Florian, Salim Roukos, Nathan Schneider
| Challenge: | Abstract Meaning Representation (AMR) graphs are compared to gold graphs by the Smatch metric, but lack a well-defined representation and evaluation. |
| Approach: | They propose an algorithm for deriving a unified graph representation using a super-sentential annotation method. |
| Outcome: | The proposed algorithm avoids the pitfalls of over-merging and lacks coherence from under merging. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024.findings-acl)
Copied to clipboard
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, Mengfei Yang
| Challenge: | Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs). |
| Approach: | They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories. |
| Outcome: | The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks. |
UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs (2024.acl-demos)
Copied to clipboard
Chaoqun He, Renjie Luo, Shengding Hu, Ranchi Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation platforms are complex and poorly modularized, hindering seamless incorporation into researcher’s workflows. |
| Approach: | They propose a lightweight evaluation framework characterized by lightweight, comprehensiveness, modularity, and efficiency that integrates models, data, and metrics into a unified evaluation workflow. |
| Outcome: | The proposed evaluation framework is lightweight, comprehensive, modular, and efficient. |
Evaluation Benchmarks for Spanish Sentence Representations (2022.lrec-1)
Copied to clipboard
Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, Alvaro Soto
| Challenge: | Existing and newly constructed datasets address different tasks from various domains. |
| Approach: | They propose to use Spanish SentEval and Spanish DiscoEval to evaluate stand-alone and discourse-aware sentence representations. |
| Outcome: | The proposed benchmarks evaluate the capabilities of stand-alone and discourse-aware sentence representations in Spanish and show that they are more robust and comparable than previous benchmarks. |
NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark (2025.findings-acl)
Copied to clipboard
Vladislav Mikhailov, Tita Enstad, David Samuel, Hans Christian Farsethås, Andrey Kutuzov, Erik Velldal, Lilja Øvrelid
| Challenge: | NorEval is a new evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). |
| Approach: | They propose a new evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs) NorEval consists of 24 high-quality human-created datasets, of which five are created from scratch. |
| Outcome: | The evaluation framework and materials are publicly available. |
FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models (2024.emnlp-demo)
Copied to clipboard
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, Shikun Zhang
| Challenge: | Large language models (LLMs) have revolutionized natural language processing with impressive performance across various tasks. |
| Approach: | They propose a framework for automated evaluations of large language models . they open-source their code at https://github.com/WisdomShell/FreeEval . |
| Outcome: | The framework is open-source and can be used to develop and validate new evaluation methods. |
SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | a new benchmark for multilingual foundation models is being developed . brittleness of foundation models in the dimensions of semantics and multilinguality is a key limitation . |
| Approach: | They propose a benchmark for multilingual foundation models, SeaEval . they examine how well these models comprehend cultural practices, nuances, and values . |
| Outcome: | The proposed model can be used to evaluate multilingual and multicultural scenarios. |