Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, Tao Gui
| Challenge: | Existing methods to train language models rely on manual design, perplexity, or careful prompt engineering. |
| Approach: | They propose a method that automatically mines criteria from human preferences for data quality with only 30 human-annotated pairs and performs efficient data selection. |
| Outcome: | The proposed method improves on human-annotated test sets and shows high accuracy on code, math, and logic domains. |
Similar Papers
AutoQual: An LLM Agent for Automated Discovery of Interpretable Features for Review Quality Assessment (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for assessing review quality are unscalable across domains and fail to adapt to evolving content patterns. |
| Approach: | They propose an LLM-based agent framework that automates the discovery of interpretable features. |
| Outcome: | The proposed framework improves on a large-scale online platform with a billion-level user base. |
Rolling Out Data Quality Overnight, without losing the plot: A Multi-Agent System for Speech Data Quality Management (2026.findings-acl)
Copied to clipboard
Rishabh Kumar, Abhinav Painuli, Chriss Philip Saji, Devesh Soni, Amrith Krishna, Ganesh Ramakrishnan
| Challenge: | Using automation to improve quality management is expensive and resource-intensive for speech datasets. |
| Approach: | They propose a natural language-driven agentic framework that compiles user requirements into dependency-aware DAG workflows over modular tools for audio, transcript, and metadata verification. |
| Outcome: | The proposed framework achieves 80-90% agreement with expert verification while requiring less than 20% of the cost and time of manual QC. |
Automatic Argument Quality Assessment - New Datasets and Methods (D19-1)
Copied to clipboard
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, Noam Slonim
| Challenge: | 6.3k arguments were collected from contributors of various levels, and are released as part of this work. |
| Approach: | They propose to use a language model to annotate arguments for argument ranking and argument-pair classification. |
| Outcome: | The proposed methods outperform state-of-the-art methods in the argument ranking task and argument-pair classification task. |
Preference Optimization for Review Question Generation Improves Writing Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Peer reviewers are overloaded and face tight deadlines, leading some to rely on large language models (LLMs) to draft questions and comments. |
| Approach: | They use open-review review datasets to train a human preference model based on human reviewer questions . human evaluations show IntelliAsk generates more grounded, substantive and effortful questions than strong baselines . |
| Outcome: | The proposed model predicts reviewer-question quality better than API-based SFT baselines and provides scalable evaluation. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
FIRE: Flexible Integration of Data Quality Ratings for Effective Pretraining (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate data quality rely on heuristic techniques or single quality signals. |
| Approach: | They propose a framework for integrating multiple data quality raters that integrates multiple quality signals into a unified space and provides a comprehensive quality signal for each data point. |
| Outcome: | The proposed framework outperforms existing methods and boosts model performance across a wide range of downstream tasks while requiring less than 37.5% tokens to reach the target performance. |
COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values (2026.findings-eacl)
Copied to clipboard
Siwei Wu, JinCheng Ren, Xeron Du, Shuyue Guo, Xingwei Qu, Yiming Liang, Jie Liu, Yunwen Li, Tyler Loakman, Tianyu Zheng, Boyu Feng, Huaqing Yuan, Zili Wang, Jiaheng Liu, Wenhao Huang, Chenglin Cai, Haoran Que, Jian Yang, Yuelin Bai, Zekun Moore Wang, Zhouliang Yu, Qunshu Lin, Ding Pan, Yuchen Eleanor Jiang, Tiannan Wang, Wangchunshu Zhou, Shenzhi Wang, Xingyuan Bu, Minghao Liu, Guoyin Wang, Ge Zhang, Chenghua Lin
| Challenge: | Existing Chinese preference datasets suffer from limited scale, restricted domain coverage, and insufficiently rigorous data validation. |
| Approach: | They propose an LLM-based data annotation pipeline with no human intervention to annotate Chinese preference datasets. |
| Outcome: | The proposed pipeline outperforms existing Chinese preference datasets on AlignBench and Chinese Reward Benchmark. |
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on summarization evaluation without a human-written reference summary have shown high correlations with human ratings. |
| Approach: | They propose to judge summary quality by learning preference rank from corrupted summaries . they use Bradley-Terry power ranking model to learn preference rank . |
| Outcome: | Experiments on several datasets show that the proposed model can produce scores highly correlated with human ratings. |
SELECting over Tokens: Curating Pre-training Data at Scale via Token Classification (2026.acl-long)
Copied to clipboard
Xin Tong, Weidong Zhang, Jiaang Li, Haibin Chen, Shilei Liu, Langming Liu, Kangtao Lv, Yujin Yuan, Wenbo Su, Bo Zheng
| Challenge: | Existing pipelines rely on expert-crafted heuristic rules, which lack content-aware, fine-grained noise detection. |
| Approach: | They propose a framework that reframes data refinement as a highly efficient token classification task. |
| Outcome: | The proposed framework outperforms existing pipelines on benchmarks and is 2.5x faster at inference. |
Dissecting Human and LLM Preferences (2024.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that human and Large Language Model preferences are important for model fine-tuning and evaluation. |
| Approach: | They dissect the preferences of human and 32 different Large Language Models to understand their quantitative composition. |
| Outcome: | The proposed model is compared with 32 different large language models using real-world user-model conversations. |