CANDY: Benchmarking LLMs’ Limitations and Assistive Potential in Chinese Misinformation Fact-Checking (2025.findings-emnlp)
Copied to clipboard
| Challenge: | CANDY is a benchmark to evaluate the capabilities and limitations of large language models (LLMs) for fact-checking misinformation. |
| Approach: | a team of researchers develop a benchmark to evaluate the capabilities and limitations of large language models in fact-checking misinformation in Chinese. |
| Outcome: | CANDY is a benchmark to evaluate the capabilities and limitations of large language models in fact-checking misinformation in China. |
Similar Papers
FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in every natural language processing task, but they are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real world. |
| Approach: | They propose a large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 17 realistic word-level and character-level perturbations and 4 types of subpopulations. |
| Outcome: | The proposed model is brittle to small input changes and exhibits performance variations across different subpopulations. |
Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations (2024.acl-long)
Copied to clipboard
| Challenge: | Currently, many benchmarks evaluate the commonsense reasoning of large language models (LLMs), but most are English-based, limiting non-English evaluations. |
| Approach: | They propose to use Chinese commonsense reasoning to evaluate LLMs' commonsensing ability. |
| Outcome: | The proposed benchmark covers both globally known and Chinese-specific commonsense reasoning abilities and can be used as a reference for future research. |
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |
Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models (2025.acl-long)
Copied to clipboard
Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Large language models have created significant safety concerns . factuality ability is crucial in determining whether they can be deployed and applied safely and compliantly within specific regions. |
| Approach: | They propose a benchmark to evaluate the factuality of large language models in China . they evaluate the models' ability to provide accurate and reliable information . |
| Outcome: | The proposed benchmark evaluates the factuality abilities of existing LLMs and compares them to LLM abilities. |
HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning (2025.findings-acl)
Copied to clipboard
Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin
| Challenge: | Existing studies show that large language models are robust in commonsense reasoning . however, some variations in questions can lead to incorrect responses . |
| Approach: | They propose a large-scale bilingual benchmark consisting of 11,200 cases . they conduct extensive experiments on 41 representative LLMs . |
| Outcome: | The proposed benchmark systematically evaluates the robustness of large language models in commonsense reasoning. |
How does Misinformation Affect Large Language Model Behaviors and Preferences? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have explored the role of Large Language Models in combating misinformation, but there is still a lack of detailed analysis on the specific aspects and extent to which LLMs are influenced by misinformation. |
| Approach: | They propose to use a benchmark to evaluate LLMs' behavior and knowledge preference toward misinformation to identify their models. |
| Outcome: | The proposed approach is based on 10,346,712 pieces of misinformation and examines knowledge conflicts and stylistic variations. |
Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language (2026.findings-acl)
Copied to clipboard
Dianqing Lin, Tian Lan, Jiali Zhu, Jiang Li, Wei Chen, Xu Liu, null Aruukhan, Xiangdong Su, Hongxu Hou, Guanglai Gao
| Challenge: | Current state-of-the-art LLMs exhibit clear limitations on multiple tasks, while performing well on tasks that involve contextual semantic understanding. |
| Approach: | They propose a mouse-based benchmark to evaluate LLMs' performance on NLP tasks involving Chouxiang Language. |
| Outcome: | The proposed benchmark evaluates the performance of LLMs on six NLP tasks involving Chouxiang Language. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models (LMs) generate false or unverifiable content, often known as hallucination, despite ongoing efforts to enhance their factuality. |
| Approach: | They propose a tool that measures LMs’ factuality in real-world user interactions by evaluating their factual accuracy and categorizing content units as Supported, Unsupported, or Undecidable based on Web-retrieved evidence. |
| Outcome: | The proposed evaluation pipeline measures language models’ factuality in real-world user interactions. |