MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models generate erroneous information and evaluations fail to assess factual correctness of models. |
| Approach: | They propose to use MoleculeQA to evaluate molecular factual correctness in large language models by organizing molecules into a taxonomy and building QA pairs through human and LLM efforts. |
| Outcome: | The proposed model improves the factual correctness of generated information and enables the development of new models. |
Similar Papers
PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025.naacl-long)
Copied to clipboard
| Challenge: | a dataset of 579 QA pairs from 208 scientific articles contains answers that reviewers raised while thoroughly examining the scientific article. |
| Approach: | They propose a dataset that contains questions that reviewers raised while thoroughly examining the scientific article. |
| Outcome: | The proposed dataset contains 579 QA pairs from 208 academic articles . the results show that decontextualization approaches improve retrieval performance . |
A Survey of Large Language Models for Text-Guided Molecular Discovery: From Molecule Generation to Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are introducing a paradigm shift in molecular discovery by enabling text-guided interaction with chemical spaces through natural language and symbolic notations. |
| Approach: | They analyze the current LLM learning paradigms to tackle four critical evaluation dimensions that have emerged as critical dimensions in recent studies. |
| Outcome: | The proposed models are able to interact with chemical spaces through natural language and symbolic notations, and have emerging extensions to incorporate multi-modal inputs. |
Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models (2025.acl-long)
Copied to clipboard
Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Dekai Sun, Zhicheng Zheng, Wenbo Su, Bo Zheng
| Challenge: | Current frontier models sometimes generate false outputs or answers that are not substantiated by evidence. |
| Approach: | They propose Chinese SimpleQA, a Chinese benchmark to evaluate LLMs' factuality . they focus on Chinese language over 6 major topics with 99 diverse subtopics . |
| Outcome: | The Chinese SimpleQA benchmark evaluates the factuality ability of LLMs . the questions and answers are short and easy-to-evaluate . |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions. |
| Approach: | They propose a benchmark to evaluate general-purpose LLMs on sequence-based genome inference tasks. |
| Outcome: | The proposed benchmark outperforms baseline models on sequence-based genome inference tasks. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models (2025.acl-long)
Copied to clipboard
Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Large language models have created significant safety concerns . factuality ability is crucial in determining whether they can be deployed and applied safely and compliantly within specific regions. |
| Approach: | They propose a benchmark to evaluate the factuality of large language models in China . they evaluate the models' ability to provide accurate and reliable information . |
| Outcome: | The proposed benchmark evaluates the factuality abilities of existing LLMs and compares them to LLM abilities. |
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing question answering datasets lack numerical reasoning and reasoning processes . current research on numerical reasoning focuses on simple calculations . |
| Approach: | They propose a conversational and bilingual question answering dataset with numerical reasoning with compound mathematical expressions. |
| Outcome: | The proposed model achieves 55.5 exact match scores while human performance is 89.7. |
Less for More: Enhanced Feedback-aligned Mixed LLMs for Molecule Caption Generation and Fine-Grained NLI Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent trends have led to the use of multimodal models to learn molecular and linguistic representations, either in separate but coordinated spaces or in a common space. |
| Approach: | They propose a novel atomic-level evaluation method leveraging off-the-shelf Natural Language Inference (NLI) models for use in the unseen chemical domain. |
| Outcome: | The proposed method surpasses state-of-the-art models in the unseen chemical domain while relying on a granularity-based evaluation method. |
Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | LLMs generate responses following user's instructions, which requires high dialogue comprehension ability. |
| Approach: | They propose to evaluate LLMs' dialogue comprehension ability using a dialogue summarization task to derive factual questions from the generated summaries and use them as a more flexible measurement of dialogue comprehension. |
| Outcome: | The proposed model reduces the error rate by 11% on the dialogue summarization task. |