| Challenge: | Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity. |
| Approach: | They propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences. |
| Outcome: | The proposed pipeline scales beyond AI to Physics, Quantitative Biology, and other computational sciences. |
Similar Papers
Pushing the Frontiers of Scientific Fact-Checking: The SCINLP Dataset (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to understand how scientific research evolves, drawing growing interest from the research community. |
| Approach: | They propose a scientific fact-checking dataset, SCINLP, tailored to the NLP domain that verifies the veracity of scientific research questions across varying rationale contexts. |
| Outcome: | The proposed framework examines scientific claims and research focus from a curated collection of influential and reputable NLP papers published between 2000 and 2024. |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. |
| Approach: | They present a dataset of 1.2K expert-verified scientific claims that require compositional reasoning for verification. |
| Outcome: | The proposed model outperforms existing models in table-based pretraining models and large language models. |
SciTrue: Evidence-Grounded Claim Verification in Science (2026.eacl-demo)
Copied to clipboard
| Challenge: | Existing systems often exhibit unverifiable attributions, shallow evidence mapping, and hallucinated citations. |
| Approach: | They propose a claim verification system that provides source-level accountability and evidence traceability. |
| Outcome: | SciTrue outperforms RAG-based baselines in summary traceability, attribution accuracy, and context alignment in a human evaluation of 300 attributions. |
RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension (2026.acl-long)
Copied to clipboard
Yelin Chen, Fanjin Zhang, Suping Sun, Yunhe Pang, Yuanchun Wang, Jian Song, XiaoYan Li, Lei Hou, Shu Zhao, Jie Tang, Juanzi Li
| Challenge: | Existing benchmarks for understanding research papers offer limited fine-grained evaluation at scale. |
| Approach: | They propose a large-scale question-answering benchmark built from review–rebuttal exchanges of high-quality computer science papers. |
| Outcome: | The proposed model is based on human-verified QA pairs and contains 15K questions. |
Unequal Scientific Recognition in the Age of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study evaluates the extent to which popular and frontier LLMs recognize scientists . recognition of scientists remains uneven across gender and geography . |
| Approach: | They evaluate the extent to which popular and frontier LLMs recognize scientists . they compare their outputs against OpenAlex and Wikipedia . |
| Outcome: | The proposed models show that they exhibit selective and inconsistent recognition patterns . women researchers and researchers from Africa, Asia, and Latin America are significantly underrecognized . |
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for fact-checking are based on verifying each piece of a model against potential evidence using an LLM. |
| Approach: | They propose a method that builds small fact-checking models that have GPT-4-level performance but 400x lower cost. |
| Outcome: | The proposed model outperforms other models and reaches GPT-4 accuracy. |
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks such as revising source code to address code reviews. |
| Approach: | They propose a benchmark to evaluate large language models' ability to bridge both technical and conversational contexts by decomposing the generation task of code refinement into three essential reasoning steps. |
| Outcome: | The proposed benchmark exposes specific model weaknesses in code review comprehension disentangled from their generative automated code refinement results. |
The student becomes the master: Outperforming GPT3 on Scientific Factual Error Correction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Factual Claim Correction rely on a verification model to guide the correction process. |
| Approach: | They propose a claim correction system that does not require a verifier but outperforms existing methods by a considerable margin. |
| Outcome: | The proposed system outperforms existing methods by a considerable margin on the SciFact dataset, 77% on SciFACT-Open and 72.75% on the CovidFact data set. |
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data. |
| Approach: | They propose a calibration method that incorporates synthetic data without compromising accuracy. |
| Outcome: | The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks. |