StructFact: Reasoning Factual Knowledge from Structured Data with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant strides in natural language processing by leveraging their ability to comprehend and reason with factual knowledge. |
| Approach: | They propose a benchmark to evaluate the ability of large language models to reason with structured data for knowledge-intensive tasks. |
| Outcome: | Extensive tests on 10 common LLMs show that they struggle with heterogeneity of structured data during reasoning. |
Similar Papers
StrucText-Eval: Evaluating Large Language Model’s Reasoning Ability in Structure-Rich Text (2025.acl-long)
Copied to clipboard
| Challenge: | Structured data has been central to corporate data strategies for decades . however, with the advancement of large language models (LLMs), there has been a significant shift towards the effective utilization of unstructured data. |
| Approach: | They propose an automatic evaluation data generation method to assess LLMs’ reasoning capabilities on structure-rich text. |
| Outcome: | The proposed method supports 8 structured languages and 29 tasks, generating data with adjustable complexity through controllable nesting and structural width. |
StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs. |
| Approach: | They propose an iterative Reading-then-Reasoning framework to solve question answering tasks based on structured data. |
| Outcome: | The proposed framework improves the reasoning ability of large language models over structured data under the few-shot and zero-shot settings. |
Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced capabilities but produce complex structured data. |
| Approach: | They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs. |
| Outcome: | The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current evaluations for large language models use a single-item assessment paradigm . current evaluations struggle to discern whether a model possesses the required capabilities or merely memorizes/guesses the answers to specific questions. |
| Approach: | They propose a framework to evaluate large language models using atomic test objectives. |
| Outcome: | The proposed evaluation framework resists data contamination and reduces interference of potential biases, and sheds light on the design of future principled and trustworthy LLM evaluation protocols. |
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focuses on accuracy and precision, but factuality evaluation is difficult due to inter-sentence dependencies. |
| Approach: | They introduce a factuality evaluation framework to enhance fact extraction . they also introduce 'factRBench' that evaluates both precision and recall . |
| Outcome: | The proposed framework enhances fact extraction by identifying incomplete and missing facts . it also evaluates precision and recall in long-form models, whereas prior work focuses on precision. |
FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMs (2025.findings-emnlp)
Copied to clipboard
Yingjia Wan, Haochen Tan, Xiao Zhu, Xinyu Zhou, Zhiwei Li, Qingsong Lv, Changxuan Sun, Jiaqi Zeng, Yi Xu, Jianqiao Lu, Yinhong Liu, Zhijiang Guo
| Challenge: | Prior evaluation pipelines fail to evaluate factuality of long-form LLMs due to inefficiency and costly human assessment. |
| Approach: | They propose a fast and strong evaluation pipeline that can evaluate factuality of long-form LLMs . they propose 'faStFact' to reduce cost of web searching and inference calling . |
| Outcome: | The proposed evaluation pipeline achieves highest alignment with human evaluation and efficiency among existing baselines. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models for complex reasoning tasks on knowledge graphs remains unexplored. |
| Approach: | They propose a multi-purpose framework leveraging large language models for complex reasoning tasks on knowledge graphs. |
| Outcome: | The proposed framework outperforms fully-supervised models in KG-based fact verification and KGQA benchmarks. |
Scalable Construction and Reasoning of Massive Knowledge Bases (N18-6)
Copied to clipboard
| Challenge: | Existing knowledge mining systems assume abundant human annotations for training high quality machine learning models, which is impractical when trying to deploy IE systems to a broad range of domains, settings and languages. |
| Approach: | They introduce how to extract structured facts from text corpora to construct knowledge bases. |
| Outcome: | The proposed methods are weakly-supervised and domain-independent for knowledge base construction across various domains. |