GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus primarily on pure graph understanding, lacking a comprehensive evaluation across all graph types and detailed capability definitions. |
| Approach: | They propose a benchmark to evaluate LLMs' graph comprehension and reasoning abilities using a three-tier hierarchical taxonomy and a granular taxonomies. |
| Outcome: | The proposed model includes 11 datasets with 5,140 graphs of varying complexity. |
Similar Papers
A Graph Talks, But Who’s Listening? Rethinking Evaluations for Graph-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for Graph-Language Models (GLMs) do not assess true multimodal integration. |
| Approach: | They propose a benchmark to evaluate multimodal reasoning over graph topology and textual semantics. |
| Outcome: | The proposed benchmarks show that strong performance is achievable using textual or structural features in isolation, bypassing the need for joint reasoning. |
Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on performance benchmarks without fully comparing LLMs to graph learning models. |
| Approach: | They evaluate off-the-shelf and instruction-tuned graph learning models across a variety of scenarios. |
| Outcome: | The proposed models outperform traditional graph learning models in few-shot settings, the authors show . their models out perform models with instruction tuning, and they show excellent generalization and robustness. |
GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated significant capabilities in processing and understanding text data. |
| Approach: | They propose a structure-based instruction-based method to enhance LLM performance on complex graph tasks. |
| Outcome: | The proposed framework outperforms open-source models on graph problem-solving, but the gap is narrowing. |
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
Can LLM Graph Reasoning Generalize beyond Pattern Memorization? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies seek to enhance the graph reasoning capabilities of Large Language Models (LLMs) by specialized instruction tuning. |
| Approach: | They propose to evaluate LLM graph reasoning generalization using in-distribution settings . they propose to use three strategies to improve LLM generalization . |
| Outcome: | The proposed benchmark evaluates LLM graph reasoning generalization with in-distribution settings only . it shows that LLMs struggle to generalize across reasoning and real-world patterns . |
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have improved reasoning abilities but are limited due to limited context length. |
| Approach: | They propose a large graph benchmark dataset and propose four tasks to evaluate LLMs' reasoning abilities. |
| Outcome: | The proposed tasks evaluate the reasoning abilities of LLMs on a large graph benchmark dataset. |
Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing understanding and reasoning abilities in graphbased tasks focus on specific graph types or tasks, posing challenges in designing versatile systems suitable for various tasks and graphs across diverse domains. |
| Approach: | They propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks. |
| Outcome: | Extensive evaluations on 14 LVLMs reveal that LVLs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. |
KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models for complex reasoning tasks on knowledge graphs remains unexplored. |
| Approach: | They propose a multi-purpose framework leveraging large language models for complex reasoning tasks on knowledge graphs. |
| Outcome: | The proposed framework outperforms fully-supervised models in KG-based fact verification and KGQA benchmarks. |
Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for integrating Knowledge Graphs with Large Language Models focus on closed-ended tasks, leaving a gap in evaluating performance on more complex, real-world scenarios. |
| Approach: | They propose a benchmark to evaluate LLMs augmented with KGs in open-ended, real-world question answering settings. |
| Outcome: | The proposed benchmark reflects practical complexities through diverse question types and incorporates metrics to quantify both hallucination rates and reasoning improvements in LLM+KG models. |
CausalGraphBench: a Benchmark for Evaluating Language Models capabilities of Causal Graph discovery (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have expanded their applications into domains not traditionally associated with natural language processing. |
| Approach: | They propose a benchmark to evaluate the ability of large language models to construct Causal Graphs (CGs) they examine various methods for CG discovery and their performance across different graph sizes and complexity levels. |
| Outcome: | The proposed benchmark comprises 35 CGs sourced from publicly available repositories and academic papers. |