Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models encode substantial factual knowledge, yet measuring and systematizing it remains challenging. |
| Approach: | They systematically analyze LLM knowledge materialization using miniGPTKBs . they find high termination rates, though model-dependent, and mixed reproducibility . |
| Outcome: | The proposed model can reliably surface core knowledge, but it has limitations. |
Similar Papers
Enabling LLM Knowledge Analysis via Extensive Materialization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have majorly advanced NLP and AI, and a major success factor is their internalized factual knowledge. |
| Approach: | They propose a method to comprehensively materialize an LLM’s factual knowledge through recursive querying and result consolidation. |
| Outcome: | The proposed method provides constructive insights into the scope and structure of LLM knowledge (or beliefs) it provides scale, accuracy, bias, cutoff and consistency at the same time. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition (2026.findings-acl)
Copied to clipboard
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
| Challenge: | Large language models have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. |
| Approach: | They propose a benchmark for evaluating large language models on a sufficient set of scientific discovery sub-tasks. |
| Outcome: | The proposed framework extracts critical components from papers across 12 disciplines with expert validation confirming its accuracy. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
From Confidence to Collapse in LLM Factual Robustness (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on performance-based metrics, often investigating from the perspective of prompt perturbations, which captures only the externally triggered side of knowledge robustness. |
| Approach: | They propose a method to measure factual robustness from the perspective of the generation process by analyzing token distribution entropy and temperature scaling sensitivity. |
| Outcome: | The proposed method measures factual robustness from the perspective of the generation process and entropy and temperature scaling sensitivity. |
Extracting structure from an LLM - how to improve on surprisal-based models of Human Language Processing (2025.coling-main)
Copied to clipboard
| Challenge: | Existing computational models capture prediction and reanalysis using Large Language Models (LLMs) and a statistical measure known as ‘surprisal’. |
| Approach: | They propose to extract structural information from Large Language Models and a statistical measure known as ‘surprisal’ to integrate it with their learnt statistics. |
| Outcome: | The proposed model achieved higher correlation with human reading times and better predicted the garden path effect and could distinguish between sentence types with different levels of difficulty. |
Domain Regeneration: How well do LLMs match syntactic properties of text domains? (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent improvements in large language models have improved their ability to approximate distributions . authors find that LLMs can suffer from model collapse due to domain considerations based on pretraining . |
| Approach: | They use open source LLMs to regenerate permissively licensed English text from Wikipedia and news text. |
| Outcome: | The proposed model can faithfully match the human-generated distributions in a semantically-controlled setting. |
The Data Frontier for Large Language Models: Selection, Synthesis, and Tools (2026.acl-tutorials)
Copied to clipboard
| Challenge: | acquiring and curating high-quality training data remains a significant bottleneck . acquiring such high-quality data is a key challenge for researchers and practitioners . |
| Approach: | This tutorial provides a comprehensive and practical guide to the state-of-the-art in data research directions for LLMs. |
| Outcome: | The tutorial covers methods for curating the most valuable information from vast, noisy datasets and the synthetic data revolution. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |