Carpe diem: On the Evaluation of World Knowledge in Lifelong Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Current language models are trained on static data, implying that the encoded knowledge could go wrong as time passes. |
| Approach: | They propose a temporally evolving question-answering benchmark for language models . they use Wikipedia databases to test language models for dynamic knowledge in ever-changing world . |
| Outcome: | The proposed task aims to model the evolution-adaptability of language models in the real world. |
Similar Papers
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022.emnlp-main)
Copied to clipboard
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo
| Challenge: | Language Models (LMs) become outdated as the world changes, a phenomenon called temporal misalignment. |
| Approach: | They propose a lifelong benchmark that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
| Outcome: | The proposed benchmark can be trained on the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based datasets are outdated due to the rapid evolution of knowledge. |
| Approach: | They propose a retrieval-interactive language model framework that evaluates and reflects on its answers for further re-retrieval. |
| Outcome: | The proposed framework performs comparably to or surpasses continuously trained language models. |
Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark (2025.coling-main)
Copied to clipboard
Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Haitao Zheng, Wenlian Lu, Pengjun Xie, Fei Huang
| Challenge: | Recent work has noted that due to the extremely high cost of iterative updates of LLMs, they are often unable to answer dynamic questions well. |
| Approach: | They propose a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest dynamic questions on the Chinese Internet. |
| Outcome: | The proposed benchmark will be one of the key data resources for improving LLMs’ Chinese question-answering ability in the future. |
EvoWiki: Evaluating LLMs on Evolving Knowledge (2025.acl-long)
Copied to clipboard
Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Yong Liao
| Challenge: | Existing knowledge evolution benchmarks are static and fail to capture the evolving nature of LLMs and knowledge. |
| Approach: | They propose an evolving dataset that categorizes information into stable, evolved, and uncharted states. |
| Outcome: | The proposed dataset is auto-updatable and enables evaluation of continuously changing knowledge and newly released LLMs. |
Static Models, Dynamic World: A Unified Perspective on Temporal Perception in Large Language Models (2026.findings-acl)
Copied to clipboard
Chenhao Li, Dandan Song, Changzhi Zhou, Jun Yang, Yuhang Tian, Huipeng Ma, Guangyuan Feng, Luan Zhang, Xudong Li, Ke Duan
| Challenge: | Large language models are trained on static corpora but deployed in a dynamic world . a foundational tension remains between time and the ability to understand it . |
| Approach: | They formalize temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers. |
| Outcome: | The proposed framework formalizes temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers . the framework induces four temporal information regimes corresponding to internal reasoning, answer recency, premise anchoring, and genuine world indeterminacy . |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences. |
| Approach: | They propose a framework which extends established benchmarks to sequential problem-solving settings and provides feedback after each round to build a demonstration memory that the models can query in future tasks. |
| Outcome: | The proposed framework can improve performance of LLMs by learning from past interactions and improve models' performance over time. |
RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to update or supplement large language models struggle under continuous knowledge drift. |
| Approach: | They propose a dynamic event benchmark and time-aware retrieval baseline that captures how knowledge evolves over time. |
| Outcome: | The proposed method enables systematic evaluation of model adaptation under continuous knowledge drift. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Knowledge Mechanisms in Large Language Models: A Survey and Perspective (2024.findings-emnlp)
Copied to clipboard
Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
| Challenge: | Using large language models, we can understand knowledge mechanisms in LLMs for learning, storage, utilization, and evolution. |
| Approach: | They propose to analyze knowledge mechanisms in Large Language Models (LLMs) they examine utilization, evolution, and the potential dark knowledge (hypothesis) they hope to help understand knowledge in LLMs and provide insights for future research . |
| Outcome: | The proposed model can be used to analyze the evolution of parametric knowledge in LLMs. |