Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
| Challenge: | Existing language models are trained on snapshots of data collected at a specific moment in time. |
| Approach: | They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. |
| Outcome: | The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods. |
Similar Papers
Can Language Models Serve as Temporal Knowledge Bases? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have only considered language models as knowledge bases in a static setting . memorizing conflicting information is still challenging for LMs and hinders memorization of other unrelated one-to-one relationships. |
| Approach: | They propose two requirements for treating language models as temporal knowledge bases . they propose a dataset which is aimed at probing temporally-scoped knowledge . |
| Outcome: | The proposed model can store conflicting information and use stored knowledge for temporal knowledge queries. |
Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to effectively retain and reason about temporal information remains limited. |
| Approach: | They propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. |
| Outcome: | The proposed methods improve performance and reduce incorrect outputs. |
Entity Cloze By Date: What LMs Know About Unseen Entities (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing literature provides benchmarks to measure LMs' knowledge about entities . |
| Approach: | They propose a framework to analyze what language models can infer about new entities that did not exist when they were pretrained. |
| Outcome: | The proposed framework shows that models more informed about the entities achieve lower perplexity on this benchmark. |
Are Large Language Model Temporally Grounded? (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent large language models lack a consistent temporal model of textual narratives . sentence ordering in unlabelled texts is only weakly correlated with event ordering . |
| Approach: | They evaluate LLMs with textual narratives and evaluate their common-sense knowledge . they find that LLM models struggle the most with self-consistency . |
| Outcome: | The proposed models lack a consistent temporal model of textual narratives. |
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve neural language models perform poorly on emerging data. |
| Approach: | They propose a lexical-level masking strategy to post-train a neural language model using static data from past years. |
| Outcome: | The proposed method outperforms existing methods on two pre-trained language models, two classification tasks, and four benchmark datasets. |
Pretraining Language Models for Diachronic Linguistic Change Discovery (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as knowledge discovery tools . historical linguistics and literary studies often construct arguments on the basis of distinctions between phenomena like time-period or genre. |
| Approach: | They propose to use LLMs to train large language models over modest historical corpora without allowing contamination from anachronistic data. |
| Outcome: | The proposed model better respects historical divisions and is more computationally efficient compared to the standard approach of fine-tuning an existing LLM. |
Set the Clock: Temporal Alignment of Pretrained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. |
| Approach: | They construct a time-sensitive question dataset and use it to examine temporal alignment methods to align their internal knowledge to a target time. |
| Outcome: | The proposed methods improve LLaMa2's performance by 62% if they are fine tuned to the year 2022 . |
Is Your LLM Outdated? A Deep Look at Temporal Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate large language models are limited due to their inherent dynamic nature and the inherent dynamicity of language and information. |
| Approach: | They introduce a new evaluation framework that employs fresh text and event prediction for assessing LLMs’ temporal adaptability. |
| Outcome: | The proposed framework shows significant temporal biases and a decline in performance over time. |
Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. |
| Approach: | They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens. |
| Outcome: | The proposed method outperforms existing temporal language models on two datasets and three tasks. |
Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information (2025.acl-long)
Copied to clipboard
| Challenge: | Temporal Heads are attention heads that primarily handle temporal knowledge. |
| Approach: | They discover Temporal Heads, specific attention heads that primarily handle temporal knowledge, through circuit analysis. |
| Outcome: | The proposed models can handle temporal knowledge without compromising time-invariant and question-answering performances. |