Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events (2021.naacl-main)
Copied to clipboard
| Challenge: | Using transformer-based language models to track entities is challenging due to dynamic nature of the world described in the text. |
| Approach: | They propose to use transformer-based language models to track entities throughout a procedure . they propose to introduce timestamp encoding to encode event information in LMs . |
| Outcome: | The proposed model improves on the state-of-the-art model with a 3.1% increase in F1 score on the Propara dataset and better results on the location prediction task on the NPN-Cooking dataset. |
Similar Papers
Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. |
| Approach: | They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens. |
| Outcome: | The proposed method outperforms existing temporal language models on two datasets and three tasks. |
Exploring Contextualized Neural Language Models for Temporal Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that deep contextualized language models (LMs) can extract temporal relations between events and time expressions. |
| Approach: | They propose a temporal relation extraction technique which extracts temporal relations between events and time expressions. |
| Outcome: | The proposed method significantly improves temporal dependency parsing, the authors show . their work compares the proposed method to other methods and shows where they may fail . |
Transformers for Tabular Data Representation: A Survey of Models and Applications (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent research efforts extend LMs by developing neural representations for structured data. |
| Approach: | They propose to extend transformer-based language models to tabular data by analyzing inputs, model training, and supported downstream tasks. |
| Outcome: | The proposed models are compared against existing models and are based on a traditional pipeline. |
A Model of the Language Process (2026.acl-long)
Copied to clipboard
| Challenge: | Language is a process that changes over time as new vocabulary emerges, word meanings shift, and narratives progress. |
| Approach: | They introduce a BERT style transformer encoder that models language by jointly learning to predict document contents and classify document publication dates. |
| Outcome: | The proposed model can predict document contents and classify document publication dates and accurately detects changes in word meanings. |
Time-Aware Language Models as Temporal Knowledge Bases (2022.tacl-1)
Copied to clipboard
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
| Challenge: | Existing language models are trained on snapshots of data collected at a specific moment in time. |
| Approach: | They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. |
| Outcome: | The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods. |
VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in interpretability suggests we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary space, a transformation that makes them more human interpretable. |
| Approach: | They propose a tool to visualize a forward pass of Generative Pre-trained Transformers as an interactive flow graph with nodes representing neurons or hidden states and edges representing interactions between them. |
| Outcome: | The proposed visualization simplifies huge amounts of data into easy-to-read graphs that can reflect the models’ internal processing, uncovering the contribution of each component to the models' final prediction. |
Temporal Attention for Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models are trained on corpora derived from the web, but ignore this information. |
| Approach: | They propose a time-aware self-attention mechanism that captures time-specific contextualized word representations and allows the transformer to capture this information. |
| Outcome: | The proposed model achieves state-of-the-art on three datasets in different languages (English, German, and Latin) that vary in time, size, and genre. |
Inferring Events from Time Series using Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work on reasoning about time series in conjunction with natural language has largely overlooked event descriptions and focused on tasks involving just numeric data like trend analysis or anomaly detection. |
| Approach: | They propose a method for generating tasks that test a model’s ability to reason about events associated with time series data based on sports data and develop a benchmarking method. |
| Outcome: | The proposed method can infer unobserved events from time series data, even when providing minimal context. |
Effective Use of Transformer Networks for Entity Tracking (D19-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models for entity-related tasks are not able to handle the nuances of procedural text. |
| Approach: | They propose to use pre-trained transformer networks to track entities in procedural text by restructuring input to focus on a particular entity. |
| Outcome: | The proposed models outperform baseline models on ingredient detection in recipes and QA over scientific processes on two different tasks. |
Neural Language Modeling for Contextualized Temporal Graph Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for temporal reasoning have been used for a number of applications, but their potential for tempor reasoning over event graphs has not been explored. |
| Approach: | They propose to use large-scale pre-trained language models to generate an event-level temporal graph from a document using existing IE/NLP tools. |
| Outcome: | The proposed method outperforms the closest existing method on several metrics on a hand-labeled, out-of-domain corpus. |