| Challenge: | Pretrained language models are trained on corpora derived from the web, but ignore this information. |
| Approach: | They propose a time-aware self-attention mechanism that captures time-specific contextualized word representations and allows the transformer to capture this information. |
| Outcome: | The proposed model achieves state-of-the-art on three datasets in different languages (English, German, and Latin) that vary in time, size, and genre. |
Similar Papers
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Improving BERT with Syntax-aware Local Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that attention-based models benefit from more focused attention over local regions. |
| Approach: | They propose a syntax-aware local attention which restrains attention over syntactically relevant words. |
| Outcome: | The proposed model performs better on all benchmark datasets, including sentence classification and sequence labeling tasks. |
Exploring Contextualized Neural Language Models for Temporal Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that deep contextualized language models (LMs) can extract temporal relations between events and time expressions. |
| Approach: | They propose a temporal relation extraction technique which extracts temporal relations between events and time expressions. |
| Outcome: | The proposed method significantly improves temporal dependency parsing, the authors show . their work compares the proposed method to other methods and shows where they may fail . |
Time-Aware Language Models as Temporal Knowledge Bases (2022.tacl-1)
Copied to clipboard
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
| Challenge: | Existing language models are trained on snapshots of data collected at a specific moment in time. |
| Approach: | They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. |
| Outcome: | The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Characterizing the Expressivity of Local Attention in Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that global and local attention are expressively complementary. |
| Approach: | They propose to restrict global attention to a fixed-size window of preceding tokens . they also propose to add local attention to local-only transformers to increase model quality . |
| Outcome: | The proposed model outperforms the global–local transformers on formal language recognition and natural language modeling. |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for entity representations do not capture information in a knowledge base, and cannot represent entities that do not exist in the KB. |
| Approach: | They propose a pretrained contextualized representation of words and entities based on the bidirectional transformer. |
| Outcome: | The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Temporal Effects on Pre-trained Models for Language Processing Tasks (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study shows that language models can be improved as time passes . a number of approaches to solving language tasks have evolved rapidly without a model . |
| Approach: | They examine temporal effects on model performance on downstream language tasks . they also examine the efficacy of two approaches for temporal domain adaptation without human annotations . |
| Outcome: | The proposed methods improve self-labeling and named entity recognition on new data. |