| Challenge: | Recent studies have shed light on the information encoded by long-term memory networks. |
| Approach: | They propose to use a neural caching model to model the role of context in an LSTM LM . they analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped . |
| Outcome: | The proposed model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context, suggesting the distant past is modeled only as a rough semantic field or topic. |
Similar Papers
Do Long-Range Language Models Actually Use Long-Range Context? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. |
| Approach: | They analyze two long-range Transformer language models that accept 8K token inputs . they find that providing long-term context only improves their predictions on a small set of tokens - not sentence-level ones . |
| Outcome: | The proposed model improves on PG-19 with only 2K tokens and does not help at all for sentence-level prediction tasks. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Putting Words in Context: LSTM Language Models and Lexical Ambiguity (P19-1)
Copied to clipboard
| Challenge: | In language, a word can contribute a very different meaning depending on the context . lexical ambiguity involves both morphosyntactic and semantic aspects . |
| Approach: | They propose a method to probe hidden representations for lexical and contextual information about words. |
| Outcome: | The proposed method shows that both types of information are represented to a large extent, but there is room for improvement for contextual information. |
Sorting through the noise: Testing robustness of information processing in pre-trained language models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. |
| Approach: | They examine how robustly pre-trained LMs retain and apply relevant context information in the face of distracting content. |
| Outcome: | The proposed models retain and use critical context information in the face of distracting content, while models are susceptible to factors of semantic similarity and word position. |
Attending to Long-Distance Document Context for Sequence Labeling (2020.findings-emnlp)
Copied to clipboard
| Challenge: | UC Berkeley researchers develop a method for incorporating global context in long documents . many of the main datasets used in NLP are comprised of relatively short documents - english OntoNotes contains 223 tokens . |
| Approach: | They propose a method for incorporating global context in long documents . they use multiple mentions of the same word type to generate a representation for each token . |
| Outcome: | The proposed model performs better at recognizing entities with high TF-IDF scores than parametric models lacking context. |
How much do contextualized representations encode long-range context? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies of contextualized representations focus on short sequences of tens to hundreds of tokens, whereas modern language models handle hundreds of thousands of token in a single context window. |
| Approach: | They use a perturbation setup and a metric to capture contextualization of long-range patterns from the perspective of representation geometry. |
| Outcome: | The proposed model can encode long-range contexts, but it's not fully recurrent, the authors say . their results suggest improvements in existing language models . |
Predicting Reference: What do Language Models Learn about Discourse Models? (2020.emnlp-main)
Copied to clipboard
| Challenge: | a growing literature that probes neural language models to assess their latent acquisition of grammatical knowledge has not investigated their acquisition of discourse modeling ability. |
| Approach: | They draw on a psycholinguistic literature that has established how different contexts affect referential biases concerning who is likely to be referred to next. |
| Outcome: | The proposed models do not resemble human language users, the authors show . their models capture the linguistic knowledge required to perform discourse modeling . |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text (2020.coling-main)
Copied to clipboard
| Challenge: | LSTMs are widely used to capture informative long-term syntactic dependencies, but how they are reflected in their internal vectors for natural text has not been adequately investigated. |
| Approach: | They analyze how syntactic dependencies are reflected in LSTM's internal gates by learning a language model where syntaktic structures are implicitly given. |
| Outcome: | The proposed model can predict whether a word is inside a phrase structure or not from a small number of components of the context-update vector. |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |