| Challenge: | Existing language models that use a large parametric neural network with episodic memory are not efficient. |
| Approach: | They propose a language model that combines a large parametric neural network with a non-parametric episodic memory component in an integrated architecture. |
| Outcome: | The proposed model can predict local context, short-term memory, or long-term memories on an ad hoc basis depending on the context. |
Similar Papers
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Semiparametric language models (LMs) use static storage, which lacks learning capability and is disconnected from the internal information flow of the parametric models. |
| Approach: | They reconceptualize the non-parametric memory represented by kNN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM) this synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks . |
| Outcome: | The proposed model is a learnable Mixture-of-neighbors induction memory (MoNIM) it synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFNs). |
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text (2020.coling-main)
Copied to clipboard
| Challenge: | LSTMs are widely used to capture informative long-term syntactic dependencies, but how they are reflected in their internal vectors for natural text has not been adequately investigated. |
| Approach: | They analyze how syntactic dependencies are reflected in LSTM's internal gates by learning a language model where syntaktic structures are implicitly given. |
| Outcome: | The proposed model can predict whether a word is inside a phrase structure or not from a small number of components of the context-update vector. |
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (P19-1)
Copied to clipboard
| Challenge: | Term memory networks (RNNs) are difficult to optimize due to gradient vanishing and explosion. |
| Approach: | They propose a neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. |
| Outcome: | The proposed method improves state-of-the-art performance on short and long sequences and generates coherent, novel text articles with thousands of tokens. |
Long-Range Language Modeling with Selective Cache (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use transformers to model language cost quadratically increase with sequence length. |
| Approach: | They propose a selective cache which stores key-value pairs from previous contexts. |
| Outcome: | The proposed selective cache outperforms XL cache and compressive cache by considerable margins. |
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Syntactic structures were deemed essential in natural language processing . but since the deep learning revolution, NLP has been dominated by neural models that do not consider syntactical structures in their design. |
| Approach: | They propose a model that models latent representations of words in a sentence . they use a conditional random field to model latent and dependency arcs . |
| Outcome: | The proposed model performs competitively to transformers on small to medium sized datasets. |
Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work on why Transformer-based large language models make predictions has made their behavior opaque due to the complexity of the computations performed within each layer. |
| Approach: | They propose a linear decomposition of final hidden states from autoregressive language models based on each initial input token, which is exact for virtually all contemporary Transformer architectures. |
| Outcome: | The proposed method analyzes the influence of input tokens on model probabilities over a sequence of upcoming words with only one forward pass from the model. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models that memorize past tokens have “flat” memory architectures that restrict the context window. |
| Approach: | They propose a framework that imitates human memorization behavior by preserving tokens from early input segments, passing memory embeddings along the sequence, and recalling relevant information from history. |
| Outcome: | The proposed framework outperforms existing models in language modeling and question-answering tasks and achieves comparable or superior generation quality to long-context models with 2 57 fewer parameters and 2.5 116 less inference memory. |
Revisiting Simple Neural Probabilistic Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have been driven not only by advances in neural architectures, but also through hardware and optimization improvements. |
| Approach: | They revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003) which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. |
| Outcome: | The proposed model performs better on word-level language model benchmarks than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |