| Challenge: | Hourglass Transformers is a computationally efficient model that can be used to reduce the sequence length in the intermediate layers. |
| Approach: | They propose a dynamic-pooling mechanism which predicts segment boundaries in an autoregressive fashion. |
| Outcome: | The proposed model is faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget. |
Similar Papers
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Retrofitting Large Language Models with Dynamic Tokenization (2025.acl-long)
Copied to clipboard
| Challenge: | Current language models use a static tokenizer, which results in degraded efficiency and language capabilities. |
| Approach: | They propose a way to dynamically decide on token boundaries based on input text . they merge frequent subword sequences in a batch and apply a hypernetwork to compute token embeddings . |
| Outcome: | The proposed method reduces token sequence lengths by >20% across 14 languages while promoting fairness across languages. |
FLEXITOKENS: Flexible Tokenization for Evolving Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Widely used subword tokenizers overfragment sequences in unseen domains, languages, and scripts . inefficient tokenizer models can cause overfragments in out-of-distribution domains if not trained properly . |
| Approach: | They propose a byte-level LM with learnable tokenizers to make tokenization adaptive . they propose 'flexitoken' which enables significantly greater flexibility during adaptation . |
| Outcome: | The proposed method significantly reduces token overfragmentation and improves on multilingual benchmarks and domains. |
Reservoir Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen. |
| Approach: | They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. |
| Outcome: | The proposed model improves on translation and language modelling tasks even when some layers are frozen. |
Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work on why Transformer-based large language models make predictions has made their behavior opaque due to the complexity of the computations performed within each layer. |
| Approach: | They propose a linear decomposition of final hidden states from autoregressive language models based on each initial input token, which is exact for virtually all contemporary Transformer architectures. |
| Outcome: | The proposed method analyzes the influence of input tokens on model probabilities over a sequence of upcoming words with only one forward pass from the model. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |
Training Hybrid Language Models by Marginalizing over Segmentations (P19-1)
Copied to clipboard
| Challenge: | Statistical language modeling is the problem of estimating a probability distribution over text data. |
| Approach: | They propose to marginalize over the segmentations efficiently to compute the true probability of a sequence. |
| Outcome: | The proposed model marginalizes over the segmentations to compute the true probability of a sequence on three datasets comprising seven languages. |
HashFormers: Towards Vocabulary-independent Pre-trained Transformers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. |
| Approach: | They propose a family of vocabulary-independent pre-trained transformers that support unlimited vocabulary . they propose to map each token to its corresponding embedding by default . |
| Outcome: | The proposed models are more memory efficient than existing models while achieving comparable performance on multiple text classification tasks. |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction (2023.acl-long)
Copied to clipboard
Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster
| Challenge: | In many NLP tasks, the input text can be seen as a sequence of related segments. |
| Approach: | They propose a layer-adjustable interactions framework that contextualizes token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. |
| Outcome: | The proposed model reduces 30-50% of attention FLOPs while maintaining high accuracy. |