| Challenge: | Adapters are small modules trained on top of a frozen language model to adapt predictions to new target languages. |
| Approach: | They propose to train transformer language adapters on top of a frozen model to adapt predictions to new target languages. |
| Outcome: | The transformer language adapters are trained on top of a frozen model to adapt predictions to new target languages. |
Similar Papers
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Jump to Conclusions: Short-Cutting Transformers with Linear Transformations (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models create hidden representations of inputs at every layer, but only use final-layer representations for prediction. |
| Approach: | They propose a method for casting hidden representations as final representations, bypassing transformer computation in-between. |
| Outcome: | The proposed method produces more accurate predictions from hidden layers across various model scales, architectures, and data distributions. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (D19-1)
Copied to clipboard
| Challenge: | a recent study has shown that deep neural networks are effective with various tasks . a new study examines how representations of tokens evolve between layers under different learning objectives . |
| Approach: | They use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers. |
| Outcome: | The proposed model outperforms untrained models on word identity prediction tasks . the model outpersforms models trained on other linguistic tasks based on the model's objective . |
On the Sub-layer Functionalities of Transformer Decoder (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation have focused on assessing the encoded representations or interpreting the multi-head self-attentions. |
| Approach: | They propose to use Transformer-based encoder-decoder architectures to analyze how information is propagated through each module of each decoder layer. |
| Outcome: | The proposed model can be dropped with minimal loss of performance on three translation datasets and can be used to train and inference faster. |
Learning Deep Transformer Models for Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation models have advanced the previous state-of-the-art by learning mappings between sequences via neural networks and attention mechanisms. |
| Approach: | They propose to use layer normalization to pass the combination of previous layers to the next layer to improve the model. |
| Outcome: | The proposed model outperforms the shallow Transformer-Big/Base baseline model on English-German and Chinese-English tasks by 0.4-2.4 BLEU points. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Analyzing Encoded Concepts in Transformer Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | a new framework to analyze how latent concepts are encoded in representations learned in pre-trained lan-guage models is proposed . conceptX uses clustering to discover the encoded concepts and align them with a large set of human-defined concepts. |
| Approach: | They propose a framework to analyze how latent concepts are encoded in representations learned within pre-trained lan-guage models. |
| Outcome: | The proposed framework explains encoded concepts by aligning with human-defined concepts. |
Explaining How Transformers Use Context to Build Predictions (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing input attributions for a model's prediction are unclear how prior words affect the model' s decision throughout the layers. |
| Approach: | They propose a procedure to analyze models for language generation using the Transformer and a comparison of their results with evidence of the linguistic phenomena. |
| Outcome: | The proposed method consistently aligns better than gradient-based and perturbation-based baselines and generates human-like source-target alignments for building predictions. |