| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Similar Papers
Do Long-Range Language Models Actually Use Long-Range Context? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. |
| Approach: | They analyze two long-range Transformer language models that accept 8K token inputs . they find that providing long-term context only improves their predictions on a small set of tokens - not sentence-level ones . |
| Outcome: | The proposed model improves on PG-19 with only 2K tokens and does not help at all for sentence-level prediction tasks. |
Improving the Transformer Translation Model with Document-Level Context (D18-1)
Copied to clipboard
| Challenge: | Existing models for document-level context translation ignore documentlevel context. |
| Approach: | They propose a document-level context encoder to represent document- level context and integrate it into the Transformer model. |
| Outcome: | Experiments on NIST Chinese-English and IWSLT French-English datasets show that the proposed translation model outperforms the Transformer model significantly. |
Explaining How Transformers Use Context to Build Predictions (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing input attributions for a model's prediction are unclear how prior words affect the model' s decision throughout the layers. |
| Approach: | They propose a procedure to analyze models for language generation using the Transformer and a comparison of their results with evidence of the linguistic phenomena. |
| Outcome: | The proposed method consistently aligns better than gradient-based and perturbation-based baselines and generates human-like source-target alignments for building predictions. |
Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. |
| Approach: | They propose to examine LLMs' long-context generalizations by probing their hidden representations. |
| Outcome: | The proposed models excel at processing extended contexts while preserving their positional bias. |
Do Transformer Modifications Transfer Across Implementations and Applications? (2021.emnlp-main)
Copied to clipboard
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
| Challenge: | Currently, the Transformer is the de facto architecture of choice for processing sequential data. |
| Approach: | They evaluate the Transformer architecture and its modifications in a shared experimental setting . they conjecture that performance improvements may strongly depend on implementation details . |
| Outcome: | The proposed improvements do not significantly improve performance, the authors find . the proposed improvements are either developed in the same codebase or are minor changes . |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Revisiting Context Choices for Context-aware Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work has cast doubt on whether context-aware machine translation models learn useful signals from context or are improvements in automatic evaluation metrics just a side-effect. |
| Approach: | They propose to use separate encoders for source sentence and context as multiple sources for one target sentence to train context-aware machine translation models. |
| Outcome: | The proposed model improves translation quality even with empty lines as context, but the correct context improves it and random out-of-domain context degrades it. |
Developmental Negation Processing in Transformer Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012). |
| Approach: | They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories. |
| Outcome: | The proposed models perform better on certain categories, suggesting clear differences in how they are processed. |
Structural Guidance for Transformer Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer language models have proven remarkably successful in learning generic transferable linguistic representations without resorting to data intensive pre-training. |
| Approach: | They propose to combine a generative parsing and a structural scaffolding idea to guide the model's representation via additional structure loss that separates the incremental constituency parse. |
| Outcome: | The proposed models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments, and provide useful linguistic representations that benefit a wide range of downstream tasks. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |