ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation

Challenge:	Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE).
Approach:	They propose a residual block of layers in Transformer that can be described as a higher-order solution to ODE.
Outcome:	The proposed architecture can gain large improvements over strong baselines at a slight cost in inference efficiency.

Similar Papers

Overview

Challenge

Approach

Outcome

A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024.findings-acl)

Copied to clipboard

Xinbo Wu, Lav Varshney

Language Models

Challenge:	Mechanisms of the Transformer architecture for causal language modeling are not well understood.
Approach:	They propose a meta-learning view of the Transformer architecture when trained for a causal language modeling task by explicating an inner optimization process that may happen within the Transformer.
Outcome:	The proposed model is based on a self-attention mechanism and has been widely used in natural language processing, computer vision, and scientific discovery.

RealFormer: Transformer Likes Residual Attention (2021.findings-acl)

Copied to clipboard

Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie

Language Models

Challenge:	Existing techniques to create Residual Attention Layer Transformer networks outperform the canonical Transformer on a wide spectrum of tasks.
Approach:	They propose a technique to create Residual Attention Layer Transformer networks that outperform the canonical Transformer on a wide spectrum of tasks.
Outcome:	The proposed technique outperforms the canonical Transformer on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQUAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP.

Modeling Recurrence for Transformer (N19-1)

Copied to clipboard

Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, Zhaopeng Tu

Language Models

Challenge:	Existing studies show that the lack of recurrence modeling hinders the development of a translation model.
Approach:	They propose to model recurrence for Transformer with an additional recurrent encoder.
Outcome:	The proposed model outperforms the deep model on EnglishGerman and ChineseEnglish translation tasks.

IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method (2025.emnlp-main)

Copied to clipboard

Xinyu Liu, Bei Li, Jiahao Liu, Junhao Ruan, Kechen Jiao, Hongyin Tang, Jingang Wang, Tong Xiao, JingBo Zhu

Green & Sustainable NLP

Challenge:	High-order numerical methods enhance performance in tasks like NLP but introduce a performance-efficiency trade-off due to increased computational overhead.
Approach:	They propose an iterative implicit Euler Transformer which simplifies high-order numerical methods by iterating implicit Eule.
Outcome:	The proposed method improves accuracy and reduces inference overhead by 55% while retaining 99.4% of the original task accuracy.

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models (2021.emnlp-main)

Copied to clipboard

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui

Language Models

Challenge:	Transformer architecture is composed of multi-head attention, which has been extensively analyzed.
Approach:	They extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization.
Outcome:	The proposed method incorporates the whole attention block, i.e., multi-head attention, residual connection, and layer normalization into the analysis.

Learning Deep Transformer Models for Machine Translation (P19-1)

Copied to clipboard

Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, Lidia S. Chao

Language Models

Challenge:	Neural machine translation models have advanced the previous state-of-the-art by learning mappings between sequences via neural networks and attention mechanisms.
Approach:	They propose to use layer normalization to pass the combination of previous layers to the next layer to improve the model.
Outcome:	The proposed model outperforms the shallow Transformer-Big/Base baseline model on English-German and Chinese-English tasks by 0.4-2.4 BLEU points.

Transformer-specific Interpretability (2024.eacl-tutorials)

Copied to clipboard

Hosein Mohebbi, Jaap Jumelet, Michael Hanna, Afra Alishahi, Willem Zuidema

Explainability & Interpretability in NLP

Challenge:	Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque.
Approach:	This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions .
Outcome:	This tutorial aims to show how a new trending approach can be applied to Transformer-based models.

Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021.emnlp-main)

Copied to clipboard

Patrick Kahardipraja, Brielen Madureira, David Schlangen

Language Models

Challenge:	Recent work attempts to apply incremental processing to NLUs but this is computationally expensive and does not scale efficiently for long sequences.
Approach:	They propose to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs.
Outcome:	The proposed model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental quality.

Transformer-Based Direct Hidden Markov Model for Machine Translation (2021.acl-srw)

Copied to clipboard

Weiyue Wang, Zijian Yang, Yingbo Gao, Hermann Ney

Language Models

Challenge:	Recent studies have found that word alignments produced by the multi-head cross-attention weights are poor.
Approach:	They propose to introduce the hidden Markov model to the transformer architecture and introduce alignment components while keeping the system monolithic.
Outcome:	The proposed model outperforms the baseline model but is slower in training and decoding.

Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)

Copied to clipboard

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush

Language Models

Challenge:	Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community.
Approach:	they propose an open-source library that aims to open up advances in machine learning to the wider community.
Outcome:	Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community.