Rationalizing Transformer Predictions via End-To-End Differentiable Self-Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks are increasingly prevalent across a wide range of applications, driving significant advancements in fields such as natural language processing, computer vision, and beyond. |
| Approach: | They propose an end-to-end differentiable training paradigm for stable training of a rationalized transformer classifier. |
| Outcome: | The proposed model is capable of classifying a sample and scoring input tokens without any explicit supervision and produces class-wise rationales without instabilities. |
Similar Papers
Effective Pretraining Objectives for Transformer-based Autoencoders (2022.findings-emnlp)
Copied to clipboard
| Challenge: | ELECTRA is more accurate than BERT, but it is not clear if this is due to its innovative architecture or to the long and extensive training, which highly increases the computation cost for obtaining the final language model. |
| Approach: | They propose to replace BERT’s Masked Language Modeling objective (MLM) with Token Detection (TD) by using a statistical approach to generate light tokens. |
| Outcome: | The proposed method can replace ELECTRA's computationally heavy generators without a significant drop in performance. |
Understanding the Difficulty of Training Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Admin (Adaptive model initialization) is more stable, converges faster, and leads to better performance. |
| Approach: | They propose a model initialization algorithm to stabilize early training and unleash its full potential in the late stage. |
| Outcome: | The proposed model initialization method stabilizes early training and unleashes full potential in late stage. |
Transformer-specific Interpretability (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque. |
| Approach: | This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions . |
| Outcome: | This tutorial aims to show how a new trending approach can be applied to Transformer-based models. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
Consistency Regularization Training for Compositional Generalization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing neural models have difficulty generalizing to unseen combinations of seen components. |
| Approach: | They propose to improve the capability of Transformer on compositional generalization by consistency regularization training without modifying model architectures. |
| Outcome: | The proposed model performs well on semantic parsing and machine translation benchmarks. |
Boosting Summarization with Normalizing Flows and Aggressive Training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that FlowSUM improves the quality of generated summaries with minimal impact on inference time. |
| Approach: | They propose a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. |
| Outcome: | The proposed model improves the quality of generated summaries and reduces inference time. |
Exploring End-to-End Differentiable Natural Logic Modeling (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to integrate natural logic with neural networks are brittle and prone to fail in the presence of noise and uncertainty. |
| Approach: | They propose to integrate natural logic with neural networks to create differentiable models that integrate natural reasoning with subsymbolic vector representations and neural components. |
| Outcome: | The proposed model can model monotonicity-based reasoning, compared to baseline models without inductive bias. |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Fine-grained Contrastive Learning for Definition Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent pre-trained transformer-based definition generation models lack effective representation learning to contain full semantic components of the given word, leading to under-specific definitions. |
| Approach: | They propose a novel contrastive learning method that encourages the model to capture more detailed semantic representations from the definition sequence encoding. |
| Outcome: | The proposed method could generate more specific definitions compared with state-of-the-art models. |
Interpretable Neural Predictions with Differentiable Binary Variables (P19-1)
Copied to clipboard
| Challenge: | Neural networks are bringing incredible performance gains on text classification tasks, but they also require interpretability. |
| Approach: | They propose a latent model that selects a rationale and a classifier that learns from the words in the rationale alone. |
| Outcome: | The proposed model can predict expected value of penalties without REINFORCE and can be directly optimised towards a pre-specified text selection rate. |