Less is More: Attention Supervision with Counterfactuals for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Specifically, we explore the advantage of counterfactual reasoning, over associative reasoning . Adding human supervision to attention has been shown to improve model predictions and explanations . |
| Approach: | They propose to use machine-augmented human attention supervision to enhance model quality. |
| Outcome: | The proposed method is more effective than existing methods requiring higher annotation cost . the proposed method can be trained to generate similar attention to human supervision . |
Similar Papers
Deriving Machine Attention from Human Rationales (D18-1)
Copied to clipboard
| Challenge: | Attention-based models are successful when trained on large amounts of data. |
| Approach: | They propose an approach to map human-annotated rationales to high-performing attention and use this to guide models trained in low-resource scenarios. |
| Outcome: | The proposed model yields over 15% error reduction on benchmark datasets. |
Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention (2020.coling-main)
Copied to clipboard
| Challenge: | SANs are an integral part of successful neural networks such as Transformer . training SAN on a task or pretraining them on language modeling requires large amounts of data and compute resources. |
| Approach: | They propose to modify SANs to enable faster learning, i.e., higher accuracies after fewer update steps. |
| Outcome: | The proposed modifications enable faster learning, i.e., higher accuracies after fewer update steps. |
Be More with Less: Hypergraph Attention Networks for Inductive Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text classification is a critical research topic with broad applications in natural language processing. graph neural networks (GNNs) have received increasing attention but their performance is jeopardized in practice. |
| Approach: | They propose a model which captures long-distance interactions between words and a graph-based model which can be used to perform text classification. |
| Outcome: | The proposed model can achieve more expressive power with less computational consumption on the text classification task. |
NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to produce counterfactuals rely on small perturbations via minimal edits, resulting in simplistic changes. |
| Approach: | They propose a novel approach to produce counterfactuals that allow for larger edits and linguistic diversity while still bearing similarity to the original document. |
| Outcome: | The proposed approach outperforms existing methods for generalizing natural language models under select settings. |
Indirectly Supervised Natural Language Processing (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on indirect supervision addresses challenges in ML for NLP . conventional approaches to NLP use taskspecific labeled examples of a large volume . indirect supervision is useful for a wide range of NLP tasks, but it is not enough for decoders . |
| Approach: | This tutorial aims to address questions about indirect supervision in machine learning . authors discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
| Outcome: | This tutorial aims to answer questions about how to provide supervision for ML tasks . it will discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
All You Need is Attention: Lightweight Attention-based Data Augmentation for Text Classification (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to augment text classification tasks require extensive dataset training. |
| Approach: | They propose a method that uses attention mechanisms to exchange semantically similar words between sentences to generate a greater diversity of synthetic sentences compared to simpler operations like random insertions. |
| Outcome: | The proposed method consistently outperforms baseline methods across diverse text classification conditions. |
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | Prior work on text generation models focused on new architectures for permuted document tasks. |
| Approach: | They propose to use a basic model architecture to improve coherence evaluation of machine generated text. |
| Outcome: | The proposed model improves on a task-independent test set and shows significant improvements in coherence evaluations of downstream tasks. |
Weaker Than You Think: A Critical Look at Weakly Supervised Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Weakly supervised learning is a popular approach for training machine learning models in low-resource settings. |
| Approach: | They propose to use weakly supervised learning to train models with noisy labels from weak sources instead of collecting expensive human annotations. |
| Outcome: | The proposed methods outperform weakly supervised methods on various NLP datasets and tasks on the test sets. |
BERT, are you paying attention? Attention regularization with human-annotated rationales (2026.eacl-long)
Copied to clipboard
| Challenge: | Attention regularisation aims to supervise the attention patterns in language models like BERT. |
| Approach: | They compare regularisation on human rationales with random tokens to find that human-annotated rationale is better at reducing model sensitivity to spurious correlations. |
| Outcome: | The proposed regularisation method improves model performance and model robustness, but not with human-annotated rationales. |
Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers (2021.acl-srw)
Copied to clipboard
| Challenge: | a growing body of research has been focused on what attention heads learn during the pre-training of visual grounded language models. |
| Approach: | They propose to use visual grounding to supervise attention directly to learn visual ground. |
| Outcome: | The proposed method improves the performance of a state-of-the-art visual grounded language model on vision-and-language tasks. |