TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches for incremental processing use RNNs or Transformers, which consume whole sequences and are by nature non-incremental. |
| Approach: | They propose a two-pass model for AdaPtIve Revision to obtain an incremental supervision signal for learning an adaptive revision policy. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to restart-incremental Transformers while showing little degradation on full sequences. |
Similar Papers
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to apply incremental processing to NLUs but this is computationally expensive and does not scale efficiently for long sequences. |
| Approach: | They propose to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental quality. |
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Learned Incremental Representations for Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | a new syntactic representation that commits to syntakic choices is proposed for humans . we use a system that uses only incremental processing of a prefix to predict the word in a sentence . |
| Approach: | They propose a syntactic representation that commits to syntakic choices incrementally . they say the system can achieve 93.72 F1 on the Penn Treebank with as few as 5 bits per word . |
| Outcome: | The proposed representation achieves 93.72 F1 on the Penn Treebank with as few as 5 bits per word . the analysis of the representations shows they have interpretable features and deferred resolution of syntactic ambiguities. |
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024.acl-long)
Copied to clipboard
| Challenge: | In incremental models, one interpretation is possible, but models that can revise can do so if the ambiguity is resolved. |
| Approach: | They propose an interpretable way to analyse incremental states in a bidirectional way . they propose to use a model that can update internal states to reflect the garden path effect . |
| Outcome: | The proposed model shows that it can perform revisions and recover if the label is incorrect. |
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing deep bidirectional language models are limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. |
| Approach: | They propose a deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA) it computes contextual language representations without repetition and shows competitive or even better accuracies than BERT . |
| Outcome: | The proposed model performs six times faster on a reranking task and twelve times faster in a semantic similarity task. |
AMR Parsing via Graph-Sequence Iterative Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsing is a broad-coverage semantic formalism that encodes the meaning of a sentence as a rooted, directed, labeled graph. |
| Approach: | They propose a model that treats AMR parsing as a series of dual decisions on the input sequence and the incrementally constructed graph. |
| Outcome: | The proposed model outperforms existing models by large margins on both input sequence and output graph. |
Active2 Learning: Actively reducing redundancies in Active Learning methods for Sequence Tagging and Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to deep learning for NLP require large amounts of labeled data. |
| Approach: | They propose an approach that iteratively selects a small number of examples for expert annotation based on their estimated utility in training the model. |
| Outcome: | The proposed approach reduces the data requirements of state-of-the-art AL strategies by 3-25% on multiple NLP tasks while achieving the same performance with virtually no additional computation overhead. |
Incremental Natural Language Processing: Challenges, Strategies, and Evaluation (C18-1)
Copied to clipboard
| Challenge: | In this survey, I consolidate and categorize the approaches, identifying similarities and differences in computation and data, and show trade-offs that have to be considered. |
| Approach: | They consolidate and categorize approaches to incremental processing and show trade-offs that have to be considered. |
| Outcome: | The proposed approaches show that they have similarities and differences in computation and data and that they are not trivial. |
PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models are pre-trained on unstructured data, which leads to poor performance when dealing with structured data. |
| Approach: | They propose a framework to train large language models to act as verifier modules and to apply iterative corrections offline. |
| Outcome: | The proposed framework improves graph-based generative capability of large language models by iterating corrective instructions on three graph-derived datasets. |
Distant Supervision from Disparate Sources for Low-Resource Part-of-Speech Tagging (D18-1)
Copied to clipboard
| Challenge: | Low-resource languages lack manual annotated data to learn basic models such as part-of-speech (POS) taggers. |
| Approach: | They propose a cross-lingual neural part-of-speech tagger that learns from disparate sources of distant supervision in a uniform framework. |
| Outcome: | The proposed model scales to hundreds of low-resource languages without access to gold annotated data. |