Papers by Brielen Madureira
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024.acl-long)
Copied to clipboard
| Challenge: | In incremental models, one interpretation is possible, but models that can revise can do so if the ambiguity is resolved. |
| Approach: | They propose an interpretable way to analyse incremental states in a bidirectional way . they propose to use a model that can update internal states to reflect the garden path effect . |
| Outcome: | The proposed model shows that it can perform revisions and recover if the label is incorrect. |
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to apply incremental processing to NLUs but this is computationally expensive and does not scale efficiently for long sequences. |
| Approach: | They propose to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental quality. |
Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset (2023.eacl-main)
Copied to clipboard
| Challenge: | In visual instruction-following dialogue games, players can engage in repair mechanisms in the face of an ambiguous or underspecified instruction. |
| Approach: | They annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. |
| Outcome: | The proposed dataset contains lexically and semantically diverse iCRs produced self-motivatedly by players deciding to clarify in order to solve the task successfully. |
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches for incremental processing use RNNs or Transformers, which consume whole sequences and are by nature non-incremental. |
| Approach: | They propose a two-pass model for AdaPtIve Revision to obtain an incremental supervision signal for learning an adaptive revision policy. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to restart-incremental Transformers while showing little degradation on full sequences. |
clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents (2023.emnlp-main)
Copied to clipboard
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, David Schlangen
| Challenge: | Recent work suggests large language models can be understood as (simulators of) such agents. |
| Approach: | They propose a method for systematic evaluation of "Situated Language Understanding Agents" they propose implementing a framework for implementing rules to be played in "self-play" |
| Outcome: | The proposed model can be evaluated in game-like settings, the authors show . they show that the model can follow game-play instructions and perform better than existing models . |
Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge (2022.acl-short)
Copied to clipboard
| Challenge: | Existing evaluation methods for visual dialogue models are not consistent . |
| Approach: | They propose a theory-based evaluation method to examine to what degree visual dialogue models incrementally build up representations that do scorekeeping. |
| Outcome: | The proposed method aims to determine to what degree models build up representations that are appropriate to do scorekeeping of shared commitments throughout a visual dialogue. |