Papers by Alvaro Soto
Tracr-Injection: Distilling Algorithms into Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent efforts to characterize symbolic abilities of the transformer architecture have shown that the tasks that can be implemented in RASP are uncommon to learn from natural unsupervised data. |
| Approach: | They propose a programming language, called RASP, which can be directly compiled into transformer weights to implement these algorithms. |
| Outcome: | The proposed method improves out-of-distribution performance compared to baselines, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. |
Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not produce suitable representations at the discourse level. |
| Approach: | They propose to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations by incorporating top-down connections that operate at the intermediate layers of the network. |
| Outcome: | The proposed approach improves in 6 out of 11 tasks by detecting discourse relationship detection. |
Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation (D18-1)
Copied to clipboard
Xiaoxue Zang, Ashwini Pokle, Marynel Vázquez, Kevin Chen, Juan Carlos Niebles, Alvaro Soto, Silvio Savarese
| Challenge: | Existing models for translating free-form natural language instructions to a high-level plan for behavioral robot navigation are difficult due to the variability in the way people describe routes. |
| Approach: | They propose an end-to-end deep learning model for translating free-form natural language instructions to a high-level plan for robot navigation. |
| Outcome: | The proposed model significantly outperforms baseline approaches on a new dataset containing 10,050 pairs of navigation instructions. |
A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing question answering methods assume that the input content can always be accessed while answering the question. |
| Approach: | They propose a model that performs rehearsal and anticipation while processing inputs to memorize important information for question answering tasks from streaming data. |
| Outcome: | The proposed model improves on short-sequence (bAbI) and large-squence textual (NarrativeQA) and video (ActivityNet-QA) question answering datasets. |
Evaluation Benchmarks for Spanish Sentence Representations (2022.lrec-1)
Copied to clipboard
Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, Alvaro Soto
| Challenge: | Existing and newly constructed datasets address different tasks from various domains. |
| Approach: | They propose to use Spanish SentEval and Spanish DiscoEval to evaluate stand-alone and discourse-aware sentence representations. |
| Outcome: | The proposed benchmarks evaluate the capabilities of stand-alone and discourse-aware sentence representations in Spanish and show that they are more robust and comparable than previous benchmarks. |
Inspecting the concept knowledge graph encoded by modern language models (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are used to solve tasks such as summarization and information retrieval. |
| Approach: | They propose to use word embeddings, text generators, context encoders to extract underlying knowledge graphs of nine influential language models. |
| Outcome: | The proposed model is able to encode word embeddings, text generators, and context encoders, but suffers from several inaccuracies. |
Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation (2024.findings-acl)
Copied to clipboard
| Challenge: | Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. |
| Approach: | They propose a Fact Extractor that leverages large language models to extract factual statements from radiology reports. |
| Outcome: | The proposed framework outperforms current state-of-the-art methods in sentence ranking, natural language inference, and label extraction tasks. |
How Relevant is Selective Memory Population in Lifelong Language Learning? (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing approaches to lifelong language learning rely on sparse experience replay to prevent catastrophic forgetting. |
| Approach: | They propose to use a selective memory population to store a uniform number of samples from the entire data stream to improve model performance. |
| Outcome: | The proposed methods show that they are relevant for lifelong language learning tasks, especially for low memory size, and consistent with computer vision studies. |