Papers by Joao Magalhaes
Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks (2024.acl-long)
Copied to clipboard
João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, Joao Magalhaes
| Challenge: | Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. |
| Approach: | They propose a method that integrates a Latent Diffusion Model (LDM) with an LLM to generate captions to maintain semantic coherence of the sequence. |
| Outcome: | The proposed method is preferred by humans in 46.6% of the cases against 26.6% for the second best method. |
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval. |
| Approach: | They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning. |
| Outcome: | The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect. |
Plan-Grounded Large Language Models for Dual Goal Conversational Settings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs can follow user instructions, but it is unclear how they can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation. |
| Approach: | They propose a dual-purpose mixed-initiative conversational setting where the LLM grounds the conversation on an arbitrary plan and seeks to satisfy both a procedural plan and user instructions. |
| Outcome: | The proposed model achieves 2.1x improvement over a strong baseline and good generalization to unseen domains. |
Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing plans-following language models (LLMs) are not capable of multimodal input and output, resulting in inconsistent performance on multimodal tasks. |
| Approach: | They propose a multimodal plan-following language model that integrates both textual plans and visual information to bring cross-modality to instructional tasks. |
| Outcome: | The proposed model performs well on multimodal and textual dialogue in a plan-grounded setting. |
Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to model conversational traits are costly and time consuming. |
| Approach: | They propose a method that generates diverse user profiles at decoding-time by sampling from trait-specific Language Models. |
| Outcome: | The proposed method generates diverse user profiles at decoding-time without fine-tuning. |
Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer. |
| Approach: | They propose a PEFT approach that modifies input embeddings without altering hidden layers. |
| Outcome: | Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters . |
VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models focus on text-only guidance or treat vision and language in isolation. |
| Approach: | They propose a multimodal dialogue model that supports grounded, plan-aware dialogue . they use a dataset with rich video dialogues aligned with cooking and DIY plans . |
| Outcome: | The proposed model outperforms existing models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA. |