Papers by Diogo Glória-Silva
Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks (2024.acl-long)
Copied to clipboard
João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, Joao Magalhaes
| Challenge: | Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. |
| Approach: | They propose a method that integrates a Latent Diffusion Model (LDM) with an LLM to generate captions to maintain semantic coherence of the sequence. |
| Outcome: | The proposed method is preferred by humans in 46.6% of the cases against 26.6% for the second best method. |
Plan-Grounded Large Language Models for Dual Goal Conversational Settings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs can follow user instructions, but it is unclear how they can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation. |
| Approach: | They propose a dual-purpose mixed-initiative conversational setting where the LLM grounds the conversation on an arbitrary plan and seeks to satisfy both a procedural plan and user instructions. |
| Outcome: | The proposed model achieves 2.1x improvement over a strong baseline and good generalization to unseen domains. |
Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing plans-following language models (LLMs) are not capable of multimodal input and output, resulting in inconsistent performance on multimodal tasks. |
| Approach: | They propose a multimodal plan-following language model that integrates both textual plans and visual information to bring cross-modality to instructional tasks. |
| Outcome: | The proposed model performs well on multimodal and textual dialogue in a plan-grounded setting. |
VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models focus on text-only guidance or treat vision and language in isolation. |
| Approach: | They propose a multimodal dialogue model that supports grounded, plan-aware dialogue . they use a dataset with rich video dialogues aligned with cooking and DIY plans . |
| Outcome: | The proposed model outperforms existing models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA. |