Papers by Gautier Dagan
CAST: Cross-modal Alignment Similarity Test for Vision Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are typically evaluated with Visual Question Answering tasks which assess a model’s understanding of scenes. |
| Approach: | They propose to use visual question answering (VQA) to assess a model's understanding of scenes to probe for self-consistency across modalities. |
| Outcome: | The proposed test does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. |
Location Attention for Extrapolation to Longer Sequences (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are surprisingly good at interpolating, but they are often unable to extrapolate patterns beyond the seen data. |
| Approach: | They propose to use a special type of extrapolation for natural language processing to generalize to sequences that are longer than the training ones. |
| Outcome: | The proposed model is more likely to extrapolate than models with common attention mechanisms. |
Learning the Effects of Physical Actions in a Multi-modal Environment (2023.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are trained on large corpora of disembodied texts. |
| Approach: | They propose a multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). They extend an LLM to model latent representations of objects to better predict action outcomes in an environment. |
| Outcome: | The proposed model can capture commonsense when augmented with visual information and generalize and learn commonsensical reasoning better. |
Co-evolution of language and agents in referential games (2021.eacl-main)
Copied to clipboard
| Challenge: | Referential games allow neural agents to learn language, but they do not take into account the learning biases of the learners. |
| Approach: | They propose to model cultural and architectural evolution in a population of agents to take into account learning biases of the language learners and let them co-evolve. |
| Outcome: | The proposed model outperforms cultural transmission in a population of agents and takes into account learning biases of the learners. |