Papers by Liangke Gui
KAT: A Knowledge Augmented Transformer for Vision-and-Language (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge retrieval and answer prediction have left open questions about the quality and relevance of the retrieved knowledge and how the reasoning processes over implicit and explicit knowledge should be integrated. |
| Approach: | They propose a Knowledge Augmented Transformer which integrates both implicit and explicit knowledge in an encoder-decoder architecture while simultaneously reasoning over both knowledge sources during answer generation. |
| Outcome: | The proposed model achieves a strong state-of-the-art (+6% absolute) on the open-domain multimodal task of OK-VQA. |
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward (2025.naacl-long)
Copied to clipboard
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander G Hauptmann, Yonatan Bisk, Yiming Yang
| Challenge: | Existing studies have demonstrated that direct preference optimization (DPO) can be effective in generalizing large language models, but its effectiveness in video domain remains limited. |
| Approach: | They propose a framework that utilizes detailed video captions as a proxy of video content to enable language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. |
| Outcome: | The proposed framework shows that it can be used to align language models with video content and improves performance on open-ended video QA tasks. |