Papers by Liangyan Gui
Aligning Large Multimodal Models with Factually Augmented RLHF (2024.findings-acl)
Copied to clipboard
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell
| Challenge: | Large Multimodal Models (LMMs) are built across modalities and the misalignment between two modality can result in "hallucination" . developing LMMs faces challenges such as a lack of data and a limited number of data sets. |
| Approach: | They propose a new algorithm that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options. |
| Outcome: | The proposed approach improves on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 and an improvement of 60% on MMHAL-BENCH over other baselines. |