Papers by Senbao Shi
Generative Multimodal Entity Linking (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing Entity Linking methods focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters. |
| Approach: | They propose a framework for multimodal entity linking based on Large Language Models (LLMs) that trains a feature mapper to enable cross-modal interactions. |
| Outcome: | The proposed framework achieves state-of-the-art on two well-established datasets with a performance gain of 7.7% on WikiDiverse and 8.8% on Wikileaks. |
MultiSkill: Evaluating Large Multimodal Models for Fine-grained Alignment Skills (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation settings for large multimodal models focus on coarse-grained evaluation without considering skill composition required by specific instructions. |
| Approach: | They propose an evaluation protocol that assesses large multimodal models across multiple fine-grained skills for alignment with human values. |
| Outcome: | The proposed evaluation protocol decomposes coarse-level scoring to fine-grained skill set-level score tailored to each instruction. |