Papers by Chaoya Jiang
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks. |
| Approach: | They propose a new framework for a semantically-aware contrastive learning that minimizes the MI between false negative and positive samples . |
| Outcome: | The proposed framework minimizes the MI between false negative samples and positive samples even though they share similar semantics. |
Exploiting Pseudo Image Captions for Multimodal Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal summarization with multimodal output (MSMO) lack reference images for training, and exposure of image captions during training is inconsistent with MSMO’s task settings. |
| Approach: | They propose a coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge. |
| Outcome: | The proposed method sets up state-of-the-art on all intermodality and intramodality metrics and improves on image recommendation precision. |
Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new paradigm for pre-trained language models that can make predictions for unseen inputs without updating parameters. |
| Approach: | They propose a method that enables a model to augmented copies of a demonstration by leveraging their deep feature distribution and a logit calibration mechanism. |
| Outcome: | The proposed method significantly improves the average and worst-case accuracy across diverse PLMs and tasks. |
TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-and-language pre-training models suffer from long visual sequences . experimental results show that TRIPS gains a speedup of 40% over previous similar VLP models . |
| Approach: | They propose an efficient vision-and-language pre-training model with text-relevant image patch selection, TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference. |
| Outcome: | The proposed model can speed up training and inference by 40% over previous models. |