Papers by Zhibin Lan
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LMM-based embedding models exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. |
| Approach: | They propose a framework that improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. |
| Outcome: | The proposed framework improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. |
AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large multimodal models typically divide high-resolution images into multiple local images and a global image, leading to a large number of visual tokens. |
| Approach: | They propose an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. |
| Outcome: | The proposed model significantly reduces visual tokens and speeds up inference on 11 benchmarks. |
PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current TIMT studies focus on providing translations for all text within an image, neglecting to provide bounding boxes and covering limited scenarios. |
| Approach: | They extend traditional TIMT into position-aware TIMt to support fine-grained translation . they introduce an Adaptive Image OCR Refinement Pipeline to refine results . |
| Outcome: | The proposed model supports fine-grained and layout-preserving translation . the experimental data highlight the scalability and generalizability of the model. |
Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-image models struggle to generate images with legible visual texts . current models lack support for Chinese texts, misspelling, and lack of diversity . |
| Approach: | They propose to empower backbone models to generate visual texts in Chinese and English . they propose to augment conventional training objective with glyph-aware training losses . |
| Outcome: | The proposed methods can generate visual texts in English and Chinese while maintaining image generation quality. |
Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image with translations in target language. |
| Approach: | They propose an end-to-end IIMT model with four modules that translate images . they propose a two-stage training framework to assist the model in learning alignment across languages . |
| Outcome: | The proposed model outperforms cascaded models with only 70.9% of parameters and is highly accurate. |
Exploring Better Text Image Translation with Multimodal Codebook (2023.acl-long)
Copied to clipboard
| Challenge: | Current studies on text image translation face bottlenecks due to lack of a publicly available dataset and poor optical character recognition. |
| Approach: | They propose a text image translation model with a multimodal codebook and an OCR dataset for Chinese-English translation. |
| Outcome: | The proposed model can associate the image with relevant texts, providing useful supplementary information for translation. |