TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models with instruction-following capabilities have revolutionized the field of artificial intelligence. |
| Approach: | They propose an annotation-free framework for empowering large language models with instruction-following capabilities. |
| Outcome: | The proposed framework generates multi-turn multimodal instruction-response conversations from a language model. |
Similar Papers
MIO: A Foundation Model on Multimodal Tokens (2025.emnlp-main)
Copied to clipboard
Zekun Moore Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jessie Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
| Challenge: | Existing models lack multimodal understanding capabilities, resulting in closed-source model that does not support multimodal interleaved sequences. |
| Approach: | They propose a foundation model built on multimodal tokens capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. |
| Outcome: | The proposed model is able to understand speech, text, images, and videos in an end-to-end, autoregressive manner. |
Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models (2024.acl-long)
Copied to clipboard
Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Xin Zhao, Fuzheng Zhang, Di Zhang, Kun Gai
| Challenge: | Existing studies overlook the multi-turn instruction following ability of large language models (LLMs) Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi- turn instruction following. |
| Approach: | They propose a method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis, and a context-aware preference optimization strategy to further enhance LLMs for complex queries. |
| Outcome: | The proposed method improves existing LLMs by up to 7.2% in multi-turn instruction following. |
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (2025.findings-acl)
Copied to clipboard
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Hamid Alinejad-Rokny, Xiaobo Xia, Jingkuan Song, Fei Huang
| Challenge: | a new framework for image-text instruction data evolution improves MLLM performance . lack of high-quality instruction data remains a major bottleneck in ML modeling . |
| Approach: | They propose a multimodal instruction data evolution framework that iteratively enhances data quality through fine-grained perception, cognitive reasoning, and interaction evolution. |
| Outcome: | The proposed approach improves MLLM performance in nine vision-language tasks while using significantly less data. |
SpiRit-LM: Interleaved Spoken and Written Language Model (2025.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, Emmanuel Dupoux
| Challenge: | SpiRit-LM is a foundation multimodal language model that freely mixes text and speech. |
| Approach: | They propose a multimodal language model that freely mixes text and speech . they extend the model to the speech modality by continuously training it on text and language units. |
| Outcome: | The proposed model can learn new tasks in a few-shot fashion across modalities. |
Can Language Models Follow Multiple Turns of Entangled Instructions? (2025.findings-emnlp)
Copied to clipboard
Chi Han, Xin Liu, Haodong Wang, Shiyang Li, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Qingyu Yin, Liang Qiu, Changlong Yu, Yifan Gao, Zheng Li, Bing Yin, Jingbo Shang, Heng Ji
| Challenge: | Despite of significant achievements in improving instruction-following capabilities of large language models, the ability to process multiple potentially entangled or conflicting instructions remains a considerable challenge. |
| Approach: | They construct multi-turn instruction with 1.1K high-quality multi-turned conversations using the human-in-the-loop approach and examine their capabilities. |
| Outcome: | The proposed model shows that it is difficult to integrate multiple turns and balance competing objectives when instructions intersect or conflict. |
Let’s Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR (2025.findings-acl)
Copied to clipboard
| Challenge: | Various fusion strategies have been explored for integration of large language models into multi-modal systems. |
| Approach: | They propose a framework for deep fusion decoding that integrates large language models into cross-modal text recognition systems. |
| Outcome: | The proposed framework surpasses cascaded methods in English and Mandarin, and significantly reduces WERs by 17.7%. |
Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often unbalanced towards English because of the imbalance in the distribution of pre-training data. |
| Approach: | They propose to empower Instruction-tuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. |
| Outcome: | The proposed model outperforms the existing model on multilingual question answering benchmarks and adapted versions of MMLU and BBH on six different languages. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing plans-following language models (LLMs) are not capable of multimodal input and output, resulting in inconsistent performance on multimodal tasks. |
| Approach: | They propose a multimodal plan-following language model that integrates both textual plans and visual information to bring cross-modality to instructional tasks. |
| Outcome: | The proposed model performs well on multimodal and textual dialogue in a plan-grounded setting. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |