Papers by Jiuxiang Gu
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes (2025.naacl-short)
Copied to clipboard
Isabel O. Gallegos, Ryan Aponte, Ryan A. Rossi, Joe Barrow, Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, Franck Dernoncourt, Nedim Lipka, Deonna Owens, Jiuxiang Gu
| Challenge: | Large language models exhibit harmful social biases, but they are often difficult to train and modify. |
| Approach: | They leverage the zero-shot capabilities of large language models to reduce stereotyping . they introduce a technique called zero- shot self-debiasing to reduce bias . |
| Outcome: | The proposed technique reduces stereotyping across nine different social groups while relying on the LLM itself and a simple prompt. |
TextLap: Customizing Language Models for Text-to-Layout Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Creating 2D graphical layouts from text alone is challenging in traditional settings. |
| Approach: | They propose to customize LLMs to allow users to generate professional looking layouts by simply inputting text instructions. |
| Outcome: | The proposed method outperforms existing benchmarks for document generation and graphical design benchmarks. |
A Survey on LLM-based Conversational User Simulation (2026.eacl-long)
Copied to clipboard
Bo Ni, Yu Wang, Leyao Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Luera, Samyadeep Basu, Subhojyoti Mukherjee, Puneet Mathur, Nesreen K. Ahmed, Junda Wu, Li Li, Huixin Zhang, Ruiyi Zhang, Tong Yu, Sungchul Kim, Jiuxiang Gu, Zhengzhong Tu, Alexa Siu, Zichao Wang, Seunghyun Yoon, Nedim Lipka, Namyong Park, Zihao Lin, Trung Bui, Yue Zhao, Tyler Derr, Ryan A. Rossi
| Challenge: | Recent advances in large language models (LLMs) have enabled high-fidelity generation of synthetic user conversation. |
| Approach: | They propose a taxonomy covering user granularity and simulation objectives . they analyze core techniques and evaluation methodologies to help them understand the latest developments . |
| Outcome: | The proposed model enables high-fidelity generation of synthetic user conversation. |
Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021.naacl-main)
Copied to clipboard
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu
| Challenge: | Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. |
| Approach: | They propose a shortcut mitigation framework to suppress NLU models from making overconfident predictions for samples with large shortcut degree. |
| Outcome: | The proposed framework suppresses the model from making overconfident predictions for samples with large shortcut degree. |
From Selection to Generation: A Survey of LLM-based Active Learning (2025.acl-long)
Copied to clipboard
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley
| Challenge: | Large Language Models (LLMs) have been used for selection and training of data for active learning. |
| Approach: | They propose an intuitive taxonomy that categorizes LLM-based active learning techniques and discuss the transformative roles they can play in the active learning loop. |
| Outcome: | The proposed model can generate entirely new data instances and provide more cost-effective annotations with fewer labeled data instances. |
DocScript: Document-level Script Event Prediction (2024.lrec-main)
Copied to clipboard
Puneet Mathur, Vlad I. Morariu, Aparna Garimella, Franck Dernoncourt, Jiuxiang Gu, Ramit Sawhney, Preslav Nakov, Dinesh Manocha, Rajiv Jain
| Challenge: | Existing script event prediction frameworks such as ChatGPT and FlanT5 lack the ability to learn long-range dependencies between events. |
| Approach: | They propose a novel script event prediction task which aims to predict the next event from a candidate list of narrative events in long-form documents. |
| Outcome: | The proposed architecture can learn sequential ordering between events at the document scale. |
MENTOR: Efficient Autoregressive Image Generation with Balanced Multimodal Control (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent text-to-image models achieve impressive visual quality but still face challenges in precise controllability, balancing multimodal inputs, and high training cost for multimodal image generation. |
| Approach: | They propose an autoregressive framework with a two-stage training paradigm for controllable multimodal image generation. |
| Outcome: | Extensive experiments on DreamBench++ and DreamBech show that the proposed framework achieves a strong balance between textual and visual guidance for controllable image generation. |
Learning the Visualness of Text Using Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual text evokes an image in a person’s mind, while non-visual text fails to do so. |
| Approach: | They propose a method to automatically detect visualness in text to enable text-to-image retrieval and generation models to augment text with relevant images. |
| Outcome: | The proposed method performs better than several baseline models and heuristics for the task. |
DocTime: A Document-level Temporal Dependency Graph Parser (2022.naacl-main)
Copied to clipboard
Puneet Mathur, Vlad Morariu, Verena Kaynig-Fittkau, Jiuxiang Gu, Franck Dernoncourt, Quan Tran, Ani Nenkova, Dinesh Manocha, Rajiv Jain
| Challenge: | Document dependency graphs (TDGs) are used to understand the temporal relations between events mentioned in a document and to improve downstream tasks such as timeline creation and time-aware summarization. |
| Approach: | They propose a temporal dependency graph parser that takes input from a text document and produces a graph that incorporates longer range dependencies. |
| Outcome: | The proposed framework outperforms existing models on three datasets and improves tasks such as timeline creation, time-aware summarization, and temporal information extraction. |
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction tuning is critical to large language models but its success heavily relies on the training data quality. |
| Approach: | They propose a paradigm that synergizes a teacher LLM’s reflection and introspection with the data selection capability of the student LLM to automatically refine existing instruction-tuning data. |
| Outcome: | The proposed method achieves much stronger and top-tier 7B and 13B LLMs without collecting brand-new data. |
Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images (2026.findings-acl)
Copied to clipboard
Shijie Zhou, Jihyung Kil, Ming Li, Jiuxiang Gu, Curtis Wigington, Rajiv Jain, Changyou Chen, Ruiyi Zhang
| Challenge: | Existing multimodal large language model (MLLM) approaches struggle to align query tokens with visual–text patches, heavily relying on lengthy OCR inputs. |
| Approach: | They propose an OCR-free approach that leverages the MLLM's inherent multi-head attention for multi-patch grounding. |
| Outcome: | Empirical results show that the proposed approach outperforms existing approaches on challenging document grounding benchmarks. |
Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train named entity recognition models on noisy data are expensive and time-intensive to accumulate. |
| Approach: | They propose to denoise noisy NER data with guidance from a small set of clean instances. |
| Outcome: | The proposed method can improve on large-scale datasets with a small guidance set. |
MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding (2022.emnlp-main)
Copied to clipboard
Zilong Wang, Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Ani Nenkova, Tong Sun, Jingbo Shang, Vlad Morariu
| Challenge: | Existing methods learn features from word-level or region-level but fail to consider both simultaneously. |
| Approach: | They propose a multi-modal multi-granular pre-training framework that encodes page-level, region-level and word-level information at the same time. |
| Outcome: | The proposed model learns features from word-level and region-level but fails to consider both simultaneously. |
CoMMIT: Coordinated Multimodal Instruction Tuning (2025.emnlp-main)
Copied to clipboard
Xintong Li, Junda Wu, Tong Yu, Rui Wang, Yu Wang, Xiang Chen, Jiuxiang Gu, Lina Yao, Julian McAuley, Jingbo Shang
| Challenge: | et al., 2024) show that multimodal instruction tuning is more effective than baselines. |
| Approach: | They propose a multimodal balance coefficient that enables quantitative measurement of the balance of learning . they propose auxiliary regularization on the gradient to promote updating with larger step sizes . |
| Outcome: | The proposed method is more effective than baselines in MLLM instruction tuning. |
METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling (2025.acl-long)
Copied to clipboard
| Challenge: | Chart generation requires strong visual design skills and precise coding capabilities that embed the desired visual properties into code. |
| Approach: | They propose a vision-language model-based multi-agent framework for effective automatic chart generation. |
| Outcome: | The proposed framework achieves a 5.2% improvement in the F1 score over the current best chart generation task. |
Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns (2022.findings-acl)
Copied to clipboard
Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, Jingbo Shang
| Challenge: | Adaptive Axis Attention learns different attention patterns for each task and model layer . sparse attention patterns do not improve the run time of the models but they reduce model memory requirements . |
| Approach: | They propose a method that learns different attention patterns for each Transformer layer . they propose 'adaptive axis attention' method that identifies important tokens . |
| Outcome: | The proposed method does not require pre-training to accommodate sparse attention patterns. |
A Critical Analysis of Document Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova
| Challenge: | Existing document understanding models focus on single-modal inputs such as images or texts. |
| Approach: | They propose to use a spatial-aware adapter to adapt transformer-based language models to document domain to exploit multi-modal information. |
| Outcome: | The proposed model significantly improves the OOD detection performance compared to using a standard language model and to competitive baselines. |