Papers by Ruiyi Zhang
Doc-React: Multi-page Heterogeneous Document Question-answering (2025.acl-short)
Copied to clipboard
Junda Wu, Yu Xia, Tong Yu, Xiang Chen, Sai Sree Harsha, Akash V Maharaj, Ruiyi Zhang, Victor Bursztyn, Sungchul Kim, Ryan A. Rossi, Julian McAuley, Yunyao Li, Ritwik Sinha
| Challenge: | Existing methods for integrating information across multiple modalities are suboptimal for multi-page, multimodal documents. |
| Approach: | They propose an adaptive iterative framework that balances information gain and uncertainty reduction at each step. |
| Outcome: | The proposed framework captures relevant multimodal content and achieves strong performance on complex QA tasks. |
CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing (2026.acl-long)
Copied to clipboard
Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Subrata Mitra, Lina Yao, Julian McAuley
| Challenge: | Existing Large Language Models exhibit critical vulnerability to indirect prompt injection attacks, where instructions injected within in the prompt context can override the user's intent. |
| Approach: | They propose a neural pruning algorithm that prunes neurons associated with instruction-following during KV cache encoding of the prompt context. |
| Outcome: | The proposed approach significantly reduces the attack success rate while preserving the model's ability to follow user instructions. |
Graphine: A Dataset for Graph-aware Terminology Definition Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Lack of large-scale terminology definition dataset hinders definition generation . lack of precise terminology definitions poses great challenges in scientific communication . |
| Approach: | They propose a large-scale terminology definition dataset Graphine that exploits the graph structure of terminologies to generate graph-aware text generation models. |
| Outcome: | The proposed model outperforms existing models by exploiting graph structure of terminologies. |
Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes (2025.naacl-short)
Copied to clipboard
Isabel O. Gallegos, Ryan Aponte, Ryan A. Rossi, Joe Barrow, Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, Franck Dernoncourt, Nedim Lipka, Deonna Owens, Jiuxiang Gu
| Challenge: | Large language models exhibit harmful social biases, but they are often difficult to train and modify. |
| Approach: | They leverage the zero-shot capabilities of large language models to reduce stereotyping . they introduce a technique called zero- shot self-debiasing to reduce bias . |
| Outcome: | The proposed technique reduces stereotyping across nine different social groups while relying on the LLM itself and a simple prompt. |
TextLap: Customizing Language Models for Text-to-Layout Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Creating 2D graphical layouts from text alone is challenging in traditional settings. |
| Approach: | They propose to customize LLMs to allow users to generate professional looking layouts by simply inputting text instructions. |
| Outcome: | The proposed method outperforms existing benchmarks for document generation and graphical design benchmarks. |
Understanding Demonstration-based Learning from a Causal Perspective (2023.acl-short)
Copied to clipboard
| Challenge: | Demonstration-based learning has shown impressive performance in exploiting pretrained language models under few-shot learning settings. |
| Approach: | They propose to construct a Structural Causal Model to understand demonstration-based learning from causal perspectives and interpret random demonstrations as interventions on the demonstration variable within the causal model. |
| Outcome: | The proposed model outperforms hand-crafted demonstrations on public sequence labeling benchmarks. |
Context-aware Information-theoretic Causal De-biasing for Interactive Sequence Labeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning models for sequence labeling are expensive and time-consuming. |
| Approach: | They propose an interactive sequence labeling that allows training directly with the user feedback . they identify context and feedback biases by formulating interactive sequence labels via a Structural Causal Model. |
| Outcome: | The proposed approach can effectively alleviate the biases and can be learnt with the user feedback. |
A Survey on LLM-based Conversational User Simulation (2026.eacl-long)
Copied to clipboard
Bo Ni, Yu Wang, Leyao Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Luera, Samyadeep Basu, Subhojyoti Mukherjee, Puneet Mathur, Nesreen K. Ahmed, Junda Wu, Li Li, Huixin Zhang, Ruiyi Zhang, Tong Yu, Sungchul Kim, Jiuxiang Gu, Zhengzhong Tu, Alexa Siu, Zichao Wang, Seunghyun Yoon, Nedim Lipka, Namyong Park, Zihao Lin, Trung Bui, Yue Zhao, Tyler Derr, Ryan A. Rossi
| Challenge: | Recent advances in large language models (LLMs) have enabled high-fidelity generation of synthetic user conversation. |
| Approach: | They propose a taxonomy covering user granularity and simulation objectives . they analyze core techniques and evaluation methodologies to help them understand the latest developments . |
| Outcome: | The proposed model enables high-fidelity generation of synthetic user conversation. |
SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents (2024.acl-long)
Copied to clipboard
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, Hao Zhu
| Challenge: | Existing studies on building language agents have not addressed this social learning gap. |
| Approach: | They propose an interactive learning method that improves the social intelligence of language agents by using behavior cloning and self-reinforcement based training on filtered social interaction data. |
| Outcome: | The proposed method allows a 7B LLM to reach the social goal completion ability of an expert model (GPT-4-based agent) without the loss of more generic abilities, such as the ability to answer knowledge-based questions. |
A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Large multimodal models struggle with text-rich images because of inadequate training data. |
| Approach: | They propose to use annotations from human annotators to generate instruction data by a hybrid approach to generate text prompts for large language models. |
| Outcome: | The proposed model improves multimodal alignment for text-rich images by using human annotations and tailored text prompts for large language models. |
Topic-Guided Variational Auto-Encoder for Text Generation (N19-1)
Copied to clipboard
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, Lawrence Carin
| Challenge: | Experimental results show that our model outperforms its competitors on both unconditional and conditional text generation. |
| Approach: | They propose a topic-guided variational auto-encoder model for text generation that specifies a Gaussian mixture model and a neural topic module to generate sentences under the topic. |
| Outcome: | The proposed model outperforms existing variational auto-encoders on unconditional and conditional text generation, and can generate semantically-meaningful sentences with various topics. |
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. |
| Approach: | They propose a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. |
| Outcome: | The proposed framework is based on a meta learning based framework that can identify the optimal rank of each LoRA layer. |
From Selection to Generation: A Survey of LLM-based Active Learning (2025.acl-long)
Copied to clipboard
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley
| Challenge: | Large Language Models (LLMs) have been used for selection and training of data for active learning. |
| Approach: | They propose an intuitive taxonomy that categorizes LLM-based active learning techniques and discuss the transformative roles they can play in the active learning loop. |
| Outcome: | The proposed model can generate entirely new data instances and provide more cost-effective annotations with fewer labeled data instances. |
Lizard: An Efficient Linearization Framework for Large Language Models (2026.acl-long)
Copied to clipboard
Chien Van Nguyen, Huy Huu Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen
| Challenge: | Existing linearization frameworks that rely on softmax attention with quadratic time and memory complexity pose significant computational and memory bottlenecks for long-context applications. |
| Approach: | They propose a linearization framework that transforms pretrained Transformer-based Large Language Models into subquadratic architectures that closely approximate softmax attention while preserving model quality. |
| Outcome: | Experiments show that the proposed framework outperforms existing methods by 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall. |
Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning (2024.eacl-long)
Copied to clipboard
Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl
| Challenge: | Existing prompt transfer techniques lack consideration for dialogue-specific information. |
| Approach: | They propose a method which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task. |
| Outcome: | The proposed method significantly outperforms baselines on two dialogue summarization benchmarks. |
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference (2020.emnlp-main)
Copied to clipboard
Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, Changyou Chen
| Challenge: | Existing studies show that multi-head attention is an effective module in deep neural networks, but there are no explicit mechanisms guaranteeing this property. |
| Approach: | They propose a non-parametric approach that explicitly improves the repulsiveness in multi-head attention and consequently strengthens model’s expressiveness. |
| Outcome: | The proposed approach improves the repulsiveness in multi-head attention and strengthens model’s expressiveness. |
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)
Copied to clipboard
Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao
| Challenge: | Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive . |
| Approach: | They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training . |
| Outcome: | The proposed framework achieves superior performance compared with baselines. |
SimpleOCR: Rendering Visual Questions to Teach MLLMs to Read (2026.findings-acl)
Copied to clipboard
Yibo Peng, Peng Xia, Ding Zhong, Kaide Zeng, Siwei Han, Yiyang Zhou, Jiaqi Liu, Ruiyi Zhang, Huaxiu Yao
| Challenge: | MLLMs lack visual grounding mechanism to read text embedded in images, or rely on parametric shortcuts . despite strong OCR capabilities, models suffer performance degradation of 12.7% in the VQ setting . |
| Approach: | They propose a plug-and-play training strategy that invalidates shortcuts in text prompts . they propose 'vq' setting where text queries are rendered directly onto images . |
| Outcome: | The proposed training strategy surpasses the base model by 5.4% and GRPO based on original images by 2.7% on four representative OOD benchmarks. |
Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images (2026.findings-acl)
Copied to clipboard
Shijie Zhou, Jihyung Kil, Ming Li, Jiuxiang Gu, Curtis Wigington, Rajiv Jain, Changyou Chen, Ruiyi Zhang
| Challenge: | Existing multimodal large language model (MLLM) approaches struggle to align query tokens with visual–text patches, heavily relying on lengthy OCR inputs. |
| Approach: | They propose an OCR-free approach that leverages the MLLM's inherent multi-head attention for multi-patch grounding. |
| Outcome: | Empirical results show that the proposed approach outperforms existing approaches on challenging document grounding benchmarks. |
Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models (2023.findings-emnlp)
Copied to clipboard
Simon Stepputtis, Joseph Campbell, Yaqi Xie, Zhengyang Qi, Wenxin Zhang, Ruiyi Wang, Sanketh Rangreji, Charles Lewis, Katia Sycara
| Challenge: | Deception and persuasion play a critical role in long-horizon multi-party dialogues, especially when the interests, goals, and motivations of the participants are not aligned. |
| Approach: | They propose a game in which players must determine each other’s hidden identities to complete their team’s objective. |
| Outcome: | The proposed model can be used to determine the true player identities of six human players in a cooperative-competitive game. |
Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used for temporal prediction tasks . however, their reliance on pretraining data can lead to contamination concerns . |
| Approach: | They investigate the capability of prompting to simulate an earlier knowledge cutoff in large language models. |
| Outcome: | The proposed model fails to induce forgetting when the forgotten content is not directly asked but causally related to the query. |
Semantic Matching for Sequence-to-Sequence Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for sequence-to-sequence semantic matching do not match target sentences . Optimal partial transport (OPT) is a technique that partially matches semantically meaningful words between source and partial target sequences. |
| Approach: | They propose a semantic matching scheme based on the Optimal Partial Transport (OPT) to match semantically meaningful words between source and partial target sequences. |
| Outcome: | Extensive experiments show that the proposed approach improves over sequence-to-sequence models. |
Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train named entity recognition models on noisy data are expensive and time-intensive to accumulate. |
| Approach: | They propose to denoise noisy NER data with guidance from a small set of clean instances. |
| Outcome: | The proposed method can improve on large-scale datasets with a small guidance set. |
Decomposition-Enhanced Training for Post-Hoc Attributions in Language Models (2026.eacl-long)
Copied to clipboard
Sriram Balasubramanian, Samyadeep Basu, Koustava Goswami, Ryan A. Rossi, Varun Manjunatha, Roshan Santhosh, Ruiyi Zhang, Soheil Feizi, Nedim Lipka
| Challenge: | Existing methods for extractive QA struggle in multi-hop, abstractive, and semi-extractive settings. |
| Approach: | They propose a method that prompts models to produce answer decompositions as intermediate reasoning steps. |
| Outcome: | The proposed method outperforms existing methods and matches or exceeds state-of-the-art frontier models. |
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | despite the success of large language models, their performance in highly specialized domains remains unsatisfactory. |
| Approach: | They propose a biomedical tool-calling dataset designed for fine-tuning LLMs . the dataset contains 34 frequently used tools from the NCBI, Ensembl, and UniProt databases . |
| Outcome: | The proposed dataset outperforms commercial LLMs on biomedical domains. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Few-Shot Class-Incremental Learning for Named Entity Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models of Named Entity Recognition (NER) are trained on large datasets with predefined entity classes, but data of new classes arrives constantly. Existing work on NER relies on the assumption that there exists abundance of labeled data for the training of new class. |
| Approach: | They propose a few-shot class-incremental learning problem where NER model is trained with only few labeled samples of the new classes without forgetting knowledge of the old ones. |
| Outcome: | The proposed model improves over existing baselines by reconstructing training data of old classes and real data from the training set. |
GUI Agents: A Survey (2025.findings-acl)
Copied to clipboard
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt
| Challenge: | Large Foundation Models (LFMs) have transformed the landscape of AI research and day-to-day life. |
| Approach: | They propose a framework that delineates GUI agents' perception, reasoning, planning, and acting capabilities. |
| Outcome: | The proposed framework delineates their perception, reasoning, planning, and acting capabilities. |
Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns (2022.findings-acl)
Copied to clipboard
Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, Jingbo Shang
| Challenge: | Adaptive Axis Attention learns different attention patterns for each task and model layer . sparse attention patterns do not improve the run time of the models but they reduce model memory requirements . |
| Approach: | They propose a method that learns different attention patterns for each Transformer layer . they propose 'adaptive axis attention' method that identifies important tokens . |
| Outcome: | The proposed method does not require pre-training to accommodate sparse attention patterns. |
A Critical Analysis of Document Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova
| Challenge: | Existing document understanding models focus on single-modal inputs such as images or texts. |
| Approach: | They propose to use a spatial-aware adapter to adapt transformer-based language models to document domain to exploit multi-modal information. |
| Outcome: | The proposed model significantly improves the OOD detection performance compared to using a standard language model and to competitive baselines. |
Defense against Prompt Injection Attacks via Mixture of Encodings (2025.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as a dominant approach for a wide range of NLP tasks, but external content embeds malicious instructions that manipulate the LLM’s output. |
| Approach: | They propose a mixture of encodings defense mechanism which utilizes multiple character encodes to degrade LLM performance on certain NLP tasks. |
| Outcome: | The proposed method achieves one of the lowest attack success rates under prompt injection attacks while maintaining high performance across all NLP tasks. |
Improving Text Generation with Student-Forcing Optimal Transport (2020.emnlp-main)
Copied to clipboard
Jianqiao Li, Chunyuan Li, Guoyin Wang, Hao Fu, Yuhchen Lin, Liqun Chen, Yizhe Zhang, Chenyang Tao, Ruiyi Zhang, Wenlin Wang, Dinghan Shen, Qian Yang, Lawrence Carin
| Challenge: | Maximum likelihood estimation (MLE) is used to train models, but during testing, the model is conditioned on previously generated tokens, resulting in exposure bias. |
| Approach: | They propose to use optimal transport to match the sequences generated in MLE and test modes to reduce exposure bias. |
| Outcome: | The proposed method is validated on machine translation, text summarization, and text generation tasks. |
Federated Domain Adaptation for Named Entity Recognition via Distilling with Heterogeneous Tag Sets (2023.findings-acl)
Copied to clipboard
| Challenge: | a federated domain adaptation approach is used to learn with NER datasets from multiple platforms while not violating data privacy. |
| Approach: | They propose to use a distillation approach to facilitate knowledge transfer across platforms. |
| Outcome: | The proposed model performs better in the clinic domain. |
Improving Adversarial Text Generation by Modeling the Distant Future (2020.acl-main)
Copied to clipboard
Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Dinghan Shen, Guoyin Wang, Zheng Wen, Lawrence Carin
| Challenge: | Recent work has shown excellent performance on text generation tasks by combining reinforcement learning (RL) and generative models. |
| Approach: | They propose a model-based imitation-learning approach to improve text generation performance by focusing on a long horizon. |
| Outcome: | The proposed model improves on a number of text-generation tasks and provides intermediate rewards for generator optimization. |
Robustness of Demonstration-based Learning Under Limited Data Scenario (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current large pretrained language models struggle to learn NLP tasks under limited data scenarios. |
| Approach: | They propose to augment input with some demonstrations to improve model performance under limited data scenarios. |
| Outcome: | The proposed demonstrations improve performance on few-shot NER tasks and show that the length of demonstrations and relevance of random tokens are the main factors affecting the model's performance. |