Papers by Jindong Chen
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs (2024.findings-naacl)
Copied to clipboard
Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, Abhanshu Sharma
| Challenge: | Visual language models (VLMs) are achieving increasingly strong performance on multimodal tasks. |
| Approach: | They propose to transfer reasoning capabilities from large-language models to VLMs by constructing a 20x larger dataset and a larger dataset to improve general reasoning capabilities. |
| Outcome: | The proposed model outperforms larger models without an upstream OCR system while keeping inference time constant. |
Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation (2025.acl-long)
Copied to clipboard
Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, Tomas Pfister
| Challenge: | Large language models have been shown to be effective in multi-turn interactions . however, their performance may be limited in complex, multi-turned interactions involving users and multiple tools. |
| Approach: | They propose a framework for synthesizing high-quality training trajectories to enhance the function calling capability of large language model agents in multi-turn conversations with humans. |
| Outcome: | The proposed model outperforms the teacher model by 68.01 on BFCL-v3 and 73.30 on ToolQuery. |
PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | PhotoChat contains 12k dialogues, each of which is paired with a user photo that is shared during the conversation. |
| Approach: | They propose to use PhotoChat to facilitate research on image-text modeling by combining a photo-sharing intent prediction task and a picture retrieval task to retrieve the most relevant photo according to the dialogue context. |
| Outcome: | The proposed tasks achieve 10.4% recall@1 and 58.1% F1 scores, indicating that the proposed dataset presents interesting yet challenging real-world problems. |
LLM Jailbreak Detection for (Almost) Free! (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detecting jailbreak prompts entail significant computational costs . |
| Approach: | They propose a free jailbreak detection method which scales logits by temperature to detect jailbreak prompts . |
| Outcome: | The proposed method detects jailbreak prompts with no additional computational costs. |
Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks (2026.acl-long)
Copied to clipboard
Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, Jindong Wang
| Challenge: | Generating synthetic datasets via large language models (LLMs) has emerged as promising approach to improve LLM performance. |
| Approach: | They propose three mitigation strategies to mitigate bias inheritance in LLMs by analyzing real and LLM-augmented data. |
| Outcome: | The proposed methods can work differently on different tasks and biases. |
PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving (2025.emnlp-main)
Copied to clipboard
Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, Chitta Baral, Chen-Yu Lee, Tomas Pfister, Hamid Palangi
| Challenge: | Existing methods for natural planning lack constraint-guided iterative verification and adaptive selection . a recent study found that LLMs are not good at such planning. |
| Approach: | They propose a model-agnostic and easily scalable agent framework with three key components: constraint, verification, and selection agents. |
| Outcome: | The proposed framework improves inference-time algorithms on NATURAL PLAN and OlympiadBench benchmarks. |
Detoxifying Large Language Models via Knowledge Editing (2024.acl-long)
Copied to clipboard
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, Huajun Chen
| Challenge: | Existing methods to detoxify Large Language Models (LLMs) are limiting, but knowledge editing can be effective. |
| Approach: | They propose a baseline method to detoxify Large Language Models (LLMs) they propose supervised fine-tuning and reinforcement learning from human feedback (RLHF) |
| Outcome: | The proposed method reduces toxicity of large language models with one instance of tuning . it reduces the toxicity, while minimizing the toxins, the authors show . |
ScreenQA: Large-Scale Question-Answer Pairs Over Mobile App Screenshots (2025.naacl-long)
Copied to clipboard
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Carbune, Jason Lin, Maria Wang, Yun Zhu, Jindong Chen
| Challenge: | Existing screen datasets focus on low-level structural and component understanding or on a much higher-level composite task such as navigation and task completion for autonomous agents. |
| Approach: | They propose to annotate 86k question-answer pairs over the RICO dataset to benchmark screen content understanding. |
| Outcome: | The proposed dataset covers full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. |
Towards an On-device Agent for Text Rewriting (2024.findings-naacl)
Copied to clipboard
Yun Zhu, Yinxiao Liu, Felix Stahlberg, Shankar Kumar, Yu-Hui Chen, Liangchen Luo, Lei Shu, Renjie Liu, Jindong Chen, Lei Meng
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting, however creating a smaller yet potent language model presents two formidable challenges: costly data collection and absence of emergent capabilities. |
| Approach: | They propose a new instruction tuning method to develop a mo-bile text rewriting model that leverages LLM-generated data and heuristic reinforcement learning, eliminating the need for human data collection. |
| Outcome: | The proposed model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size using public benchmark EditEval and our new benchmark. |
Reimagining Safety Alignment with An Image (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models face inefficiency, fragility, or architectural constraints, resulting in inefficient performance and heightened over-refusal in cross-modal tasks. |
| Approach: | They propose an optimization-driven visual prompt framework that enhances security and reduces over-refusal at the same time. |
| Outcome: | The proposed framework enhances security and reduces over-refusal while maintaining robust safety while reducing unnecessary denials. |
Towards Better Semantic Understanding of Mobile Interfaces (2022.coling-1)
Copied to clipboard
Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, James W. W. Stout
| Challenge: | a dataset of 500k unique annotations is released to improve mobile accessibility and automation capabilities. |
| Approach: | They propose to use an annotation dataset to improve the accessibility of mobile UIs . they use images and view hierarchies to augment annotations for icons and their semantics - and use multimodal inputs to build models. |
| Outcome: | The proposed dataset shows that it can be used to improve UIs and categories on unseen apps. |
Multimodal Pragmatic Jailbreak on Text-to-image Models (2025.acl-long)
Copied to clipboard
Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu
| Challenge: | Existing jailbreaks for diffusion-based text-to-image models generate unsafe content . experimental results show that all tested models suffer from unsafe generation . |
| Approach: | They propose a jailbreak that triggers diffusion-based text-to-image models to generate the image with visual text, resulting in unsafe content. |
| Outcome: | The proposed model generates image with visual text, but the model is unsafe under such jailbreak. |
Visual Question Decomposition on Multimodal Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for question decomposition focus on unimodal language models, but question decomposing capability of Multimodal Large Language Models (MLLMs) has yet to be explored. |
| Approach: | They propose a finetuning dataset and a training objective for selective decomposition to enhance the model's question decomposing capability. |
| Outcome: | The proposed dataset shows that existing models struggle to produce high-quality sub-questions. |
Fusion-Eval: Integrating Assistant Evaluators with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent studies have employed large language models (LLMs) as reference-free metrics for NLG evaluation, enhancing adaptability to new tasks tasks. |
| Approach: | They propose a method that leverages large language models to integrate insights from various assistant evaluators. |
| Outcome: | The proposed approach achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.7444 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. |
AgentReview: Exploring Peer Review Dynamics with LLM Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of peer review analysis do not address multivariate nature of the process, account for latent variables, and are constrained by privacy concerns due to the sensitive nature of data. |
| Approach: | They propose a large language model based peer review simulation framework which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. |
| Outcome: | The proposed framework disentangles the impacts of multiple latent factors and addresses privacy concerns. |
RESTful-Llama: Connecting User Queries to RESTful APIs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated exceptional performance in zero-shot learning and reasoning tasks. |
| Approach: | They propose a framework that transforms natural language instructions into effective RESTful API calls and a method to generate fine-tuning datasets from public API documentation. |
| Outcome: | The proposed framework improves performance in a 31.9% improvement in robustness and 2.33x increase in efficiency compared to existing methods. |
An Efficient Conversational Smart Compose System (2023.acl-demo)
Copied to clipboard
Yun Zhu, Xiayu Chen, Lei Shu, Bowen Tan, Xinying Song, Lijuan Liu, Maria Wang, Jindong Chen, Ning Ruan
| Challenge: | a cloud-based smart compose system is designed to improve human-to-human conversation efficiency. |
| Approach: | They propose a cloud-based smart compose system to improve conversation efficiency . they propose heuristics to achieve the best trade-off between quality and latency . |
| Outcome: | The proposed system reduces latency without losing composing quality further. |
You Only Need One Single Token to Refine Safety Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Excessive safety can lead to over-refusal, where models reject harmful-looking yet benign queries, severely limiting utility. |
| Approach: | They propose a lightweight training-based approach that reshapes the distributions of harmful and benign samples within the model’s decision space by using a single-token prefix. |
| Outcome: | The proposed approach can distinguish between harmful and benign samples while keeping the model frozen. |
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs (2024.naacl-demo)
Copied to clipboard
| Challenge: | Recent advances in machine learning (ML) are attributed to large language models (LLMs), but their escalating memory requirements require developers to partition a large model to distribute it across multiple GPUs or TPUs. |
| Approach: | They propose a lightweight and user-friendly tool to automate distributed training and inference for LLMs and to simplify ML pipeline development. |
| Outcome: | The proposed tool automates distributed training and inference for LLMs, and simplifies ML pipeline development. |
Can an Individual Manipulate the Collective Decisions of Multi-Agents? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that coordinated multi-agent systems exhibit enhanced decision-making and reasoning abilities through collaboration. |
| Approach: | They propose a framework that simulates agent interactions within a multi-agent system to generate adversarial samples and use them to manipulate the target agent in the target system. |
| Outcome: | The proposed framework generates adversarial samples that are used to manipulate the target agent in the target system, misleading the system’s decision-making process. |