Papers by Renjie Pi

MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)

Copied to clipboard

Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang

Challenge:	MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning.
Approach:	They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones.
Outcome:	The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs.

DetGPT: Detect What You Need via Reasoning (2023.emnlp-main)

Copied to clipboard

Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang

Language Models

Challenge:	Recent advances in the field of computer vision have enabled more effective and sophisticated interactions between humans and machines.
Approach:	They propose a reasoning-based object detection paradigm that leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user’s instructions and the visual scene.
Outcome:	The proposed method enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity.

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs (2024.emnlp-main)

Copied to clipboard

Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, Tong Zhang

Multimodality

Challenge:	Existing multi-modal large language models (MLLMs) are able to process visual inputs by converting them into visual tokens that share the same latent space as language tokens in LLMs.
Approach:	They propose a benchmark that assesses the visual illusion level given spurious images and a pipeline that converts visual inputs into visual tokens.
Outcome:	The proposed benchmark shows that MLLMs suffer from an instinctive bias to varying degrees when presented with spurious images.

Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning (2026.acl-long)

Copied to clipboard

Xuchen Li, Xuzhao Li, Jiahui Gao, Renjie Pi, Shiyu Hu, Wentao Zhang

Multimodality

Challenge:	Recent work has shown promise by incorporating pixel-level visual information into the reasoning process, enabling VLMs to access high-resolution visual details during their thought process.
Approach:	They propose a framework that dynamically determines necessary pixel-level operations based on the input query.
Outcome:	The proposed model achieves 73.4% accuracy on HR-Bench 4K while maintaining a tool usage ratio of only 20.1%, improving accuracy and reducing tool usage by 66.5% compared to the previous methods.

Mitigating the Alignment Tax of RLHF (2024.emnlp-main)

Copied to clipboard

Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang

Language Models

Challenge:	Large Language Models (LLMs) acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax.
Approach:	They propose to use a model averaging technique to find the most powerful alignment-forging Pareto front among RLHF algorithms.
Outcome:	The proposed method achieves the strongest alignment-forging Pareto front among competing methods.

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark (2024.acl-short)

Copied to clipboard

Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, Xiangliang Zhang

Natural Language Interfaces

Challenge:	SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology.
Approach:	They propose to use SceMQA to evaluate multimodal question answering at college entrance level.
Outcome:	The proposed model provides specific knowledge points for each problem and detailed explanations for each answer.

LongVideoAgent: Multi-Agent Reasoning with Long Videos (2026.acl-long)

Copied to clipboard

Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, Renjie Pi, Jipeng Zhang, Qifeng Chen

Multimodality

Challenge:	a key emerging challenge is robust long video understanding, authors say . current methods compress content into lossy summaries or rely on limited toolsets .
Approach:	They propose a multi-agent framework where a master LLM coordinates a grounding agent and a vision agent to extract targeted textual observations.
Outcome:	The proposed model outperforms strong non-agent baselines on episode-level datasets . the proposed model significantly outperformed existing models on other datasets.

Plum: Prompt Learning using Metaheuristics (2024.findings-acl)

Copied to clipboard

Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, KaShun Shum, Jipeng Zhang, Renjie Pi, Tong Zhang

Low-Resource NLP

Challenge:	Recent advances in prompt learning have led to a need for general prompt optimization methods.
Approach:	They propose a branch of discrete non-convex optimization methods with over 100 options as a promising approach to prompt learning.
Outcome:	The proposed methods can be used to discover more human-understandable prompts that were previously unknown in reasoning and image generation tasks.

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts (2024.emnlp-main)

Copied to clipboard

Ruida Wang, Jipeng Zhang, Yizhen Jia, Rui Pan, Shizhe Diao, Renjie Pi, Tong Zhang

Reasoning

Challenge:	a framework for formal proof writing using formal languages like Lean4 is needed to prove mathematical theorems using formal language.
Approach:	They propose a framework that trains a general-purpose LLM to be a Lean4 expert.
Outcome:	The proposed framework achieves cumulative accuracies of 36.48% and 33.61% on MiniF2F-Valid and Test datasets.

MR. Judge: Multimodal Reasoner as a Judge (2025.emnlp-main)

Copied to clipboard

Renjie Pi, Haoping Bai, Qibin Chen, Xiaoming Simon Wang, Jiulong Shan, Xiaojiang Liu, Meng Cao

Language Models

Challenge:	Effective reward modeling is especially valuable in reinforcement learning (RLHF) .
Approach:	They propose a paradigm for empowering general-purpose MLLMs judges with strong reasoning capabilities by using multiple-choice problem models instead of directly assigning scores.
Outcome:	The proposed model surpasses GPT-4o on VL-RewardBench and improves performance on MM-Vet by up to 7.7%.

Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models (2025.emnlp-main)

Copied to clipboard

Renjie Pi, Kehao Miao, Li Peihang, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou

Multimodality

Challenge:	Multimodal large language models exhibit a pronounced form of visual sycophantic behavior when they process image inputs.
Approach:	They propose a technique that allows multimodal large language models to engage in reflective reasoning and determine whether a user’s instruction is misleading or corrective.
Outcome:	The proposed model resists misleading instructions but is stubborn even if it is wrong.

Bridge-Coder: Transferring Model Capabilities from High-Resource to Low-Resource Programming Language (2025.findings-acl)

Copied to clipboard

Jipeng Zhang, Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Zheng Ziqiang, Tong Zhang

Multimodality

Challenge:	Large Language Models (LLMs) excel at generating code for high-resource programming languages (HRPLs) however, they struggle significantly with low-resourced programming languages such as D, exacerbating the digital divide.
Approach:	They propose a method to generate LRPL data using LLM's general knowledge, HRPL proficiency, and in-context learning capabilities.
Outcome:	The proposed method improves on R, D, Racket, and Bash, while maintaining the same quality.

TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data (2025.findings-naacl)

Copied to clipboard

Jipeng Zhang, Yaxuan Qin, Renjie Pi, Weizhong Zhang, Rui Pan, Tong Zhang

Semantic Text Processing

Challenge:	Existing methods for instruction tuning are limited due to the increasing volume of instruction datasets and the increased computational costs.
Approach:	They propose to extract a small and highly informative subset of training samples from a large dataset that achieves comparable performance to the full dataset.
Outcome:	The proposed algorithm outperforms other unsupervised methods and achieves comparable performance to the full dataset.

GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis (2024.acl-long)

Copied to clipboard

Yueqi Xie, Minghong Fang, Renjie Pi, Neil Gong

Language Models

Challenge:	Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs.
Approach:	They propose a method which scrutinizes the gradients of safety-critical parameters in large LLMs to detect jailbreak prompts.
Outcome:	The proposed method outperforms Llama Guard in detecting jailbreak prompts despite extensive finetuning with a large dataset.

ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting (2025.acl-long)

Copied to clipboard

Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, Tong Zhang

Green & Sustainable NLP

Challenge:	Existing paradigms for bilevel optimization require second-order information, making it difficult to scale them up.
Approach:	They propose a scalable instantiation of a bilevel optimization paradigm for large-scale LLMs by using a memory-efficient training technique.
Outcome:	The proposed paradigm scales to 30B-sized LLMs on 8H100 GPUs.

VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues (2025.acl-long)

Copied to clipboard

Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, Yi R. Fung

Visual Data in NLP

Challenge:	Existing vision-language models lack the ability to visually link matching visual cues across images or frames.
Approach:	They propose a benchmark to assess whether vision-language models can Visually Link Matching cues with 9 subtasks and over 3,000 test cases.
Outcome:	The proposed benchmarks on multiple images and videos do not demonstrate that vision-language models can link visual cues across images or frames.

ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects (2025.findings-emnlp)

Copied to clipboard

Jipeng Zhang, Haolin Yang, Kehao Miao, Ruiyuan Zhang, Renjie Pi, Jiahui Gao, Xiaofang Zhou

Programming Languages in NLP

Challenge:	Existing text-to-SQL models are limited to SQLite due to dataset limitations . data generated through static prompting is noisy and unreliable, authors say .
Approach:	They propose a text-to-SQL framework with execution-driven, agentic bootstrapping . ExeSQl bridges the dialect gap in text- to-Sql, achieving average improvements .
Outcome:	ExeSQL bridges the dialect gap in text-to-SQl, with average improvements of 15.2%, 10.38%, and 4.49% over GPT-4o on PostgreSQLE, MySQL, and Oracle.