Papers by Jianshu Zhang
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)
Copied to clipboard
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
| Challenge: | MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning. |
| Approach: | They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones. |
| Outcome: | The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. |
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the confidence calibration of LLMs have not explored the effects of different prompting strategies on LLM performance. |
| Approach: | They propose Fact-and-Reflection prompting which improves LLM confidence calibration . they propose to use human cognition to elicit known "facts" and ask model to "reflect" over them . |
| Outcome: | The proposed method lowers the expected calibration error by 23.5% on multi-purpose QA tasks. |
How do Words Contribute to Sentence Semantics? Revisiting Sentence Embeddings with a Perturbation Method (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on sentence representation learning focus on human annotation, but they neglect the critical property that essential contents should contribute to sentence semantics more than non-essential contents when encoding a sentence. |
| Approach: | They propose a perturbation method for unsupervised semantic analysis that uses a sentence compression metric to adapt sentence compression datasets for automatic evaluation. |
| Outcome: | The proposed method can capture the main semantics of sentences better than several SOTA unsupervised sentence embedding models. |
Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pretrained language models suffer from reporting bias, describing the lack of explicit commonsense knowledge in written text. |
| Approach: | They propose to endow language models with visual imagination capabilities by recalling existing images and synthesizing nonexistent images via text-to-image generation. |
| Outcome: | The proposed model improves the performance of existing language models across a diverse set of language tasks. |
ProgressLM: Towards Progress Reasoning in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models for task progress estimation lack long-horizon and dynamic reasoning . estimating how much of a task has been completed requires long-term reasoning based on partial information. |
| Approach: | They propose a benchmark for evaluating progress reasoning from a single observation . they instantiate a two-stage paradigm that combines episodic retrieval with mental simulation . |
| Outcome: | The proposed benchmark improves on 14 VLMs on a small scale and shows common failure patterns. |
UniTabNet: Bridging Vision and Language Models for Enhanced Table Structure Recognition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Table structure recognition technology is a critical tool for processing and analyzing large volumes of tabular data. |
| Approach: | They propose a framework for table structure parsing based on the image-to-text model and a vision guider to refine the model’s capability to understand textual semantics in table images. |
| Outcome: | The proposed framework improves on a dataset of PubTabNet, PubTables1M, WTW, and iFLYTAB and will be made publicly available. |
WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models (2026.acl-long)
Copied to clipboard
Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong
| Challenge: | Existing agentic systems are retrieval-heavy but reasoning-light . current systems lack compositional reasoning, a key component of deep research . |
| Approach: | They propose a data synthesis pipeline WebAggregator to shift agentic paradigm . they use Proactive Explorer to collect interconnected knowledge and Compositional Logic Proposer to weave knowledge into complex questions . |
| Outcome: | The proposed pipeline surpasses GPT-4.1 and matches Claude-3.7-Sonnet on GAIA, WebWalkerQA, and XBench. |
Bridge-Coder: Transferring Model Capabilities from High-Resource to Low-Resource Programming Language (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at generating code for high-resource programming languages (HRPLs) however, they struggle significantly with low-resourced programming languages such as D, exacerbating the digital divide. |
| Approach: | They propose a method to generate LRPL data using LLM's general knowledge, HRPL proficiency, and in-context learning capabilities. |
| Outcome: | The proposed method improves on R, D, Racket, and Bash, while maintaining the same quality. |
WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback (2025.findings-emnlp)
Copied to clipboard
Minda Hu, Tianqing Fang, Jianshu Zhang, Jun-Yu Ma, Zhisong Zhang, Jingyan Zhou, Hongming Zhang, Haitao Mi, Dong Yu, Irwin King
| Challenge: | Web agents powered by Large Language Models lack the ability to perform in uncertain web environments. |
| Approach: | They propose to reconstruct web agents' reasoning skills into chain-of-thought rationales by fine-tuning their LLM backbone into a web-based model. |
| Outcome: | The proposed approach significantly improves the agent self-improving benchmark OpenWebVoyager, demonstrating that it can be used to improve the agent's reasoning skills. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |
VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues (2025.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models lack the ability to visually link matching visual cues across images or frames. |
| Approach: | They propose a benchmark to assess whether vision-language models can Visually Link Matching cues with 9 subtasks and over 3,000 test cases. |
| Outcome: | The proposed benchmarks on multiple images and videos do not demonstrate that vision-language models can link visual cues across images or frames. |
ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Zero pronoun recovery and resolution aim at recovering the dropped pronounce and pointing out its anaphoric mentions. |
| Approach: | They propose to solve two tasks together to recover the dropped pronoun and point out its anaphoric mentions. |
| Outcome: | The proposed model outperforms previous state of the arts benchmarks on two benchmarks. |
Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to RLVR provide sparse supervision since reward arrives only after the full generation is complete. |
| Approach: | They propose a step-level reward system that extracts confidence and correctness and combines them into a Step Potential signal that explicitly estimates reasoning state at each step. |
| Outcome: | The proposed method outperforms existing methods on multiple benchmarks and improves accuracy while reducing response length. |
CEO: Corpus-based Open-Domain Event Ontology Induction (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing event-centric NLP models restrict their generalization capabilities by limiting the pre-defined ontology. |
| Approach: | They propose a Corpus-based Event Ontology induction model to relax the restriction imposed by pre-defined ontologies. |
| Outcome: | The proposed model can induce a hierarchical event ontology with meaningful names on eleven open-domain corpora, making it more trustworthy and easier to be further curated. |
EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing semantic similarity methods struggle to accurately identify semantically equivalent steps in domain-specific contexts like mathematical reasoning. |
| Approach: | They propose a simple yet effective approach that identifies and prunes semantically equivalent actions during LLM reasoning search. |
| Outcome: | The proposed approach reduces token consumption while improving reasoning efficiency and accuracy on Qwen2.5-Math-7B-Instruct on GSM8K. |
PIVOINE: Instruction Tuning for Open-world Entity Profiling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for information extraction focus on a closed-world setting, but PIVOINE is a promising solution to tackle the open-world problem of entity profiling. |
| Approach: | They propose to develop an LLM that performs Open-world Entity Profiling with instruction tuning to extract desirable entity profiles . they construct INSTRUCTOPENWIKI, a substantial instruction-tuning dataset for Open-World Entity Profiles . |
| Outcome: | The proposed model outperforms existing methods and ChatGPT-based baselines on unseen and out-of-ontology cases. |