Papers by James Zou
ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior studies assess instruction adherence in the model’s main responses, but it is also critical for large reasoning models to follow user instructions throughout their reasoning process. |
| Approach: | They propose a systematic benchmark for assessing reasoning instruction following to assess the model's adherence to instructions. |
| Outcome: | The proposed benchmark reduces the risk of undesirable shortcuts, hallucinations, or reward hacking within reasoning traces. |
Science Across Languages: Assessing LLM Multilingual Translation of Scientific Papers (2026.findings-eacl)
Copied to clipboard
| Challenge: | a large number of scientific journals are published exclusively in English . this creates barriers for non-native English speakers to access scientific knowledge . |
| Approach: | They propose a way to translate scientific articles while preserving native JATS XML formatting. |
| Outcome: | The proposed approach shows that the key scientific details are accurately conveyed. |
Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Medical reasoning in large language models is a complex cognitive process through which clinicians interpret patient data and make diagnostic and therapeutic decisions. |
| Approach: | They propose an evaluation framework that disentangles knowledge recall from reasoning by training a PubMedBERT-based classifier and applying it to 11 widely used biomedical QA benchmarks. |
| Outcome: | The proposed evaluation framework disentangles knowledge recall from reasoning by training a PubMedBERT-based classifier and applying it to 11 widely used biomedical QA benchmarks. |
ALICE: Active Learning with Contrastive Natural Language Explanations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Annotating a large dataset with annotations is costly and infeasible. |
| Approach: | They propose an expert-in-the-loop training framework that utilizes contrastive natural language explanations to improve data efficiency in learning. |
| Outcome: | The proposed framework outperforms baseline models trained with 40-100% more training data on bird species classification and social relationship classification tasks. |
Inefficiencies of Meta Agents for Agent Design (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has automated the design of agentic systems using meta-agents . authors examine three key challenges in a common class of meta-gents. |
| Approach: | They examine how meta-agents learn across iterations and show performance improves with evolutionary approach. |
| Outcome: | The proposed meta-agents perform worse when iterating on multiple agents than human-designed agents. |
SEAL: Interactive Tool for Systematic Error Analysis and Labeling (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing models that fail on tail data or rare groups are difficult to identify due to lack of explicit labels. |
| Approach: | They propose a systematic error analysis and labeling tool that uses a two-step approach to identify high-error slices of data and then give human-understandable semantics to those underperforming slices. |
| Outcome: | The proposed tool identifies high-error slices of data and gives human-understandable semantics to those underperforming slices. |
OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing prompting methods for large language models (LLMs) are restricted to specialized domains, limited tool types, or require additional training data. |
| Approach: | They propose a training-free, user-friendly, and easily extensible multi-agent framework designed to tackle complex reasoning across diverse domains. |
| Outcome: | The proposed framework outperforms AutoGen, GPT-Functions, and LangChain by up to 10.6% when given the same set of tools. |
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models (2023.findings-acl)
Copied to clipboard
Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z. HaoChen, James Zou, Percy Liang, Serena Yeung
| Challenge: | Recent studies show that some tasks exhibit inverse scaling, or U-shaped scaling, where the performance degrades as models are scaled up. |
| Approach: | They propose a task that asks questions with negation to show positive scaling . they hypothesize that solving NeQA depends on question answering and negation understanding . |
| Outcome: | The proposed task can exhibit inverse scaling, U-shaped scaling, or positive scaling, and the scaling trends shift as the task is more powerful. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) is of importance in multimodal retrieval applications. |
| Approach: | They propose a two-stage pruning and merging frameworks that combine pruning and merge techniques to achieve higher compression rates. |
| Outcome: | The proposed framework outperforms existing methods on 29 visual document retrieval datasets. |
Protein Large Language Models: A Comprehensive Survey (2025.findings-emnlp)
Copied to clipboard
Yijia Xiao, Wanjia Zhao, Junkai Zhang, Yiqiao Jin, Han Zhang, Zhicheng Ren, Renliang Sun, Haixin Wang, Guancheng Wan, Pan Lu, Xiao Luo, Yu Zhang, James Zou, Yizhou Sun, Wei Wang
| Challenge: | Existing studies focus on specific aspects or applications, but this study provides a comprehensive overview of Protein-specific large language models. |
| Approach: | This paper proposes a structured taxonomy of state-of-the-art ProteinLLMs . they analyze how they leverage large-scale protein sequence data for improved accuracy . |
| Outcome: | The proposed model covers their architectures, training datasets, evaluation metrics, and diverse applications. |
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2026.eacl-long)
Copied to clipboard
| Challenge: | Unlike fine-tuning or static retrieval methods, DC adapts LMs’ problem-solving skills on the fly, without modifying their underlying parameters. |
| Approach: | They propose a lightweight framework that endows a black-box LM with a persistent, evolving memory. |
| Outcome: | The proposed framework enables models to store and reuse accumulated strategies, code snippets, and general problem-solving insights at inference time. |
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic dialog evaluation metrics are mostly reference-based . Existing models that measure self-reported user ratings are biased and variance among different users. |
| Approach: | They propose an automatic evaluation model that automatically cleans self-reported user ratings as it trains on them. |
| Outcome: | The proposed model achieves 89.2% accuracy in the dialog comparison task. |
Zero-Shot Open-Schema Entity Structure Discovery (2026.eacl-long)
Copied to clipboard
Xueqiang Xu, Jinfeng Xiao, James Barry, Mohab Elkaref, Jiaru Zou, Pengcheng Jiang, Yunyi Zhang, Maxwell J Giammona, Geeth De Mel, Jiawei Han
| Challenge: | Existing methods based on large language models (LLMs) rely heavily on predefined entity attribute schemas or annotated datasets, often leading to incomplete extraction results. |
| Approach: | They propose a novel approach to entity structure extraction that does not require any schema or annotated datasets. |
| Outcome: | Experiments show that ZOES improves LLMs’ ability to extract more complete entity structures across three different domains, showcasing both the effectiveness and generalizability of the method. |
Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Small shifts in user behavior can cause sharp drops in agent performance . prior work has shown that LLMs lack robustness to real-world noise and small input perturbations. |
| Approach: | They propose a model-agnostic method for systematically stress testing AI agents that learns directions in activation space corresponding to steerable user traits. |
| Outcome: | The proposed method can be used to stress test AI agents in airline, retail, telecom, and telehealth domains. |
Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings (N19-1)
Copied to clipboard
| Challenge: | a new framework for studying political polarization in social media is needed to understand how group divisions manifest in language. |
| Approach: | They propose to cluster tweet embeddings to uncover four dimensions of political polarization in social media . their results apply existing lexical methods to analyze 4.4M tweets on 21 mass shootings . |
| Outcome: | The proposed framework generates more cohesive topics than traditional models. |