Papers by Jiaxuan You
TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents (2025.emnlp-demos)
Copied to clipboard
Haofei Yu, Keyang Xuan, Fenghai Li, Kunlun Zhu, Zijie Lei, Jiaxun Zhang, Ziheng Qi, Kyle Richardson, Jiaxuan You
| Challenge: | Existing research systems often design and use agentic workflows to perform research tasks such as ideation, scientific coding, review writing, and tree-based search. |
| Approach: | They propose an open-source codebase, an interactive web demonstration, and a PyPI Python package to make state-of-the-art auto-research pipelines broadly accessible to every researcher and developer. |
| Outcome: | The proposed framework adapts easily to new tools and supports iterative growth. |
In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in artificial intelligence have led to the creation of highly capable large language models (LLMs) that can perform tasks in a human-like manner, but lack infant-level cognitive abilities in certain areas. |
| Approach: | They designed a text-based multi-choice QA scenario similar to the A-Not-B error to test their inhibitory control abilities. |
| Outcome: | The proposed model shows that state-of-the-art LLMs perform well with in-context learning but make errors and show a drop of as many as 83.3% in reasoning tasks when the context changes trivially. |
LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences. |
| Approach: | They propose a framework which extends established benchmarks to sequential problem-solving settings and provides feedback after each round to build a demonstration memory that the models can query in future tasks. |
| Outcome: | The proposed framework can improve performance of LLMs by learning from past interactions and improve models' performance over time. |
GUIDE: Towards Scalable Advising for Research Ideas (2026.acl-long)
Copied to clipboard
| Challenge: | Existing systems that provide detailed, constructive feedback on academic papers struggle with review fidelity. |
| Approach: | They explore factors that underlie the development of robust advising systems . large language models have shown remarkable progress in tasks from text generation to code synthesis . |
| Outcome: | The proposed model outperforms general-purpose language models in acceptance rates for self-ranked top-30% submissions to ICLR 2025. |
Arxiv Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing solutions for document QA fail to provide personalized and up-to-date information efficiently. |
| Approach: | They propose to deploy a self-evolving, efficient LLM system that can offer personalized research services, maintaining a real-time updated database. |
| Outcome: | The proposed system saves 69.92% of time after efficient deployment. |
Beyond Facts: Evaluating Intent Hallucination in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) produce unsatisfactory results when faced with complex queries containing multiple conditions. |
| Approach: | They propose a benchmark for intent hallucination that covers 20,068 problems and an automatic LLM generation evaluation metric for detecting intent hallucinosis. |
| Outcome: | The proposed benchmark covers query-only and retrieval-augmented generation (RAG) setups with varying topics and difficulty. |
ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional self-consistency methods fail to capture subtle semantic errors in multi-step tasks. |
| Approach: | They propose a tree-based evaluation framework that measures LLMs’ ability to preserve semantic consistency during reversible transformations. |
| Outcome: | The proposed framework measures generalization abilities across models from 1.5B to 72B and can be used to benchmark LLMs without constructing new datasets. |
MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, Jiaxuan You
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. |
| Approach: | They propose a benchmark to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. |
| Outcome: | The proposed framework measures task completion and quality of collaboration and competition using novel, milestone-based key performance indicators. |
Evaluating LLM-Generated Diagrams as Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods on diagram generation with LLMs rely heavily on proprietary LLM systems. |
| Approach: | They propose a new evaluation metric to assess demonstration diagrams generated by large language models. |
| Outcome: | The proposed evaluation metric evaluates diagrams produced by state-of-the-art LLMs on recent research literature. |
DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to support academic rebuttal rely on off-the-shelf LLMs or simple pipelines that struggle with long-context understanding. |
| Approach: | They propose an agentic framework for automatic academic rebuttal generation that operates through four steps: Decompose reviews into atomic concerns, Retrieve relevant evidence from the paper, Plan refortations, and Generate responses accordingly. |
| Outcome: | The proposed framework outperforms existing rebuttal pipelines and achieves 98% accuracy beyond the average human level using only an 8B model. |
Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieving historical LLM responses are lacking in long-context summarization tasks. |
| Approach: | They propose a graph of records which leverages historical LLM responses to enhance RAG for long-context global summarization. |
| Outcome: | The proposed method improves on four long-context summarization datasets. |
SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific Discovery (2025.emnlp-main)
Copied to clipboard
Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, Jiaxuan You
| Challenge: | Recent advances in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet raised critical ethical and safety concerns. |
| Approach: | They propose a framework to enhance safety and ethical responsibility in AI-driven scientific exploration. |
| Outcome: | The proposed framework significantly improves safety performance by 35% compared to traditional frameworks. |