Papers by Yilun Zhao

TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity (2026.findings-acl)

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Copied to clipboard

Zheyuan Yang, Liqiang Shang, Junjie Chen, Xun Yang, Chenglong Xu, Bo Yuan, Chenyuan Jiao, Yaoru Sun, Yilun Zhao

Yifei Shen, Yilun Zhao, Justice Ou, Tinglin Huang, Arman Cohan

Challenge:	TableVista evaluates multimodal table reasoning under visual and structural complexity . current models struggle to maintain reasoning consistency when structural complexity combined with visually integrated presentations.
Approach:	They propose a benchmark for evaluating multimodal table reasoning under visual and structural complexity.
Outcome:	The proposed model performs poorly on visual and structural complexity.

Copied to clipboard

A Survey of Reasoning-Intensive Retrieval: Progress and Challenges (2026.acl-long)

Challenge:	Despite recent advances, performance remains far from clinically reliable . specialized medical terminology and fine-grained temporal reasoning are key to executing clinical data analysis.
Approach:	They propose a benchmark for clinical text-to-SQL that demands multi-table joins, clinically meaningful filters, and executable SQL.
Outcome:	The proposed benchmark performs well on a set of 20 proprietary and open-source models . it scores 74.7% execution, while DeepSeek-R1 leads open-sourced at 69.2% .

Copied to clipboard

Yiyang Wei, Tingyu Song, Siyue Zhang, Yilun Zhao

TAIL: A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation (2024.emnlp-demo)

Challenge:	Reasoning-Intensive Retrieval (RIR) targets retrieval settings where relevance is mediated by latent inferential links between a query and supporting evidence, rather than semantic similarity.
Approach:	They propose a taxonomy that categorizes methods based on where and how reasoning is integrated into the retrieval pipeline.
Outcome:	The proposed method framework provides a detailed analysis of the current landscape and its trade-offs and practical applications.

Copied to clipboard

Gefei Gu, Yilun Zhao, Ruoxi Ning, Yanan Zheng, Arman Cohan

Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026.acl-long)

Challenge:	Existing evaluation methods for long-context large language models are overly simplistic and require extensive human annotations.
Approach:	They propose an automatic toolkit to create realistic evaluation benchmarks . they use a document-grounded benchmark to generate question-answer pairs .
Outcome:	The proposed toolkit provides a way to create realistic evaluation benchmarks and visualize performance metrics of evaluated models.

Copied to clipboard

Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao, Shu Wu

Latent Inter-User Difference Modeling for LLM Personalization (2025.emnlp-main)

Challenge:	Composed Image Retrieval (CIR) is a complex task in multimodal understanding . current CIR benchmarks lack a robust evaluation pipeline and limited query categories .
Approach:	They construct a fine-grained CIR benchmark that allows for precise control over modification types and content.
Outcome:	The proposed benchmark covers 5,000 high-quality queries structured across five main categories and fifteen subcategories.

Copied to clipboard

Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng

Semantic Text Processing

Challenge:	Large language models (LLMs) are increasingly integrated into users’ daily lives, leading to a growing demand for personalized outputs.
Approach:	They propose a framework that models inter-user differences in the latent space instead of relying on language-based prompts.
Outcome:	The proposed framework outperforms baseline methods on personalized review generation.

MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024.findings-acl)

Copied to clipboard

Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, Mark Gerstein

OpenRT: An Open-source Framework for Reasoning Over Tabular Data (2023.acl-demo)

Challenge:	Large language models face unique challenges such as domain-specific terminologies and reasoning over specialized knowledge.
Approach:	They propose a multi-disciplinary collaboration framework that leverages LLM-based agents in a role-playing setting.
Outcome:	The proposed framework excels at mining and harnessing medical expertise within LLMs, as well as extending its reasoning abilities.

Copied to clipboard

Yilun Zhao, Boyu Mi, Zhenting Qi, Linyong Nan, Minghao Guo, Arman Cohan, Dragomir Radev

Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions (2026.acl-long)

Challenge:	Existing table pre-training methods are benchmarked on a limited number of datasets with varying configurations, resulting in a lack of unified, standardized, fair, and comprehensive comparison between methods.
Approach:	They propose to use OpenRT to reproduce existing table pre-training models and develop new models quickly.
Outcome:	The proposed framework reproduces existing table pre-training models and compares them against four question answering, one fact checking, and one faithful text generation datasets.

Copied to clipboard

Minda Zhao, Yilun Du, Mengyu Wang

P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)

Challenge:	Existing large language models lack a functional internal sampler to faithfully sample from specified probability distributions . lack of robust sampling mechanisms across diverse application scenarios is a critical functional requirement .
Approach:	They propose to use a dual-protocol design to disentangle failure modes . batch generation achieves only modest statistical validity, while independent requests collapse almost entirely .
Outcome:	The proposed model fails to enforce uniform answer-position constraints and violates demographic targets in attribute-constrained text-to-image prompt synthesis.

Copied to clipboard

Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan

Information Extraction & Text Mining

Challenge:	Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales.
Approach:	They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans.
Outcome:	The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more.

FinDVer: Explainable Claim Verification over Long and Hybrid-content Financial Documents (2024.emnlp-main)

Copied to clipboard

Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, Arman Cohan

Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers (2025.emnlp-industry)

Challenge:	FinDVer is a benchmark to evaluate the explainable claim verification capabilities of LLMs . financial documents are typically long, intricate and dense, and they include both quantita and numerical reasoning.
Approach:	They propose a benchmark to evaluate the explainable claim verification capabilities of LLMs . they assess 25 LLM systems under long-context and RAG settings .
Outcome:	The proposed benchmark can be used to evaluate the explainable claim verification capabilities of LLMs in financial documents.

Copied to clipboard

Zhiyuan Peng, Ting-Ruen Wei, Tingyu Song, Yilun Zhao

FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain (2025.emnlp-main)

Challenge:	Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency and the number of forward passes.
Approach:	They propose to use a large language model to evaluate the efficiency of LLM-based rerankers . they propose to measure ranking quality and query processing efficiency using an interpretable FLOPs estimator .
Outcome:	The proposed metrics evaluate LLM-based rerankers with different architectures without running any experiments.

Copied to clipboard

Tiansheng Hu, Tongyan Hu, Liuyang Bai, Yilun Zhao, Arman Cohan, Chen Zhao

OpenT2T: An Open-Source Toolkit for Table-to-Text Generation (2024.emnlp-demo)

Challenge:	Recent LLMs have demonstrated promising ability in solving finance related problems, but applying them in real-world finance applications remains challenging due to its high risk and high stakes property.
Approach:	They propose a benchmark specifically designed for evaluating the trustworthiness of LLMs in finance applications.
Outcome:	The proposed benchmark outperforms proprietary models in most tasks while open-source models have advantage in specific areas like industry-level fairness.

Copied to clipboard

Haowei Zhang, Shengyun Si, Yilun Zhao, Lujing Xie, Zhijian Xu, Lyuhao Chen, Linyong Nan, Pengcheng Wang, Xiangru Tang, Arman Cohan

Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025.naacl-long)

Challenge:	Existing methods for table-to-text generation are limited and benchmarked on a limited number of datasets.
Approach:	They propose to use open-source tools to reproduce existing large language models for performance comparison and expedite the development of new models.
Outcome:	The proposed toolkit compares existing large language models on 9 table-to-text generation datasets and maintains a leaderboard to provide insights for future work.

Copied to clipboard

Yilun Zhao, Guo Gan, Chen Zhao, Arman Cohan

SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers (2025.emnlp-demos)

Challenge:	Recent-released MLLMs have shown remarkable performance on various multimodal math reasoning benchmarks.
Approach:	They introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models in handling multimodal math reasoning.
Outcome:	The proposed model performs well on a broad spectrum of 17 MLLMs and demonstrates that they are robust to adversarial perturbations.

Copied to clipboard

Zihang Wang, Yilun Zhao, Kaiyan Zhang, Chen Zhao, Manasi Patwardhan, Arman Cohan

MIMIR: A Customizable Agent Tuning Platform for Enhanced Scientific Applications (2024.emnlp-demo)

Challenge:	SCISKETCH is an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Approach:	They propose an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Outcome:	The open-source framework outperforms several state-of-the-art foundation models in generating schematic diagrams for scientific papers.

Copied to clipboard

Xiangru Tang, Chunyuan Deng, Hanminwang Hanminwang, Haoran Wang, Yilun Zhao, Wenqi Shi, Yi Fung, Wangchunshu Zhou, Jiannan Cao, Heng Ji, Arman Cohan, Mark Gerstein

TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction (2026.acl-long)

Challenge:	Large language models (LLMs) have evolved into interactive agents capable of planning, tool use, and task execution across various tasks.
Approach:	They propose a platform that leverages large language models to generate agent-tuning data for fine-tuneing smaller, specialized models.
Outcome:	MIMIR enables large models to simulate various roles and create interaction data, which can then be used to fine-tune open-source models like LLaMA2.

Copied to clipboard

Chengye Wang, Lin Fu, Zexi Kuang, Yilun Zhao

MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data (2022.acl-long)

Challenge:	Existing document OCR largely targets plain text or Markdown, discarding structural and executable properties that make LaTeX essential for scientific publishing.
Approach:	They propose a benchmark and a training corpus for document reconstruction . they train a 2B-parameter model using supervised fine-tuning and reinforcement learning .
Outcome:	The proposed model improves on existing models using supervised fine-tuning and reinforcement learning with verifiable rewards.

Copied to clipboard

Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang

ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples (2022.emnlp-main)

Challenge:	Existing benchmarks for numerical reasoning over hybrid data only include a single flat table in each document .
Approach:	They propose a new benchmark with QA pairs over multi hierarchical tabular and textual data.
Outcome:	The proposed model is more complex and challenging than existing benchmarks and is available on github . it uses facts retrieving to extract relevant facts from both tables and text and symbolic reasoning over retrieved facts.

Copied to clipboard

Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, Dragomir Radev

AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research (2025.acl-long)

Challenge:	Existing models with table-specific architectures and pre-training methods perform well on understanding table structures but lack table reasoning skills.
Approach:	They propose to pre-train tables with table reasoning skills without complex architectures . they define 7 table reasoning skill, and then pre-teach them to generate tables .
Outcome:	The proposed model improves on four tasks and is available on github.

Copied to clipboard

Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Chengye Wang, Yixin Liu, Lovekesh Vig, Arman Cohan

Table-R1: Inference-Time Scaling for Table Reasoning Tasks (2025.emnlp-main)

Challenge:	a benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research is available online.
Approach:	They propose to use a benchmark to evaluate LLMs' ability to design ablation studies . they investigate whether current automated evaluation methods are not reliable .
Outcome:	The benchmark compared leading LLMs with human experts on generating detailed ablation study designs . the results show that current evaluation methods are not reliable for the task .

Copied to clipboard

Zheyuan Yang, Lyuhao Chen, Arman Cohan, Yilun Zhao

Structured Data in NLP

Challenge:	In this study, we explore inference-time scaling on table reasoning tasks.
Approach:	They propose a large-scale dataset of reasoning traces and a reinforcement learning with verifiable rewards approach to enable inference-time scaling on table reasoning tasks.
Outcome:	The proposed model matches or exceeds GPT-4.1 and DeepSeek-R1 models on diverse table reasoning tasks.

KnowledgeFMath: A Knowledge-Intensive Math Reasoning Dataset in Finance Domains (2024.acl-long)

Copied to clipboard

Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, Arman Cohan

SciMDR: Advancing Scientific Multimodal Document Reasoning (2026.acl-long)

Challenge:	Existing benchmarks for large language models (LLMs) are only 56.6% accurate, leaving room for improvement.
Approach:	They propose a benchmark to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems using a finance-domain knowledge bank and expert-annotated solution references.
Outcome:	The proposed system achieves only 56.6% accuracy, leaving room for improvement.

Copied to clipboard

Ziyu Chen, Yilun Zhao, Chengye Wang, Rilyn R. Han, Manasi Patwardhan, Arman Cohan

Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024.emnlp-main)

Challenge:	Current models struggle to provide reliable assistance in real-world scientific workflows because evidence is distributed across long, multimodal documents.
Approach:	They propose a framework for QA Synthesis and document-scale regrounding that generates faithful, isolated QA pairs and reasoning on focused segments.
Outcome:	The proposed framework achieves significant improvements across multiple QA benchmarks, particularly in tasks requiring complex document-level reasoning.

Copied to clipboard

Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, JingBo Zhu

Revisiting Automated Evaluation for Long-form Table Question Answering (2024.emnlp-main)

Challenge:	Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset.
Approach:	They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR).
Outcome:	The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations.

Copied to clipboard

Yuqi Wang, Lyuhao Chen, Songcheng Cai, Zhijian Xu, Yilun Zhao

IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025.naacl-long)

Challenge:	Existing automated metrics for long-form table question answering (LFTQA) are poorly correlated with human judgments and fail to distinguish between factually accurate responses and those that are factual incorrect.
Approach:	They propose to use a meta-evaluation dataset to assess the effectiveness of LLM-based LFTQA systems.
Outcome:	The proposed meta-evaluation dataset includes 2,988 human-annotated examples.

Copied to clipboard

Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao

R2D2: Robust Data-to-Text with Replacement Detection (2022.emnlp-main)

Challenge:	Current information retrieval systems struggle to handle complex instructions, despite its critical importance . current models struggle to follow complex instructions in real-world applications, resulting in user-specific tasks.
Approach:	They propose a benchmark to evaluate instruction-following information retrieval in expert domains.
Outcome:	The proposed method improves on existing models and provides valuable insights to guide future advancements in retrieval.

Copied to clipboard

Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, Dragomir Radev

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)

Challenge:	Existing methods to mitigate unfaithful text generation are inadequate . data-to-text generation requires a structured input format .
Approach:	They propose a training framework that addresses unfaithful Data-to-Text generation by training a system as a generator and faithfulness discriminator with additional replacement detection and unlikelihood learning tasks.
Outcome:	The proposed training framework improves FeTaQA, LogicNLG, and ToTTo fidelity on D2T systems.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QA (2026.acl-long)

Challenge:	Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality.
Approach:	They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence.
Outcome:	The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability.

Copied to clipboard

Justice Ou, Tinglin Huang, Yilun Zhao, Ziyang Yu, Peiqing Lu, Yifei Shen, Rex Ying

ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)

Challenge:	Existing methods to improve the reliability of Large Language Models (LLMs) in clinical applications require factual knowledge from open-ended datasets and clinical case-based knowledge to provide context grounded in real-world patient experiences.
Approach:	They propose a retrieval-augmented generation framework based on the electronic health record to offer contextual information from other patients’ discharge reports.
Outcome:	The proposed framework outperforms a text-based ranker in a clinical QA dataset with 1,280 discharge-related questions .

Copied to clipboard

Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan

Robustness in NLP

Challenge:	Existing evaluations of large language models (LLMs) for instruction following are incomplete.
Approach:	They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets.
Outcome:	The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Z1: Efficient Test-time Scaling with Code (2025.emnlp-industry)

Copied to clipboard

Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

SciRAG: Adaptive, Citation-Aware, and Outline-Guided Retrieval and Synthesis for Scientific Literature (2026.eacl-long)

Challenge:	Large Language Models (LLMs) can achieve enhanced complex problem-solving through test-time computing scaling, but this often entails longer contexts and numerous reasoning token costs.
Approach:	They propose an efficient test-time scaling method that trains LLMs on code-related reasoning trajectories and a novel Shifted Thinking Window to mitigate overthinking overhead.
Outcome:	The proposed method reduces overthinking overhead while maintaining performance.

Copied to clipboard

Hang Ding, Yilun Zhao, Tiansheng Hu, Zihang Wang, Manasi Patwardhan, Arman Cohan

TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024.emnlp-industry)

Challenge:	Existing retrieval-augmented generation methods overlook citation graph structure, adapt poorly to complex queries, and yield fragmented, hard-to-verify syntheses.
Approach:	They propose a retrieval-augmented generation framework that addresses these gaps by combining adaptive retrieval and symbolic reasoning.
Outcome:	Extensive experiments show that SciRAG outperforms prior systems in factual accuracy and synthesis quality.

Copied to clipboard

Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang

Semantic Text Processing

Challenge:	Large language models have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools.
Approach:	They propose a framework to enhance the task planning and tool usage abilities of LLMs in industrial systems.
Outcome:	The proposed framework enhances the task planning and tool usage abilities of LLM-based agents in industrial systems.

WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code (2025.findings-acl)

Copied to clipboard

Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, Xuelong Li

Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers (2025.findings-acl)

Challenge:	Existing benchmarks for large language models focus on webpage generation outcomes.
Approach:	They propose a multi-view evaluation framework to evaluate MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code.
Outcome:	The proposed framework evaluates MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code.

Copied to clipboard

Yilun Zhao, Chengye Wang, Chuhan Li, Arman Cohan

Explainability & Interpretability in NLP

Challenge:	MISS-QA is the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature.
Approach:	They propose an automated evaluation protocol powered by open-source LLMs trained on human-scored data to ensure reliable evaluation.
Outcome:	The proposed protocol is powered by open-source LLMs trained on human-scored data.

Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization (2025.findings-acl)

Copied to clipboard

Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, Tat-Seng Chua

Challenge:	Extensive experiments on real-world datasets demonstrate that DPL significantly enhances LLM personalization.
Approach:	They propose a novel approach that emphasizes extracting inter-user differences to enhance LLM personalization.
Outcome:	The proposed approach extracts inter-user differences to enhance LLM personalization.

Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)

Copied to clipboard

Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan

Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline.
Approach:	They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews.
Outcome:	The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms.

Copied to clipboard

Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu, Chen Zhao

Semantic Text Processing

Challenge:	Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks.
Approach:	They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training.
Outcome:	The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks.

Part Represents Whole: Improving the Evaluation of Machine Translation System Using Entropy Enhanced Metrics (2022.findings-aacl)

Copied to clipboard

Yilun Liu, Shimin Tao, Chang Su, Min Zhang, Yanqing Zhao, Hao Yang

Machine Translation

Challenge:	Existing machine translation metrics have poor correlations with human assessments . entropy-based evaluations are often limited to a limited number of samples .
Approach:	They propose a fast and unsupervised approach to enhance machine translation metrics using entropy by introducing sentence-level difficulty.
Outcome:	The proposed method outperforms existing metrics on five sub-tracks in the WMT19 Metrics shared tasks.

RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations (2023.acl-long)

Copied to clipboard

Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, Dragomir Radev

OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems (2024.emnlp-industry)

Challenge:	Existing Table QA models are vulnerable to task-specific perturbations, such as replacing key question entities or shuffling table columns.
Approach:	They propose to use large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.
Outcome:	The proposed model significantly improves on existing Table QA models against human-annotated adversarial perturbations.

Copied to clipboard

Linyong Nan, Weining Fang, Aylin Rasteh, Pouya Lahabi, Weijin Zou, Yilun Zhao, Arman Cohan

Natural Language Interfaces

Challenge:	Existing approaches to QA require multiple modalities and a broad pool of information sources to generate coherent answers.
Approach:	They propose a new resource to evaluate the effectiveness of question answering systems that perform retrieval augmented generation in scenarios that demand reasoning on multi-modal, multi-document contexts.
Outcome:	The proposed method evaluates question answering systems that perform retrieval augmented generation (RAG) in open-domain questions . it requires systems to navigate diverse modalities and a broad pool of information sources, making it uniquely challenging.

MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application (2026.acl-long)

Copied to clipboard

Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Vincent Jim Zhang, Yuqing Guo, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Zhiyang Deng, Heming Fu, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Guojun Xiong, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)

Challenge:	Existing evaluations of LLMs in finance are text-only, monolingual, and largely saturated by current models.
Approach:	They propose a multilingual and multimodal benchmark for evaluating LLMs in real financial contexts.
Outcome:	The first expert-annotated multilingual and multimodal benchmark is released . it evaluates 21 leading LLMs and shows they perform better in multilingual settings .

Copied to clipboard

Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

Challenge:	a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision.
Approach:	They introduce Physics, a benchmark for university-level physics problem solving.
Outcome:	The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems .

A Survey on Evaluation of LLM-based Agents (2026.findings-acl)

Copied to clipboard

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer

Challenge:	This paper provides the first comprehensive survey of evaluation methods for LLM-based agents . LLMs are static, having fixed knowledge, and confined to text-to-text interaction.
Approach:	They analyze the evaluation of LLM-based agents across five perspectives . they identify current trends and key gaps in evaluation methods .
Outcome:	The proposed evaluation frameworks and tools are based on five perspectives . the results highlight current trends and identify gaps in future research .

TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024.acl-long)

Copied to clipboard

Yilun Zhao, Lyuhao Chen, Arman Cohan, Chen Zhao

Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023.emnlp-industry)

Challenge:	Long-form table question answering often generates paragraph long and complex answers . a prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context.
Approach:	They propose a modular framework that decomposes the whole process into three sub-modules . framework produces a QA-based plan first, followed by generating an answer conditioned on this plan . human evaluation results indicate the framework improves strong baselines on accuracy and truthfulness .
Outcome:	The proposed framework improves accuracy and truthfulness on the FeTaQA and QTSumm datasets.

Copied to clipboard

Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, Arman Cohan

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)

Challenge:	Existing table-to-text generation techniques that transform complex tabular data into comprehensible narratives are lacking in real-world applications.
Approach:	They investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios.
Outcome:	The proposed models can generate table-to-text data in two real-world information seeking scenarios and perform better than existing models.

Copied to clipboard

Zhaojian Yu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

Programming Languages in NLP

Challenge:	Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks.
Approach:	They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks .
Outcome:	The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks.

On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering (2024.findings-naacl)

Copied to clipboard

Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, Arman Cohan

Natural Language Interfaces

Challenge:	a new study evaluates how Large Language Models interact with a SQL interpreter . the model is limited in context and is stochastic, making it less suited for tasks requiring high precision and extensive computations.
Approach:	They propose and evaluate two interaction strategies to evaluate how LLMs interact with a SQL interpreter.
Outcome:	The proposed framework improves the accuracy and reliability of the evaluations.

LoFT: Enhancing Faithfulness and Diversity for Table-to-Text Generation via Logic Form Control (2023.eacl-main)

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Lorenzo Jaime Flores, Dragomir Radev

Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026.acl-long)

Challenge:	Existing models for LT2T generation focus on surface-level realizations without much logical inference.
Approach:	They propose a model that uses logic forms as fact verifiers and content planners to control LT2T generation.
Outcome:	Experimental results show that the proposed model addresses unfaithfulness and diversity issues simultaneously.

Copied to clipboard

Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao, Arman Cohan

Web Fraud Attacks Against LLM-Driven Multi-Agent Systems (2026.findings-acl)

Challenge:	Existing evaluation benchmarks for retrievers are narrow and evaluate them in isolation . existing evaluation benchmarking frameworks focus on evaluating retrievers in isolation, obscuring their value in real-world applications.
Approach:	They propose an evaluation framework that evaluates retrievers in agentic search systems . they provide expert-annotated reasoning aspects, positive documents, a reference response and evaluation rubrics .
Outcome:	The proposed framework assesses retrievers in agentic search systems.

Copied to clipboard

Dezhang Kong, Hujin Peng, Yilun Zhang, Lele Zhao, Zhenhua Xu, Shi Lin, Changting Lin, Meng Han

Robustness in NLP

Challenge:	Large Language Model (LLM)-driven multi-agent systems (MAS) are rapidly gaining popularity, and its inherent security risks are rapidly becoming a concern.
Approach:	They propose a novel attack manipulating unique structures of web links to deceive MAS by using homoglyph deception, sub-directory nesting, and parameter obfuscation.
Outcome:	The proposed attacks exploit unique structures of web links to deceive MAS . they exhibit significant destructive potential across different MAS architectures .

Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)

Copied to clipboard

Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, Mark Gerstein

FinMath: Injecting a Tree-structured Solver for Question Answering over Financial Reports (2022.lrec-1)

Challenge:	Large Language Models (LLMs) have advanced capabilities but produce complex structured data.
Approach:	They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs.
Outcome:	The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats.

Copied to clipboard

Chenying Li, Wenbo Ye, Yilun Zhao

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)

Challenge:	Existing models for answering complex questions require multiple-step numerical reasoning.
Approach:	They propose a framework that injects a tree-structured neural model into a model to perform multi-step numerical reasoning.
Outcome:	The proposed framework improves the previous best model by 8.5% absolute for Exact Match (EM) score and 6.1% absolute for numeracy-focused F1 score.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

Information Extraction & Text Mining

Challenge:	Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied.
Approach:	They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance .
Outcome:	The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs.

SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification (2025.acl-long)

Copied to clipboard

Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, Yilun Zhao

LimRank: Less is More for Reasoning-Intensive Information Reranking (2025.emnlp-main)

Challenge:	Existing scientific claim verification benchmarks focus on textual content alone or on verifying claims based on a single table.
Approach:	They propose to use SciVer to evaluate the ability of foundation models to verify claims within a multimodal scientific context.
Outcome:	The proposed model outperforms 21 state-of-the-art models and human experts on SciVer.

Copied to clipboard

Tingyu Song, Yilun Zhao, Siyue Zhang, Chen Zhao, Arman Cohan

Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge (2025.findings-emnlp)

Challenge:	Existing approaches to rerank information require large-scale fine-tuning, which is computationally expensive.
Approach:	They propose an open-source pipeline for generating diverse, challenging, and realistic reranking examples.
Outcome:	The proposed model performs competitively on two benchmarks, while being trained on less than 5% of the data typically used in prior work.

Copied to clipboard

Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, Soroush Vosoughi

A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning (2026.acl-long)

Challenge:	LLM-as-Judge frameworks provide scalable alternative to human evaluation . but the question of how intrinsic biases manifest in these settings remains unexplored .
Approach:	They conduct systematic analysis of four bias types in multi-agent LLM-as-Judge frameworks . they find debate framework amplifies biases sharply after initial debate .
Outcome:	The proposed frameworks amplify biases after debate and show they are stronger in meta-judge scenarios.

Copied to clipboard

Tianyu Yang, Sihong Wu, Yilun Zhao, Zhenwen Liang, Lisen Dai, Chen Zhao, Minhao Cheng, Arman Cohan, Xiangliang Zhang

Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation (2024.findings-acl)

Challenge:	Multimodal mathematical Reasoning (MMR) has attracted increasing attention for its ability to solve mathematical problems involving both textual and visual modalities.
Approach:	They review the theoretical frameworks of multimodal reasoning and examine the challenges they face in visual math tasks.
Outcome:	The proposed models can solve problems involving both textual and visual modalities.

Copied to clipboard

Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, Arman Cohan

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)

Challenge:	Data contamination is a problem in Large language models due to the reliance on extensive internet-derived training corpora.
Approach:	They present a survey on the topic of data contamination in large language models.
Outcome:	The results of the first survey on data contamination in large language models provide a comprehensive guide for NLP researchers seeking a systematic understanding of the issue.

Copied to clipboard

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

Text Clustering

Challenge:	Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks.
Approach:	They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters .
Outcome:	The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data.

Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)

Copied to clipboard

Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev

Programming Languages in NLP

Challenge:	In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory.
Approach:	They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs.
Outcome:	The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset.

MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding (2026.acl-long)

Copied to clipboard

Xue Xia, Zheyuan Yang, Arman Cohan, Yilun Zhao

Programming Languages in NLP

Challenge:	MMSciCode is a benchmark for evaluating foundation models in scientific code generation.
Approach:	They propose a multilingual, multi-discipline benchmark for evaluating foundation models in scientific code generation that integrates domain-specific knowledge with algorithmic reasoning.
Outcome:	The new benchmark is annotated by domain experts and features rigorous quality controls to ensure dataset integrity and authenticity.

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models (2026.acl-long)

Copied to clipboard

Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui HE, Chenxin Liu, Zhang Li, null Mahongxia, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao

Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers (2025.acl-long)

Challenge:	Existing multilingual evaluation benchmarks neglect cultural nuances and lack language coverage in subjective tasks.
Approach:	They propose a framework that categorizes evaluation tasks into three cultural layers and nine cognitive sub-layers.
Outcome:	The proposed framework surpasses prior coverage by up to 111% on 20+ LLMs.

Copied to clipboard

Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, Arman Cohan

Challenge:	Recent advances in large language models (LLMs) have demonstrated remarkable capabilities across a variety of scientific tasks, such as answering questions about scientific papers, writing scientific papers and retrieving related works.
Approach:	They propose a taxonomy of limitation types in scientific research with a focus on AI to evaluate their ability to support early-stage feedback and complement human peer review.
Outcome:	The proposed model enhances the ability of LLM systems to generate limitations in research papers, enabling them to provide more concrete and constructive feedback.

A Corpus of Adpositional Supersenses for Mandarin Chinese (2020.lrec-1)

Copied to clipboard

Siyao Peng, Yang Liu, Yilun Zhu, Austin Blodgett, Yushi Zhao, Nathan Schneider

Challenge:	Adpositions are frequent markers of semantic relations, but they are highly ambiguous and vary significantly from language to language.
Approach:	They propose to annotate Chinese adpositions in a corpus with all aforementioned supersenses . they adapt a framework that defined a set of supersens according to ostensibly language-independent criteria .
Outcome:	The proposed corpus is the first to be broadly annotated with adposition semantics in Chinese . it shows that the supersense categories are well-suited to Chinese adepositions despite syntactic differences from English .

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)

Copied to clipboard

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Robustness in NLP

Challenge:	Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale.
Approach:	They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark.
Outcome:	The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement.

Anchor: Branch-Point Data Generation for GUI Agents (2026.acl-long)

Copied to clipboard

Jinbiao Wei, Yilun Zhao, Kangqi Ni, Arman Cohan

Challenge:	Existing GUI agents for real desktop environments require large amounts of high-quality interaction data, but collecting human demonstrations is expensive.
Approach:	They propose a framework that bootstraps scalable desktop supervision from seed demonstrations.
Outcome:	Experiments on standard desktop benchmarks show that the framework improves on zero-shot agents and representative synthesis baselines.

Investigating Data Contamination in Modern Benchmarks for Large Language Models (2024.naacl-long)

Copied to clipboard

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, Arman Cohan

Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure (2025.acl-short)

Challenge:	Existing evaluation benchmarks for large language models are inflated and inconsistent with actual performance.
Approach:	They propose a retrieval-based system to explore potential overlaps between benchmarks and pretraining corpora and a protocol to investigate testset slot guessing.
Outcome:	The proposed method exploits overlaps between evaluation benchmarks and pretraining corpora and masks a wrong answer in a multiple choice question and prompts the model to fill in the gap.

Copied to clipboard

Zheyuan Yang, Zexi Kuang, Xue Xia, Yilun Zhao

Challenge:	TestCase-Eval focuses on Fault Coverage and Fault Exposure tasks . authors provide insights into their strengths and limitations in generating effective test cases . correctness and robustness of algorithmic solutions hinge on quality of test suites .
Approach:	They introduce TestCase-Eval, a benchmark for systematic evaluation of LLMs in test-case generation.
Outcome:	The new benchmark measures the performance of LLMs in test-case generation.

QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation (2026.findings-acl)

Challenge:	Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights.
Approach:	They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary.
Outcome:	The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input.

Copied to clipboard

Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, Arman Cohan

Challenge:	Prior studies show that large language models can draft fluent reviews but they miss specific issues, show shallow analysis, and produce generic phrasing.
Approach:	They propose a task that targets actionable review feedback generation and places existing peer review rebuttal at the center of learning.
Outcome:	The proposed model improves on a large dataset that maps review segments to rebuttal segments that address them, with perspective labels and impact categories that order author uptake.

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs (2026.acl-long)

Copied to clipboard

Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs (2026.findings-acl)

Challenge:	NVFP4 supports fine-grained block isolation, 4-bit quantization errors and mixed-precision approaches . ARCQuant boosts NVFO4 performance via Augmented Residual Channels .
Approach:	They propose a framework that boosts NVFP4 performance via Augmented Residual Channels.
Outcome:	ARCQuant boosts NVFP4 performance via Augmented Residual Channels . the proposed framework achieves state-of-the-art accuracy comparable to full-precision baselines compared to FP16 .

Copied to clipboard

Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi

Information Extraction & Text Mining

Challenge:	Reinforcement learning (RL) is a paradigm for post-training large language models, but it suffers from exploration collapse . a new study finds that RL fails to reward correct solutions that exhibit rare high-level strategies .
Approach:	They propose a method that rewards correct solutions that exhibit rare high-level strategies by clustering rollouts according to their high- level solution strategies.
Outcome:	The proposed approach improves pass@k across large sampling budgets and increases area under the pass@K curve (AUC@K) without sacrificing pass@1.

VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025.acl-long)

Copied to clipboard

Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao

MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search (2025.findings-emnlp)

Challenge:	Multimodal large language models (MLLMs) are used for video quality assessment, image captioning and video analysis.
Approach:	They propose a benchmark to evaluate MLLMs on AIGC videos using coherence validation, error awareness, error type detection and reasoning evaluation tasks.
Outcome:	The proposed benchmark evaluates 13 frontier MLLMs on AIGC videos.

Copied to clipboard

Yunhai Hu, Yilun Zhao, Chen Zhao, Arman Cohan

SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering (2025.emnlp-main)

Challenge:	MCTS-RAG combines structured reasoning with adaptive retrieval . compared to conventional MCTLs, MCTR-RAg relies on internal model knowledge without external facts .
Approach:	a new approach integrates retrieval-augmented generation and Monte Carlo Tree Search to enhance reasoning capabilities of small language models.
Outcome:	MCTS-RAG integrates retrieval-augmented generation and Monte Carlo Tree Search to improve reasoning paths.

Copied to clipboard

Kaiyue Feng, Siyue Zhang, Bingsen Chen, Yilun Zhao, Chen Zhao