Papers by Arman Cohan

SCIURus: Shared Circuits for Interpretable Uncertainty Representations in Language Models (2025.naacl-long)

Challenge:	Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks.
Approach:	They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search.
Outcome:	The proposed model outperforms existing models by over 2 points absolute.

Copied to clipboard

Carter Teplica, Yixin Liu, Arman Cohan, Tim G. J. Rudner

FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)

Challenge:	Existing methods for uncertainty quantification in large language models provide little insight into factors responsible for an uncertainty estimate, limiting their usefulness as practical tools for improving trustworthiness and understanding uncertainty reasoning.
Approach:	They adapt causal tracing and zero-ablation techniques to study the effect of different circuits on LLM generation to identify whether factuality of generated responses and uncertainty originate in separate or shared circuits.
Outcome:	The proposed methods use the well-established methods of causal tracing and zero-ablation to study the effect of different circuits on LLM generation.

Copied to clipboard

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev

On Learning to Summarize with Large Language Models as References (2024.naacl-long)

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Copied to clipboard

Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan

SciBERT: A Pretrained Language Model for Scientific Text (D19-1)

Challenge:	Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets.
Approach:	They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved.
Outcome:	The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance.

Copied to clipboard

Iz Beltagy, Kyle Lo, Arman Cohan

Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering (2023.acl-long)

Challenge:	SciBERT is a pretrained language model based on BERT to improve performance on scientific NLP tasks.
Approach:	They propose a pretrained language model based on BERT to improve NLP performance . they evaluate on sequence tagging, sentence classification and dependency parsing .
Outcome:	The proposed model improves on sequence tagging, sentence classification and dependency parsing tasks with datasets from a variety of scientific domains.

Copied to clipboard

Avi Caciularu, Matthew Peters, Jacob Goldberger, Ido Dagan, Arman Cohan

ABNIRML: Analyzing the Behavior of Neural IR Models (2022.tacl-1)

Challenge:	Among recent NLP research, multi-document processing is gaining increasing attention due to the need to handle and process an increasing amount of textual data and available documents online.
Approach:	They propose to pre-train a generic multi-document model from a cross-document question answering pre-training objective by generating salient sentences from one document and challenging it to recover the sentence from which it was generated.
Outcome:	The proposed model outperforms zero-shot GPT-3.5 and GPT-4 in multiple document tasks and generates the correct answer and the salient sentence from a salient document.

Copied to clipboard

Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, Arman Cohan

On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization (2024.eacl-short)

Challenge:	Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc ranking.
Approach:	They propose a framework for Analyzing the Behavior of Neural IR ModeLs that includes new types of diagnostic probes that allow us to test several characteristics that are not addressed by previous techniques.
Outcome:	The proposed framework tests writing styles, factuality, sensitivity to paraphrasing and word order, and can be used to identify unintended biases.

Copied to clipboard

Lorenzo Jaime Flores, Arman Cohan

Yifei Shen, Yilun Zhao, Justice Ou, Tinglin Huang, Arman Cohan

Challenge:	Text summarization and simplification are among the most widely used applications of NLP, but they are prone to hallucination due to training on unaligned data.
Approach:	They propose a loss-truncation approach to modify the standard log loss to adaptively remove noisy examples during training to improve model performance.
Outcome:	The proposed approach yields a considerable number of hallucinated entities on various datasets.

Copied to clipboard

CDLM: Cross-Document Language Modeling (2021.findings-emnlp)

Challenge:	Despite recent advances, performance remains far from clinically reliable . specialized medical terminology and fine-grained temporal reasoning are key to executing clinical data analysis.
Approach:	They propose a benchmark for clinical text-to-SQL that demands multi-table joins, clinically meaningful filters, and executable SQL.
Outcome:	The proposed benchmark performs well on a set of 20 proprietary and open-source models . it scores 74.7% execution, while DeepSeek-R1 leads open-sourced at 69.2% .

Copied to clipboard

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, Ido Dagan

TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)

Challenge:	Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks.
Approach:	They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective.
Outcome:	The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks.

Copied to clipboard

Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan

Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires (2022.acl-long)

Challenge:	Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text.
Approach:	They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space.
Outcome:	The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models.

Copied to clipboard

Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, Arman Cohan

Challenge:	Existing approaches to identify mental health conditions using social media are limited by the presence of symptoms described in a questionnaire used by clinicians.
Approach:	They propose to ground a model in PHQ9's symptoms to improve generalization . they also show that this approach can still perform competitively on in-domain data.
Outcome:	The proposed approach can perform competitively on in-domain data while improving generalizability and generalisability.

MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting (2022.naacl-main)

Copied to clipboard

Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, Arman Cohan, David Jurgens, Kyle Lo

Challenge:	Citation context analysis (CCA) is an important task in natural language processing that studies how and why scholars discuss each other’s work.
Approach:	They propose to use a dataset of 12.6K citation contexts from 1.2K computational linguistics papers to model three important CCA phenomena.
Outcome:	The proposed dataset contains 12.6K citation contexts from 1.2K computational linguistics papers and can model these phenomena.

TAIL: A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation (2024.emnlp-demo)

Copied to clipboard

Gefei Gu, Yilun Zhao, Ruoxi Ning, Yanan Zheng, Arman Cohan

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)

Challenge:	Existing evaluation methods for long-context large language models are overly simplistic and require extensive human annotations.
Approach:	They propose an automatic toolkit to create realistic evaluation benchmarks . they use a document-grounded benchmark to generate question-answer pairs .
Outcome:	The proposed toolkit provides a way to create realistic evaluation benchmarks and visualize performance metrics of evaluated models.

Copied to clipboard

Martin Riddell, Ansong Ni, Arman Cohan

Generating Scientific Claims for Zero-Shot Scientific Fact Checking (2022.acl-long)

Challenge:	Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks.
Approach:	They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length.
Outcome:	The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training.

Copied to clipboard

Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, Lucy Lu Wang

MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024.findings-acl)

Challenge:	Existing methods for scientific fact checking require domain expertise and time consuming.
Approach:	They propose a new supervised method for generating claims from scientific sentences and a novel method for negating claims.
Outcome:	The proposed method improves on existing methods on biomedical claims and negations.

Copied to clipboard

Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, Mark Gerstein

OpenRT: An Open-source Framework for Reasoning Over Tabular Data (2023.acl-demo)

Challenge:	Large language models face unique challenges such as domain-specific terminologies and reasoning over specialized knowledge.
Approach:	They propose a multi-disciplinary collaboration framework that leverages LLM-based agents in a role-playing setting.
Outcome:	The proposed framework excels at mining and harnessing medical expertise within LLMs, as well as extending its reasoning abilities.

Copied to clipboard

Yilun Zhao, Boyu Mi, Zhenting Qi, Linyong Nan, Minghao Guo, Arman Cohan, Dragomir Radev

LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023.eacl-main)

Challenge:	Existing table pre-training methods are benchmarked on a limited number of datasets with varying configurations, resulting in a lack of unified, standardized, fair, and comprehensive comparison between methods.
Approach:	They propose to use OpenRT to reproduce existing table pre-training models and develop new models quickly.
Outcome:	The proposed framework reproduces existing table pre-training models and compares them against four question answering, one fact checking, and one faithful text generation datasets.

Copied to clipboard

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, Kyle Lo

Information Extraction & Text Mining

Challenge:	Human evaluation is labor-intensive, expensive to scale, and difficult to design.
Approach:	They propose a set of guidelines for human evaluation of faithfulness in long-form summaries that address the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can our annotator minimize workload while maintaining accurate faithfulness?
Outcome:	The proposed framework reduces inter-annotator variance in faithfulness scores while minimizing annotator workload while maintaining accuracy.

P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)

Copied to clipboard

Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan

Information Extraction & Text Mining

Challenge:	Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales.
Approach:	They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans.
Outcome:	The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more.

FinDVer: Explainable Claim Verification over Long and Hybrid-content Financial Documents (2024.emnlp-main)

Copied to clipboard

Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, Arman Cohan

FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain (2025.emnlp-main)

Challenge:	FinDVer is a benchmark to evaluate the explainable claim verification capabilities of LLMs . financial documents are typically long, intricate and dense, and they include both quantita and numerical reasoning.
Approach:	They propose a benchmark to evaluate the explainable claim verification capabilities of LLMs . they assess 25 LLM systems under long-context and RAG settings .
Outcome:	The proposed benchmark can be used to evaluate the explainable claim verification capabilities of LLMs in financial documents.

Copied to clipboard

Tiansheng Hu, Tongyan Hu, Liuyang Bai, Yilun Zhao, Arman Cohan, Chen Zhao

Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding (2023.findings-emnlp)

Challenge:	Recent LLMs have demonstrated promising ability in solving finance related problems, but applying them in real-world finance applications remains challenging due to its high risk and high stakes property.
Approach:	They propose a benchmark specifically designed for evaluating the trustworthiness of LLMs in finance applications.
Outcome:	The proposed benchmark outperforms proprietary models in most tasks while open-source models have advantage in specific areas like industry-level fairness.

Copied to clipboard

Lorenzo Jaime Yu Flores, Heyuan Huang, Kejian Shi, Sophie Chheang, Arman Cohan

OpenT2T: An Open-Source Toolkit for Table-to-Text Generation (2024.emnlp-demo)

Challenge:	Text simplification has emerged as an increasingly useful application of AI for bridging the communication gap in specialized fields such as medicine, where the lexicon is often dominated by technical jargon and complex constructs.
Approach:	They propose a unlikelihood loss that encourages generation of simpler terms and a reranked beam search decoding method that optimizes for simplicity.
Outcome:	The proposed methods achieve better performance on readability metrics on three datasets.

Copied to clipboard

Haowei Zhang, Shengyun Si, Yilun Zhao, Lujing Xie, Zhijian Xu, Lyuhao Chen, Linyong Nan, Pengcheng Wang, Xiangru Tang, Arman Cohan

Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025.naacl-long)

Challenge:	Existing methods for table-to-text generation are limited and benchmarked on a limited number of datasets.
Approach:	They propose to use open-source tools to reproduce existing large language models for performance comparison and expedite the development of new models.
Outcome:	The proposed toolkit compares existing large language models on 9 table-to-text generation datasets and maintains a leaderboard to provide insights for future work.

Copied to clipboard

Yilun Zhao, Guo Gan, Chen Zhao, Arman Cohan

SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers (2025.emnlp-demos)

Challenge:	Recent-released MLLMs have shown remarkable performance on various multimodal math reasoning benchmarks.
Approach:	They introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models in handling multimodal math reasoning.
Outcome:	The proposed model performs well on a broad spectrum of 17 MLLMs and demonstrates that they are robust to adversarial perturbations.

Copied to clipboard

Zihang Wang, Yilun Zhao, Kaiyan Zhang, Chen Zhao, Manasi Patwardhan, Arman Cohan

Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity (2022.naacl-main)

Challenge:	SCISKETCH is an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Approach:	They propose an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Outcome:	The open-source framework outperforms several state-of-the-art foundation models in generating schematic diagrams for scientific papers.

Copied to clipboard

Sheshera Mysore, Arman Cohan, Tom Hope

Representation Learning

Challenge:	Using co-citations, we can train a model that matches aspects of papers to document level similarity.
Approach:	They propose a model that matches fine-grained aspects of papers and aggregates them into a document level similarity model using a naturally-occurring source of supervision: co-citations.
Outcome:	The proposed model improves performance on document similarity tasks in four datasets and achieves competitive results.

MIMIR: A Customizable Agent Tuning Platform for Enhanced Scientific Applications (2024.emnlp-demo)

Copied to clipboard

Xiangru Tang, Chunyuan Deng, Hanminwang Hanminwang, Haoran Wang, Yilun Zhao, Wenqi Shi, Yi Fung, Wangchunshu Zhou, Jiannan Cao, Heng Ji, Arman Cohan, Mark Gerstein

SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search (2020.emnlp-main)

Challenge:	Large language models (LLMs) have evolved into interactive agents capable of planning, tool use, and task execution across various tasks.
Approach:	They propose a platform that leverages large language models to generate agent-tuning data for fine-tuneing smaller, specialized models.
Outcome:	MIMIR enables large models to simulate various roles and create interaction data, which can then be used to fine-tune open-source models like LLaMA2.

Copied to clipboard

Sean MacAvaney, Arman Cohan, Nazli Goharian

From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)

Challenge:	Existing search methods for COVID-19 are not based on scientific data, but use a neural re-ranking model pre-trained on scientific text.
Approach:	They propose a zero-shot ranking algorithm that adapts to COVID-related scientific literature . they use a neural re-ranking model pre-trained on scientific text and filters the target document .
Outcome:	The proposed algorithm outperforms models on the TREC COVID Round 1 leaderboard . it outperformed models that do not rely on TREC-COVID data .

Copied to clipboard

Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao

TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)

Challenge:	Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments.
Approach:	They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation.
Outcome:	The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%.

Copied to clipboard

Jaesung Tae, Hamish Ivison, Sachin Kumar, Arman Cohan

MIR: Methodology Inspiration Retrieval for Scientific Research Problems (2025.acl-long)

Challenge:	Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm.
Approach:	They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models.
Outcome:	The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks.

Copied to clipboard

Aniketh Garikaparthi, Manasi Patwardhan, Aditya Sanjiv Kanade, Aman Hassan, Lovekesh Vig, Arman Cohan

ParsiNLU: A Suite of Language Understanding Challenges for Persian (2021.tacl-1)

Challenge:	Existing methods for generating ideas rely on grounding the discovery process within the literature, but their effectiveness varies significantly with the quality and nature of the retrieved literature.
Approach:	They construct a methodological inspiration retrieval task using a citation-based methodology adjacency graph and embed an "intuitive prior'' into dense retrievers.
Outcome:	The proposed method achieves significant gains in Recall@3 and mAP over strong baselines.

Copied to clipboard

Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, Yadollah Yaghoobzadeh

Challenge:	Despite progress in natural language understanding, most progress is concentrated on resource-rich languages like English . despite high-quality benchmarks, there are few available NLU datasets for Persian language .
Approach:	They propose a benchmark for Persian language that includes a range of language understanding tasks . they present their results on monolingual and multilingual pre-trained language models .
Outcome:	The proposed benchmarks compare human performance with monolingual and multilingual models on Persian language with high quality evaluation datasets.

AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research (2025.acl-long)

Copied to clipboard

Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Chengye Wang, Yixin Liu, Lovekesh Vig, Arman Cohan

Table-R1: Inference-Time Scaling for Table Reasoning Tasks (2025.emnlp-main)

Challenge:	a benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research is available online.
Approach:	They propose to use a benchmark to evaluate LLMs' ability to design ablation studies . they investigate whether current automated evaluation methods are not reliable .
Outcome:	The benchmark compared leading LLMs with human experts on generating detailed ablation study designs . the results show that current evaluation methods are not reliable for the task .

Copied to clipboard

Zheyuan Yang, Lyuhao Chen, Arman Cohan, Yilun Zhao

Structured Data in NLP

Challenge:	In this study, we explore inference-time scaling on table reasoning tasks.
Approach:	They propose a large-scale dataset of reasoning traces and a reinforcement learning with verifiable rewards approach to enable inference-time scaling on table reasoning tasks.
Outcome:	The proposed model matches or exceeds GPT-4.1 and DeepSeek-R1 models on diverse table reasoning tasks.

KnowledgeFMath: A Knowledge-Intensive Math Reasoning Dataset in Finance Domains (2024.acl-long)

Copied to clipboard

Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, Arman Cohan

A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (N18-2)

Challenge:	Existing benchmarks for large language models (LLMs) are only 56.6% accurate, leaving room for improvement.
Approach:	They propose a benchmark to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems using a finance-domain knowledge bank and expert-annotated solution references.
Outcome:	The proposed system achieves only 56.6% accuracy, leaving room for improvement.

Copied to clipboard

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, Nazli Goharian

Summarization

Challenge:	Existing abstractive summarization models focus on summarizing sentences and short documents.
Approach:	They propose a hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary.
Outcome:	The proposed model significantly outperforms state-of-the-art models on two large-scale datasets of scientific papers.

SciMDR: Advancing Scientific Multimodal Document Reasoning (2026.acl-long)

Copied to clipboard

Ziyu Chen, Yilun Zhao, Chengye Wang, Rilyn R. Han, Manasi Patwardhan, Arman Cohan

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2022.acl-long)

Challenge:	Current models struggle to provide reliable assistance in real-world scientific workflows because evidence is distributed across long, multimodal documents.
Approach:	They propose a framework for QA Synthesis and document-scale regrounding that generates faithful, isolated QA pairs and reasoning on focused segments.
Outcome:	The proposed framework achieves significant improvements across multiple QA benchmarks, particularly in tasks requiring complex document-level reasoning.

Copied to clipboard

Wen Xiao, Iz Beltagy, Giuseppe Carenini, Arman Cohan

CPTCoder: A Reliable LLM System for Medical Procedure Code Prediction (2026.acl-demo)

Challenge:	Existing pretrained models require domain-specific additional information to be effective.
Approach:	They propose a pre-trained model for multi-document representation with a focus on summarization that uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents.
Outcome:	PRIMERA outperforms current state-of-the-art models on most datasets with large margins . PRImerA uses efficient encoder-decoder transformers to simplify processing of concatenated input documents.

Copied to clipboard

Benlu Wang, Ziyao Shangguan, Kyle Tegtmeyer, Zhenyu Zhang, Sophie Chheang, Arman Cohan

Challenge:	Clinical procedure coding is an extreme multi-label classification problem . CPTCoder predicts standardized medical procedure codes from clinical text .
Approach:	a new human-in-the-loop system predicts standardized medical procedure codes from clinical text.
Outcome:	CPTCoder outperforms baseline system by 12 and 5 points in a clinical procedure classification problem.

Understanding Reference Policies in Direct Preference Optimization (2025.findings-naacl)

Copied to clipboard

Yixin Liu, Pengfei Liu, Arman Cohan

Beyond Paragraphs: NLP for Long Sequences (2021.naacl-tutorials)

Challenge:	Direct Preference Optimization (DPO) is a widely used training method for the fine-tuning of large language models (LLMs).
Approach:	They explore the dependency on the reference model or policy in DPO and examine the necessity of the KL-constraint from the reference policies in D PO.
Outcome:	The proposed model is sensitive to the KL divergence constraint, and the reference policy can penalize deviations from the reference constraint.

Copied to clipboard

Iz Beltagy, Arman Cohan, Hannaneh Hajishirzi, Sewon Min, Matthew E. Peters

ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)

Challenge:	In this tutorial, we will introduce document-level representation learning techniques . document-based learning is challenging due to the limited sequence length of many models .
Approach:	They will provide an overview of established long sequence NLP techniques and discuss memory-saving methods that are key to processing long sequences.
Outcome:	The tutorial will introduce the latest and ongoing techniques for document-level representation learning.

Copied to clipboard

Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan

Robustness in NLP

Challenge:	Existing evaluations of large language models (LLMs) for instruction following are incomplete.
Approach:	They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets.
Outcome:	The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Z1: Efficient Test-time Scaling with Code (2025.emnlp-industry)

Copied to clipboard

Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

SciRAG: Adaptive, Citation-Aware, and Outline-Guided Retrieval and Synthesis for Scientific Literature (2026.eacl-long)

Challenge:	Large Language Models (LLMs) can achieve enhanced complex problem-solving through test-time computing scaling, but this often entails longer contexts and numerous reasoning token costs.
Approach:	They propose an efficient test-time scaling method that trains LLMs on code-related reasoning trajectories and a novel Shifted Thinking Window to mitigate overthinking overhead.
Outcome:	The proposed method reduces overthinking overhead while maintaining performance.

Copied to clipboard

Hang Ding, Yilun Zhao, Tiansheng Hu, Zihang Wang, Manasi Patwardhan, Arman Cohan

Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (2024.findings-acl)

Challenge:	Existing retrieval-augmented generation methods overlook citation graph structure, adapt poorly to complex queries, and yield fragmented, hard-to-verify syntheses.
Approach:	They propose a retrieval-augmented generation framework that addresses these gaps by combining adaptive retrieval and symbolic reasoning.
Outcome:	Extensive experiments show that SciRAG outperforms prior systems in factual accuracy and synthesis quality.

Copied to clipboard

Rilyn Han, Jiawen Chen, Yixin Liu, Arman Cohan

SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)

Challenge:	a limited number of human annotations are required to evaluate multilingual summarization evaluation metrics.
Approach:	They propose a multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual metaevaluations dataset into multilingual versions.
Outcome:	The proposed framework outperforms classical text-matching-based metrics in non-English languages.

Copied to clipboard

Shruti Singh, Nandan Sarkar, Arman Cohan

Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024.emnlp-main)

Challenge:	SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles.
Approach:	They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs.
Outcome:	The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering.

Copied to clipboard

Yicheng Gao, Gonghan Xu, Zhe Wang, Arman Cohan

Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers (2025.findings-acl)

Challenge:	Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for text quality evaluation.
Approach:	They propose two methods to improve the accuracy of LLM evaluators by Bayesian inference.
Outcome:	The proposed methods improve the accuracy of the win rate estimation using LLMs . the proposed methods are based on six datasets covering story generation, summarization, and instruction following tasks .

Copied to clipboard

Yilun Zhao, Chengye Wang, Chuhan Li, Arman Cohan

Explainability & Interpretability in NLP

Challenge:	MISS-QA is the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature.
Approach:	They propose an automated evaluation protocol powered by open-source LLMs trained on human-scored data to ensure reliable evaluation.
Outcome:	The proposed protocol is powered by open-source LLMs trained on human-scored data.

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference (2025.findings-naacl)

Copied to clipboard

Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan

Challenge:	Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences.
Approach:	They propose a system-level evaluation framework that ranks LLMs based on their alignment with human preferences.
Outcome:	The proposed framework aims to rank LLMs based on their performance and alignment with human preferences.

Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)

Copied to clipboard

Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan

Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline.
Approach:	They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews.
Outcome:	The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms.

Copied to clipboard

Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu, Chen Zhao

SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions (C18-1)

Challenge:	Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks.
Approach:	They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training.
Outcome:	The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks.

Copied to clipboard

Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, Nazli Goharian

OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems (2024.emnlp-industry)

Challenge:	Existing methods to label mental health conditions are based on high-precision diagnosis patterns and carefully selected control users.
Approach:	They propose to use high-precision diagnosis patterns to identify self-reported diagnoses of nine different mental health conditions and obtain high-quality labeled data without manual labelling.
Outcome:	The proposed dataset is two orders of magnitude larger than the largest published similar resource.

Copied to clipboard

Linyong Nan, Weining Fang, Aylin Rasteh, Pouya Lahabi, Weijin Zou, Yilun Zhao, Arman Cohan

Natural Language Interfaces

Challenge:	Existing approaches to QA require multiple modalities and a broad pool of information sources to generate coherent answers.
Approach:	They propose a new resource to evaluate the effectiveness of question answering systems that perform retrieval augmented generation in scenarios that demand reasoning on multi-modal, multi-document contexts.
Outcome:	The proposed method evaluates question answering systems that perform retrieval augmented generation (RAG) in open-domain questions . it requires systems to navigate diverse modalities and a broad pool of information sources, making it uniquely challenging.

Structural Scaffolds for Citation Intent Classification in Scientific Publications (N19-1)

Copied to clipboard

Arman Cohan, Waleed Ammar, Madeleine van Zuylen, Field Cady

MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application (2026.acl-long)

Challenge:	Existing methods for identifying intent of citations are limited by external linguistic resources and hand-engineered features.
Approach:	They propose a multitask model to incorporate structural information of scientific papers into citations for effective classification of citation intents.
Outcome:	The proposed model achieves a 13.3% increase in F1 score on an existing ACL anthology dataset without external linguistic resources or hand-engineered features as done in existing methods.

Copied to clipboard

Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Vincent Jim Zhang, Yuqing Guo, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Zhiyang Deng, Heming Fu, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Guojun Xiong, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)

Challenge:	Existing evaluations of LLMs in finance are text-only, monolingual, and largely saturated by current models.
Approach:	They propose a multilingual and multimodal benchmark for evaluating LLMs in real financial contexts.
Outcome:	The first expert-annotated multilingual and multimodal benchmark is released . it evaluates 21 leading LLMs and shows they perform better in multilingual settings .

Copied to clipboard

Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

Challenge:	a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision.
Approach:	They introduce Physics, a benchmark for university-level physics problem solving.
Outcome:	The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems .

A Survey on Evaluation of LLM-based Agents (2026.findings-acl)

Copied to clipboard

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer

Challenge:	This paper provides the first comprehensive survey of evaluation methods for LLM-based agents . LLMs are static, having fixed knowledge, and confined to text-to-text interaction.
Approach:	They analyze the evaluation of LLM-based agents across five perspectives . they identify current trends and key gaps in evaluation methods .
Outcome:	The proposed evaluation frameworks and tools are based on five perspectives . the results highlight current trends and identify gaps in future research .

TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024.acl-long)

Copied to clipboard

Yilun Zhao, Lyuhao Chen, Arman Cohan, Chen Zhao

Calibrating Long-form Generations From Large Language Models (2024.findings-emnlp)

Challenge:	Long-form table question answering often generates paragraph long and complex answers . a prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context.
Approach:	They propose a modular framework that decomposes the whole process into three sub-modules . framework produces a QA-based plan first, followed by generating an answer conditioned on this plan . human evaluation results indicate the framework improves strong baselines on accuracy and truthfulness .
Outcome:	The proposed framework improves accuracy and truthfulness on the FeTaQA and QTSumm datasets.

Copied to clipboard

Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, Bhuwan Dhingra

TLDR: Extreme Summarization of Scientific Documents (2020.findings-emnlp)

Challenge:	Conventional calibration methods treat answer correctness as binary and do not work for long-form generation where an answer can be partially correct.
Approach:	They propose a framework where correctness of LLMs' responses and associated confidence levels are treated as distributions across a range of scores.
Outcome:	The proposed framework treats the correctness of the LLMs’ responses and their associated confidence levels as distributions across a range of scores.

Copied to clipboard

Isabel Cachola, Kyle Lo, Arman Cohan, Daniel Weld

LocAgent: Graph-Guided LLM Agents for Code Localization (2025.acl-long)

Challenge:	TLDR generation requires expert background knowledge and understanding of complex domain-specific language.
Approach:	They propose a learning strategy that exploits titles as an auxiliary training signal.
Outcome:	The proposed method improves upon strong baselines under both automated metrics and human evaluations.

Copied to clipboard

Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, Xingyao Wang

Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023.emnlp-industry)

Challenge:	Existing approaches struggle to efficiently navigate complex codebases when identifying relevant code snippets.
Approach:	They propose a graph-guided agent framework that addresses code localization through a distributed graph-based agent.
Outcome:	The proposed framework improves accuracy on real-world benchmarks and can be used to locate code snippets at a cost of 86%.

Copied to clipboard

Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, Arman Cohan

MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs (2025.emnlp-main)

Challenge:	Existing table-to-text generation techniques that transform complex tabular data into comprehensible narratives are lacking in real-world applications.
Approach:	They investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios.
Outcome:	The proposed models can generate table-to-text data in two real-world information seeking scenarios and perform better than existing models.

Copied to clipboard

Gabrielle Kaili-May Liu, Gal Yona, Avi Caciularu, Idan Szpektor, Tim G. J. Rudner, Arman Cohan

Low-Resource NLP

Challenge:	Existing methods for faithful calibration of large language models (LLMs) are insufficient and can harm faithful calibration.
Approach:	They propose a new prompt-based calibration approach inspired by human metacognition that measures faithfulness across diverse models and task domains and enables up to 61% improvement in faithfulness.
Outcome:	The proposed approach improves faithfulness across diverse models and task domains and achieves an 83% win rate over original generations as judged by humans.

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)

Copied to clipboard

Zhaojian Yu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering (2024.findings-naacl)

Challenge:	Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks.
Approach:	They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks .
Outcome:	The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks.

Copied to clipboard

Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, Arman Cohan

Natural Language Interfaces

Challenge:	a new study evaluates how Large Language Models interact with a SQL interpreter . the model is limited in context and is stochastic, making it less suited for tasks requiring high precision and extensive computations.
Approach:	They propose and evaluate two interaction strategies to evaluate how LLMs interact with a SQL interpreter.
Outcome:	The proposed framework improves the accuracy and reliability of the evaluations.

Embedding Recycling for Language Models (2023.findings-eacl)

Copied to clipboard

Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D’Arcy, Arman Cohan, Doug Downey

Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026.acl-long)

Challenge:	Existing studies on embedding recycling have not adequately account for overhead costs.
Approach:	They propose to reuse contextualized embeddings from previous runs to speed training and inference of future ones.
Outcome:	The proposed technique speeds training and inference with no impact on accuracy.

Copied to clipboard

Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao, Arman Cohan

Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)

Challenge:	Existing evaluation benchmarks for retrievers are narrow and evaluate them in isolation . existing evaluation benchmarking frameworks focus on evaluating retrievers in isolation, obscuring their value in real-world applications.
Approach:	They propose an evaluation framework that evaluates retrievers in agentic search systems . they provide expert-annotated reasoning aspects, positive documents, a reference response and evaluation rubrics .
Outcome:	The proposed framework assesses retrievers in agentic search systems.

Copied to clipboard

Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, Mark Gerstein

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics (2026.acl-long)

Challenge:	Large Language Models (LLMs) have advanced capabilities but produce complex structured data.
Approach:	They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs.
Outcome:	The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats.

Copied to clipboard

Jinu Lee, Kyoung-Woon On, Sophia Simeng Han, Arman Cohan, Julia Hockenmaier

SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020.acl-main)

Challenge:	Evaluating the quality of LLM-generated reasoning traces in expert domains is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks.
Approach:	They propose a large-scale legal reasoning dataset with an emphasis on reasoning trace evaluation that converts court judgments into hierarchical trees of opposing parties’ arguments and the court’s conclusions.
Outcome:	The proposed model improves the quality of LLM-generated reasoning traces in legal domains, whereas RL improves correctness albeit with reduced coverage.

Copied to clipboard

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel Weld

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)

Challenge:	Recent Transformer language models do not leverage information on inter-document relatedness, which limits their document-level representation power.
Approach:	They propose a method to generate document-level embeddings using citation graphs.
Outcome:	The proposed method outperforms baselines on document-level tasks.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

Information Extraction & Text Mining

Challenge:	Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied.
Approach:	They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance .
Outcome:	The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs.

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions (2025.naacl-long)

Copied to clipboard

Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini

SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification (2025.acl-long)

Challenge:	Modern language models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests.
Approach:	They propose a dataset that contains an instruction evaluation benchmark and a training set to help IR models learn to follow instructions.
Outcome:	The proposed model improves after fine-tuning on a training set and rigorous instruction evaluation benchmark.

Copied to clipboard

Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, Yilun Zhao

LimRank: Less is More for Reasoning-Intensive Information Reranking (2025.emnlp-main)

Challenge:	Existing scientific claim verification benchmarks focus on textual content alone or on verifying claims based on a single table.
Approach:	They propose to use SciVer to evaluate the ability of foundation models to verify claims within a multimodal scientific context.
Outcome:	The proposed model outperforms 21 state-of-the-art models and human experts on SciVer.

Copied to clipboard

Tingyu Song, Yilun Zhao, Siyue Zhang, Chen Zhao, Arman Cohan

Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge (2025.findings-emnlp)

Challenge:	Existing approaches to rerank information require large-scale fine-tuning, which is computationally expensive.
Approach:	They propose an open-source pipeline for generating diverse, challenging, and realistic reranking examples.
Outcome:	The proposed model performs competitively on two benchmarks, while being trained on less than 5% of the data typically used in prior work.

Copied to clipboard

Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, Soroush Vosoughi

Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)

Challenge:	LLM-as-Judge frameworks provide scalable alternative to human evaluation . but the question of how intrinsic biases manifest in these settings remains unexplored .
Approach:	They conduct systematic analysis of four bias types in multi-agent LLM-as-Judge frameworks . they find debate framework amplifies biases sharply after initial debate .
Outcome:	The proposed frameworks amplify biases after debate and show they are stronger in meta-judge scenarios.

Copied to clipboard

Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, Sameer Singh

A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning (2026.acl-long)

Challenge:	a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques .
Approach:	They aim to introduce techniques for learning from little-to-no data using pretrained language models.
Outcome:	This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods .

Copied to clipboard

Tianyu Yang, Sihong Wu, Yilun Zhao, Zhenwen Liang, Lisen Dai, Chen Zhao, Minhao Cheng, Arman Cohan, Xiangliang Zhang

Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval (2023.findings-emnlp)

Challenge:	Multimodal mathematical Reasoning (MMR) has attracted increasing attention for its ability to solve mathematical problems involving both textual and visual modalities.
Approach:	They review the theoretical frameworks of multimodal reasoning and examine the challenges they face in visual math tasks.
Outcome:	The proposed models can solve problems involving both textual and visual modalities.

Copied to clipboard

John Giorgi, Luca Soldaini, Bo Wang, Gary Bader, Kyle Lo, Lucy Wang, Arman Cohan

Long Context Question Answering via Supervised Contrastive Learning (2022.naacl-main)

Challenge:	Multi-document summarization (MDS) assumes a set of topic-related documents is provided as input.
Approach:	They formalize the task and bootstrap it using existing datasets, retrievers and summarizers.
Outcome:	The proposed method reduces the sensitivity of summarizers to imperfect retrieval, but is highly sensitive to other errors.

Copied to clipboard

Avi Caciularu, Ido Dagan, Jacob Goldberger, Arman Cohan

Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation (2024.findings-acl)

Challenge:	Long-context question answering tasks often require identifying evidence spans (e.g., sentences) prior work showed that jointly training models to perform evidence extraction and question answering is important for achieving high performance.
Approach:	They propose a method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence.
Outcome:	The proposed method exhibits consistent improvements on three different strong long-context transformer models, across two challenging question answering benchmarks – HotpotQA and QAsper.

Copied to clipboard

Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, Arman Cohan

CourtReasoner: Can LLM Agents Reason Like Judges? (2025.emnlp-main)

Challenge:	Data contamination is a problem in Large language models due to the reliance on extensive internet-derived training corpora.
Approach:	They present a survey on the topic of data contamination in large language models.
Outcome:	The results of the first survey on data contamination in large language models provide a comprehensive guide for NLP researchers seeking a systematic understanding of the issue.

Copied to clipboard

Sophia Simeng Han, Yoshiki Takashima, Shannon Zejiang Shen, Chen Liu, Yixin Liu, Roque K. Thuo, Sonia Knowlton, Ruzica Piskac, Scott J Shapiro, Arman Cohan

MedTutor: A Retrieval-Augmented LLM System for Case-Based Medical Education (2025.emnlp-demos)

Challenge:	Large language models (LLMs) have achieved impressive results across mathematical, logical, and commonsense reasoning tasks.
Approach:	They propose a novel expert-annotated judicial reasoning benchmark to measure LLMs' ability to construct goal-oriented legal reasoning.
Outcome:	The proposed benchmark measures the LLM agent’s ability to construct goal-oriented legal reasoning.

Copied to clipboard

Dongsuk Jang, Ziyao Shangguan, Kyle Tegtmeyer, Anurag Gupta, Jan T Czerminski, Sophie Chheang, Arman Cohan

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)

Challenge:	Existing educational tools for medical residents are time-consuming and inconsistent.
Approach:	They propose a system that generates educational content and multiple-choice questions from clinical case reports and a pipeline that takes clinical case report input and produces targeted educational materials.
Outcome:	The system generates educational content and multiple-choice questions from clinical case reports and synergizes with local knowledge base to ensure it is foundationally sound and current.

Copied to clipboard

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

Text Clustering

Challenge:	Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks.
Approach:	They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters .
Outcome:	The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data.

Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)

Copied to clipboard

Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev

Fact or Fiction: Verifying Scientific Claims (2020.emnlp-main)

Challenge:	In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory.
Approach:	They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs.
Outcome:	The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset.

Copied to clipboard

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi

Challenge:	SciFact is a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales.
Approach:	They construct a dataset of 1.4K scientific claims paired with evidence-containing abstracts annotated with labels and rationales to test their system.
Outcome:	The proposed system can verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus.

MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding (2026.acl-long)

Copied to clipboard

Xue Xia, Zheyuan Yang, Arman Cohan, Yilun Zhao

When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (2024.findings-eacl)

Challenge:	MMSciCode is a benchmark for evaluating foundation models in scientific code generation.
Approach:	They propose a multilingual, multi-discipline benchmark for evaluating foundation models in scientific code generation that integrates domain-specific knowledge with algorithmic reasoning.
Outcome:	The new benchmark is annotated by domain experts and features rigorous quality controls to ensure dataset integrity and authenticity.

Copied to clipboard

Orion Weller, Kyle Lo, David Wadden, Dawn Lawrie, Benjamin Van Durme, Arman Cohan, Luca Soldaini

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021.naacl-main)

Challenge:	Using large language models (LMs) for query or document expansion can improve generalization in information retrieval.
Approach:	They conduct the first comprehensive analysis of large language models (LMs) for query or document expansion.
Outcome:	The proposed expansions improve retrieval performance for weaker models but harm stronger models.

Copied to clipboard

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner

Question Answering

Challenge:	Existing information-seeking question answering datasets do not perform well on answering these questions . existing models that do well on other QA tasks do not do well answering these tasks .
Approach:	They present a dataset of 5049 questions over 1585 NLP papers . they use a question-seeking QA model that seeks information in the full text .
Outcome:	The proposed dataset underperforms existing models on other QA tasks by 27 F1 points . the focus is on document-grounded, information-seeking QA .

SUPP.AI: finding evidence for supplement-drug interactions (2020.acl-demos)

Copied to clipboard

Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg, Carissa Schoenick, Nick Botner, Waleed Ammar

Challenge:	Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete.
Approach:	They propose an application to search evidence sentences extracted from the literature to identify supplement-drug interactions.
Outcome:	The proposed model extracts supplement information and identifies interactions using labeled DDI data.

A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents (2023.emnlp-main)

Copied to clipboard

Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, Kyle Lo

Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers (2025.acl-long)

Challenge:	snippets are not meant to be read outside their original document.
Approach:	They propose a framework that decomposes the task into three stages: question generation, question answering, and rewriting.
Outcome:	The proposed framework decomposes the task into three stages: question generation, question answering, and rewriting.

Copied to clipboard

Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, Arman Cohan

Challenge:	Recent advances in large language models (LLMs) have demonstrated remarkable capabilities across a variety of scientific tasks, such as answering questions about scientific papers, writing scientific papers and retrieving related works.
Approach:	They propose a taxonomy of limitation types in scientific research with a focus on AI to evaluate their ability to support early-stage feedback and complement human peer review.
Outcome:	The proposed model enhances the ability of LLM systems to generate limitations in research papers, enabling them to provide more concrete and constructive feedback.

IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery (2025.acl-demo)

Copied to clipboard

Aniketh Garikaparthi, Manasi Patwardhan, Lovekesh Vig, Arman Cohan

Challenge:	Recent work on automated hypothesis generation focuses on multi-agent frameworks and extending test-time compute, but none incorporates human-in-the-loop (HITL) integration.
Approach:	They propose an open-source platform to enable researchers to leverage LLM-assisted scientific ideation.
Outcome:	The proposed system empowers researchers with greater control throughout ideation process.

Anchor: Branch-Point Data Generation for GUI Agents (2026.acl-long)

Copied to clipboard

Jinbiao Wei, Yilun Zhao, Kangqi Ni, Arman Cohan

Challenge:	Existing GUI agents for real desktop environments require large amounts of high-quality interaction data, but collecting human demonstrations is expensive.
Approach:	They propose a framework that bootstraps scalable desktop supervision from seed demonstrations.
Outcome:	Experiments on standard desktop benchmarks show that the framework improves on zero-shot agents and representative synthesis baselines.

Investigating Data Contamination in Modern Benchmarks for Large Language Models (2024.naacl-long)

Copied to clipboard

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, Arman Cohan

QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)

Challenge:	Existing evaluation benchmarks for large language models are inflated and inconsistent with actual performance.
Approach:	They propose a retrieval-based system to explore potential overlaps between benchmarks and pretraining corpora and a protocol to investigate testset slot guessing.
Outcome:	The proposed method exploits overlaps between evaluation benchmarks and pretraining corpora and masks a wrong answer in a multiple choice question and prompts the model to fill in the gap.

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan

SciFact-Open: Towards open-domain scientific claim verification (2022.findings-emnlp)

Challenge:	Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights.
Approach:	They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary.
Outcome:	The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input.

Copied to clipboard

David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, Hannaneh Hajishirzi

MDCure: A Scalable Pipeline for Multi-Document Instruction-Following (2025.acl-long)

Challenge:	Current scientific claim verification systems can achieve very strong performance on limited contexts, in some cases approaching human agreement.
Approach:	They propose to pool and annotate top predictions from four state-of-the-art scientific claim verification models to evaluate their performance against large corpora.
Outcome:	The proposed system performs well on a corpus of 500K scientific abstracts.

Copied to clipboard

Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, Arman Cohan

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation (2026.findings-acl)

Challenge:	Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents.
Approach:	They propose a framework that generates high-quality synthetic MD instruction data over sets of articles via targeted prompts.
Outcome:	MDCure generates high-quality synthetic MD instruction data over sets of articles . evaluations show it improves over pre-trained models by up to 75.1% .

Copied to clipboard

Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, Arman Cohan

Challenge:	Prior studies show that large language models can draft fluent reviews but they miss specific issues, show shallow analysis, and produce generic phrasing.
Approach:	They propose a task that targets actionable review feedback generation and places existing peer review rebuttal at the center of learning.
Outcome:	The proposed model improves on a large dataset that maps review segments to rebuttal segments that address them, with perspective labels and impact categories that order author uptake.

Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models (2025.acl-long)

Copied to clipboard

Junjie Wu, Gefei Gu, Yanan Zheng, Dit-Yan Yeung, Arman Cohan

MultiVerS: Improving scientific claim verification with weak supervision and full-document context (2022.findings-naacl)

Challenge:	Long-context language models have impressive capabilities in long-contrast understanding tasks, but long-text referencing remains underexplored.
Approach:	They propose a benchmark to assess long-context referencing capability of LCLMs . they use three subsets to test the model's ability to identify key indexes based on contextual relationships .
Outcome:	The proposed benchmark assesses the long-context referencing capability of LCLMs.

Copied to clipboard

David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, Hannaneh Hajishirzi

MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search (2025.findings-emnlp)

Challenge:	a new approach to scientific claim verification uses a document-level fact-checking label to label scientific documents . a multitask approach combines a shared encoding of the claim and document context .
Approach:	They propose a system which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context.
Outcome:	The proposed approach outperforms baselines on three scientific claim verification datasets . it can learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales based on the datasets.

Copied to clipboard

Yunhai Hu, Yilun Zhao, Chen Zhao, Arman Cohan

SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)

Challenge:	MCTS-RAG combines structured reasoning with adaptive retrieval . compared to conventional MCTLs, MCTR-RAg relies on internal model knowledge without external facts .
Approach:	a new approach integrates retrieval-augmented generation and Monte Carlo Tree Search to enhance reasoning capabilities of small language models.
Outcome:	MCTS-RAG integrates retrieval-augmented generation and Monte Carlo Tree Search to improve reasoning paths.

Copied to clipboard

David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan

Pretrained Language Models for Sequential Sentence Classification (D19-1)

Challenge:	ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs.
Approach:	They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions .
Outcome:	The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions.

Copied to clipboard

Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, Dan Weld