Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

SciClaims: An End-to-End Generative System for Biomedical Claim Analysis (2025.emnlp-demos)

Challenge:	Existing solutions for information extraction (IE) require specialized models for different tasks or require expensive large language models.
Approach:	They propose a framework that enhances the original GLiNER architecture to support named entity recognition, text classification, and hierarchical structured data extraction within a single efficient model.
Outcome:	The proposed framework improves performance across diverse IE tasks and accessibility compared to LLM-based alternatives.

Copied to clipboard

Raúl Ortega, Jose Manuel Gomez-Perez

Challenge:	SciClaims is an interactive web-based system for scientific claim analysis in the biomedical domain.
Approach:	They present SciClaims, an interactive web-based system for scientific claim analysis in the biomedical domain.
Outcome:	The system extracts factual claims from scientific texts and retrieves evidence from PubMed . it also verifies the validity of each claim using large language models . the system is optimized to run efficiently on a single GPU and is publicly available .

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning (2025.emnlp-demos)

Copied to clipboard

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Xie Xie, Wei Zhou, Wang Xu, Zhou Su, Zhongwu Zhai, Xiaoming Liu, null Meiyudong, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

Marcel: A Lightweight and Open-Source Conversational Agent for University Student Support (2025.emnlp-demos)

Challenge:	Large language model agents have enabled GUI-based automation, but their deployment is limited by noisy data, poor generalization, and lack of support for non-English GUIs.
Approach:	They propose an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction.
Outcome:	The proposed GUI agent achieves promising performance on five public benchmarks and proposed Chinese benchmark CAGUI.

Copied to clipboard

Jan Trienes, Anastasiia Derzhanskaia, Roland Schwarzkopf, Markus Mühling, Jörg Schlötterer, Christin Seifert

Natural Language Interfaces

Challenge:	Existing systems that provide contextually relevant information are difficult to deploy in a university setting . a number of universities are developing or using chatbots to support prospective students .
Approach:	They propose a conversational agent called Marcel that uses retrieval-augmented generation to provide contextually relevant information.
Outcome:	The proposed system is designed to provide fast and personalized responses while reducing workload.

Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment (2025.emnlp-demos)

Copied to clipboard

Saizhuo Wang, Hang Yuan, Leon Zhou, Lionel Ni, Heung-Yeung Shum, Jian Guo

AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories (2025.emnlp-demos)

Challenge:	Traditional methods of alpha mining have inherent limitations, especially in implementing the ideas of quant researchers.
Approach:	They propose a new alpha mining paradigm by introducing human-AI interaction and a prompt engineering algorithmic framework to implement this paradigm by using large language models.
Outcome:	The proposed framework is based on human-AI interaction and large language models and is comparable to human participants in the WorldQuant International Quant Championship.

Copied to clipboard

Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, Xiang Yue

Challenge:	Large Language Model (LLM) agents produce rich, multi-step trajectories that interleave observations, internal reasoning, and tool actions.
Approach:	They propose an open-source framework for diagnosing agent trajectories that quantifies five core agentic competencies and a visualization module that highlights trajectory semantics.
Outcome:	The proposed framework is extensible and compatible with most agent trajectories.

Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization (2025.emnlp-demos)

Copied to clipboard

Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi

ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning (2025.emnlp-demos)

Challenge:	Existing studies on text anonymization prioritize privacy preservation at the expense of utility, relying on reference-based metrics like ROUGE, BERTScore, or METEOR to measure textual fidelity.
Approach:	They propose an open-source framework for benchmarking text anonymization methods through the lens of privacy and utility task sensitivity.
Outcome:	The proposed framework is open-source and provides a Python library, documentation and tutorials.

Copied to clipboard

Yichen Lu, Wei Dai, Jiaen Liu, Ching Wing Kwok, Zongheng Wu, Xudong Xiao, Ao Sun, Sheng Fu, Jianyuan Zhan, Yian Wang, Takatomo Saito, Sicheng Lai

Machine Translation

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in Machine Translation (MT) tasks.
Approach:	They propose a translation agent system designed for multimodal input that leverages visual and contextual background information to enhance the translation process.
Outcome:	The proposed translation agent achieves significantly higher translation quality in subtitle generation and general translation tasks compared to previous state-of-the-art systems.

Sanskrit Voyager: Unified Web Platform for Interactive Reading and Linguistic Analysis of Sanskrit Texts (2025.emnlp-demos)

Copied to clipboard

Giacomo De Luca, Danilo Croce, Roberto Basili

Challenge:	Sanskrit Voyager enables users to search for words and phrases as they actually appear in texts . evaluation shows over 92% parsing accuracy on complex compounds compared to BuddhaNexus .
Approach:	Sanskrit Voyager is a web application for searching, reading and analyzing the Sanskrt literary corpus.
Outcome:	Sanskrit Voyager is a web application for searching, reading, and analyzing the Sanskrt literary corpus.

PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation (2025.emnlp-demos)

Copied to clipboard

Eliya Habba, Noam Dahan, Gili Lior, Gabriel Stanovsky

LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderators (2025.emnlp-demos)

Challenge:	Recent studies have demonstrated that LLMs are highly sensitive to small, meaning-preserving variations in task formulation.
Approach:	They propose a framework that enables the automatic generation of various prompts.
Outcome:	The proposed framework provides meaningful variations to support strong evaluation practices.

Copied to clipboard

Leanne Tan, Gabriel Chua, Ziyu Ge, Roy Ka-Wei Lee

GraphMind: Interactive Novelty Assessment System for Accelerating Scientific Discovery (2025.emnlp-demos)

Challenge:	a new study shows that moderation systems that ignore localisation and low-resource variants risk degraded performance and exploitation in real-world deployments.
Approach:	They propose a lightweight, multilingual moderation classifier tailored to Singapore's context . it uses pre-trained OpenAI embeddings and a multi-head ordinal classifier .
Outcome:	The proposed classifier outperforms commercial and open-source models across 17 benchmarks.

Copied to clipboard

Italo Luis da Silva, Hanqi Yan, Lin Gui, Yulan He

Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research (2025.emnlp-demos)

Challenge:	Existing approaches to literature analysis lack transparency and information retrieval module.
Approach:	GraphMind is an easy-to-use interactive web tool designed to assist users in evaluating novelty of scientific papers or drafted ideas.
Outcome:	GraphMind enables users to capture the main structure of a scientific paper, explore related ideas through various perspectives, and assess novelty via providing verifiable contextual insights.

Copied to clipboard

Richard Diehl Martinez, David Demitri Africa, Yuval Weiss, Suchir Salhan, Ryan Daniels, Paula Buttery

DistaLs: a Comprehensive Collection of Language Distance Measures (2025.emnlp-demos)

Challenge:	Recent advances in large language models (LLMs) have enabled strong performance across diverse tasks, but small enough to train on modest budgets.
Approach:	They propose a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development.
Outcome:	The proposed framework enables systematic, hypothesis-driven research for small and medium-scale language model development.

Copied to clipboard

Rob Van Der Goot, Esther Ploeger, Verena Blaschke, Tanja Samardzic

Challenge:	Existing work on how to measure distances between languages has focused on intuition and typological distance.
Approach:	They propose a toolkit that provides users with easy access to language distance measures.
Outcome:	The proposed toolkit provides easy access to a wide variety of language distance measures.

MedTutor: A Retrieval-Augmented LLM System for Case-Based Medical Education (2025.emnlp-demos)

Copied to clipboard

Dongsuk Jang, Ziyao Shangguan, Kyle Tegtmeyer, Anurag Gupta, Jan T Czerminski, Sophie Chheang, Arman Cohan

Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification (2025.emnlp-demos)

Challenge:	Existing educational tools for medical residents are time-consuming and inconsistent.
Approach:	They propose a system that generates educational content and multiple-choice questions from clinical case reports and a pipeline that takes clinical case report input and produces targeted educational materials.
Outcome:	The system generates educational content and multiple-choice questions from clinical case reports and synergizes with local knowledge base to ensure it is foundationally sound and current.

Copied to clipboard

Chenfei Xiong, Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Zhijing Jin, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Mennatallah El-Assady, Elliott Ash

Information Extraction & Text Mining

Challenge:	Social scientists often need to develop codebooks that can be reliable but require significant human effort.
Approach:	They propose a mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models.
Outcome:	The proposed framework integrates human expertise with automatic annotation guided by large language models.

DVAGen: Dynamic Vocabulary Augmented Generation (2025.emnlp-demos)

Copied to clipboard

Wei Du, Nuowei Liu, Jie Wang, Jiahao Kuang, Tao Ji, Xiaoling Wang, Yuanbin Wu

MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models (2025.emnlp-demos)

Challenge:	Existing dynamic vocabulary approaches struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations.
Approach:	They propose an open-source framework for training, evaluation, and visualization of dynamic vocabulary-augmented language models.
Outcome:	The proposed framework validates the effectiveness of dynamic vocabulary-augmented language models on modern LLMs and shows support for batch inference significantly improving inference throughput.

Copied to clipboard

Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong

Challenge:	Existing evaluation frameworks suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications.
Approach:	They propose an open-source, Model Context Protocol-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents.
Outcome:	The proposed framework uncovers nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics.

SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers (2025.emnlp-demos)

Copied to clipboard

Zihang Wang, Yilun Zhao, Kaiyan Zhang, Chen Zhao, Manasi Patwardhan, Arman Cohan

MALLM: Multi-Agent Large Language Models Framework (2025.emnlp-demos)

Challenge:	SCISKETCH is an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Approach:	They propose an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Outcome:	The open-source framework outperforms several state-of-the-art foundation models in generating schematic diagrams for scientific papers.

Copied to clipboard

Jonas Becker, Lars Benedikt Kaesberg, Niklas Bauer, Jan Philip Wahle, Terry Ruas, Bela Gipp

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks (2025.emnlp-demos)

Challenge:	Multi-agent debate (MAD) has demonstrated the ability to augment collective intelligence by scaling test-time compute and leveraging expertise.
Approach:	They propose an open-source framework that enables systematic analysis of multi-agent debates.
Outcome:	The proposed framework enables systematic analysis of multi-agent debate components.

Copied to clipboard

Adamenko Pavel, Ivanov Mikhail, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitrii Babaev, Alena Fenogenova, Valentin Malykh

Open-Theatre: An Open-Source Toolkit for LLM-based Interactive Drama (2025.emnlp-demos)

Challenge:	SWE-bench is a static benchmark that collects only once and never updates.
Approach:	They propose a dynamic, continuously updated benchmark to address data contamination issues by collecting real-world GitHub issues and rigorous quality validation.
Outcome:	The proposed benchmarks are based on a dataset of 2,294 GitHub issues and their corresponding pull requests (PRs) the static nature of the benchmarks makes it hard to distinguish meaningful progress.

Copied to clipboard

Tianyang Xu, Hongqiu Wu, Weiqi Wu, Hai Zhao

Dialogue Systems & Conversational Agents

Challenge:	Existing tools for creating, modifying, and experimenting with interactive dramas are limited.
Approach:	They propose an open-source toolkit for creating configurable LLM-based interactive drama.
Outcome:	The proposed toolkit enhances narrative coherence and realistic behavior in interactions with agents.

CafGa: Customizing Feature Attributions to Explain Language Models (2025.emnlp-demos)

Copied to clipboard

Alan David Boyle, Furui Cheng, Vilém Zouhar, Mennatallah El-Assady

Explainability & Interpretability in NLP

Challenge:	Feature attribution methods, such as SHAP and LIME, quantify the influence of each input component in a model.
Approach:	They propose a tool for generating and evaluating feature attribution explanations at customizable granularities.
Outcome:	The proposed tool is compared with two baseline methods: PartitionSHAP and MExGen.

UnityAI Guard: Pioneering Toxicity Detection Across Low-Resource Indian Languages (2025.emnlp-demos)

Copied to clipboard

Himanshu Beniwal, Reddybathuni Venkat, Rohit Kumar, Birudugadda Srivibhav, Daksh Jain, Pavan Deekshith Doddi, Eshwar Dhande, Adithya Ananth, null Kuldeep, Mayank Singh

BioGraphia: A LLM-Assisted Biological Pathway Graph Annotation Platform (2025.emnlp-demos)

Challenge:	Existing systems target high-resource languages, but UnityAI-Guard addresses this gap by developing state-of-the-art models for binary toxicity classification targeting low-resourced Indian languages.
Approach:	They propose a framework for binary toxicity classification targeting low-resource Indian languages.
Outcome:	The proposed framework achieves an impressive average F1-score of 84.23% across seven languages, leveraging a dataset of 567k training instances and 30k manually verified test instances.

Copied to clipboard

Xi Xu, Sumin Jo, Adam Officer, Angela Chen, Yufei Huang, Lei Li

SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains (2025.emnlp-demos)

Challenge:	Existing methods for obtaining pathway information from biomedical literature rely on simplifying assumptions that limit their ability to capture true complexity of biological reactions.
Approach:	They propose a web-based platform to facilitate collaborative pathway graph annotation.
Outcome:	The platform supports multi-user collaboration with real-time monitoring, curation, and interactive pathway graph visualization.

Copied to clipboard

Krithika Ramesh, Daniel Smolyak, Zihao Zhao, Nupoor Gandhi, Ritu Agarwal, Margrét V. Bjarnadóttir, Anjalie Field

Quest2DataAgent: Automating End-to-End Scientific Data Collection (2025.emnlp-demos)

Challenge:	SynthTextEval is a toolkit for conducting comprehensive evaluations of synthetic text.
Approach:	They propose a toolkit for conducting comprehensive evaluations of synthetic text using large language models.
Outcome:	The proposed toolkit can be run over any dataset, but it is aimed at two high-stakes domains: healthcare and law.

Copied to clipboard

Tianyu Yang, Yuhan Liu, Sobin Alosious, Ethan A. Brown, Jason R. Rohr, Tengfei Luo, Xiangliang Zhang

End-to-End Multilingual Automatic Dubbing via Duration-based Translation with Large Language Models (2025.emnlp-demos)

Challenge:	Existing approaches for data collection are labor-intensive and dependent on domain expertise.
Approach:	They propose a general-purpose multi-agent framework for automating scientific data collection workflows.
Outcome:	The proposed framework improves data relevance, usability, and time efficiency over existing methods.

Copied to clipboard

Hyun-Sik Won, DongJin Jeong, Hyunkyu Choi, Jinwon Kim

Multilinguality

Challenge:	Automatic dubbing (AD) aims to replace the original speech with translated speech that maintains precise temporal alignment (isochrony).
Approach:	They propose an end-to-end automatic dubbing framework that leverages large language models to integrate translation and timing control seamlessly.
Outcome:	The proposed framework achieves up to 24% relative gains on English, Spanish, and Korean language pairs while maintaining competitive translation quality measured by COMET scores.

EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models (2025.emnlp-demos)

Copied to clipboard

Ziwen Xu, Shuxun Wang, Kewei Xu, Haoming Xu, Mengru Wang, Xinle Deng, Yunzhi Yao, Guozhou Zheng, Huajun Chen, Ningyu Zhang

AERA Chat: An Interactive Platform for Automated Explainable Student Answer Assessment (2025.emnlp-demos)

Challenge:	Large Language Models (LLMs) have demonstrated extraordinary capabilities, however, they may still generate unreliable or unsafe outputs.
Approach:	They propose a framework that allows plug-and-play adjustability for controlling Large Language Model (LLM) behaviors.
Outcome:	The framework is designed to enable plug-and-play adjustability for controlling Large Language Model (LLM) behaviors.

Copied to clipboard

Jiazheng Li, Artem Bobrov, Runcong Zhao, Cesare Aloisi, Yulan He

Explainability & Interpretability in NLP

Challenge:	Existing systems that use pretrained language models to score student answers are noisy and unreliable.
Approach:	They propose a visualization platform for automated student answer assessment that leverages multiple LLMs to generate rationales.
Outcome:	The proposed platform enables educators to mark tasks and researchers to evaluate rationale quality from different models.

RadEval: A framework for radiology text evaluation (2025.emnlp-demos)

Copied to clipboard

Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, David W Eyre, Jean-Benoit Delbrouck

TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents (2025.emnlp-demos)

Challenge:	Evaluating automated radiology report generation systems remains a fundamental challenge in the development of safe, accurate, and clinically useful medical AI.
Approach:	They propose a unified, open-source framework for evaluating radiology texts that consolidates a diverse range of metrics from classic ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (GREEN).
Outcome:	The framework consolidates a diverse range of metrics from ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLMbased evaluators (GREEN).

Copied to clipboard

Haofei Yu, Keyang Xuan, Fenghai Li, Kunlun Zhu, Zijie Lei, Jiaxun Zhang, Ziheng Qi, Kyle Richardson, Jiaxuan You

KMatrix-2: A Comprehensive Heterogeneous Knowledge Collaborative Enhancement Toolkit for Large Language Model (2025.emnlp-demos)

Challenge:	Existing research systems often design and use agentic workflows to perform research tasks such as ideation, scientific coding, review writing, and tree-based search.
Approach:	They propose an open-source codebase, an interactive web demonstration, and a PyPI Python package to make state-of-the-art auto-research pipelines broadly accessible to every researcher and developer.
Outcome:	The proposed framework adapts easily to new tools and supports iterative growth.

Copied to clipboard

Shun Wu, Di Wu, Wangtao Sun, Ziyang Huang, Xiaowei Yuan, Kun Luo, XueYou Zhang, Shizhu He, Jun Zhao, Kang Liu

GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)

Challenge:	Existing studies on K-LLMs systems focus on declarative knowledge and procedural knowledge (rules) .
Approach:	They propose to build a toolkit that supports comprehensive heterogeneous knowledge collaborative enhancement for Large Language Models (LLMs).
Outcome:	The proposed toolkit provides unified knowledge integration and joint knowledge retrieval methods to achieve more comprehensive heterogeneous knowledge collaborative enhancement.

Copied to clipboard

Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann

Multilinguality

Challenge:	Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios.
Approach:	They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks.
Outcome:	The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks.

MASA: LLM-Driven Multi-Agent Systems for Autoformalization (2025.emnlp-demos)

Copied to clipboard

Lan Zhang, Marco Valentino, Andre Freitas

Reasoning

Challenge:	This paper presents a framework for building multi-agent systems for autoformalization driven by Large Language Models.
Approach:	They propose a framework for building multi-agent systems for autoformalization driven by Large Language Models.
Outcome:	The proposed framework leverages collaborative agents to convert natural language statements into formal representations.

LearnLens: LLM-Enabled Personalised, Curriculum-Grounded Feedback with Educators in the Loop (2025.emnlp-demos)

Copied to clipboard

Runcong Zhao, Artem Bobrov, Jiazheng Li, Cesare Aloisi, Yulan He

Challenge:	Existing systems that provide personalised, curriculum-aligned feedback are time-intensive and time-consuming.
Approach:	They propose a modular, LLM-based system that generates personalised, curriculum-aligned feedback in science education.
Outcome:	The proposed system generates personalised, curriculum-aligned feedback in science education.

o-MEGA: Optimized Methods for Explanation Generation and Analysis (2025.emnlp-demos)

Copied to clipboard

Ľuboš Kriš, Jaroslav Kopčan, Qiwei Peng, Andrej Ridzik, Marcel Veselý, Martin Tamajka

EvoAgentX: An Automated Framework for Evolving Agentic Workflows (2025.emnlp-demos)

Challenge:	a growing number of transformer-based language models have created challenges for model transparency and trustworthiness.
Approach:	They propose a tool to automatically identify the most effective explainable AI methods . they evaluate o-mega on a post-claim matching pipeline using a curated dataset .
Outcome:	The proposed tool shows that the most effective explainable AI methods can be implemented in semantic matching tasks.

Copied to clipboard

Yingxu Wang, Siwei Liu, Jinyuan Fang, Zaiqiao Meng

Numerical Reasoning

Challenge:	Existing MAS frameworks often require manual workflow configuration and lack native support for dynamic evolution and performance optimization.
Approach:	They propose an open-source platform that automates generation, execution, and evolutionary optimization of multi-agent workflows.
Outcome:	The proposed platform automates generation, execution, and evolutionary optimization of multi-agent workflows.

OpenRLHF: A Ray-based Easy-to-use, Scalable and High-performance RLHF Framework (2025.emnlp-demos)

Copied to clipboard

Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Wenkai Fang, null Xianyu, Yu Cao, Haotian Xu, Yiming Liu

ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses (2025.emnlp-demos)

Challenge:	Existing RLHF frameworks face inference bottlenecks and complexity barriers restricting their accessibility for newcomers.
Approach:	They propose an open-source RLHF framework that can be used to train large language models.
Outcome:	The proposed framework achieves superior training efficiency with speedups ranging from 1.22 to 1.68 across different model sizes compared to state-of-the-art frameworks, while requiring significantly fewer lines of code for implementation.

Copied to clipboard

Michael Galarnyk, Rutwik Routu, Vidhyakshaya Kannan, Kosha Bheda, Prasun Banerjee, Agam Shah, Sudheer Chava

TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability (2025.emnlp-demos)

Challenge:	ARR Responsible NLP Research checklist is designed to encourage best practices for responsible research . previous research has shown that self-reported checklist responses don't always accurately represent papers .
Approach:	They propose a retrieval-augmented generation application that can be used to assist authors with conference checklists.
Outcome:	The proposed application can be used to help authors with conference checklists and review their work.

Copied to clipboard

Mohammad Aflah Khan, Ameya Godbole, Johnny Wei, Ryan Yixiang Wang, James Flemings, Krishna P. Gummadi, Willie Neiswanger, Robin Jia

LLM×MapReduce-V3: Enabling Interactive In-Depth Survey Generation through a MCP-Driven Hierarchically Modular Agent System (2025.emnlp-demos)

Challenge:	Existing workflows for pretraining large language models are cumbersome, fragmented and inaccessible.
Approach:	They propose an open-source library for editing, inspection, and analysis of large language model datasets.
Outcome:	TokenSmith is an open-source library for editing, inspection, and analysis of large language model datasets.

Copied to clipboard

Yu Chao, Siyu Lin, Xiaorong Wang, Zhu Zhang, Zihan Zhou, Haoyu Wang, Shuo Wang, Jie Zhou, Zhiyuan Liu, Maosong Sun

Challenge:	Generating high-quality long-form survey articles poses significant challenges to AI Agent systems.
Approach:	They propose a hierarchically modular agent system for long-form survey generation . they use atomic models to implement skeleton initialization, digest construction, and skelet refinement . human evaluations demonstrate system surpasses representative baselines .
Outcome:	The proposed system surpasses representative baselines in both content depth and length, highlighting the strength of MCP-based modular planning.

GraDeT-HTR: A Resource-Efficient Bengali Handwritten Text Recognition System utilizing Grapheme-based Tokenizer and Decoder-only Transformer (2025.emnlp-demos)

Copied to clipboard

Md. Mahmudul Hasan, Ahmed Nesar Tahsin Choudhury, Mahmudul Hasan, Md Mosaddek Khan

AutoIntent: AutoML for Text Classification (2025.emnlp-demos)

Challenge:	Bengali is the sixth most spoken language in the world, but handwritten text recognition systems for the language are underdeveloped.
Approach:	They propose a Bengali handwritten text recognition system that uses a decoder-only transformer to address the unique challenges of Bengali script.
Outcome:	The proposed system significantly improves on existing tokenizers on Bengali script.

Copied to clipboard

Ilya Alekseev, Roman Solomatin, Darina Rustamova, Denis Kuznetsov

TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs (2025.emnlp-demos)

Challenge:	Existing solutions for text classification tasks lack comprehensive support for hyperparameter optimization.
Approach:	They propose to automate text classification tasks using a modular, sklearn-like interface.
Outcome:	The proposed framework shows superior performance on intent classification datasets and enables users to balance effectiveness and resource consumption.

Copied to clipboard

Duygu Nur Yaldiz, Yavuz Faruk Bakman, Sungmin Kang, Alperen Öziş, Hayrettin Eren Yildiz, Mitash Ashish Shah, Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu, Sai Praneeth Karimireddy, Salman Avestimehr

Challenge:	Generative Large Language Models (LLMs) produce untruthful outputs, referred to as hallucinations, which are often referred as false positives.
Approach:	They propose an open-source Python library with over 30 truthfulness prediction methods.
Outcome:	The proposed methods span diverse trade-offs in computational cost, access level, grounding document requirements, and supervision type (self-supervised or supervised).

The Dangers of Indirect Prompt Injection Attacks on LLM-based Autonomous Web Navigation Agents: A Demonstration (2025.emnlp-demos)

Copied to clipboard

Sam Johnson, Viet Pham, Thai Le

LaTeXMT: Machine Translation for LaTeX Documents (2025.emnlp-demos)

Challenge:	Large Language Model (LLM)-integrated applications are becoming more popular to support, augment, and automate tasks.
Approach:	They propose to embed universal adversarial triggers in webpage HTML to hijack agents . they also use a browser-gym agent powered by Llama-3.1 to test their system .
Outcome:	The proposed system software is released under the MIT License .

Copied to clipboard

Calvin Hoy, Samuel Frontull, Georg Moser

LangVAE and LangSpace: Building and Probing for Language Model VAEs (2025.emnlp-demos)

Challenge:	LaTeXMT is a software solution for structure-preserving, source-to-source translation of LaTex documents.
Approach:	They propose a software solution for structure-preserving, source-to-source translation of LaTeX documents . authors propose transformer-based language models which can be trained on plain text .
Outcome:	The proposed software is available under the LGPL-3.0 open-source licence and a web version is publicly available.

Copied to clipboard

Danilo Carvalho, Yingji Zhang, Harriet Unsworth, Andre Freitas

PresentAgent: Multimodal Agent for Presentation Video Generation (2025.emnlp-demos)

Challenge:	a novel framework for modular construction of variational autoencoders (VAEs) on pre-trained large language models (LLMs) is presented.
Approach:	They propose a modular framework for variational autoencoders on top of pre-trained large language models.
Outcome:	The proposed framework can encode pre-trained language models into more compact and semantically disentangled representations.

Copied to clipboard

Jingwei Shi, Zeyu Zhang, Biao Wu, Yanjie Liang, Meng Fang, Ling Chen, Yang Zhao

PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization (2025.emnlp-demos)

Challenge:	Existing methods for generating static slides or text summaries are limited to producing narrated presentations.
Approach:	They propose a multimodal agent that transforms long-form documents into narrated presentations.
Outcome:	The present agent produces fully synchronized visual and spoken content that closely mimics human-style presentations.

Copied to clipboard

Dawei Xiang, Wenyan Xu, Kexin Chu, Tianqi Ding, Zixu Shen, Yiming Zeng, Jianchang Su, Wei Zhang

Low-Resource NLP

Challenge:	PromptSculptor automates the iterative prompt optimization process for Text-to-Image models . previous work focused on generating detailed, high-quality prompts based on user feedback .
Approach:	They propose a framework that decomposes a task into four specialized agents . they use Chain-of-Thought reasoning to transform a short, vague user prompt into a comprehensive, refined prompt.
Outcome:	The proposed framework significantly improves output quality and reduces iterations needed for user satisfaction.

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models (2025.emnlp-demos)

Copied to clipboard

Chengyu Wang, Junbing Yan, Wenrui Cai, Yuanhao Yue, Jun Huang

AM4DSP: Argumentation Mining in Structured Decentralized Discussion Platforms for Deliberative Democracy (2025.emnlp-demos)

Challenge:	In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models.
Approach:	They propose a toolkit for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs).
Outcome:	The framework offers data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning techniques specifically tailored for KD scenarios.

Copied to clipboard

Sofiane Elguendouze, Lucas Anastasiou, Erwan Hain, Elena Cabrio, Anna De Liddo, Serena Villata

Reasoning

Challenge:	Argument mining is the automated process of identification and extraction of argumentative structures in natural language.
Approach:	They propose to use argument mining to extract arguments from online discussions in the context of deliberative democracy.
Outcome:	The proposed system enables the extraction and analysis of arguments from online discussions in the context of deliberative democracy.

TRACE: Training and Inference-Time Interpretability Analysis for Language Models (2025.emnlp-demos)

Copied to clipboard

Nura Aljaafari, Danilo Carvalho, Andre Freitas

Explainability & Interpretability in NLP

Challenge:	Existing tools for interpretability analysis of transformer models are post hoc, rely on scalar metrics or require nontrivial integration effort.
Approach:	They propose a modular toolkit for training and inference-time interpretability analysis of transformer models.
Outcome:	Experiments with autoregressive transformers show that TRACE reveals developmental phenomena overlooked by traditional scalar metrics such as loss or accuracy.

MathBuddy: A Multimodal System for Affective Math Tutoring (2025.emnlp-demos)

Copied to clipboard

Debanjana Kar, Leopold Böss, Dacia Braca, Sebastian Maximilian Dennerlein, Nina Christine Hubig, Philipp Wintersberger, Yufang Hou

Sentiment Analysis

Challenge:	Existing LLM-based conversational systems do not take into account the student’s affective states.
Approach:	They propose an emotionally aware LLM-powered math tutor that models student emotions and maps them to relevant pedagogical strategies.
Outcome:	The proposed model improves student engagement and learning effectiveness by 23 points using win rate and 3 points at an overall level using DAMR scores.

PledgeTracker: A System for Monitoring the Fulfilment of Pledges (2025.emnlp-demos)

Copied to clipboard

Yulong Chen, Michael Sejr Schlichtkrull, Zhenyun Deng, David Corney, Nasim Asl, Joshua Salisbury, Andrew Dudfield, Andreas Vlachos

Interactive Training: Feedback-Driven Neural Network Optimization (2025.emnlp-demos)

Challenge:	Existing methods simplify pledge verification into document classification task, overlooking its dynamic temporal and multi-document nature.
Approach:	They propose a system that reformulates pledge verification into structured event timeline construction.
Outcome:	The proposed system shows that it can be used in real-world workflows and reduces human verification effort.

Copied to clipboard

Wentao Zhang, Yang Young Lu, Yuntian Deng

Metamo: Empowering Large Language Models with Psychological Distortion Detection for Cognition-aware Coaching (2025.emnlp-demos)

Challenge:	In traditional neural network training, static optimization methods lack flexibility and responsiveness . authors demonstrate that Interactive Training provides superior training stability and reduced sensitivity to initial hyperparameters .
Approach:	They propose an open-source framework that enables real-time feedback-driven optimization of neural networks by human experts or automated AI agents.
Outcome:	The proposed framework achieves superior training stability, reduced sensitivity to initial hyperparameters, and improved adaptability to evolving user needs.

Copied to clipboard

Hajime Hotta, Huu-Loi Le, Manh-Cuong Phan, Minh-Tien Nguyen

InTriage: Intelligent Telephone Triage in Pre-Hospital Emergency Care (2025.emnlp-demos)

Challenge:	Metamo is a browser-based dialogue system that transforms an off-the-shelf large language model into an empathetic coach for everyday workplace concerns.
Approach:	They propose a browser-based dialogue system that first identifies the cognitive distortion behind an emotion, then recognizes the user’s emotion, and finally produces a question-centered reply that invites reflection.
Outcome:	Empirical tests on public corpora showed that the proposed system improved emotionrecognition quality and response diversity without sacrificing latency.

Copied to clipboard

Kai He, Qika Lin, Hao Fei, Eng Siong Chng, Dehan Hong, Marcus Eng Hock Ong, Mengling Feng

Green & Sustainable NLP

Challenge:	Existing TT processes face challenges such as incomplete data collection, communication barriers, and manual errors, leading to high over-triage and under-triages rates.
Approach:	They propose to use an AI-driven multilingual TT system to provide decision support for triage.
Outcome:	The proposed system achieves word error rate of 14.57% for speech recognition and an F1 score of 73.34% for key information extraction.

From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens (2025.emnlp-demos)

Copied to clipboard

Hala Sheta, Eric Haoran Huang, Shuyu Wu, Ilia Alenabi, Jiajun Hong, Ryker Lin, Ruoxi Ning, Daniel Wei, Jialin Yang, Jiawei Zhou, Ziqiao Ma, Freda Shi

ResearStudio: A Human-intervenable Framework for Building Controllable Deep Research Agents (2025.emnlp-demos)

Challenge:	Existing vision-language models are based on exactmatch based accuracy and its derivations to evaluate performance.
Approach:	They propose a toolkit that supports systematic benchmarking, analysis, and interpretation of vision-language models by extracting intermediate outputs from any layer during the forward pass of open-source VLMs.
Outcome:	The proposed toolkit supports 16 state-of-the-art base VLMs and their over 30 variants and is extensible to accommodate new models without changing the core logic.

Copied to clipboard

Linyi Yang, Yixuan Weng

Challenge:	Existing deep-research agents run in a 'fire-and-forget' mode, with no way to fix errors or add expert knowledge during execution.
Approach:	They propose to use a planner-executor to write every step to a live 'plan-as-document' a fast communication layer streams each action, file change, and tool call to . web interface.
Outcome:	The proposed framework surpasses OpenAI’s DeepResearch and Manus in the GAIA benchmark.

OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model (2025.emnlp-demos)

Copied to clipboard

Chen Wang, Tianyu Peng, Wen Yang, YiNan Bai, Guangfu Wang, Jun Lin, Lanpeng Jia, Lingxiang Wu, Jinqiao Wang, Chengqing Zong, Jiajun Zhang

PDFMathTranslate: Scientific Document Translation Preserving Layouts (2025.emnlp-demos)

Challenge:	Empathetic speech models are increasingly closed off, leaving details about the architecture, data and development opaque to researchers.
Approach:	They propose an open-source empathetic speech-to-text model with a streaming interleaved decoding architecture and a data pipeline to enable end-to end training.
Outcome:	The proposed model is open-source and transparent, with no data or data required to build it.

Copied to clipboard

Rongxin Ouyang, Chang Chu, Zhikuang Xin, Xiangyao Ma

Machine Translation

Challenge:	Prior efforts in translating scientific documents overlooked layouts . PDFMathTranslate is open-source with more than 222k downloads - a record for the first time ever.
Approach:	They propose PDFMathTranslate, the world's first open-source software for translating scientific documents while preserving layouts.
Outcome:	The work is open-sourced at https://github.com/byaidu/pdfmathtranslate with more than 222k downloads.

CrowdAgent: Multi-Agent Managed Multi-Source Annotation System (2025.emnlp-demos)

Copied to clipboard

Maosheng Qin, Renyu Zhu, Mingxuan Xia, null Chenchenkai, Zhen Zhu, Minmin Lin, Junbo Zhao, Lu Xu, Changjie Fan, Runze Wu, Haobo Wang

Bratly: A Python Extension for BRAT Functionalities (2025.emnlp-demos)

Challenge:	Recent approaches to annotate data focus on labeling, but lack holistic process control . a novel system that integrates task assignment, data annotation, and quality/cost management is needed .
Approach:	They propose a multi-agent system that integrates task assignment, data annotation, and quality/cost management.
Outcome:	The proposed system automates human management by using a collaborative multi-agent system.

Copied to clipboard

Jamil Zaghir, Jean-Philippe Goldman, Nikola Bjelogrlic, Mina Bjelogrlic, Christian Lovis

Challenge:	BRAT is a widely used web-based text annotation tool, but lacks robust Python support for effective annotation management and processing.
Approach:	They propose an open-source extension of BRAT that introduces a solid Python backend and enables advanced annotation functions such as annotation typings, collection typings with statistical insights, corpus and annotation handling, object modifications, and entity-level evaluation.
Outcome:	The proposed extension streamlines annotation workflows, improves usability, and facilitates high-quality NLP research.

BAREC Demo: Resources and Tools for Sentence-level Arabic Readability Assessment (2025.emnlp-demos)

Copied to clipboard

Kinda Altarbouch, Khalid N. Elmadani, Ossama Obeid, Hanada Taha, Nizar Habash

Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents (2025.emnlp-demos)

Challenge:	Existing efforts to assess the readability of Arabic text are limited due to its rich morphology, complex syntax, and ambiguous orthography.
Approach:	They propose a web-based system for fine-grained, sentence-level Arabic readability assessment.
Outcome:	The demo provides two main functionalities for educators, content creators, language learners, and researchers: (1) a Search interface to explore the annotated dataset for text selection and resource development; (2) an Analyze interface to assign detailed readability labels to Arabic texts at the sentence level.

Copied to clipboard

Ziyang Miao, Qiyu Sun, Jingyuan Wang, Yuchen Gong, Yaowei Zheng, Shiqi Li, Richong Zhang

SlackAgents: Scalable Collaboration of AI Agents in Workspaces (2025.emnlp-demos)

Challenge:	Existing data synthesis tools struggle to extract reliable fine-tuning data from heterogeneous documents.
Approach:	They propose a framework for synthesizing fine-tuning data from unstructured documents via an intuitive graphical user interface.
Outcome:	The proposed framework can extract reliable data from unstructured documents via an intuitive graphical user interface (GUI) it leverages persona-driven prompting approach to generate diverse question-answer pairs using public-available LLMs.

Copied to clipboard

Zhiwei Liu, Weiran Yao, Zuxin Liu, Juntao Tan, Jianguo Zhang, Frank Wang, Sukhandeep Nahal, Huan Wang, Shelby Heinecke, Silvio Savarese, Caiming Xiong

Robustness in NLP

Challenge:	Existing open-source frameworks like LangChain and LlamaIndex fail to integrate into daily workflows, resulting in limited daily usage for work.
Approach:	They propose a multi-agent library for scalable management and collaboration of AI agents on Slack.
Outcome:	The proposed framework offers instant AI integration into organizational workflows and facilitates scalable collaboration, allowing for effective communication and task orchestration.

Open Political Corpora: Structuring, Searching, and Analyzing Political Text Collections with PoliCorp (2025.emnlp-demos)

Copied to clipboard

Nina Smirnova, Muhammad Ahsan Shahid, Philipp Mayr