API

NLP Web Services for Resource-Scarce Languages (P18-4)

Challenge:	Data privacy is an important issue for “machine learning as a service” providers.
Approach:	They propose an attack on membership inference attacks using a sequence-to-sequence model and a machine translation dataset to investigate the feasibility of a privacy attack.
Outcome:	The proposed model can infer sentence-level membership from the output of the model, but it is difficult to infer it.

Copied to clipboard

Martin Puttkammer, Roald Eiselen, Justin Hocking, Frederik Koen

Challenge:	Existing text-based core technologies were ported to Java-based web services for 10 resource-scarce languages spoken in South Africa.
Approach:	They propose to port existing text-based core technologies to Java-based web services from various architectures for 10 resource-scarce languages spoken in South Africa.
Outcome:	The proposed technologies were developed over a period of eight years for 10 resource-scarce languages spoken in South Africa.

GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture (2025.acl-demo)

Copied to clipboard

Valentin Mamedov, Evgenii Kosarev, Gregory Leleytner, Ilya Shchuckin, Valeriy Berezovskiy, Daniil Smirnov, Dmitry Kozlov, Sergei Averkiev, Lukyanenko Ivan, Aleksandr Proshunin, Ainur Israfilova, Ivan Baskov, Artem Chervyakov, Emil Shakirov, Mikhail Kolesov, Daria Khomich, Daria Latortseva, Sergei Porkhun, Yury Fedorov, Oleg Kutuzov, Polina Kudriavtseva, Sofiia Soldatova, Kolodin Egor, Stanislav Pyatkin, Dzmitry Menshykh, Grafov Sergei IUrevich, Eldar Damirov, Vladimir Karlov, Ruslan Gaitukiev, Arkadiy Shatenov, Alena Fenogenova, Nikita Savushkin, Fedor Minkin

StructMem: Structured Memory for Long-Horizon Behavior in LLMs (2026.acl-short)

Challenge:	generative large language models have become crucial for modern NLP research and applications across multiple languages.
Approach:	They introduce the GigaChat family of Russian LLMs, available in various sizes . they evaluate their performance on Russian and English benchmarks and compare them with multilingual analogs .
Outcome:	The proposed model family is available in various sizes and is tested on Russian and English benchmarks.

Copied to clipboard

Buqiang Xu, Yijun Chen, Jizhan Fang, Ruobin Zhong, Yunzhi Yao, Yuqi Zhu, Lun Du, Shumin Deng

ActiveAnno: General-Purpose Document-Level Annotation Tool with Active Learning Integration (2021.naacl-demos)

Challenge:	Existing memory systems lack structure and efficiency in capturing relationships between events.
Approach:	They propose a structure-enriched hierarchical memory framework that preserves event-level bindings and induces cross-event connections.
Outcome:	The proposed framework preserves event-level bindings and induces cross-event connections while reducing token usage, API calls, and runtime compared to prior memory systems.

Copied to clipboard

Max Wiechmann, Seid Muhie Yimam, Chris Biemann

Low-Resource NLP

Challenge:	Existing tools for document-level annotation lack document-based quality and flexibility.
Approach:	a new annotation tool is being developed for industry and research use cases . a configurable user interface and a RESTful API are included . authors propose to use ACTIVEANNO as default for document-level annotation .
Outcome:	ACTIVEANNO is an annotation tool for industry and research use cases.

EasyTurk: A User-Friendly Interface for High-Quality Linguistic Annotation with Amazon Mechanical Turk (2021.eacl-demos)

Copied to clipboard

Lorenzo Bocchi, Valentino Frasnelli, Alessio Palmero Aprosio

Challenge:	Amazon Mechanical Turk (AMT) is one of the most popular crowd-sourcing platforms, allowing researchers from all over the world to create linguistic datasets quickly and at a relatively low cost.
Approach:	They propose to improve the potential of Amazon Mechanical Turk by adding some new features to the tool.
Outcome:	The proposed tool improves the performance of Amazon Mechanical Turk by adding new features.

Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking (2021.emnlp-demo)

Copied to clipboard

Dirk Väth, Pascal Tilli, Ngoc Thang Vu

ASAD: Arabic Social media Analytics and unDerstanding (2021.eacl-demos)

Challenge:	Existing evaluation tools for general Visual Question Answering (VQA) systems are limited to answering accuracy, but they can be used to evaluate performance in real-world scenarios.
Approach:	They propose a browser-based benchmarking tool with an API for easy integration of new models and datasets to keep up with the fast-changing landscape of VQA.
Outcome:	The proposed tool tests generalization capabilities of models across multiple datasets and includes metrics that measure biases and uncertainty to further explain model behavior.

Copied to clipboard

Sabit Hassan, Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish

Sentiment Analysis

Challenge:	Currently, there are no publicly available tools for analyzing Arabic social media, such as ADIDA and CAMeL, which are not trained with Twitter data.
Approach:	They propose to use Arabic social media analysis and unDerstanding to analyze tweets using a web API and a user interface.
Outcome:	The proposed system allows users to determine dialects, sentiment, news category, offensiveness, hate speech, adult content, and spam in Arabic tweets.

A description and demonstration of SAFAR framework (2021.eacl-demos)

Copied to clipboard

Karim Bouzoubaa, Younes Jaafar, Driss Namly, Ridouane Tachicart, Rachida Tajmout, Hakima Khamar, Hamid Jaafar, Lhoussain Aouragh, Abdellah Yousfi

Challenge:	Existing NLP infrastructures are naming them "toolkit", "platform" and "framework" authors present a monolingual framework dedicated to Arabic language .
Approach:	They propose a monolingual framework dedicated to Arabic language . they propose namings for existing infrastructures: "toolkit", "platform" and "framework"
Outcome:	The proposed framework is dedicated to Arabic language, especially the modern standard Arabic and Moroccan dialect.

GR-NLP-TOOLKIT: An Open-Source NLP Toolkit for Modern Greek (2025.coling-demos)

Copied to clipboard

Lefteris Loukas, Nikolaos Smyrnioudis, Chrysa Dikonomaki, Spiros Barbakos, Anastasios Toumazatos, John Koutsikakis, Manolis Kyriakakis, Mary Georgiou, Stavros Vassos, John Pavlopoulos, Ion Androutsopoulos

Syntactic Text Processing

Challenge:	GR-NLP-TOOLKIT is an open-source natural language processing toolkit for modern Greek.
Approach:	They present GR-NLP-TOOLKIT, an open-source natural language processing toolkit for Greek.
Outcome:	The toolkit provides state-of-the-art performance in five core NLP tasks . it can be easily installed in Python and is accessible through a demonstration platform on HuggingFace .

FastFit: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024.naacl-demo)

Copied to clipboard

Asaf Yehudai, Elron Bandel

Low-Resource NLP

Challenge:	Few-shot prompting of large language models (LLMs) via API calls presents a unique challenge when dealing with a multitude of classes that share similar semantic meanings.
Approach:	They present a Python package that integrates batch contrastive learning and token-level similarity score to provide fast few-shot classification.
Outcome:	The proposed method significantly improves multi-class classification speed and accuracy across English and Multilingual datasets.

Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning (2024.findings-naacl)

Copied to clipboard

Yinger Zhang, Hui Cai, Xierui Song, Yicheng Chen, Rui Sun, Jing Zheng

Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers (2025.naacl-short)

Challenge:	Existing approaches to enable large language models to implement function calling are limited in their tool-use capabilities.
Approach:	They propose a controllable, target-driven approach to empower LLMs to operate external APIs only via prompts.
Outcome:	The proposed approach limits LLMs to executing simple tasks, e.g., API Selection and Argument Completion.

Copied to clipboard

Akshit Achara, Anshuman Chhabra

MixingBoard: a Knowledgeable Stylized Integrated Text Generation Platform (2020.acl-demos)

Challenge:	ASM classifiers are designed to moderate content on social media platforms and serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs.
Approach:	They examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API.
Outcome:	The classifiers do not unfairly classify content belonging to minority groups as unsafe compared to those belonging to majority groups and their behavior remains robust and consistent across similar inputs.

Copied to clipboard

Xiang Gao, Michel Galley, Bill Dolan

string2string: A Modern Python Library for String-to-String Algorithms (2024.acl-demos)

Challenge:	Neural text generation algorithms have seen great improvements over the past several years.
Approach:	They propose a platform for quickly building demos with a focus on knowledge grounded stylized text generation.
Outcome:	The proposed framework unifies existing text generation algorithms in a shared codebase and further adapts earlier algorithms for constrained generation.

Copied to clipboard

Mirac Suzgun, Stuart Shieber, Dan Jurafsky

Daniel Baumartz, Tolga Uslu, Alexander Mehler

Challenge:	Notable algorithms include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, and the Wagner-Fischer algorithm for edit distance.
Approach:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.
Outcome:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.

LTV: Labeled Topic Vector (C18-2)

Copied to clipboard

Information Extraction & Text Mining

Challenge:	Using nnDDC, we generate labeled topic classifications based on the Dewey Decimal Classification (DDC) Unlike related approaches, we use classifiers to define the dimensions of CISS, which are directly labeles by the underlying target class.
Approach:	They propose a website and API that generates labeled topic classifications based on the Dewey Decimal Classification (DDC) they propose nnDDC, a largely language-independent natural network-based classifier for DDC, which is language-dependent .
Outcome:	The proposed model is language-independent and performs well in 40 languages.

Prta: A System to Support the Analysis of Propaganda Techniques in the News (2020.acl-demos)

Copied to clipboard

Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, Preslav Nakov

Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective (2023.emnlp-industry)

Challenge:	recent events have brought the public attention to the dangers of online disinformation.
Approach:	a new tool helps users analyze propaganda using specific rhetorical and psychological techniques. a prta system identifies the spans in which propaganda techniques occur and compares them.
Outcome:	a new tool can analyze articles crawled on a regular basis and compare them on the basis of their use of propaganda techniques.

Copied to clipboard

Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, Shashi Bhushan TN

ExplainaBoard: An Explainable Leaderboard for NLP (2021.acl-demo)

Challenge:	a study examines how to build meeting summarization systems using large language models . closed-source models are generally better in terms of performance, but open-source ones are more advantageous for industrial use .
Approach:	They compare closed-source and open-source meeting summarization models for real-world use . they find that closed-sourced models are generally better in terms of performance . however, smaller open-sourced LLMs could still achieve comparable performance if they are open .
Outcome:	The proposed model is more efficient for industrial use than closed-source models due to privacy concerns and high cost.

Copied to clipboard

Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Graham Neubig

Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction (2023.acl-demo)

Challenge:	Using leaderboards, researchers can track the performance of various systems on various NLP tasks.
Approach:	They propose a new conceptualization and implementation of NLP evaluation using a leaderboard.
Outcome:	The ExplainaBoard is an evaluation tool for natural language processing (NLP) it covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks.

Copied to clipboard

Gabriele Picco, Marcos Martinez Galindo, Alberto Purpura, Leopold Fuchs, Vanessa Lopez, Thanh Lam Hoang

LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing (2021.acl-demo)

Challenge:	ZSL is a machine learning field that uses textual descriptions of entities or relations to perform tasks that are not seen during training.
Approach:	They propose a framework that allows researchers to compare state-of-the-art ZSL methods with standard benchmark datasets.
Outcome:	The proposed framework compares state-of-the-art methods with benchmark datasets and provides APIs for production under the standard SpaCy NLP pipeline.

Copied to clipboard

Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, Zhou Yu

Natural Language Interfaces

Challenge:	Currently, researchers use automatic metrics and human evaluation to evaluate dialogue systems.
Approach:	They propose to use a Python API to easily evaluate dialogue systems using Amazon Mechanical Turk.
Outcome:	The open-source toolkit provides a fast, consistent method for reproducing human evaluation results.

Kompetencer: Fine-grained Skill Classification in Danish Job Postings via Distant Supervision and Transfer Learning (2022.lrec-1)

Copied to clipboard

Mike Zhang, Kristian Nørgaard Jensen, Barbara Plank

LongGenBench: Long-context Generation Benchmark (2024.findings-emnlp)

Challenge:	Several studies focus on Skill Identification, but there is little work in further categorizing the identified skills.
Approach:	They propose a Danish job posting dataset annotated for nested spans of competences . they use the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy API to obtain fine-grained labels via distant supervision.
Outcome:	The proposed dataset outperforms existing models in the Danish job postings.

Copied to clipboard

Xiang Liu, Peijie Dong, Xuming Hu, Xiaowen Chu

Evaluating Embedding APIs for Information Retrieval (2023.acl-industry)

Challenge:	Current long-context benchmarks focus on retrieval-based tests, requiring Large Language Models to locate specific information within extensive input contexts.
Approach:	They propose a long-context generation benchmark that allows for flexible configurations of customized generation context lengths.
Outcome:	The proposed benchmark improves performance on NIAH and other retrieval-based tests.

Copied to clipboard

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-hermelo, Mehdi Rezagholizadeh, Jimmy Lin

Towards Reliable Agents: Benchmarking Customized LLM-Based Retrieval-Augmented Generation Frameworks with Deployment Validation (2025.naacl-industry)

Challenge:	a growing number of language models are limiting their access to the community . we evaluate existing APIs for domain generalization and multilingual retrieval .
Approach:	They evaluate semantic embedding APIs in retrieval scenarios to assess their capabilities . they use BEIR and MIRACL to re-rank BM25 results using the APIs .
Outcome:	The proposed model is based on semantic embedding APIs that build vector representations of a given text.

Copied to clipboard

Kevin Shukang Wang, Karel Joshua Harjono, Ramon Lawrence

LLMs on interactive feature collections with implicit dynamic decision strategy (2025.coling-main)

Challenge:	Existing benchmarks for general-purpose RAG systems, such as CRAG, RGB, MultiHop-RAG, and CRUD-RAGG, are limited and lack a benchmark specifically tailored to evaluate frameworks.
Approach:	They evaluated OpenAI’s Assistants API versus a RAG assistant built with Langchain and deployed a system based on benchmark insights as a course assistant over a two-year span.
Outcome:	The proposed benchmarks show that domain-specific retrieval impacts response accuracy and highlight key challenges in real-world deployment.

Copied to clipboard

Juyeon Heo, Vihari Piratla, Kyunghyun Lee, Hyonkeun Joh, Adrian Weller

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) struggle to efficiently narrow down the search space . external engineered systems may not fully utilize the inherent problem-solving capabilities of LLMs .
Approach:	They propose to implicitly guide Large Language Models to enhance their interactive feature collection abilities within a single prompt.
Outcome:	The proposed approach improves the performance of large language models in real-world scenarios.

TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2021.acl-long)

Copied to clipboard

Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, Mihir Kale

ATGen: A Framework for Active Text Generation (2025.acl-demo)

Challenge:	TicketTalk dataset with 23,789 annotated dialogs is a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy.
Approach:	They propose a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy.
Outcome:	The proposed model generates verbal responses and API call predictions on a movie ticketing dialog dataset with 23,789 annotated conversations.

Copied to clipboard

Akim Tsvigun, Daniil Vasilev, Ivan Tsvigun, Ivan Lysenko, Talgat Bektleuov, Aleksandr Medvedev, Uliana Vinogradova, Nikita Severin, Mikhail Mozikov, Andrey Savchenko, Ilya Makarov, Grigorev Rostislav, Ramil Kuleev, Fedor Zhdanov, Artem Shelmanov

Metamo: Empowering Large Language Models with Psychological Distortion Detection for Cognition-aware Coaching (2025.emnlp-demos)

Challenge:	Despite the surging popularity of natural language generation tasks, the application of active learning (AL) to NLG has been limited.
Approach:	They propose a framework that bridges AL with text generation tasks and provides a unified platform for smooth implementation and benchmarking of novel AL strategies tailored to NLG tasks.
Outcome:	The proposed framework simplifies AL-empowered annotation in NLG tasks using both human annotators and automatic annotation agents based on large language models (LLMs).

Copied to clipboard

Hajime Hotta, Huu-Loi Le, Manh-Cuong Phan, Minh-Tien Nguyen

EVE: A Domain-Specific LLM Framework for Earth Intelligence (2026.acl-industry)

Challenge:	Metamo is a browser-based dialogue system that transforms an off-the-shelf large language model into an empathetic coach for everyday workplace concerns.
Approach:	They propose a browser-based dialogue system that first identifies the cognitive distortion behind an emotion, then recognizes the user’s emotion, and finally produces a question-centered reply that invites reflection.
Outcome:	Empirical tests on public corpora showed that the proposed system improved emotionrecognition quality and response diversity without sacrificing latency.

Copied to clipboard

Àlex R. Atrio, Antonio Lopez, Jino Rohit, Yassine El Ouahidi, Marcello Politi, Vijayasri Iyer, Umar Jamil, Sébastien Bratières, Nicolas Longépé

Question Answering

Challenge:	Earth Virtual Expert (EVE) is the first open-source, end-to-end initiative for developing and deploying domain-specialized LLMs for Earth Intelligence.
Approach:	They introduce Earth Virtual Expert, an open-source initiative for developing and deploying domain-specialized LLMs for Earth Intelligence.
Outcome:	The proposed model outperforms existing models on Earth Observation and Earth Sciences benchmarks while maintaining general capabilities.

Adapting LLMs for Structured Natural Language API Integration (2024.emnlp-industry)

Copied to clipboard

Robin Chan, Katsiaryna Mirylenka, Thomas Gschwind, Christoph Miksovic, Paolo Scotton, Enrico Toniato, Abdel Labbi

Named Entity Recognition

Challenge:	API integration is crucial for enterprise systems, but there are challenges in combining APIs based on user intent.
Approach:	They propose a framework that leverages large language models to integrate APIs based on natural language input.
Outcome:	The proposed framework improves performance over existing methods and RAGs based on open APIs . it can learn structural API constraints implicitly during training and retain structured knowledge .

In-Context Retrieval-Augmented Language Models (2023.tacl-1)

Copied to clipboard

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

Exploring efficient zero-shot synthetic dataset generation for Information Retrieval (2024.findings-eacl)

Challenge:	Existing RALM methods focus on modifying the LM architecture to facilitate incorporation of external information, complicating deployment.
Approach:	They propose to condition a language model on relevant documents from a grounding corpus during generation by conditioning on external knowledge sources.
Outcome:	The proposed method significantly improves language modeling performance and provides natural source attribution mechanism.

Copied to clipboard

Tiago Almeida, Sérgio Matos

GenDLN: Evolutionary Algorithm-Based Stacked LLM Framework for Joint Prompt Optimization (2025.acl-srw)

Challenge:	Recent advances in large language models offer a new avenue of generating synthetic training data to train neural retrieval models for unlabelled data collections.
Approach:	They propose a method to generate high-quality synthetic datasets using a small language model and a filtering mechanism to ensure the quality of generated questions.
Outcome:	The proposed method outperforms unsupervised retrieval methods such as BM25 and pretrained monoT5.

Copied to clipboard

Pia Chouayfati, Niklas Herbster, Ábel Domonkos Sáfrán, Matthias Grabmair

RESTful-Llama: Connecting User Queries to RESTful APIs (2024.emnlp-industry)

Challenge:	Large Language Models (LLMs) are increasingly replacing traditional classification and inference models due to their generality, ability to perform a wide range of tasks, and seemingly advanced "reasoning" prompt optimization is a promising alternative to manual/human prompt engineering, but the cost of using LLMs for prompt optimization via commercial APIs remains high.
Approach:	They propose an open-source, efficient genetic algorithm-based prompt pair optimization framework that leverages commercial APIs.
Outcome:	The proposed approach allows teams with limited resources to efficiently use commercial LLMs for prompt optimization.

Copied to clipboard

Han Xu, Ruining Zhao, Jindong Wang, Haipeng Chen

Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QA (2026.acl-industry)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated exceptional performance in zero-shot learning and reasoning tasks.
Approach:	They propose a framework that transforms natural language instructions into effective RESTful API calls and a method to generate fine-tuning datasets from public API documentation.
Outcome:	The proposed framework improves performance in a 31.9% improvement in robustness and 2.33x increase in efficiency compared to existing methods.

Copied to clipboard

Ummar Abbas, Mourad Ouzzani, Mohamed Y. Eltabakh, Omar Sinan, Gagan Bhatia, Hamdy Mubarak, Majd Hawasly, Mohammed Qusay Hashim, Kareem Mohamed Darwish, Firoj Alam

Information Retrieval

Challenge:	Large language models (LLMs) can answer religious knowledge queries fluently, but they often hallucinate and misattribute sources.
Approach:	They propose a bilingual Arabic-English Islamic QA system that uses a multi-agent, tool-augmented architecture to route Islamic queries to specialized modules.
Outcome:	The proposed system is based on a multi-agent, tool-augmented architecture and has received over 1.9M accesses in less than a year.

Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification (L18-1)

Copied to clipboard

Wajdi Zaghouani, Anis Charfi

Challenge:	Existing corpus of Arabic textual data is limited to English or other European languages.
Approach:	They present a large-scale and multi-dialectal corpus of Tweets from 11 regions and 16 countries in the arab world representing the major Arabic dialectal varieties.
Outcome:	The provided corpus will enrich the limited set of available language resources for Arabic and be invaluable enabler for developing author profiling tools and NLP tools for Arabic.

Efficient Answer Retrieval System (EARS): Combining Local DB Search and Web Search for Generative QA (2024.emnlp-industry)

Copied to clipboard

Nikita Krayko, Ivan Sidorov, Fedor Laputin, Daria Galimzianova, Vasily Konovalov

Question Answering

Challenge:	Developing a virtual assistant is crucial for supporting clients as it provides 24/7 assistance . factual questionanswering system is capable of handling all user queries .
Approach:	They propose a production-ready factual question answering system that combines local knowledge base search with generative, context-based QA.
Outcome:	The proposed system boosts local knowledge base retrieval by 23% . the system is language-agnostic and can be applied to any data domain .

Declarative Techniques for NL Queries over Heterogeneous Data (2025.emnlp-industry)

Copied to clipboard

Elham Khabiri, Jeffrey O. Kephart, Fenno F. Heath Iii, Srideepika Jayaraman, Yingjie Li, Fateh A. Tipu, Dhruv Shah, Achille Fokoue, Anu Bhamidipaty

Challenge:	In many industrial settings, users wish to ask questions in natural language . however, these applications do not cope with data source heterogeneity that typifies such environments.
Approach:	They propose a declarative approach to handling data heterogeneity in industrial settings . they simulate the heterogenity of industrial environments by adding two extensions of the popular Spider benchmark dataset .
Outcome:	The proposed approach copes with data source heterogeneity better than state-of-the-art systems.

Hierarchical and Dynamic Prompt Compression for Efficient Zero-shot API Usage (2024.findings-eacl)

Copied to clipboard

Yichen Jiang, Marco Vecchio, Mohit Bansal, Anders Johannsen

Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation (2022.naacl-main)

Challenge:	Long prompts present a significant challenge for LLM-based systems that learn to use unseen APIs directly in-context from their documentation.
Approach:	They propose a method that learns to compress a prompt into a few "gist token" activations during finetuning but is ineffective in compressing API documentation.
Outcome:	The proposed model keeps the simplicity, efficiency, and large compression factor (20x on SGD) of the gist token approach while achieving significantly better accuracy.

Copied to clipboard

Guangyi Liu, Zichao Yang, Tianhua Tao, Xiaodan Liang, Junwei Bao, Zhen Li, Xiaodong He, Shuguang Cui, Zhiting Hu

DispatchQA: A Benchmark for Small Function Calling Language Models in E-Commerce Applications (2025.emnlp-industry)

Challenge:	Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss.
Approach:	They propose an Edit-Invariant Sequence Loss method which computes the matching loss of a target sequence with all n-grams in the generated sequence.
Outcome:	The proposed method outperforms the common CE loss and strong baselines on a wide range of tasks.

Copied to clipboard

Joachim Daiber, Victor Maricato, Ayan Sinha, Andrew Rabinovich

<SYNTACT>: Structuring Your Natural Language SOPs into Tailored Ambiguity-Resolved Code Templates (2025.emnlp-industry)

Challenge:	DispatchQA is a benchmark to evaluate how well small language models (SLMs) translate openended search queries into executable API calls via explicit function calling.
Approach:	They propose a benchmark to evaluate how well small language models translate openended search queries into executable API calls via explicit function calling.
Outcome:	The proposed benchmark aims to evaluate how well small language models (SLMs) translate openended search queries into executable API calls via explicit function calling.

Copied to clipboard

Sachin Kumar Giroh, Pushpendu Ghosh, Aryan Jain, Harshal Giridhari Paunikar, Aditi Rastogi, Promod Yenigalla, Anish Nediyanchath

Challenge:	Unstructured and ambiguous Standard Operating Procedures suffer from ambiguity, missing information, and inconsistency, all of which hinder automation.
Approach:	They propose a three-stage LLM framework that transforms unstructured SOPs into a structured plan and an executable code template.
Outcome:	The proposed framework shows an 88.4% accuracy and significant reduction in inconsistency on real-world SOPs and synthetic variants.

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2023.emnlp-main)

Copied to clipboard

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li

Natural Language Interfaces

Challenge:	Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools.
Approach:	They propose a runnable evaluation system consisting of 73 API tools and an annotation system for 314 tool-use dialogues with 753 API calls.
Outcome:	The proposed benchmark assesses the effectiveness of existing LLMs by analyzing 314 tool-use dialogues with 753 API calls.

Your Semantic-Independent Watermark is Fragile: A Semantic Perturbation Attack against EaaS Watermark (2025.findings-emnlp)

Copied to clipboard

Zekun Fei, Biao Yi, Jianing Geng, He Ruiqi, Lihai Nie, Zheli Liu

HoneyComb: A Flexible LLM-Based Agent System for Materials Science (2024.findings-emnlp)

Challenge:	Embedding-as-a-Service (EaaS) is a successful business pattern but faces significant challenges related to various forms of copyright infringement.
Approach:	They propose a semantic-independent watermarking scheme that exploits semantic perturbation tests to bypass verification.
Outcome:	The proposed watermarking schemes possess semantic-independent characteristics and exploit semantic perturbation tests to bypass verification.

Copied to clipboard

Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, Bang Liu

Verifiable Format Control for Large Language Model Generations (2025.findings-naacl)

Challenge:	specialized large language models (LLMs) have shown promise in materials science but often struggle with the distinct complexities of materials science tasks.
Approach:	They propose a new LLM-based agent system specifically designed for materials science that leverages a reliable materials science knowledge base and a sophisticated tool hub.
Outcome:	The proposed system outperforms baseline models across tasks in materials science while ensuring accuracy and relevance.

Copied to clipboard

Zhaoyang Wang, Jinqi Jiang, Huichi Zhou, Wenhao Zheng, Xuchao Zhang, Chetan Bansal, Huaxiu Yao

ARL2: Aligning Retrievers with Black-box Large Language Models via Self-guided Adaptive Relevance Labeling (2024.acl-long)

Challenge:	Existing methods focus on benchmarking general instruction following while overlooking how to improve specific format following ability for small LLMs.
Approach:	They propose to synthesize massive datasets to improve LLMs' format following abilities by using a verifiable format following feature.
Outcome:	The proposed method improves the format following ability of small LLMs with about 7B parameters.

Copied to clipboard

LingXi Zhang, Yue Yu, Kuan Wang, Chao Zhang

Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants’ API Invocation Capabilities (2024.lrec-main)

Challenge:	Existing retrievers are misaligned with large language models due to separate training processes and inherent black-box nature of LLMs.
Approach:	They propose a retriever learning technique that harnesses LLMs as labelers to annotate and score adaptive relevance evidence.
Outcome:	Extensive experiments show that ARL2 improves accuracy and reduces the cost of API calls.

Copied to clipboard

Honglin Mu, Yang Xu, Yunlong Feng, Xiaofeng Han, Yitong Li, Yutai Hou, Wanxiang Che

Challenge:	Existing evaluation methods for human-machine interactions are static and can be misleading.
Approach:	They propose to use a LLM-based user agent to assess an assistant's API call capability without human involvement.
Outcome:	The proposed method mirrors real human conversation patterns in human-machine interactions, and shows that it aligns more closely with human assessment.

ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context (2024.findings-emnlp)

Copied to clipboard

Zirui Wu, Yansong Feng

Interpreting User Requests in the Context of Natural Language Standing Instructions (2024.findings-naacl)

Challenge:	Tables are a crucial tool for organizing and presenting information in various domains.
Approach:	They propose a Plan-then-Reason framework to answer different types of user queries over tables with sentence context.
Outcome:	The proposed framework outperforms existing frameworks without self-consistency while using less API calls and in-context demonstrations.

Copied to clipboard

Nikita Moghe, Patrick Xia, Jacob Andreas, Jason Eisner, Benjamin Van Durme, Harsh Jhamtani

Universal Semantic Annotator: the First Unified API for WSD, SRL and Semantic Parsing (2022.lrec-1)

Challenge:	Existing approaches to LLM-based dialogue modeling provide additional context for users to make requests.
Approach:	They propose an approach to LLM-based dialogue modeling where persistent user constraints and preferences are provided as additional context for such interfaces.
Outcome:	The proposed model achieves a maximum of 46% exact match on the prediction of 2.4K English dialogues with a language-to-program dataset.

Copied to clipboard

Riccardo Orlando, Simone Conia, Stefano Faralli, Roberto Navigli

Digital Socrates: Evaluating LLMs through Explanation Critiques (2024.acl-long)

Challenge:	Existing approaches to understanding textual information are still far from achieving true natural language understanding (NLU).
Approach:	They propose a unified API for high-quality automatic annotations of texts in 100 languages through state-of-the-art systems for Word Sense Disambiguation, Semantic Role Labeling and Semantics Parsing.
Outcome:	The proposed system can provide users with rich and diverse semantic information, help second-language learners, and integrate explicit semantic knowledge into downstream tasks and real-world applications.

Copied to clipboard

Yuling Gu, Oyvind Tafjord, Peter Clark

FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking (2024.findings-emnlp)

Challenge:	Large language models (LLMs) can provide reasoned explanations, but the nature and quality of those explanations are still poorly understood.
Approach:	They propose to define a task of explanation critiquing and train an open-source automatic critique model using this data.
Outcome:	The proposed model can provide high-quality, nuanced evaluations without expensive API calls or human annotations.

Copied to clipboard

Zhuoer Wang, Leonardo Ribeiro, Alexandros Papangelis, Rohan Mukherjee, Tzu-Yen Wang, Xinyan Zhao, Arijit Biswas, James Caverlee, Angeliki Metallinou

Programming in Natural Language with fuSE: Synthesizing Methods from Spoken Utterances Using Deep Natural Language Understanding (2020.acl-main)

Challenge:	Existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user’s request.
Approach:	They propose an output-side optimization approach that uses state-tracked constrained decoding and a lightweight model as the discriminator to rerank the beam-searched candidate generations of the large language model.
Outcome:	The proposed approach outperforms existing supervised and in-context learning approaches in the DSTC8 and API Bank datasets in terms of call generation accuracy, inference efficiency, and context efficiency.

Copied to clipboard

Sebastian Weigelt, Vanessa Steurer, Tobias Hey, Walter F. Tichy

A Framework for Shared Agreement of Language Tags beyond ISO 639 (2020.lrec-1)

Challenge:	a new approach to teach new functions from natural language is needed to make intelligent systems programmable in everyday language.
Approach:	They propose to use natural language to teach intelligent systems new functions . fuSE synthesizes method signatures and API calls from spoken utterances .
Outcome:	The proposed system synthesizes 84.6% of method signatures and 79.2% of API calls correctly on unseen dataset.

Copied to clipboard

Frances Gillis-Webber, Sabine Tittel

Challenge:	Identification and annotation of languages in an unambiguous and standardized way is essential for the description of linguistic data.
Approach:	They propose a pattern that extends the BCP 47 sub-tag ‘privateuse’ and is able to overcome the limits of BCP47 and ISO 639.
Outcome:	The proposed pattern overcomes the limitations of BCP 47 and ISO 639 for the identification of lesser-known languages, endangered languages, regional varieties or historical stages of a language.

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024.findings-acl)

Copied to clipboard

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

What data should I include in my POS tagging training set? (2025.findings-emnlp)

Challenge:	Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants.
Approach:	They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs.
Outcome:	The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting.

Copied to clipboard

Zoey Liu, Masoud Jasbi, Christan Grant, Kenji Sagae, Emily Prud’hommeaux

Syntactic Text Processing

Challenge:	POS tagging is a crucial task for descriptive linguistics and language documentation . POS tags are not available in all languages, but are used for training sets for understudied languages .
Approach:	They compare POS tagging with in-context learning, active learning, and random sampling . they find that POS can deliver reasonable results for communities with limited resources .
Outcome:	The proposed training set for Indigenous and endangered languages performs better than random sampling.

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (D19-1)

Copied to clipboard

Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik

Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)

Challenge:	a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs .
Approach:	They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach .
Outcome:	The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains.

Copied to clipboard

Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)

Challenge:	In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API.
Approach:	They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API.
Outcome:	The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting.

Copied to clipboard

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker

Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel (2024.emnlp-main)

Challenge:	Perception of toxicity evolves over time and differs between geographies and cultural backgrounds.
Approach:	They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results.
Outcome:	The Perspective API has been updated to reflect the changes in toxicity scores.

Copied to clipboard

Brendan King, Jeffrey Flanigan

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing (2023.findings-emnlp)

Challenge:	a task-oriented dialogue system requires turn-level annotations for interacting with their APIs.
Approach:	They propose an unsupervised approach that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent.
Outcome:	The proposed method doubles the success rate of a strong GPT-3.5 benchmark.

Copied to clipboard

Shufan Wang, Sébastien Jean, Sailik Sengupta, James Gung, Nikolaos Pappas, Yi Zhang

Scalable and Domain-General Abstractive Proposition Segmentation (2024.findings-emnlp)

Challenge:	In task-oriented semantic parsing, the system aims to translate users’ utterances in natural language to machine-interpretable programs (API calls) However, Large Language Models (LLMs) are known to hallucinate and therefore pose a formidable challenge in constraining generated content.
Approach:	They propose to use large language models to translate user's utterances to machine-interpretable programs (API calls) they identify constraints violations in task-oriented utterrances and define fine-grained metrics that complement traditional ones.
Outcome:	The proposed methods reduce constraints violations and improve quality of the generated API calls, but require careful consideration given their implementation complexity and latency.

Copied to clipboard

Mohammad Javad Hosseini, Yang Gao, Tim Baumgärtner, Alex Fabrikant, Reinald Kim Amplayo

Text Segmentation

Challenge:	Several recent studies have demonstrated the utility of proposition segmentation for downstream tasks.
Approach:	They propose a scalable, yet accurate, proposition segmentation model that can be supervised by LLMs.
Outcome:	The proposed model improves on training on annotated datasets and shows that it is easy to use.

Template Guided Text Generation for Task-Oriented Dialogue (2020.emnlp-main)

Copied to clipboard

Mihir Kale, Abhinav Rastogi

The Critique of Critique (2024.findings-acl)

Challenge:	a new study examines the use of templates to generate natural language utterances for a large number of APIs.
Approach:	They propose a schema-guided approach which conditions the generation on a natural language schema.
Outcome:	The proposed method improves over strong baselines, is robust to out-of-domain inputs and shows improved sample efficiency.

Copied to clipboard

Shichao Sun, Junlong Li, Weizhe Yuan, Ruifeng Yuan, Wenjie Li, Pengfei Liu

Challenge:	MetaCritique builds specific quantification criteria to evaluate the quality of critique . a systematic method to evaluate critique is lacking.
Approach:	They propose a critique of critique, termed MetaCritique, which builds specific quantification criteria and aggregates each AIU's judgment for the overall score.
Outcome:	The proposed method can achieve near-human performance across 16 datasets.

WikiDragon: A Java Framework For Diachronic Content And Network Analysis Of MediaWikis (L18-1)

Copied to clipboard

Rüdiger Gleim, Alexander Mehler, Sung Y. Song

Information Extraction & Text Mining

Challenge:	WikiDragon is a Java Framework designed to give developers in computational linguistics an intuitive API to build, parse and analyze instances of MediaWikis.
Approach:	They introduce WikiDragon, a Java Framework that allows developers to build, parse and analyze instances of MediaWikis on their computers.
Outcome:	The framework is based on the Wikipedia, Wiktionary, WikiSource or WikiNews and evaluates link extraction, diachronic network analysis and the impact of different frameworks to text analysis.

Socratic Style Chain-of-Thoughts Help LLMs to be a Better Reasoner (2025.findings-acl)

Copied to clipboard

Jiangbo Pei, Peiyu Liu, Xin Zhao, Aidong Men, Yang Liu

Cultivating Gaming Sense for Yourself: Making VLMs Gaming Experts (2025.acl-long)

Challenge:	Synthetic data generation is a promising approach to enhance reasoning capabilities of large language models.
Approach:	They propose a multi-agent debate framework based on the Socratic questioning strategy . they use socratic questions to deepen the thinking process and broaden it to motivate self-reflection .
Outcome:	The proposed framework outperforms existing methods on math and code generation tasks while maintaining affordable costs.

Copied to clipboard

Wenxuan Lu, Jiangyang He, Zhanqiu Zhang, Steven Y. Guo, Tianning Zang

TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023.findings-emnlp)

Challenge:	Recent efforts leverage Vision Language Models (VLMs) as direct controllers, often pausing the game to analyze screens and plan action through language reasoning.
Approach:	They propose a paradigm shift in gameplay agent design that uses Vision Language Models as a developer instead of direct control.
Outcome:	The proposed framework achieves fluent gameplay in diverse genres, including ACT, FPS, and Flappy Bird, setting a new benchmark for game-playing agents.

Copied to clipboard

Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, Yiming Qian

Information Extraction & Text Mining

Challenge:	ChatGPT and GPT-4 are commercial large language models (LLMs) however, they may produce vague responses or incorrect answers in certain specialized domains.
Approach:	They propose a token compression scheme that uses summarization and semantic compression to reduce the token size of LLMs.
Outcome:	The proposed method reduces token size by doing summarization and semantic compression while reducing token size with only 1.6% accuracy drop.

German Parliamentary Corpus (GerParCor) Reloaded (2024.lrec-main)

Copied to clipboard

Giuseppe Abrami, Mevlüt Bagci, Alexander Mehler

Challenge:	In 2022, the largest German-speaking corpus of parliamentary protocols from three different centuries has been published - GerParCor.
Approach:	They propose to update the largest German-speaking corpus of parliamentary protocols from three different centuries, on a national and federal level, from Germany, Austria, Switzerland and Liechtenstein, and to make them available in XMI format.
Outcome:	The updated corpus includes all new parliamentary protocols and adds and preprocesses further parliamentary protocol not covered in the previous version.

Retrieving Information from the French Lexical Network in RDF/OWL Format (L18-1)

Copied to clipboard

Alexsandro Fonseca, Fatiha Sadat, François Lareau

Information Retrieval

Challenge:	a Java API to retrieve lexical information from the French Lexical Network is presented . RDF/OWL languages are not sufficient for a more detailed representation of linguistic information.
Approach:	They propose a Java API to retrieve lexical information from the French Lexical Network . this API was used in the identification of collocations in a french corpus of 1.8 million sentences .
Outcome:	The proposed API was used to identify collocations in a French corpus of 1.8 million sentences and in the semantic classification of these collocation.

Android in the Zoo: Chain-of-Action-Thought for GUI Agents (2024.findings-emnlp)

Copied to clipboard

Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang

Challenge:	Existing studies on large language models (LLMs) focus on the semantics of smartphone operations.
Approach:	They propose a large language model (LLM) which predicts a sequence of actions of API by analyzing past actions and visual observations.
Outcome:	The proposed model improves the prediction of actions on a zero-shot Android-In-The-Zoo dataset compared to previous models .

Cache & Distil: Optimising API Calls to Large Language Models (2024.findings-acl)

Copied to clipboard

Guillem Ramírez, Matthias Lindemann, Alexandra Birch, Ivan Titov

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion (2024.findings-emnlp)

Challenge:	Large Language Models are expensive to run and expose the entire request stream to external providers.
Approach:	They propose to locally train a small private language model on the LLM's predictions to minimise the costs and data exposure associated with calling the API.
Outcome:	The proposed model can handle an increasing number of user requests independently and is able to perform better than other policies and baselines across tasks and budgets.

Copied to clipboard

Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

Humanitarian Corpora for English, French and Spanish (2024.lrec-main)

Challenge:	Large Language Models (LLMs) are increasingly used for task completion in real-world situations.
Approach:	They propose a PowerPoint Task Completion-Robustness (PPTC-R) benchmark to measure LLMs’ robustness to the user PPT task instruction and software version (Powerpoint).
Outcome:	The proposed benchmark compares 3 closed-source and 4 open-source LLMs to the PowerPoint task instruction and software version (Powerpoint) .

Copied to clipboard

Loryn Isaacs, Santiago Chambó, Pilar León-Araúz

Text Classification

Challenge:	et al., a leading database of humanitarian documents, compiled with ReliefWeb reports . documents selected with language identification and noise reduction techniques . authors present corpora of English, French and Spanish humanitarian documents .
Approach:	They present three corpora of English, French and Spanish humanitarian documents compiled with ReliefWeb reports . documents were tokenized, lemmatized, tagged by part of speech, and enriched with metadata . authors propose a project to develop a humanitarian dictionary with a focus on conceptual variation .
Outcome:	The corpora were compiled to satisfy the research needs of the Humanitarian Encyclopedia project with a focus on conceptual variation.

Grammar Pruning: Enabling Low-Latency Zero-Shot Task-Oriented Language Models for Edge AI (2025.emnlp-main)

Copied to clipboard

Octavian Alexandru Trifan, Jason Lee Weber, Marc Titus Trifan, Alexandru Nicolau, Alexander Veidenbaum

CoCo: A Tool for Automatically Assessing Conceptual Complexity of Texts (2020.lrec-1)

Challenge:	Existing approaches to task-oriented semantic parsers require high latency and extensive resource requirements.
Approach:	They propose a framework that couples a rule-based entity extractor with an iterative grammar-constrained decoder.
Outcome:	The proposed framework achieves an average execution accuracy of over 90% while sustaining at least 2x lower end-to-end latency than existing methods.

Copied to clipboard

Sanja Stajner, Sergiu Nisioi, Ioana Hulpuș

Text Complexity

Challenge:	Traditional text complexity assessment only takes into account lexical and lexiconal complexity.
Approach:	They propose a tool for automatic assessment of conceptual text complexity based on the current state-of-the-art unsupervised approach . they compare the current implementation with the state of the art and discuss the influence of the choice of entity linker on the performance of the tool.
Outcome:	The proposed tool can be personalized and adapted to the needs of struggling readers.

DeepNLPF: A Framework for Integrating Third Party NLP Tools (2020.lrec-1)

Copied to clipboard

Francisco Rodrigues, Rinaldo Lima, William Domingues, Robson Fidalgo, Adrian Chifu, Bernard Espinasse, Sébastien Fournier

Challenge:	Many third-party NLP tools perform distinct NLP subtasks, but integration is difficult . authors present a framework that enables easy integration of third-parties into a pipeline .
Approach:	They propose a framework that enables easy integration of third-party NLP tools . it provides an API for complete pipeline customization including definition of input/output formats .
Outcome:	The proposed framework reduces runtime processing time compared to executing the same pipeline in a sequential manner.

HIPO: A Hierarchical Prompt Optimization Framework with Task Awareness and Fine-Grained Debugging (2026.findings-acl)

Copied to clipboard

Lu Qi, Lei Chai, Hongrui Yu, Binhang Qi, Hailong Sun

PromptWizard: Optimizing Prompts via Task-Aware, Feedback-Driven Self-Evolution (2025.findings-acl)

Challenge:	Existing methods for prompt optimization apply the same prompt across all samples . existing methods ignore variation in sample difficulty .
Approach:	They propose a framework that shifts the paradigm from dataset-level to sample-level optimization.
Outcome:	The proposed framework outperforms baselines on 27 tasks and reduces API calls, token consumption and overall cost by 1.2 to 80.

Copied to clipboard

Eshaan Agarwal, Raghav Magazine, Joykirat Singh, Vivek Dani, Tanuja Ganu, Akshay Nambi

Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks (2023.findings-emnlp)

Challenge:	Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs.
Approach:	They propose a framework for discrete prompt optimization that generates human-readable prompts using feedback-driven critique and synthesis process.
Outcome:	The proposed framework improves prompt quality across 45 tasks and reduces API calls, token usage and overall cost.

Copied to clipboard

Ruixiang Tang, Gord Lueck, Rodolfo Quispe, Huseyin Inan, Janardhan Kulkarni, Xia Hu

GOAT: A Training Framework for Goal-Oriented Agent with Tools (2026.findings-acl)

Challenge:	Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks.
Approach:	They investigate the membership inference attack by using model's API to determine if a sample was part of the training data.
Outcome:	The proposed model is able to identify if a sample was part of the training data and exploits its similarity and resistance to document modifications as potential MI signals on widely used datasets.

Copied to clipboard

Hyunji Min, Sangwon Jung, Junyoung Sung, Dosung Lee, Leekyeung Han, Paul Hongsuck Seo

PyRater: A Python Toolkit for Annotation Analysis (2024.lrec-main)

Challenge:	Recent advances in large language models have led to remarkable progress across a wide range of natural language processing tasks.
Approach:	They propose a training framework that enables fine-tuning LLM agents without human annotation.
Outcome:	The proposed framework enables fine-tuning LLM agents without human annotation.

Copied to clipboard

Angelo Basile, Marc Franco-Salvador, Paolo Rosso

Challenge:	PyRater is an open-source Python toolkit for analysing corpora annotations.
Approach:	They propose to use PyRater to analyse corpora annotations.
Outcome:	The proposed model can be used to identify the best annotations and retrieve the gold standard.

ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering (2026.acl-long)

Copied to clipboard

Yindong Zhang, Wenmian Yang, Yiquan Zhang, Weijia Jia

Natural Language Interfaces

Challenge:	Real estate agents are labor-intensive, difficult to scale, and prone to interest-driven bias.
Approach:	They propose a large-scale benchmark of 29,270 real-estate instances with machine-verifiable supervision for intermediate steps . they propose 'hIRE-Agent' framework that integrates heterogeneous evidence into an understand–plan–execute architecture as a strong baseline .
Outcome:	Experiments show that HIRE-Agent integrates heterogeneous evidence . the framework is able to integrate a front-end parser, planning Supervisor, and execution Specialists .

CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation (2026.findings-acl)

Copied to clipboard

Duy Tung Doan, Quang Huy Phung, Dzung Nguyen, Khac-Hoai Nam Bui

Programming Languages in NLP

Challenge:	Existing multi-agent code generation frameworks are constrained by static planning, isolated execution, high computational overhead, and limited adaptability to complex tasks.
Approach:	They propose a plan-code co-evolution framework that allows dynamic multi-agent collaboration to improve code quality and robustness across tasks.
Outcome:	The proposed framework improves code quality and robustness across tasks while reducing the number of API calls by an average of 4-10 per execution.

Weaver: Interweaving SQL and LLM for Table Reasoning (2025.emnlp-main)

Copied to clipboard

Rohit Khoja, Devanshu Gupta, Yanjie Fu, Dan Roth, Vivek Gupta

EET: Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents (2026.findings-acl)

Challenge:	Existing approaches that combine SQL and LLM rely on rigid workflows . Tables play a critical role across various domains such as finance, healthcare and scientific research .
Approach:	Weaver is a modular pipeline that integrates SQL and LLM for table-based question answering.
Outcome:	Weaver outperforms state-of-the-art methods on four Table QA datasets.

Copied to clipboard

Yaoqi Guo, Ying Xiao, Jie M. Zhang, Mark Harman, Yiling Lou, Yang Liu, Zhenpeng Chen