BERTSCORE

Evaluating Question Answering Evaluation (D19-58)

Challenge:	Creating a synthetic parallel corpus from noisy data is also difficult due to its noisy nature.
Approach:	They propose a training methodology that fine-tunes the NMT system only using source-side data to balance adequacy and fluency.
Outcome:	The proposed method surpasses the MLE-based fine-tuning approach by achieving a 1.9 BLEU improvement.

Copied to clipboard

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction (2025.acl-industry)

Challenge:	Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks.
Approach:	They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA .
Outcome:	The proposed BERTScore metric fails to provide stronger correlation with human judgements .

Copied to clipboard

Harsh Maheshwari, Srikanth Tenneti, Alwarappan Nakkiran

string2string: A Modern Python Library for String-to-String Algorithms (2024.acl-demos)

Challenge:	Retrieval Augmented Generation (RAG) is a powerful application of Large Language Models (LLMs).
Approach:	They propose to use BERTScore to fine-tune Large Language Models on domain-specific data to improve citation accuracy.
Outcome:	The proposed approach improves citation accuracy by 15.46% with minimal latency and cost.

Copied to clipboard

Mirac Suzgun, Stuart Shieber, Dan Jurafsky

UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning (2021.acl-short)

Challenge:	Notable algorithms include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, and the Wagner-Fischer algorithm for edit distance.
Approach:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.
Outcome:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.

Copied to clipboard

Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, Kyomin Jung

SmartMatch: Real-Time Semantic Retrieval for Translation Memory Systems (2026.eacl-demo)

Challenge:	BERTScore and other text generation metrics do not use reference captions to evaluate image captions.
Approach:	They propose a new metric which does not require reference captions to evaluate image captions . they train UMIC to discriminate negative captions via contrastive learning .
Outcome:	The proposed metric has higher correlation than previous metrics that require multiple references.

Copied to clipboard

Ernesto L. Estevanell-Valladares, Salima Lamsiyah, Alicia Picazo-Izquierdo, Tharindu Ranasinghe, Ruslan Mitkov, Rafael Muñoz

RadEval: A framework for radiology text evaluation (2025.emnlp-demos)

Challenge:	Translation Memory (TM) systems are core components of computer-aided translation tools . however, they fail to retrieve semantically relevant content when surface similarity is low.
Approach:	They propose an open-source demo and evaluation toolkit for TM retrieval that connects modern sentence encoders and strong lexical/fuzzy baselines with a vector database.
Outcome:	The proposed toolkit exposes the end-to-end retrieval pipeline through a web-based UI for qualitative inspection and preference logging.

Copied to clipboard

Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, David W Eyre, Jean-Benoit Delbrouck

VizSeq: a visual analysis toolkit for text generation tasks (D19-3)

Challenge:	Evaluating automated radiology report generation systems remains a fundamental challenge in the development of safe, accurate, and clinically useful medical AI.
Approach:	They propose a unified, open-source framework for evaluating radiology texts that consolidates a diverse range of metrics from classic ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (GREEN).
Outcome:	The framework consolidates a diverse range of metrics from ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLMbased evaluators (GREEN).

Copied to clipboard

Changhan Wang, Anirudh Jain, Danlu Chen, Jiatao Gu

Visual Data in NLP

Challenge:	Several softwares for text evaluation are available that do not provide detailed examples.
Approach:	They propose a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks.
Outcome:	The proposed toolkit covers most common n-gram metrics and latest embedding-based metrics such as BERTScore.

Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words (2022.acl-short)

Copied to clipboard

Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, Dan Jurafsky

Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)

Challenge:	We find that word similarities estimated by cosine over contextual embeddings are understated and trace this effect to training data frequency.
Approach:	They propose to use cosine similarity to estimate word similarities in contextual embeddings to trace this effect to training data frequency.
Outcome:	The proposed model underestimates similarity between frequent and low frequency words even after controlling for polysemy and other factors.

Copied to clipboard

Frederic Thomas Kirstein, Terry Lima Ruas, Bela Gipp

Challenge:	Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors.
Approach:	They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment.
Outcome:	The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines.

Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation (2022.coling-1)

Copied to clipboard

Yihe Wang, Yitong Li, Yasheng Wang, Fei Mi, Pingyi Zhou, Xin Wang, Jin Liu, Xin Jiang, Qun Liu

Dialogue Systems & Conversational Agents

Challenge:	Existing methods for generating open-domain dialogue systems underutilize training data.
Approach:	They propose a retrieval-generation training framework that takes advantage of heterogeneous training data by considering them as "evidence" they use BERTScore retrieval framework which gives better qualities of the training data, they show .
Outcome:	The proposed method performs well on zero-shot experiments and is more robust to real-world data.

Balancing Lexical and Semantic Quality in Abstractive Summarization (2023.acl-short)

Copied to clipboard

Jeewoo Sul, Yong Suk Choi

Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment (2026.acl-short)

Challenge:	Existing methods to reduce exposure bias in sequence-to-sequence models are underexplored.
Approach:	They propose a method to re-rank sequence-to-sequence neural models to reduce exposure bias.
Outcome:	The proposed method achieves an 89.67 BERTScore on the CNN/DailyMail and XSum datasets.

Copied to clipboard

Anuj Kumar, Satyadev Ahlawat, Yamuna Prasad, Virendra Singh

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021.tacl-1)

Challenge:	Evaluating Question Answering systems in low-resource Indic languages remains challenging due to the scarcity of annotated data and the lack of reliable evaluation metrics.
Approach:	They propose a language-based multi-aspect evaluation framework for question answering systems . the framework integrates semantic similarity, factual completeness, numerical accuracy and contextual relevance .
Outcome:	The proposed metric is evaluated across eight Indic-language QA tasks using multiple LLMs . Across all settings, it shows stronger agreement with human evaluation .

Copied to clipboard

Daniel Deutsch, Rotem Dror, Dan Roth

NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023.acl-long)

Challenge:	Existing methods for summarization evaluations that approximate human judgments are lacking for accuracy and reliability.
Approach:	They propose methods for calculating confidence intervals and running hypothesis tests for correlations using bootstrapping and permutation.
Outcome:	The proposed methods show that the confidence intervals are wide, demonstrating high uncertainty in the reliability of automatic metrics.

Copied to clipboard

Iftitahu Nimah, Meng Fang, Vlado Menkovski, Mykola Pechenizkiy

Evaluating the Efficacy of Summarization Evaluation across Languages (2021.findings-acl)

Challenge:	a systematic review of automatic evaluation metrics for Natural Language Generation (NLG) shows that task-agnostic metrics have a weak correlation with human .
Approach:	They propose a framework to assess the effectiveness of automatic metrics in three NLG tasks . they propose task-agnostic and human-aligned metrics to be used for evaluation .
Outcome:	The proposed framework provides access to the evaluation tools for three NLG tasks.

Copied to clipboard

Fajri Koto, Jey Han Lau, Timothy Baldwin

Rewarding Semantic Similarity under Optimized Alignments for AMR-to-Text Generation (2022.acl-short)

Challenge:	Using multilingual summarization evaluation methods is more reliable and interpretable than manual methods.
Approach:	They propose to use multilingual BERT within BERTScore to evaluate summarization evaluation metrics . they use English datasets that are not representative of modern summarizing systems .
Outcome:	The proposed methods perform well across all languages, at a level above that for English.

Copied to clipboard

Lisa Jin, Daniel Gildea

Location-Aware Visual Question Generation with Lightweight Models (2023.emnlp-main)

Challenge:	Automatic evaluation metrics score natural language generation systems based on how well they lexically align to humanannotated references.
Approach:	They propose to replace greedy alignments in BERTScore with optimized ones that replace the n-gram matching BERTAcore metrics with a token embedding to prevent domain mismatch.
Outcome:	The proposed metrics outperform cross-entropy and BLEU reward baselines on AMR-to-text generation.

Copied to clipboard

Nicholas Suwono, Justin Chen, Tun Hung, Ting-Hao Huang, I-Bin Liao, Yung-Hui Li, Lun-Wei Ku, Shao-Hua Sun

Visual Data in NLP

Challenge:	a novel task aims to generate engaging questions from location-aware information . a lightweight model can be used to generate such questions .
Approach:	They propose a task to generate engaging questions from location-aware data . they represent location-based information with surrounding images and a GPS coordinate .
Outcome:	The proposed method outperforms baselines regarding human evaluation and evaluation metrics.

Analyzing the Performance of Large Language Models on Code Summarization (2024.lrec-main)

Copied to clipboard

Rajarshi Haldar, Julia Hockenmaier

Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies (2024.eacl-long)

Challenge:	Large language models perform very well on tasks that involve both natural language and source code.
Approach:	They show that large language models perform very well on tasks that involve both natural language and source code.
Outcome:	The proposed models perform very well on tasks that involve both natural language and source code.

Copied to clipboard

Yassir Fathullah, Puria Radmard, Adian Liusie, Mark Gales

FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2022.acl-long)

Challenge:	Autoregressive decoding is expensive for many sequence-to-sequence tasks, but for some downstream tasks, the actual decoding output is not needed, just attributes of the sequence.
Approach:	They propose non-autoregressive proxy models that can efficiently predict scalar-valued sequence-level attributes from the encodings, avoiding the expensive decoding stage.
Outcome:	The proposed models outperform ensembles in machine translation (MT) and automatic speech recognition (ASR) while being significantly faster.

Copied to clipboard

Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, Michalis Vazirgiannis

Improving Image Captioning Evaluation by Considering Inter References Variance (2020.acl-main)

Challenge:	Existing evaluation metrics are not reliable, but require significant computational resources.
Approach:	They propose a method to learn a fixed, low cost version of any expensive NLG metric while retaining most of its original performance.
Outcome:	The proposed approach retains most of the original performance while running faster and faster.

Copied to clipboard

Yanzhi Yi, Hangyu Deng, Jinglu Hu

ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering (2026.acl-long)

Challenge:	Existing one-to-one metrics penalize mismatches without considering the intrinsic variance between ground truth captions.
Approach:	They propose a one-to-one metric based on BERTScore that could be extended to include new features for image captioning evaluation.
Outcome:	The proposed metric achieves state-of-the-art human judgment correlation while improving performance.

Copied to clipboard

Simon Lupart, Mohammad Aliannejadi, Evangelos Kanoulas

Natural Language Interfaces

Challenge:	Unlike static ‘rewrite, retrieve, and generate’ pipelines, ChatR1 interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through RL.
Approach:	They propose a reasoning framework based on reinforcement learning (RL) for conversational question answering that interleaves search and reasoning across turns and provides turn-level feedback.
Outcome:	The proposed framework outperforms competing models on five CQA datasets, measured by different metrics (F1, BERTScore, and LLM-as-judge).

Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation (2024.findings-naacl)

Copied to clipboard

Yifu Qiu, Varun Embar, Shay Cohen, Benjamin Han

Reasoning

Challenge:	Knowledge-to-text generators often struggle to faithfully generate descriptions for input facts . we propose a decoding-only method to reduce hallucinations .
Approach:	They propose a decoding-only method to generate accurate descriptions for input facts . they use a Natural Language Inference model as the model and replace it with a task-specific HVM .
Outcome:	The proposed method improves faithfulness with minimal impact on quality and in/out-of-distribution evaluations.

SKGSum: Structured Knowledge-Guided Document Summarization (2024.findings-acl)

Copied to clipboard

Qiqi Wang, Ruofan Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, Jiamou Liu, Xianda Zheng, Zijian Huang

Multi-perspective Preference Alignment of LLMs for Programming-Community Question Answering (2025.coling-main)

Challenge:	Existing summarization methods ignore the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections.
Approach:	They propose a method that uses automatically extracted summary points to generate summaries.
Outcome:	The proposed methods improve quality and BERTScore of summaries and broaden the types of documents that can be effectively summarized.

Copied to clipboard

Hongyu Yang, Jiahui Hou, Liyang He, Rui Li

Natural Language Interfaces

Challenge:	Extensive experiments on a high-quality, real-world PCQA dataset validate its accuracy and preference.
Approach:	They propose a multi-perspective preference alignment for programming-community question answering to generate user-centric responses.
Outcome:	Experiments on a high-quality, real-world PCQA dataset validate the proposed model's accuracy and preference.

BERTTune: Fine-Tuning Neural Machine Translation with BERTScore (2021.acl-short)

Copied to clipboard

Inigo Jauregi Unanue, Jacob Parnell, Massimo Piccardi

TempLM: Distilling Language Models into Template-Based Generators (2023.findings-acl)

Challenge:	Neural machine translation models are biased toward limited translation references . BERTScore is a scoring function based on contextual embeddings that overcomes the limitations of n-gram-based metrics.
Approach:	They propose to fine-tune models with a new evaluation metric based on contextual embeddings to overcome the limitations of n-gram-based metrics.
Outcome:	The proposed training objective improves translations that are different from the translations but close in the contextual embedding space.

Copied to clipboard

Tianyi Zhang, Mina Lee, Xiang Lisa Li, Ende Shen, Tatsunori Hashimoto

“Tell me who you are and I tell you how you argue”: Predicting Stances and Arguments for Stakeholder Groups (2024.findings-naacl)

Challenge:	Pretrained language models (PLMs) have greatly improved text generation, but they have also been known to produce unfaithful or inappropriate content.
Approach:	They propose a pretrained language model that is a template-based generator and uses it to generate a text.
Outcome:	The proposed model is more faithful than the original PLM and more fluent than prior template systems.

Copied to clipboard

Philipp Heinisch, Lorik Dumani, Philipp Cimiano, Ralf Schenkel

Reasoning

Challenge:	Argument mining has focused on the identification, extraction, and formalization of arguments.
Approach:	They propose a framework that relies on a recommender-based architecture to predict stances and argumentative main points on societally controversial topics for a given stakeholder.
Outcome:	The proposed framework predicts arguments on a debate topic based on BERTScore and debate.org datasets.

DETECT: Determining Ease and Textual Clarity of German Text Simplifications (2026.eacl-long)

Copied to clipboard

Maria Korobeynikova, Alessia Battisti, Lukas Fischer, Yingqiang Gao

Accuracy is not enough: Evaluating Personalization in Summarizers (2023.findings-emnlp)

Challenge:	Current evaluation of German automatic text simplification relies on general-purpose metrics such as SARI, BLEU, and BERTScore.
Approach:	They propose a German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency.
Outcome:	The proposed metric achieves higher correlations with human judgments than widely used ATS metrics.

Copied to clipboard

Rahul Vansh, Darsh Rank, Sourish Dasgupta, Tanmoy Chakraborty

LatticeGen: Hiding Generated Text in a Lattice for Privacy-Aware Large Language Model Generation on Cloud (2024.findings-naacl)

Challenge:	Existing accuracy measures cannot evaluate the degree of personalization of summarization models.
Approach:	They propose to use a PENS dataset to analyze the degree of personalization of ten different summarization models.
Outcome:	The proposed measure can evaluate the degree of personalization of summarization models using the PENS dataset.

Copied to clipboard

Mengke Zhang, Tianxing He, Tianle Wang, Lu Mi, Niloofar Mireshghallah, Binyi Chen, Hao Wang, Yulia Tsvetkov

Responsible & Trustworthy NLP

Challenge:	Currently, the server controls the generated text, but users can't keep it private . prompted generation is a common interaction paradigm for large language models on cloud .
Approach:	They propose a protocol where the server handles most of the computation while the client controls the sampling operation.
Outcome:	The proposed protocol protects both prompt and generation under strong attacks.

A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021.eacl-main)

Copied to clipboard

Miruna-Adriana Clinciu, Arash Eshghi, Helen Hastie

Explainability & Interpretability in NLP

Challenge:	a lack of transparency is a key issue for robotics and AI.
Approach:	They propose to map existing automatic evaluation methods for natural language generation onto explanations.
Outcome:	The proposed model shows that embedding-based evaluation methods have higher correlations with human ratings than word-overlap metrics.

Paraphrase Generation and Evaluation on Colloquial-Style Sentences (2020.lrec-1)

Copied to clipboard

Eetu Sjöblom, Mathias Creutz, Yves Scherrer

BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation (2022.emnlp-main)

Challenge:	a new study investigates the quality and novelty of generated paraphrases . paraphrase models can be used for information retrieval and data mining .
Approach:	They use state-of-the-art neural machine translation models trained on the Opusparcus corpus to generate paraphrases in six languages.
Outcome:	The proposed model outperforms the existing model on human evaluation in five of the six languages.

Copied to clipboard

Tianxiang Sun, Junliang He, Xipeng Qiu, Xuanjing Huang

SentSim: Crosslingual Semantic Evaluation of Machine Translation (2021.naacl-main)

Challenge:	Currently, pre-trained language model (PLM) based metrics are widely adopted in text generation tasks.
Approach:	They propose to use PLMs to encode stereotypical societal biases in PLM-based metrics . they show that popular metrics exhibit higher social bias than traditional metrics based on 6 attributes .
Outcome:	The proposed method shows that PLM-based metrics exhibit higher social bias than traditional metrics on 6 attributes.

Copied to clipboard

Yurun Song, Junchen Zhao, Lucia Specia

R-GDA: Reflective Guidance Data Augmentation with Multi-Agent Feedback for Domain-Specific Named Entity Recognition (2026.findings-eacl)

Challenge:	Machine translation (MT) is currently evaluated in one of two ways: monolingually or trained crosslingually by building a supervised model to predict quality scores from human-labeled data.
Approach:	They propose an unsupervised model that directly compares the source and machine translated sentence using strong pretrained multilingual word and sentence representations.
Outcome:	The proposed model outperforms glass-box approaches to quality estimation that rely on a supervised model.

Copied to clipboard

Hyeonseok Kang, Hyuk Namgoong, Goun Pyeon, Sangkeun Jung

Responsible & Trustworthy NLP

Challenge:	Named Entity Recognition (NER) tasks require data augmentation due to the scarcity of annotated corpora.
Approach:	They propose a framework that introduces a multi-agent feedback loop to enhance augmentation quality.
Outcome:	The proposed framework improves on SciERC and NCBI-disease datasets and achieves low BERTScore in most cases.

Summarizing Patients’ Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models (2022.coling-1)

Copied to clipboard

Yanjun Gao, Dmitriy Dligach, Timothy Miller, Dongfang Xu, Matthew M. M. Churpek, Majid Afshar

COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)

Challenge:	Problem list summarization requires a model to understand, abstract, and generate clinical documentation.
Approach:	They propose a task that summarises patients' main problems from daily progress notes using input from the provider's progress notes during hospitalization.
Outcome:	The proposed model outperforms two state-of-the-art seq2seq transformer architectures in summarizing patients' main problems from daily progress notes in the medical information mart for Intensive Care (MIMIC)-III.

Copied to clipboard

Mert Inan, Piyush Sharma, Baber Khalid, Radu Soricut, Matthew Stone, Malihe Alikhani

Evaluating the Validity of Word-level Adversarial Attacks with Large Language Models (2024.findings-acl)

Challenge:	Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric.
Approach:	They propose a first discourse-aware learned generation metric for evaluating image descriptions.
Outcome:	The proposed metric predicts human ratings of captions on out-of-domain images.

Copied to clipboard

Huichi Zhou, Zhaoyang Wang, Hongtao Wang, Dongping Chen, Wenhan Mu, Fangyuan Zhang

Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics (2022.findings-acl)

Challenge:	Existing adversarial examples can generate invalid adversarials due to significant changes in semantic meanings compared to their originals.
Approach:	They propose to use a large language model to evaluate adversarial examples by semantic constraints.
Outcome:	The proposed method can generate valid adversarial examples even when they are not equipped with semantic constraints.

Copied to clipboard

Daniel Deutsch, Dan Roth

Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering (2025.coling-main)

Challenge:	Existing QA-based summarization metrics must automatically determine whether the QA model’s prediction is correct or not.
Approach:	They benchmark lexical answer verification methods used by current QA-based metrics and two more sophisticated text comparison methods, BERTScore and LERC.
Outcome:	The proposed methods outperform the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others.

Copied to clipboard

Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, Fei Wu

Reasoning

Challenge:	Hallucination remains a critical challenge in large language models (LLMs) in high-stake domains such as legal question answering.
Approach:	They propose a method to mitigate hallucination in legal question answering by using behavior cloning and a novel Hard Sample-aware Direct Preference Optimization.
Outcome:	The proposed method improves non-hallucinated Statute Rate, Statute Relevance Rate, Legal Claim Truthfulness, and traditional metrics.

Layer or Representation Space: What Makes BERT-based Evaluation Metrics Robust? (2022.coling-1)

Copied to clipboard

Doan Nam Long Vu, Nafise Sadat Moosavi, Steffen Eger

Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? (2022.emnlp-main)

Challenge:	Recent embedding-based evaluation metrics for text generation are based on measuring correlation with human evaluations on standard benchmarks.
Approach:	They examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation.
Outcome:	The embedding-based metrics that have the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases.

Copied to clipboard

Pei Zhang, Baosong Yang, Hao-Ran Wei, Dayiheng Liu, Kai Fan, Luo Si, Jun Xie

Machine Translation

Challenge:	Neural machine translation models are often criticized for failures that happen without competency awareness.
Approach:	They propose a method that extends conventional NMT with a self-estimator to translate a source sentence and estimate its competency.
Outcome:	The proposed method performs on translation tasks intact and on quality estimation tasks better than existing methods.

Identifying Reliable Evaluation Metrics for Scientific Text Revision (2025.acl-long)

Copied to clipboard

Leane Jourdan, Nicolas Hernandez, Florian Boudin, Richard Dufour

Challenge:	Effective revision is a critical step in scientific writing, ensuring clarity, coherence, and adherence to academic standards.
Approach:	They propose to use ROUGE and BERTScore to assess revision quality . they also examine LLM-as-a-judge approaches to assess instruction-following revisions .
Outcome:	The proposed method improves the accuracy of revision tasks with and without a gold reference.

Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks (2023.findings-emnlp)

Copied to clipboard

Yichen Huang, Timothy Baldwin

Abstractive Summarization of Bengali Academic Videos Based on Audio Subtitles (2026.findings-eacl)

Challenge:	BERTScore, BLEURT, and COMET are automatic evaluation metrics that are often underperformed on adversarially-synthesized texts.
Approach:	They examine MT evaluation metric performance on adversarially-synthesized texts . they validate that automatic metrics tend to overpenalize adversarial-degraded translations .
Outcome:	The results show that automatic metrics tend to overpenalize adversarially-degraded translations.

Copied to clipboard

Lamisa Bintee Mizan Deya, Farhatun Shama, Abdul Aziz, Md Kaykobad Reza, Md Shahidul Salim

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems (2020.coling-main)

Challenge:	Existing methods for summarizing educational videos in Bengali are limited due to the rapid growth of educational video content.
Approach:	They propose an end-to-end pipeline for the abstractive summarization of Bengali videos . they fine-tuned the BanglaT5 model on a new benchmark dataset .
Outcome:	The proposed system preprocesses audio and converts speech to text using Google's Speech Recognition API.

Copied to clipboard

Vitou Phy, Yang Zhao, Akiko Aizawa

Natural Language Interfaces

Challenge:	Existing evaluation metrics are not designed to cope with this flexibility.
Approach:	They propose to group the qualities into three groups to obtain a single metric called USL-H.
Outcome:	The proposed metric achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics.

Cross-lingual Contextualized Phrase Retrieval (2024.findings-emnlp)

Copied to clipboard

Huayang Li, Deng Cai, Zhi Qu, Qu Cui, Hidetaka Kamigaito, Lemao Liu, Taro Watanabe

Structuring Radiology Reports: Challenging LLMs with Lightweight Models (2025.emnlp-main)

Challenge:	Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks.
Approach:	They propose a task formulation of dense retrieval, cross-lingual contextualized phrase retrieval . they extract pairs of cross-linguistic phrases using word alignment information .
Outcome:	The proposed task formulation surpasses baselines on the phrase retrieval task and a downstream task, i.e., machine translation, and achieves top-1 accuracy 13 points higher.

Copied to clipboard

Johannes Moll, Louisa Fay, Asfandyar Azhar, Sophie Ostmeier, Sergios Gatidis, Tim C. Lueth, Curtis Langlotz, Jean-Benoit Delbrouck

Responsible & Trustworthy NLP

Challenge:	Radiology reports lack a standardized format, limiting both interpretability and machine learning applications.
Approach:	They propose to use lightweight encoder-decoder models for structuring radiology reports . they compare models with eight open-source LLMs with prompting and in-context learning .
Outcome:	The proposed models outperform eight open-source LLMs on a human-annotated test set.

Large Language Models for Persian-English Idiom Translation (2025.naacl-long)

Copied to clipboard

Sara Rezaeimanesh, Faezeh Hosseini, Yadollah Yaghoobzadeh

LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs (2024.findings-emnlp)

Challenge:	Large language models have shown superior capabilities in translating figurative language compared to neural machine translation systems.
Approach:	They evaluate LLMs, NMTs and their combinations using PersianIdioms datasets . they find that automatic evaluation methods like BLEU and BERTScore are effective .
Outcome:	The proposed model performs better in both directions than other models.

Copied to clipboard

Sihui Yang, Keping Bi, Wanqing Cui, Jiafeng Guo, Xueqi Cheng

Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2021.naacl-main)

Challenge:	Non-factoid (NF) question answering is challenging to evaluate due to diverse potential answers and no objective criterion.
Approach:	They propose a listwise NFQA evaluation approach that uses Large Language Models to rank candidate answers in a descending list of reference answers sorted by descending quality.
Outcome:	The proposed method has higher correlations with human annotations than standard methods.

Copied to clipboard

Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, Dan Jurafsky

RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023.findings-acl)

Challenge:	Existing report generation systems suffer from incomplete and inconsistent generation, despite achieving high performance on natural language metrics such as CIDEr and BLEU.
Approach:	They propose two new rewards that encourage the generation of factually complete and consistent radiology reports by using an existing semantic equivalence metric.
Outcome:	The proposed system significantly improves the F1 score of a clinical information extraction performance on two open radiology report datasets.

Copied to clipboard

Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi

Interactive Post-Editing for Verbosity Controlled Translation (2022.coling-1)

Challenge:	Existing metrics for evaluating the quality of automatically generated questions are expensive and penalise valid questions that may not have high lexical or semantic similarity to the reference questions.
Approach:	They propose a question-answering and span scorer metric based on the answerability of the candidate question given the context.
Outcome:	The proposed metric has higher correlation with human judgment without relying on the reference question.

Copied to clipboard

Prabhakar Gupta, Anil Nelakanti, Grant M. Berry, Abhishek Sharma

Improving Accessibility of SCOTUS Opinions: A Benchmark Study and a New Dataset for Generic Heading Prediction and Specific Heading Generation (2025.coling-main)

Challenge:	Recent machine translation models have shown to excel with aspects of translation quality like adequacy and fluency but these models still suffer notable shortcomings like out-of-domain data, low-resource languages, rare words and longer sentences.
Approach:	They propose to use human-in-loop interactive post-editing models to improve translation quality and rephrase the text with a desired style variation.
Outcome:	The proposed model achieves BERTScore over state-of-the-art machine translation models while maintaining the desired token-level and verbosity preference.

Copied to clipboard

Malek Yaich, Nicolas Hernandez

Revisiting Grammatical Error Correction Evaluation and Beyond (2022.emnlp-main)

Challenge:	SCOTUS opinions are notoriously long and use specialised language, making them laborious to read and understand.
Approach:	They propose generic and specific headings for each section to be trained automatically . they compare the performance of different systems trained for each subtask .
Outcome:	The proposed system can achieve a score of 0.90% in predicting generic headings . the proposed system also achieves similar scores in generating specific headings.

Copied to clipboard

Peiyuan Gong, Xuebo Liu, Heyan Huang, Min Zhang

Syntactic Text Processing

Challenge:	Pretraining-based (PT) evaluation metrics are not effective for training grammatical error correction systems.
Approach:	They propose a pretraining-based GEC evaluation metric which only uses PT-based metrics to score the corrected parts of the system.
Outcome:	The proposed evaluation metric outperforms existing methods on a CoNLL14 evaluation task.

NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization (2025.acl-long)

Copied to clipboard

Hyuntak Kim, Byung-Hak Kim

Information Extraction & Text Mining

Challenge:	Summarizing long-form narratives requires capturing intricate plotlines, character interactions, and thematic coherence over tens of thousands of tokens.
Approach:	They propose a multi-agent LLM framework for narrative summarization that processes long-form text through a structured pipeline without fine-tuning.
Outcome:	The proposed framework achieves 30.0% improvement in BERTScore across books, movies, and TV scripts.

Language Model Augmented Relevance Score (2021.acl-long)

Copied to clipboard

Ruibo Liu, Jason Wei, Soroush Vosoughi

QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)

Challenge:	Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements.
Approach:	They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements.
Outcome:	The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree.

Copied to clipboard

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari

EmailSum: Abstractive Email Thread Summarization (2021.acl-long)

Challenge:	Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments.
Approach:	They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document.
Outcome:	The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions.

Copied to clipboard

Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, Mohit Bansal

PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)

Challenge:	Recent years have brought about interest in the task of summarizing conversation threads.
Approach:	They develop an email thread summarization dataset that contains human-annotated short and long email threads over a wide variety of topics.
Outcome:	The proposed dataset contains human-annotated short (30 words) and long (100 words) summaries of 2,549 email threads over a wide variety of topics.

Copied to clipboard

Zongxia Li, Ishani Mondal, Huy Nghiem, Yijun Liang, Jordan Boyd-Graber

Explainability & Interpretability in NLP

Challenge:	Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs.
Approach:	They propose a rubric for machine QA that is more stable than an exact match and neural methods.
Outcome:	The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community.

Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)

Copied to clipboard

Angelina Parfenova, Jürgen Pfeffer

Don’t Go Far Off: An Empirical Study on Neural Poetry Translation (2021.emnlp-main)

Challenge:	Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability.
Approach:	They propose a framework that combines ensemble methods with code refinement methodology to address these challenges.
Outcome:	The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus.

Copied to clipboard

Tuhin Chakrabarty, Arkadiy Saakyan, Smaranda Muresan

Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)

Challenge:	despite improvements in machine translation quality, automatic poetry translation remains a challenging problem . et al., a study of automatic poetry translators shows that multilingual fine-tuning on poetic data outperforms bilingual fine-timing on non-poetic text .
Approach:	They propose to use poetic parallel corpora for 6 languages to study poetry translation . they find that multilingual fine-tuning on poetic data outperforms bilingual fine-uning .
Outcome:	The proposed model outperforms bilingual and multilingual models on poetic data . the proposed model is based on a parallel dataset of poetry translations for several languages .

Copied to clipboard

Ramya Akula, Ivan Garibay

DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)

Challenge:	Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity.
Approach:	They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods.
Outcome:	The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset.

Copied to clipboard

Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, Mohit Iyyer

Machine Translation

Challenge:	BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics.
Approach:	They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics .
Outcome:	The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena .

Evaluating and Improving Factuality in Multimodal Abstractive Summarization (2022.emnlp-main)

Copied to clipboard

David Wan, Mohit Bansal

On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)

Challenge:	Current factuality metrics do not account for vision modality, thus are not adequate for vision-and-language summarization.
Approach:	They propose a weighted combination of CLIPScore and BERTScore to evaluate factuality for abstractive document summarization.
Outcome:	The proposed metric outperforms existing factuality metrics on four factuity metric-evaluation benchmarks and is robust to human judgments.

Copied to clipboard

Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov

COSMIC: Mutual Information for Task-Agnostic Summarization Evaluation (2024.acl-long)

Challenge:	Existing methods for text generation evaluation metrics are lacking in robustness analysis.
Approach:	They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization .
Outcome:	The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization.

Copied to clipboard

Maxime Darrin, Philippe Formont, Jackie Cheung, Pablo Piantanida

Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)

Challenge:	Existing methods for summarizing text are not well aligned with human judgments.
Approach:	They propose a task-oriented evaluation approach that assesses the quality of summarizers based on their capacity to produce summaries while preserving task outcomes.
Outcome:	The proposed method is able to predict task performance in a variety of contexts and tasks.

Copied to clipboard

Marvin Kaster, Wei Zhao, Steffen Eger

Explainability & Interpretability in NLP

Challenge:	Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored .
Approach:	They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE .
Outcome:	The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show .

HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs (2024.acl-long)

Copied to clipboard

Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, Prathosh Ap

Listening Like Humans: Semantics-Guided Noise-Robust Multimodal Speech Recognition (2026.acl-long)

Challenge:	Existing approaches to answer multi-hop questions are query-agnostic and the extracted facts are ambiguous as they lack context.
Approach:	They propose to use a knowledge graph to extract query-relevant information from unstructured text.
Outcome:	The proposed method achieves performance improvements on two popular datasets.

Copied to clipboard

Yan Fang, Jun Chen, Yian Yao, Shuxin Zhong, Min Sun, Kaishun Wu

Extrinsic Evaluation of Machine Translation Metrics (2023.acl-long)

Challenge:	Severe acoustic degradation results in unreliable ASR outputs . et al., 2024b): critical concerns regarding reliability and fairness of ASR .
Approach:	They propose a multimodal framework that reframes ASR as semantics-guided speech reconstruction.
Outcome:	The proposed framework achieves an average reduction in WER while also attaining 98.71% BERTScore and 96.7% USE over advanced baselines.

Copied to clipboard

Nikita Moghe, Tom Sherborne, Mark Steedman, Alexandra Birch

ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization (2021.emnlp-main)

Challenge:	MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets.
Approach:	They evaluate the segment-level performance of the most widely used MT metrics by correlating them with how useful they are for downstream tasks.
Outcome:	The MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets.

Copied to clipboard

Alireza Salemi, Emad Kebriaei, Ghazal Neisi Minaei, Azadeh Shakery

Information Extraction & Text Mining

Challenge:	Abstractive summarization is one of the areas influenced by pre-trained language models.
Approach:	They propose a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue.
Outcome:	The proposed model outperforms previous models on six Persian summarization tasks . it also outperformed previous models in textual entailment, question paraphrasing, and question answering .

Topic Modeling: Contextual Token Embeddings Are All You Need (2024.findings-emnlp)

Copied to clipboard

Dimo Angelov, Diana Inkpen

Information Extraction & Text Mining

Challenge:	Current neural approaches to topic modeling have not been able to solve all of the problems.
Approach:	They propose a topic modeling approach that uses document contextual token embeddings to find topics and find topic spans within documents.
Outcome:	The proposed model outperforms the current state-of-the-art models on a comprehensive set of topic model evaluation metrics.

Reference-based Metrics Disprove Themselves in Question Generation (2024.findings-emnlp)

Copied to clipboard

Bang Nguyen, Mengxia Yu, Yun Huang, Meng Jiang

UPTON: Preventing Authorship Leakage from Public Text Release via Data Poisoning (2023.findings-emnlp)

Challenge:	Existing metrics for question generation are based on human-written references . however, the results of the metrics on our study disprove the metrics themselves .
Approach:	They propose to use human-written references to evaluate question generation . they propose to combine criteria such as naturalness, answerability, and complexity .
Outcome:	The proposed model is based on multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models.

Copied to clipboard

Ziyao Wang, Thai Le, Dongwon Lee

Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech (2024.findings-acl)

Challenge:	Recent authorship attribution models can reveal the true authorship of unseen texts with high accuracies, with some cases up to 95% accuracy.
Approach:	They propose a solution that weakens authorship features in training samples and makes released texts unlearnable by exploiting black-box data poisoning methods.
Outcome:	The proposed model weakens authorship features in training samples and makes released texts unlearnable.

Copied to clipboard

Neemesh Yadav, Sarah Masud, Vikram Goyal, Md Shad Akhtar, Tanmoy Chakraborty

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)

Challenge:	Existing language models to generate implicit hate explanations are lacking in many fields.
Approach:	They propose to use language models to generate explicit hate posts to make it clear . they find that simpler models incorporating external toxicity signals outperform KG-infused models .
Outcome:	The proposed setup produces more precise explanations than zero-shot GPT-3.5, highlighting the intricate nature of the task.

Copied to clipboard

Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig

MATCHA: Matching Text via Contrastive Semantic Alignment (2026.findings-acl)

Challenge:	NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge .
Approach:	They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code .
Outcome:	The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages.

Copied to clipboard

Siran Li, Ece Sena Etoglu, Carsten Eickhoff, Seyed Ali Bahrainian

BiMol-Diff: A Unified Diffusion Framework for Molecular Generation and Captioning (2026.acl-long)

Challenge:	MATCHA is an automatic metric that rewards semantic agreement with a reference and penalizes contradictions.
Approach:	They introduce a metric that jointly rewards semantic agreement with a reference and penalizes contradictions.
Outcome:	The proposed metric outperforms popular metrics on eight public benchmarks compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks.

Copied to clipboard

Aditya Hemant Shahane, Anuj Kumar Sirohi, Devansh Arora, Nitin Kumar, Prathosh AP, Sandeep Kumar

Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)

Challenge:	Autoregressive models struggle with long-range dependencies, while standard diffusion processes apply uniform corruption across positions.
Approach:	They propose a framework that bridges molecular structures and natural language . it assigns position-dependent corruption based on token recovery difficulty .
Outcome:	The proposed framework improves molecule reconstruction and captioning performance on two datasets.

Copied to clipboard

Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick

Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarization (2026.findings-acl)

Challenge:	Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages.
Approach:	They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries.
Outcome:	The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian.

Copied to clipboard

Xiaoyong Mei, Tingting Zuo, Da Chen, Guangyu Hu, Xiangyu Wen, Chao Duan, Mingyan Zhang, Fudan Zheng

Dialogue Systems & Conversational Agents

Challenge:	Existing methods for multi-role dialogue summarization favor surface-level imitation of references rather than genuine gains in faithfulness or alignment with human preferences.
Approach:	They propose a framework that couples explicit cognitive-style reasoning with reward-based optimization for multi-role dialogue summarization.
Outcome:	The proposed framework matches strong baselines on ROUGE and BERTScore, while in-depth analysis on SAMSum shows clear gains in factual faithfulness and model-based preference alignment.

Agent-as-Judge for Factual Summarization of Long Narratives (2025.emnlp-main)

Copied to clipboard

Yeonseok Jeong, Minsoo Kim, Seung-won Hwang, Byung-Hak Kim

SGCM: Salience-Guided Context Modeling for Question Generation (2024.lrec-main)

Challenge:	Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore . however, these metrics do not adequately capture critical aspects of summarizing quality, such as factual accuracy, especially for long narratives.
Approach:	They propose a framework that evaluates and refines factuality in narrative summarization by leveraging a Character Knowledge Graph extracted from input narrative.
Outcome:	The proposed framework evaluates factuality and provides actionable guidance for refinement.

Copied to clipboard

Chuyao Ding, Yu Hong, Jianmin Yao

Multilingual Definition Modeling (2025.findings-acl)

Challenge:	Identifying relevant sentences to answers is crucial for reasoning the possible questions before generation.
Approach:	They propose a salience-guided approach to enhance Paragraph-level Question Generation by identifying salient sentences that manifest relevance.
Outcome:	The proposed approach achieves Rouge-L, BLEU4, BERTScore, Q-BLUE-3 and F1-scores compared to baseline on FairytaleQA.

Copied to clipboard

Edison Marrese-Taylor, Erica K. Shimomoto, A. Solano, Enrique Reid

QUIDS: Query Intent Description for Exploratory Search via Dual Space Modeling (2025.emnlp-main)

Challenge:	Existing definition modeling tasks are mainly encoder-decoder-based, with no explicit definitions.
Approach:	They propose a multilingual study on definition modeling using monolingual dictionary data for four new languages.
Outcome:	The proposed task is based on monolingual dictionary data for four new languages . results show that multilingual models can perform on-pair with English but cannot leverage potential cross-lingual synergies .

Copied to clipboard

Yumeng Wang, Xiuying Chen, Suzan Verberne

Information Retrieval

Challenge:	Using QUIDS, we generate user-facing query intent descriptions that surface what the search engine likely inferred the query to mean based on post-retrieval evidence.
Approach:	They propose a method that leverages dual-space contrastive learning to isolate intent-relevant information while suppressing irrelevant content.
Outcome:	The proposed method outperforms state-of-the-art methods across ROUGE, BERTScore, and human/LLM evaluations.

Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation (2026.acl-long)

Copied to clipboard

Tathagata Raha, Clement Christophe, Nada Saadi, Hamza A Javed, Marco AF Pimentel, Ronnie Rajan, Praveenkumar Kanithi