Papers by Chrysoula Zerva
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset (2022.lrec-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, André F. T. Martins
| Challenge: | Existing datasets for machine translation quality estimation and post-editing have several shortcomings. |
| Approach: | They propose a dataset for machine translation quality estimation and automatic post-editing . they report the performance of baseline systems trained on the MLQE-PE dataset . |
| Outcome: | The proposed dataset contains human labels for up to 10,000 translations per language pair. |
Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors (2023.findings-acl)
Copied to clipboard
George Filandrianos, Edmund Dervakos, Orfeas Menis Mastromichalakis, Chrysoula Zerva, Giorgos Stamou
| Challenge: | Existing explanations for classifiers are counterfactual or contrastive . lack of universal ground truth for counterf actual edits hinders their evaluation . |
| Approach: | They propose a back translation-inspired evaluation methodology that utilises earlier outputs of the explainer as ground truth proxies to investigate the consistency of explainers. |
| Outcome: | The proposed method can provide valuable insights into the behaviour of predictor and explainer models and infer patterns that would otherwise be obscured. |
Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decoding (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle to adequately handle discourse phenomena, such as pronoun resolution and lexical cohesion at the document level. |
| Approach: | They propose to use minimum Bayes risk decoding to extract discourse knowledge from LLMs and propose to apply QAD to enhance the semantic richness of translations. |
| Outcome: | The proposed method outperforms other methods and enhances the semantic richness of translations and aligns them more closely with human preferences. |
”I Never Said That”: A dataset, taxonomy and baselines on response clarity classification (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Equivocation and ambiguity in public speech are well-studied discourse phenomena . a new taxonomy aims to detect and classify response clarity in political interviews . |
| Approach: | They propose a taxonomy that uses Large Language Models and human annotations to detect and classify response clarity in political interviews. |
| Outcome: | The proposed taxonomy combines ChatGPT and human annotations to identify clarity in political questions . it provides a fine-grained taxonomies for evasion techniques related to unclear, ambiguous responses . |
Rejected Dialects: Biases Against African American Language in Reward Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Preference alignment via reward models can introduce new biases, hindering reward models’ fairness and equity. |
| Approach: | They propose a framework for evaluating dialect biases in reward models and conduct a case study on biase . they compare reward models' preferences and behavior on paired White Mainstream English and machine-translated and human-written AAL corpora. |
| Outcome: | The proposed framework evaluates dialect biases in reward models and compares them with paired White Mainstream English (WME) and machine-translated and human-written AAL corpora. |
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate image captions are English-centric, despite improvements in the CLIPScore metric . however, there are no available benchmarks for multilingual captioning evaluation . |
| Approach: | They propose to use machine-translated and machine-repurposed datasets to evaluate CLIPScore variants in multilingual settings. |
| Outcome: | The proposed evaluation strategies are based on machine-translated and human judgements. |
Non-Exchangeable Conformal Language Generation with Nearest Neighbors (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate reliability of generated text are lacking in natural language generation. |
| Approach: | They propose a non-exchangeable conformal prediction method that provides bounds on coverage . they validated their method with k-NN retrieval and show that it produces encouraging results . |
| Outcome: | The proposed method produces encouraging results in machine translation and language modeling tasks. |
Uncertainty-Aware Machine Translation Evaluation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Several neural-based metrics have been proposed to evaluate machine translation quality, but they are trained on noisy, biased and scarce human judgements. |
| Approach: | They propose a method to evaluate machine translation quality using point estimates . they combine COMET framework with Monte Carlo dropout and deep ensembles . |
| Outcome: | The proposed methods perform well across multiple language pairs and with references. |
A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing caption evaluation metrics lack granular assessments for errors within captions . lack of uncertainty quantification can give way to misleading scores, reducing user trust . |
| Approach: | They propose a conformal risk control framework to generate and calibrate CLIPScore distributions . they use a model-agnostic conformal framework to detect erroneous words . |
| Outcome: | The proposed method detects erroneous words while providing formal guarantees aligned with desired risk levels. |
Disentangling Uncertainty in Machine Translation Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. |
| Approach: | They propose to use Monte Carlo dropout and deep ensembles to quantify uncertainty in machine translation and assess their ability to target different sources of aleatoric and epistemic uncertainty. |
| Outcome: | The proposed measures can target different sources of aleatoric and epistemic uncertainty, with a reduction in computational costs. |
Learning Disentangled Representations of Negation and Uncertainty (2022.acl-long)
Copied to clipboard
| Challenge: | Negation and uncertainty modeling are long-standing tasks in natural language processing. |
| Approach: | They propose to disentangle negation, uncertainty, and content using a Variational Autoencoder by supervising latent representations using auxiliary objectives. |
| Outcome: | The proposed model can disentangle negation, uncertainty, and content using a Variational Autoencoder. |