Papers by Vilém Zouhar
A Diachronic Perspective on User Trust in AI under Uncertainty (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern NLP systems are rarely calibrated and are often confidently incorrect about their predictions, which violates users’ mental model and erodes their trust. |
| Approach: | They propose to use a mental model to bet on the correctness of an NLP system and to study how trust is rebuilt as a function of time after these events. |
| Outcome: | The proposed model shows that even a few highly inaccurate confidence estimation instances damage users’ trust in the system and performance, which does not easily recover over time. |
Enhancing Textbooks with Visuals from the Web for Improved Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Textbooks lack visuals that support student learning, but many lack them . e-textbooks lack such visuals, and many lack these visuals . |
| Approach: | They propose to use vision-language models to automatically enhance textbooks with images from the web. |
| Outcome: | The proposed model improves textbooks with images from the web while allowing for better pedagogical value. |
A Formal Perspective on Byte-Pair Encoding (2023.findings-acl)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, but the underlying optimization problem that BPE seeks to solve has not yet been laid down. |
| Approach: | They propose an algorithm which is a 1/sigma*(1-e(-sigma))-approximation of an optimal merge sequence. |
| Outcome: | The proposed algorithm improves the runtime complexity from O(NM) to O(N log M) and the lower bound of the approximation is approx0.37. |
AI-Assisted Human Evaluation of Machine Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Annotation metrics are misaligned with the ideal measure of text quality and human evaluation remains the most accurate, reliable, and ultimate standard. |
| Approach: | They propose an annotation protocol that helps annotators mark erroneous parts of the translation and assign a final score. |
| Outcome: | The proposed protocol reduces the time per span annotation by half . the method reduces annotation budget by 25% with filtering of examples that the AI deems to be likely to be correct. |
Early-Exit and Instant Confidence Translation Quality Estimation (2026.eacl-long)
Copied to clipboard
| Challenge: | Quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines. |
| Approach: | They propose an uncertainty-aware quality estimation model that matches previous approaches at a fraction of their costs. |
| Outcome: | The proposed method reduces evaluation costs by 50% and improves reranking performance. |
PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers in Overleaf (2026.acl-demo)
Copied to clipboard
Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner, Xuanqiang Angelo Huang, Vilém Zouhar, Dominik Glandorf, Isabel Dahlgren, Rishit Dagli, Yuen Chen, Felix Leeb, Van Q. Truong, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Mrinmaya Sachan, Bernhard Schölkopf, Mona T. Diab, Zhijing Jin
| Challenge: | Emerging AI-powered writing assistants focus on grammar fixes or simulating peer review with final scores, yet they fall short of providing concrete, actionable suggestions that help students improve their papers during drafting. |
| Approach: | They propose a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments while leaving the actual writing entirely to human authors. |
| Outcome: | The proposed system outperforms a baseline with the skill library and provides actionable suggestions while leaving the actual writing to human authors. |
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024.acl-long)
Copied to clipboard
| Challenge: | a decade ago a single metric, BLEU, governed progress in machine translation research. |
| Approach: | They investigate the "dynamic range" of a number of modern machine translation metrics to provide a collective understanding of differences in scores . they use a large dataset to discover deltas at which metrics achieve system-level differences that are meaningful to humans . |
| Outcome: | The proposed method is more stable than statistical p-values in regards to testset size. |
Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (2026.eacl-long)
Copied to clipboard
Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Miserlis Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Elliott Ash
| Challenge: | Variation in human annotation (i.e., disagreements) is common in NLP, but it is unclear whether it is possible to model this variation in LLMs. |
| Approach: | They evaluate the influence of different reasoning settings on LLM disagreement modeling . RLVR-style reasoning degrades performance in disagreement modeling, they find . |
| Outcome: | The proposed reasoning settings improve LLM disagreement modeling, while RLVR-style reasoning degrades it. |
How to Engage your Readers? Generating Guiding Questions to Promote Active Reading (2024.acl-long)
Copied to clipboard
| Challenge: | Using questions in written text is an effective strategy to enhance readability, but what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied. |
| Approach: | They present a dataset of 10K in-text questions from textbooks and scientific articles and explore various approaches to generate such questions using language models. |
| Outcome: | The generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers’ memorization and comprehension. |
Neural Machine Translation Quality and Post-Editing Performance (2021.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that MT post-editing can reduce translation quality and speed . a large-scale study involving 30 professional translators examined the relationship between MT performance and post-edited outputs. |
| Approach: | They examine the relationship between MT performance and post-editing time and quality . they use neural MT of high quality to improve translation quality based on phrase-based MT . |
| Outcome: | The proposed model is not stable predictor of time or quality, the authors say . they find that better MT systems lead to fewer changes in the sentences . |
CafGa: Customizing Feature Attributions to Explain Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Feature attribution methods, such as SHAP and LIME, quantify the influence of each input component in a model. |
| Approach: | They propose a tool for generating and evaluating feature attribution explanations at customizable granularities. |
| Outcome: | The proposed tool is compared with two baseline methods: PartitionSHAP and MExGen. |
Sampling and Filtering of Neural Machine Translation Distillation Data (2021.naacl-srw)
Copied to clipboard
| Challenge: | In most of neural machine translation distillation or stealing scenarios, the highest-scoring hypothesis of the target model is used to train a new model. |
| Approach: | They propose to use the highest-scoring hypothesis of the target model (teacher) to train a new model (student). |
| Outcome: | The proposed method improves the performance of MT models in English to Czech and with reference translations. |
PWESuite: Phonetic Word Embeddings and Tasks They Facilitate (2024.lrec-main)
Copied to clipboard
Vilém Zouhar, Kalvin Chang, Chenxuan Cui, Nate B. Carlson, Nathaniel Romney Robinson, Mrinmaya Sachan, David R. Mortensen
| Challenge: | Existing word embedding methods overlook phonetic information that is crucial for many tasks. |
| Approach: | They propose three methods that use articulatory features to build phonetically informed word embeddings. |
| Outcome: | The proposed methods improve word retrieval and correlation with sound similarity and on rhyme and cognate detection tasks. |
Estimating Machine Translation Difficulty (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the high-quality outputs, it is difficult to distinguish between state-of-the-art models and identify areas for future improvement. |
| Approach: | They propose a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches. |
| Outcome: | The proposed models outperform both heuristic-based methods and LLM-as-a-judge approaches, with sentinel-src achieving the best performance. |
Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification (2025.emnlp-demos)
Copied to clipboard
Chenfei Xiong, Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Zhijing Jin, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Mennatallah El-Assady, Elliott Ash
| Challenge: | Social scientists often need to develop codebooks that can be reliable but require significant human effort. |
| Approach: | They propose a mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models. |
| Outcome: | The proposed framework integrates human expertise with automatic annotation guided by large language models. |
Outbound Translation User Interface Ptakopět: A Pilot Study (2020.lrec-1)
Copied to clipboard
| Challenge: | a task called outbound translation is not uncommon for Internet users to have to produce a text in a foreign language they have very little knowledge of and are unable to verify the translation quality. |
| Approach: | They propose an open-source modular system to inspect human interaction with machine translation systems enhanced with additional subsystems such as backward translation and quality estimation. |
| Outcome: | The proposed system is able to produce a text in a foreign language with minimal knowledge and is compared with MT systems of mid-range quality. |
A Bayesian Optimization Approach to Machine Translation Reranking (2025.naacl-long)
Copied to clipboard
| Challenge: | reranking is a method of improving prediction quality but can add computational cost. |
| Approach: | They propose to score a list of prediction candidates with an external scoring model and return the highest-scoring candidate. |
| Outcome: | The proposed method achieves the same CometKiwi score using 70 evaluations on average compared to scoring a subset of 180 candidates. |
How Important is ‘Perfect’ English for Machine Translation Prompts? (2026.findings-eacl)
Copied to clipboard
Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Kathy Hämmerl, Vilém Zouhar
| Challenge: | Large language models (LLMs) are largely trained on and respond best to English prompts, but are also sensitive to errors in user prompts. |
| Approach: | They propose to model a range of error types exhibited by second language English speakers and quantify their impact on LLM performance. |
| Outcome: | The proposed model is brittle to natural spelling errors but not to errors at the phrasal level, but the variance in quality caused by these errors is lower than the variance over the initial prompt choice. |
Two Counterexamples to Tokenization and the Noiseless Channel (2024.lrec-main)
Copied to clipboard
| Challenge: | Nevertheless, Rényi efficiency is not perfect and the metric is difficult to evaluate because training multiple tokenizers can be prohibitively expensive and takes days or weeks. |
| Approach: | They propose to use Rényi efficiency as an intrinsic mechanism to evaluate a tokenizer for NLP tasks without the expensive step of training multiple models with different tokenizers. |
| Outcome: | The proposed metric is better correlated to downstream model performance than a percentile frequency metric. |
Biased Tales: Cultural and Topic Bias in Generating Children’s Stories (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalized stories are often preferred because they reflect a child's interests, experiences, and developmental needs. |
| Approach: | They analyze a dataset to examine how biases influence protagonists’ attributes and story elements in LLM-generated stories. |
| Outcome: | The proposed dataset shows that gender stereotypes influence protagonist attributes and story elements in LLM-generated stories. |
Tokenization and the Noiseless Channel (2023.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance. |
| Approach: | They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords . |
| Outcome: | The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task . |
Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains (2024.acl-short)
Copied to clipboard
| Challenge: | a new dataset examines whether fine-tuned metrics are robust to domain shifts between training and inference. |
| Approach: | They use an annotated multidimensional quality metrics dataset to examine whether they are robust to domain shifts between training and inference. |
| Outcome: | The proposed metrics exhibit a substantial performance drop in the unseen domain scenario compared to metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments. |
Backtranslation Feedback Improves User Confidence in MT, Not Quality (2021.naacl-main)
Copied to clipboard
Vilém Zouhar, Michal Novák, Matúš Žilinec, Ondřej Bojar, Mateo Obregón, Robin L. Hill, Frédéric Blain, Marina Fomicheva, Lucia Specia, Lisa Yankovskaya
| Challenge: | Inbound translation is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility. |
| Approach: | They propose to provide cues that indicate the quality of MT output as well as suggest possible rephrasing of the source language. |
| Outcome: | The proposed feedback module increases user confidence in the produced translation, but not the objective quality. |
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern WQE techniques rely on expensive inference with large language models or ad-hoc training with large amounts of human-labeled data. |
| Approach: | They propose to use word-level quality estimation to identify translation errors from the inner workings of translation models to quantify the impact of human label variation on metric performance. |
| Outcome: | The proposed methods identify translation errors from the inner workings of translation models using human labels. |
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023.eacl-main)
Copied to clipboard
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan
| Challenge: | State-of-the-art machine translation quality estimation systems have been achieving remarkable correlations with human judgements yet they require human annotations, which are expensive and computationally heavy. |
| Approach: | They propose a problem where one predicts automated metric scores without the reference. |
| Outcome: | The proposed model can estimate automated metrics at the sentence-level without the reference. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |
Distributional Properties of Subword Regularization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training. |
| Approach: | They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word. |
| Outcome: | The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus. |