Papers by Patrícia Schmidtová
Real-World Summarization: When Evaluation Reaches Its Limits (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examines the evaluation of hotel highlights in the context of hotel data. |
| Approach: | They examine evaluation of faithfulness to input data in the context of hotel highlights . they compare traditional metrics, trainable methods, and LLM-as-a-judge approaches . |
| Outcome: | The results show that simple metrics outperform human judgments on LLM-generated summaries . the results also highlight challenges in crowdsourced evaluations. |
How Important is ‘Perfect’ English for Machine Translation Prompts? (2026.findings-eacl)
Copied to clipboard
Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Kathy Hämmerl, Vilém Zouhar
| Challenge: | Large language models (LLMs) are largely trained on and respond best to English prompts, but are also sensitive to errors in user prompts. |
| Approach: | They propose to model a range of error types exhibited by second language English speakers and quantify their impact on LLM performance. |
| Outcome: | The proposed model is brittle to natural spelling errors but not to errors at the phrasal level, but the variance in quality caused by these errors is lower than the variance over the initial prompt choice. |
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs (2024.eacl-long)
Copied to clipboard
| Challenge: | Lack of access to model details has raised concerns about data contamination among researchers. |
| Approach: | They conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. |
| Outcome: | The proposed models have been exposed to 4.7M samples from 263 benchmarks during the first year after their release. |