| Challenge: | Hallucinations in machine translation (MT) outputs are prone to hallucination, authors say . lack of high-quality benchmarks for halluciation detection has hindered MT deployments . |
| Approach: | They propose a dataset that provides annotated hallucination distributions and benchmarks . they use 350,959 span-level annotations across 38 language pairs to analyze hallucis a MT output . |
| Outcome: | The proposed dataset provides high-quality benchmarks for hallucination detection in machine translation . the dataset includes 350,959 span-level annotated samples across 38 language pairs . |
Similar Papers
HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation (2023.emnlp-main)
Copied to clipboard
David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loic Barrault, Marta Costa-jussà
| Challenge: | Previously available quality assessments do not distinguish between hallucinations and omissions. |
| Approach: | They propose to annotate hallucinations and omissions in machine translation using a single language pair. |
| Outcome: | The proposed dataset covers 18 translation directions with varying resource levels and scripts. |
Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models (2024.findings-emnlp)
Copied to clipboard
Kenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs, Joshua Darmon, Pontus Stenetorp, David Adelani, Eduardo Sánchez
| Challenge: | Existing methods for detecting hallucinations in machine translation are limited for low-resource languages. |
| Approach: | They evaluate sentence-level hallucination detection approaches using Large Language Models (LLMs) they find that the choice of model is essential for performance. |
| Outcome: | The proposed models outperform the existing models in HRLs and LRLs on average by 0.16 MCC. |
ANAH: Analytical Annotation of Hallucinations in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | a comprehensive and fine-grained measurement of the hallucination is crucial for LLMs' wide applications. |
| Approach: | They propose a dataset that offers ANalytical Annotation of Hallucinations in Large Language Models. |
| Outcome: | The proposed dataset can be used to train and evaluate hallucination annotators. |
The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive Remediations (2023.emnlp-main)
Copied to clipboard
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | Recent advances in Large Language Models have generated widespread acclaim, but hallucination has also emerged as a by-product. |
| Approach: | They propose a fine-grained discourse on profiling hallucination based on its degree, orientation, and category . they categorize hallucines into six types: acronym ambiguity, generated golem, virtual voice, geographic erratum, time wrap . |
| Outcome: | The proposed method categorizes hallucination into six types based on their degree, orientation, and category . |
HalluLens: LLM Hallucination Benchmark (2025.acl-long)
Copied to clipboard
Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, Pascale Fung
| Challenge: | Large language models (LLMs) generate responses that deviate from user input or training data, a phenomenon known as "hallucination" . |
| Approach: | They propose a hallucination benchmark HalluLens that includes both extrinsic and intrinsic evaluation tasks to distinguish between extrindic and intrinsic hallucines. |
| Outcome: | The proposed framework disentangles LLM hallucination from "factuality" and distinguishes between extrinsic and intrinsic hallucines to promote consistency and facilitate research. |
Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Machine Translation (MT) systems based on fine-tuned large language models (LLMs) are at a higher risk of generating hallucinations, which can severely undermine user’s trust and safety. |
| Approach: | They propose a method that intrinsically learns to mitigate hallucinations during the model training phase. |
| Outcome: | The proposed method reduces hallucinations by 89% on an average across three unseen target languages while preserving translation quality. |
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) are prone to generate hallucinations . a recent study shows that LLMs are able to generate content that conflicts with the source or cannot be verified by factual knowledge. |
| Approach: | They propose a framework to evaluate the performance of large language models (LLMs) they propose to use a sample of generated and human-annotated hallucinated samples to evaluate their performance . |
| Outcome: | The proposed framework generates and annotates hallucinated samples from ChatGPT . the results show that existing LLMs face great challenges in recognizing hallucines . |
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2021.findings-acl)
Copied to clipboard
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, Marjan Ghazvininejad
| Challenge: | Neural sequence models can generate fluent sentences, but they can also hallucinate additional content not supported by the input. |
| Approach: | They propose a task to predict whether each token in the output sequence is hallucinated and collect manually annotated evaluation sets for this task. |
| Outcome: | The proposed method outperforms baseline methods on machine translation and abstractive summarization datasets and achieves significant improvements in both supervised and unsupervised settings. |
When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA (2025.findings-emnlp)
Copied to clipboard
Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, Julia Belikova
| Challenge: | Existing hallucination detection benchmarks operate at the sequence level and are limited to English . Existing methods lacking fine-grained, multilingual supervision are limited in English based on the sequence . |
| Approach: | They propose a large-scale, multilingual dataset annotated with span-level hallucinations across 14 languages. |
| Outcome: | The proposed dataset annotated with span-level hallucinations across 14 languages is scalable and cost-efficient. |
On the Hallucination in Simultaneous Machine Translation (2024.acl-short)
Copied to clipboard
| Challenge: | Currently, there are no studies which systematically analyze hallucination in SiMT. |
| Approach: | They conduct a comprehensive analysis of hallucination in simultaneous machine translation (SiMT) they find that halluciation is extremely severe, especially as latency increases . |
| Outcome: | The results show that it is possible to alleviate hallucination by decreasing the over usage of target-side information for SiMT. |