Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior research on AI mistrust focused primarily on AI's bias towards different human pop-ups. |
| Approach: | They examine how bias shapes the perception of AI versus human generated content . they found that raters favored content labeled "Human Generated" even when labels were deliberately swapped . |
| Outcome: | The findings highlight the limitations of human judgment in interacting with AI and offer a foundation for improving human-AI collaboration. |
Similar Papers
Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI (2026.acl-long)
Copied to clipboard
Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyan Su, Mervat Abassy, Saadeldine Eletter, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
| Challenge: | Prior studies have shown that distinguishing text generated by Large Language Models from human-written text is challenging for humans and often no better than random guessing. |
| Approach: | They conduct extensive case study to determine the upper bound of human detection accuracy. |
| Outcome: | The findings challenge previous conclusions on human detection accuracy across languages and domains. |
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021.acl-long)
Copied to clipboard
| Challenge: | evaluators distinguish between human- and machine-authored text in three domains without training . evals' accuracy improved up to 55%, but it did not significantly improve across the three domain. |
| Approach: | They examine the role untrained human evaluations play in NLG evaluation and propose ways to improve their evaluations. |
| Outcome: | The evaluators distinguished between human- and machine-authored text at random chance level without training, but their accuracy did not improve across the three domains. |
Identifying Bias in Machine-generated Text Detection (2026.acl-long)
Copied to clipboard
| Challenge: | a growing number of generative AI systems are detecting text generated by a model or written by . humans perform poorly at the detection task, but show no significant biases on the studied attributes. |
| Approach: | They examine gender, race/ethnicity, English-language learner status, and economic status . they find several models tend to classify disadvantaged groups as machine-generated . |
| Outcome: | The proposed models show strong performance but can cause negative impacts . the models classify disadvantaged groups as machine-generated, while economically disadvantaged students' essays are less likely to be classified as machine generated . |
Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs. |
| Approach: | They propose a framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia's** on LLM and human judges. |
| Outcome: | The proposed framework investigates **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia' on LLM and human judges. |
Automatic Detection of Generated Text is Easiest when Humans are Fooled (2020.acl-main)
Copied to clipboard
| Challenge: | Recent advances in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. |
| Approach: | They compare decoding methods with popular sampling-based decoding strategies . they show that multi-sentence excerpts can fool expert human raters over 30% of the time . |
| Outcome: | The proposed methods improve with longer excerpt length, but multi-sentence excerpts fool human raters over 30% of the time. |
Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can generate highly persuasive text, raising concerns about misuse for propaganda, manipulation, and other harmful purposes. |
| Approach: | They propose a multilingual benchmark to compare LLM-generated persuasive texts with human-written ones. |
| Outcome: | The proposed benchmark compares human-authored and LLM-generated persuasive texts . it finds that overtly persuasive LLMs are easier to detect than human-written ones . |
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as automated evaluators . et al., 2024: strong labels can foster trust but also undermine it . |
| Approach: | They show that LLMs' source labels bias trust judgments by humans . they use eye-tracking data to analyze LLM internal states during judgment . |
| Outcome: | The proposed model is biased by disclosed source labels, the authors show . eye-tracking data show humans rely heavily on source labels for judgments . |
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation are inadequate . distinguishing machine-generated text is challenging even for human evaluators . |
| Approach: | They compare human-based evaluators with automated evaluation procedures . they find human evaluers do not correlate well with discriminative evalators . |
| Outcome: | The proposed evaluation methods are compared with a dozen state-of-the-art generators for online product reviews. |
How Does the Disclosure of AI Assistance Affect the Perceptions of Writing? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in generative AI technologies like large language models have boosted the incorporation of AI assistance in writing workflows. |
| Approach: | They conduct an experimental study to determine whether disclosure of AI assistance in the writing process would affect people's evaluation on the quality of the writing and ranking of different writings. |
| Outcome: | The disclosure of AI assistance decreases the average quality ratings for argumentative essays and creative stories, and increases the quality of the writings. |
Limitations of Human Identification of Automatically Generated Text (2024.lrec-main)
Copied to clipboard
Nadège Alavoine, Maximin Coavoux, Emmanuelle Esperança-Rodier, Romane Gallienne, Carlos Gonzalez Gallardo, Jérôme Goulian, Jose G. Moreno, Aurélie Névéol, Didier Schwab, Vincent Segonne, Johanna Simoens
| Challenge: | Neural text generation tools such as ChatGPT are gaining popularity . human annotations are considered gold standard labels for multiple tasks . |
| Approach: | They propose a new corpus in French and English for recognising automatically generated texts . they propose 'incontext' setup which makes explicit the interaction between two parties . |
| Outcome: | The proposed model generates fluent text, which requires much closer reading than the current model. |