Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that loophole-seeking is frequent and intuitive in children . a large number of models capture the pragmatic understanding required for loopholes, says a researcher . |
| Approach: | a study compares large language models to humans to examine loophole behavior . they found that models struggle to recognize humor in creative exploitation of loopholes . |
| Outcome: | a study compares state-of-the-art models to humans to examine loophole behavior in humans . a large language model can generate loopholes, but only two are capable of generating them . |
Similar Papers
Language Models Identify Ambiguities and Exploit Loopholes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that exploit loopholes identify and reason about ambiguity and conflicting goals, presenting a potential safety risk. |
| Approach: | They propose to study the responses of large language models to loopholes by examining ambiguity and pragmatics in LLMs. |
| Outcome: | The proposed models can identify ambiguities and exploit loopholes to satisfy their given goals as opposed to the goals of the user. |
How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Slang is a commonly used type of informal language that poses a daunting challenge to NLP systems. |
| Approach: | They compare human-attested slang and swiss-generated slurs with machine-generated ones . they find that LLMs have significant knowledge about the creative aspects of sling . |
| Outcome: | The proposed model compares human and machine-generated slang usages to find biases in human perceptions of sling . the results suggest that human-attested slms have significant knowledge about the creative aspects of a language . |
Pun Unintended: LLMs and the Illusion of Humor Understanding (2025.emnlp-main)
Copied to clipboard
Alessandro Zangari, Matteo Marcuzzo, Andrea Albarelli, Mohammad Taher Pilehvar, Jose Camacho-Collados
| Challenge: | Existing models for pun detection lack nuanced grasp typical of human interpretation. |
| Approach: | They analyze existing pun detection benchmarks and human evaluation across recent LLMs to find subtle changes in puns that mislead LLM. |
| Outcome: | The proposed models lack the nuance typical of human interpretation and lack the depth of their analysis to detect puns. |
Investigating Human and LLMs’ Decisions in Unverifiable Environments: A Case Study with GitHub Activity Overview (2026.findings-acl)
Copied to clipboard
| Challenge: | examining the behaviors of Large Language Models as artificial social actors is underexplored, especially in unverifiable scenarios where conventional benchmarking has little to help improve their abilities. |
| Approach: | They propose a method to collect, compare, and reason about human and LLMs' decisions in an unverifiable scenario and use it to examine their behaviors. |
| Outcome: | The proposed method compared human and LLM decisions in an unverifiable scenario on GitHub and found that proprietary LLMs behave more like humans than open-source LLM systems. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
“A good pun is its own reword”: Can Large Language Models Understand Puns? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the understanding of puns in large language models (LLMs) have not explored the use of pun in creative writing and humor creation. |
| Approach: | They propose to use pun recognition, explanation and generation tasks to evaluate the capabilities of large language models (LLMs) they adopt automated evaluation metrics from prior research and introduce new evaluation methods and metrics that align more closely with human cognition. |
| Outcome: | The proposed methods align more closely with human cognition than previous evaluation metrics. |
Can Large Language Models Be an Alternative to Human Evaluations? (2023.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is indispensable for assessing the quality of texts generated by machine learning models or written by humans. |
| Approach: | They propose to use large language models to evaluate unseen texts using the same instructions and samples . they also use LLMs to generate responses to questions that are used to conduct human evaluation . |
| Outcome: | The proposed model can be used to evaluate texts in open-ended story generation and adversarial attacks. |
“What do you call a dog that is incontrovertibly true? Dogma”: Testing LLM Generalization through Humor (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong performance in NLP tasks like text summarization and question answering. |
| Approach: | They propose a new humor-based question-answering benchmark to assess LLMs’ reasoning through carefully crafted puns. |
| Outcome: | Experiments on pun comprehension, resolution, and generation reveal that most LLMs struggle with generalization, even on simple tasks, consistently underperforming the human baseline. |
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation approaches for large language models (LLMs) rely on existing tasks and benchmarks, raising concerns about test set contamination and the genuine comprehension abilities of LLMs. |
| Approach: | They propose to evaluate LLMs by designing new tasks, automatically generating evaluation datasets for the tasks, and conducting detailed error analyses to scrutinize LLM's adaptability to new tasks. |
| Outcome: | The proposed method examines LLMs’ adaptability to new tasks, their sensitivity to prompt variations, and their error tendencies. |
Awes, Laws, and Flaws From Today’s LLM Research (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a powerful technology that can follow instructions and output coherent, persuasive text. |
| Approach: | They examine the scientific methodology behind large language model (LLM) research and cross-validate it with arguments at the centre of controversy. |
| Outcome: | The authors cross-validate 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research and find that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. |