DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a recent study of controversy-handling in large language models (LLMs) has shown that people may become increasingly dependent on such systems for information. |
| Approach: | They propose to construct a controversial questions dataset using a subset of a publicly available dataset. |
| Outcome: | The proposed dataset presents challenges concerning knowledge recency, safety, fairness, and bias. |
Similar Papers
DebateQA: Evaluating Question Answering on Debatable Knowledge (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing QA benchmarks that provide fixed answers to debatable questions are inadequate for evaluating their performance. |
| Approach: | They propose to use a dataset of 2,941 debatable questions to assess their ability to provide comprehensive answers to inherently debatably asked questions. |
| Outcome: | The proposed model performs well on 2,941 debatable questions accompanied by human-annotated partial answers that capture a variety of perspectives. |
Awes, Laws, and Flaws From Today’s LLM Research (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a powerful technology that can follow instructions and output coherent, persuasive text. |
| Approach: | They examine the scientific methodology behind large language model (LLM) research and cross-validate it with arguments at the centre of controversy. |
| Outcome: | The authors cross-validate 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research and find that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. |
LLM Tropes: Revealing Fine-Grained Values and Opinions in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to evaluate latent values and opinions in large language models suffer from three notable shortcomings. |
| Approach: | They propose to analyze 156k LLM responses to 62 propositions of the Political Compass Test (PCT) generated by 6 LLMs using 420 prompt variations. |
| Outcome: | The proposed analysis of 156k LLM responses to the Political Compass Test (PCT) generated by 6 LLMs shows that tropes are recurrent and consistent across prompts. |
Hate Personified: Investigating the role of LLMs in content moderation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Our work provides preliminary guidelines and highlights the nuances of applying Large Language models in culturally sensitive cases. |
| Approach: | They propose to use large language models to help with content moderation to assess how well the needs of diverse groups are reflected in annotated posts. |
| Outcome: | The proposed model is able to leverage community-based flagging efforts and exposure to adversaries. |
Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Rapid medical concept drift can lead LLMs to provide incorrect or outdated advice. |
| Approach: | They propose to evaluate how large language models manage knowledge conflicts in clinical guidelines. |
| Outcome: | The proposed benchmark evaluates how LLMs manage varied knowledge conflicts in clinical guidelines. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
Bias in the Mirror : Are LLMs opinions robust to their own adversarial attacks (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on large language models lacks robustness, highlighting the limitations of such models. |
| Approach: | They propose a novel approach where two LLMs engage in self-debate to persuade a neutral version of the model. |
| Outcome: | The proposed approach examines whether large language models are robust during interactions and whether they are susceptible to reinforcing misinformation or shifting to harmful viewpoints. |
Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on evaluating large language models' ability to handle disagreement cases. |
| Approach: | They evaluate the performance of large language models in detecting offensive language at varying levels of agreement. |
| Outcome: | The proposed model improves detection accuracy and model alignment with human judgment by using disagreement samples in training. |
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are now commonplace in conversation applications, but their misuse for generating harmful responses has raised serious societal concerns. |
| Approach: | They provide a comprehensive overview of recent studies covering attacks, defenses, and evaluations of Large Language Models (LLMs) . |
| Outcome: | The proposed review summarizes three aspects of LLM conversation safety: attacks, defenses, and evaluations. |
Arabic Dataset for LLM Safeguard Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on large language models have focused on English, but the safety of LLMs in Arabic remains under-explored. |
| Approach: | They propose to use Arabic-region-specific questions to evaluate LLMs' safety . they use a dual-perspective evaluation framework to examine differences between LLM responses . |
| Outcome: | The proposed framework assesses the LLM responses from both governmental and opposition viewpoints. |