Papers by Ali Emami
NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive performance on various benchmarks, yet their ability to engage in deliberate reasoning remains questionable. |
| Approach: | They propose to penalize quick, intuitive "System 1" thinking by combining linguistic isolation with resistance to intuitive shortcuts to assess model's reasoning abilities. |
| Outcome: | The proposed model penalizes quick, intuitive “System 1” thinking, isolating fundamental reasoning skills. |
Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index (2025.coling-main)
Copied to clipboard
| Challenge: | Prompt engineering is a growing subdiscipline of natural language processing . a lack of appropriate consideration for the financial constraints of computationally burdensome methods can limit their adoption and impact. |
| Approach: | They propose a new metric that combines accuracy scores with token consumption to reflect different resource constraints. |
| Outcome: | The economic prompting index (EPI) measures the performance of 6 prompting techniques across 10 widely-used language models and 4 diverse datasets. |
Fine-Tuned LLMs are “Time Capsules” for Tracking Societal Bias Through Books (2025.naacl-long)
Copied to clipboard
| Challenge: | We develop a corpus comprising 593 fictional books across seven decades (1950-2019) to track bias evolution. |
| Approach: | They develop a method to trace and quantify bias evolution using fine-tuned LLMs on fictional books across seven decades to track bias evolution. |
| Outcome: | The proposed method traces and quantifies bias evolution in a corpus of 593 fictional books across seven decades. |
Reasoning Traces Shape Outputs but Models Won’t Say So (2026.acl-long)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) generate explicit reasoning traces before producing answers, offering a window into their decisionmaking. |
| Approach: | They propose a method that injects synthetic reasoning snippets into a model’s reasoning trace and measures whether the model follows the injected reasoning and acknowledges doing so. |
| Outcome: | The proposed method reveals that models refuse to disclose their influence when asked to explain their changed answers. |
The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution (P19-1)
Copied to clipboard
| Challenge: | Existing methods for coreference resolution exploit the number and gender of antecedents or have been handcrafted and do not reflect the diversity of naturally occurring text. |
| Approach: | They propose a trick to improve resolution by antecedent switching to target common-sense understanding and world knowledge. |
| Outcome: | The proposed method achieves state-of-the-art results on the GAP coreference task. |
A Knowledge Hunting Framework for Common Sense Reasoning (D18-1)
Copied to clipboard
| Challenge: | a new system that uses common sense to solve a common sense problem is developed . a winograd schema challenge and a choice of plausible alternatives are popular tests . |
| Approach: | They propose an automatic system that achieves state-of-the-art results on the Winograd Schema Challenge . they use a knowledge hunting module to gather web text for problem resolutions . |
| Outcome: | The proposed system achieves state-of-the-art on the Winograd Schema Challenge . it improves F1 performance on the full WSC by 0.21 over the previous best . |
STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that assess explicit and implicit biases are based on a single scenario . a dataset of 450 offensive progressions contains 2,700 sentences of varying severity . |
| Approach: | They evaluate a dataset of offensive progressions that contain 2,700 sentences . they find that even the best-performing models detect bias inconsistently . |
| Outcome: | The proposed dataset shows that even the best-performing models detect bias inconsistently . aligning models with human judgments on STOP can improve answer rates on sensitive tasks by 191% . |
MirrorStories: Reflecting Diversity through Personalized Narrative Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used to create personalized “mirror stories” that reflect and resonate with individual readers’ identities. |
| Approach: | They propose to use Large Language Models to create personalized “mirror stories” that reflect and resonate with individual readers’ identities. |
| Outcome: | The proposed models outperform generic human-written and LLM-generated narratives on all metrics of engagement and textual diversity while preserving the intended moral. |
The Turing Quest: Can Transformers Make Good NPCs? (2023.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained Transformer-based language models have demonstrated impressive conversational abilities, but their use in real-world applications remains unexplored. |
| Approach: | They propose a pipeline for automatic construction of NPC scripts using Transformer-based believable scripts for a variety of game genres and specifications. |
| Outcome: | The proposed pipeline generates scripts that fool judges in a variety of game genres and contexts, and can be easily compared to human-written scripts. |
WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at answering WSC questions, but their ability to generate such questions remains less explored. |
| Approach: | They propose a tree-of-experts prompting method which enhances the generation of WSC instances by incorporating new 'ambiguous' and 'offensive' categories. |
| Outcome: | The proposed method enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods) it extends the framework by incorporating new ‘ambiguous’ and ‘offensive’ categories, providing a deeper insight into model overconfidence and bias. |
Personality Matters: User Traits Predict LLM Preferences in Multi-Turn Collaborative Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly integrated into everyday workflows . a recent study found that LLMs exhibit distinct personality-like traits that affect user engagement . |
| Approach: | They evaluated 32 LLM users for four collaborative tasks and found significant preferences . they found that rationalists preferred GPT-4, while idealists favored Claude 3.5 . |
| Outcome: | The results show that users with different personality traits prefer certain LLMs over others. |
Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used in high-stakes areas such as healthcare, law, and education. |
| Approach: | They propose a concept of Confidence-Probability Alignment that connects an LLM’s internal confidence to the confidence conveyed in the model’s response when explicitly asked about its certainty. |
| Outcome: | The proposed model shows the strongest confidence-probability alignment across a wide range of tasks. |
Memory Dial: A Training Framework for Controllable Memorization in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to memorization detection are post-hoc . large language models can reproduce training data verbatim, complicating accuracy estimates . |
| Approach: | They propose a training framework that makes memorization an explicit variable. |
| Outcome: | The proposed framework produces models identical in architecture, data, and optimization, but varying in memorization pressure. |
If Only My CGM Could Speak: A Privacy-Preserving Agent for Question Answering over Continuous Glucose Data (2026.findings-acl)
Copied to clipboard
| Challenge: | Current patient platforms only offer static summaries which do not support inquisitive user queries. |
| Approach: | They propose a framework for question answering over personal glucose data that uses large language models to provide a reasoning engine that selects analytical functions. |
| Outcome: | The proposed framework achieves 94% value accuracy on synthetic queries and 88% on ambiguous real-world queries. |
A Generalized Knowledge Hunting Framework for the Winograd Schema Challenge (N18-4)
Copied to clipboard
| Challenge: | a new system that performs well on common-sense reasoning tasks is developed . the Winograd Schema Challenge (WSC) is a popular alternative to the Turing test . |
| Approach: | They propose an automatic system that performs well on two common-sense reasoning tasks. |
| Outcome: | The proposed system improves performance on the Winograd Schema Challenge and COPA by 0.16 over the previous best. |
Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to the creation of large Transformer-based language models (LMs). |
| Approach: | They propose a protocol which distinguishes methods that yield desirable results . they apply this protocol to a popular debiasing method, Self-Debiase, and compare it to one called Instructive Debiaser. |
| Outcome: | The proposed protocol provides essential insights into the generalizability and interpretability of debiasing methods that may otherwise go overlooked. |
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (D19-1)
Copied to clipboard
| Challenge: | a recent study has improved the state-of-the-art on common-sense reasoning benchmarks . a san francisco-based approach to common-ense reasoning is challenging . |
| Approach: | They propose to use common-sense reasoning benchmarks to test machine learning's common-sentence inference task SWAG to test common-mind systems. |
| Outcome: | a new study shows that improved performance on common-sense reasoning benchmarks is genuine . the proposed task is more difficult than the current one, but it is more efficient than the previous one. |
Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | grammatical gender significantly influences image generation in text-to-image models . masculine grammatikal markers increase male representation to 73% on average . feminine grammatological markers increase female representation to 38% . |
| Approach: | They propose a cross-linguistic benchmark examining words where grammatical gender contradicts stereotypical gender associations. |
| Outcome: | The proposed benchmark examines words where grammatical gender contradicts stereotypical gender associations. |
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are prone to hallucination and rely on static, pre-annotated references for evaluation. |
| Approach: | They propose a framework to assess large language models without fixed ground-truth answers by iteratively generating web queries and synthesizing external evidence. |
| Outcome: | The proposed framework achieves substantial to perfect agreement with human evaluations on multiple free-form QA benchmarks. |
Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations (2025.findings-acl)
Copied to clipboard
| Challenge: | addressing gender bias and maintaining logical coherence in machine translation remains challenging, especially when translating between natural gender languages, like English, and genderless languages, such as Persian, Indonesian, and Finnish. |
| Approach: | They propose a dataset to assess translation systems' performance in six low- to mid-resource languages and a translation dataset to examine gender bias and logical coherence. |
| Outcome: | The Translate-with-Care dataset, comprising 3,950 challenging scenarios across six low- to mid-resource languages, reveals a universal struggle in translating genderless content, resulting in gender stereotyping and reasoning errors. |
Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition (2024.acl-srw)
Copied to clipboard
| Challenge: | Trace-of-Thought Prompting allows small neural networks to emulate larger, teacher models with reduced computational demands. |
| Approach: | They propose a framework to distill critical reasoning capabilities from teacher models to student models . they use problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions . |
| Outcome: | a new framework enables small neural networks to emulate the performance of larger, teacher models . it leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions . the proposed framework is available on github.com/trace-of-thought/trac-of_thought-prompting/main . |
Common to Whom? Regional Cultural Commonsense and LLM Bias in India (2026.acl-long)
Copied to clipboard
Sangmitra Madhusudan, Trush Shashank More, Steph Buongiorno, Renata Dividino, Jad Kabbara, Ali Emami
| Challenge: | Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. |
| Approach: | They evaluate eight state-of-the-art LLMs and find two critical gaps . commonsense knowledge is fundamentally long-tailed, with most facts rare in training data . |
| Outcome: | The proposed model achieves only 13.4%–20.9% accuracy on region-specific questions and exhibits geographic bias over-selecting Central and North India as the "default" while under-representing East and West. |
An Analysis of Dataset Overlap on Winograd-Style Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | a large number of test instances overlap considerably with pretraining corpora, a study finds . for a number of years, models struggled to exceed chance-level performance . |
| Approach: | They analyze the effects of varying degrees of overlaps that occur between pretraining corpora and test instances in WSC-style tasks. |
| Outcome: | The WSC-Web dataset is the largest to date and has lower overlaps with current pretraining corpora. |
We Politely Insist: Your LLM Must Learn the Persian Art of Taarof (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the cultural competence of large language models to navigate complex social interactions. |
| Approach: | They propose a benchmark to evaluate LLM understanding of Persian taarof . they evaluate 450 role-play scenarios covering 12 common social interaction topics . their results show significant gaps in cultural competence . |
| Outcome: | The proposed model improves on Persian *taarof*, a social norm in Iranian interactions . the model improve despite culturally specific responses and gender-based asymmetries . |
ADEPT: An Adjective-Dependent Plausibility Task (2021.acl-long)
Copied to clipboard
| Challenge: | ADEPT is a large-scale semantic plausibility task that requires a significant degree of world knowledge and common-sense reasoning. |
| Approach: | They propose a large-scale semantic plausibility task that pairs 16 thousand sentences with slightly modified versions obtained by adding an adjective to a noun. |
| Outcome: | The proposed task is easier for humans (85% accuracy), but more difficult for transformer-based models (71% accuracy). |
Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Representative bias is a tendency of Large Language Models to generate outputs that mirror the experiences of certain identity groups, and affinity bias is an evaluative preference for specific narratives. |
| Approach: | They propose two new metrics to measure representative bias and affinity bias within large language models and present a new set of tasks designed with customized rubrics to detect these biases. |
| Outcome: | The proposed model identifies representative biases in prominent LLMs, with a preference for identities associated with being white, straight, and men. |
Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. |
| Approach: | They propose a framework to isolate models' ability in pronoun disambiguation from other visual processing challenges. |
| Outcome: | The proposed framework isolates the models’ ability in pronoun disambiguation from other visual processing challenges. |
EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models excel at the Winograd Schema Challenge, but struggle with instances that feature minor alterations or rewording. |
| Approach: | They propose an open-source platform that harnesses a human-in-the-loop approach to create a dynamic dataset tailored to such altered WSC instances. |
| Outcome: | The proposed model outperforms existing models in the Winograd Schema Challenge (WSC) a human-in-the-loop approach allows for a dynamic dataset tailored to such altered instances. |
The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can explain quantum mechanics and write sophisticated code, yet fail to parse sentences like "The cat that the mouse feared chased meowed" |
| Approach: | They propose a framework to distinguish structural understanding from semantic pattern matching . they use a set of 9,720 comprehension questions on center-embedded sentences . |
| Outcome: | a new framework shows that models lose performance when they abandon structural analysis for semantic associations. |
DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) tuned for safety often avoid acknowledging demographic differences . current safety alignment forces LLMs to default to identity-blindness even when demographic differences are factually correct or contextually justified. |
| Approach: | They propose a tool to classify whether a correct answer requires recognizing group differences . they use label-conditioned reasoning from a teacher to audit outputs for harm drift cases . |
| Outcome: | The proposed model improves accuracy and safety on eight benchmarks. |