Papers by Rachel Rudinger

45 papers
Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution (2022.naacl-main)

Copied to clipboard

Challenge: a new benchmark evaluates coreference resolution systems' ability to recognize singular personal "they" we find that current systems overwhelmingly choose to resolve "they's" correctly to a singular entity or to 'a group'
Approach: They propose to evaluate coreference resolution systems for singular personal "they" they use WinoNB schemas to evaluate whether they can correctly resolve singular "they".
Outcome: The proposed benchmark evaluates coreference resolution systems for singular personal "they" they show that they are biased toward resolving "they", not "them"
Thinking Like a Skeptic: Defeasible Inference in Natural Language (2020.findings-emnlp)

Copied to clipboard

Challenge: Defeasible inference is a mode of reasoning in which an inference may be weakened or overturned in light of new evidence.
Approach: They propose a dataset for defeasible inference in natural language that includes extensions to existing inference datasets.
Outcome: Defeasible NLI extends existing datasets for defeaasibility inference in natural language . generative models can weaken or strengthen inferences up to 68% of the time, it shows .
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022.naacl-main)

Copied to clipboard

Challenge: Pre-trained language models encode correlations between social groups and traits, like associating the group with the group.
Approach: They adapt the Agency-Belief-Communion (ABC) stereotype model to a language model and introduce the sensitivity test (SeT) to measure stereotypical associations.
Outcome: The proposed framework is used to measure stereotyping of intersectional identities in language models.
Analyzing Stereotypes in Generative Text Inference Tasks (2021.findings-acl)

Copied to clipboard

Challenge: Social psychology studies how social stereotypes are shared as part of cultural knowledge .
Approach: They study how stereotypes manifest when potential targets are situated in neutral contexts . they collect human judgments on the presence of stereotypes in generated inferences based on annotator positionality .
Outcome: The results show that the annotators' positions differ depending on the type of inferences they generate .
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (D18-1)

Copied to clipboard

Challenge: a plethora of new natural language inference datasets has been created in recent years . however, these datasets do not provide clear insight into what type of reasoning or inference a model may be performing.
Approach: They propose to recast 13 existing natural language inference datasets into a common structure.
Outcome: The proposed datasets provide insight into how well a sentence representation captures distinct types of reasoning.
Multiple LLM Agents Debate for Equitable Cultural Alignment (2025.acl-long)

Copied to clipboard

Challenge: Recent efforts focus on single-LLM, single-turn generation approaches, but it can be challenging for any single model to support all cultures equally well.
Approach: They propose to exploit the complementary strengths of multiple LLMs to promote cultural adaptability.
Outcome: The proposed model improves accuracy and cultural group parity over single-LLM models.
Lexicosyntactic Inference in Neural Models (D18-1)

Copied to clipboard

Challenge: lexicosyntactic inferences are triggered by surprising aspects of the syntactical context that a word occurs in.
Approach: They build a factuality judgment dataset for English clause-embedding verbs in various syntactic contexts and use it to probe the behavior of current state-of-the-art neural systems.
Outcome: The proposed model makes systematic errors that are visible through the lens of factuality prediction.
On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models (2024.emnlp-main)

Copied to clipboard

Challenge: We show that models are less likely to predict romantic relationships for same-gender character pairs than different-grace character pairs.
Approach: They perform name-replacement experiments to examine gender biases in large language models . they hypothesize that models mirror heteronormative biase and prejudice against interracial romantic relationships .
Outcome: The results suggest that models may mirror heteronormative biases and prejudice against interracial romantic relationships in human and society.
Gender Bias in Coreference Resolution (N18-2)

Copied to clipboard

Challenge: a study of coreference resolution systems that resolve gender differences in pairs is aimed at examining implicit gender biases.
Approach: They propose a Winograd schema-style set of minimal pair sentences that differ only by gender . they evaluate and confirm systematic gender bias in three publicly-available coreference resolution systems .
Outcome: The proposed system resolves a male and neutral pronoun as coreferent with "The surgeon" but does not resolve the female pronounce.
It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)

Copied to clipboard

Challenge: Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored.
Approach: They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting.
Outcome: The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets.
MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain (2021.acl-short)

Copied to clipboard

Challenge: a large number of crowdworker-constructed datasets have been used to conduct natural language inference (NLI) on unstructured, domainspecific texts such as patient notes, pathology reports, and scientific papers.
Approach: They investigate whether MedNLI contains lexical and syntactic annotation artifacts associated with annotation process that allow hypothesis-only classifiers to achieve better-than-random performance.
Outcome: The proposed model outperforms a majority-class baseline model on a physician-annotated dataset with premises extracted from clinical notes.
On Measuring Social Biases in Sentence Encoders (N19-1)

Copied to clipboard

Challenge: Word embeddings such as word2vec and GloVe exhibit human-like implicit biases based on gender, race, and other social constructs.
Approach: They propose a simple generaliza test to measure bias in word embeddings by comparing two sets of target-concept words to two sets .
Outcome: The proposed test shows that word2vec and word2Ve exhibit human-like implicit biases based on gender, race, and other social constructs.
Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US (2024.emnlp-main)

Copied to clipboard

Challenge: Recent work has highlighted the culturally-contingent nature of commonsense knowledge . a multi-stage process is used to evaluate the commonsence of English LLMs .
Approach: They propose a test set of 525 multiple-choice questions to evaluate commonsense knowledge of English LLMs in Ghana and the u.s. They use existing commonsensible datasets to rewrite them in a multi-stage process.
Outcome: The proposed model improves on the culturally-contingent commonsense knowledge of English LLMs in Ghana and the United States.
Partial-input baselines show that NLI models can ignore context, but they don’t. (2022.naacl-main)

Copied to clipboard

Challenge: Researchers have shown that many datasets contain statistical biases, or "annotation artifacts" that systems leverage to correctly predict entailment.
Approach: They propose to use edited contexts to examine RoBERTa models' sensitivity to edited context to examine their model's sensitivity.
Outcome: The proposed model can learn to condition on context, despite being trained on artifact-ridden datasets.
Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms (2026.findings-acl)

Copied to clipboard

Challenge: Neologisms and emerging slang are central to daily conversation, yet challenging for non-native speakers (NNS) to interpret and use appropriately in cross-cultural communication with native speakers (NS).
Approach: They use AI to learn English neologisms and write messages using the learned word to an NS friend.
Outcome: The proposed model shows that AI Explanation yields the largest gains over no support in NS-rated competence, while contextual appropriateness judgments show indifference across support.
Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility (2026.acl-long)

Copied to clipboard

Challenge: Experiments with LLMs reveal similar patterns of influence on human plausibility judgments of commonsense benchmark answers.
Approach: They find that human plausibility judgments of commonsense benchmark answers are affected by implausibility arguments for or against an answer.
Outcome: The results show that human judges find LLM rationales convincing and that human annotators agree on the most plausible answer when the plausibility gap is wide.
Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering (2024.naacl-long)

Copied to clipboard

Challenge: a question-answering system must address pragmatic inferences to answer usefully, says a new study . human information needs are often inferred from the surface form, but answers must address the pragmatic needs of the question.
Approach: They examine assumptions and implications made when mothers ask questions . they find that incorporating these inferences into QA pipelines produces more complete answers .
Outcome: a study shows that incorporating inferences from questions helps to address harmful beliefs . human needs vary when asking questions, but a complete answer can address them . a QA pipeline can be more effective in addressing these needs, the study finds .
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can’t Answer? (2025.naacl-short)

Copied to clipboard

Challenge: Question answering (QA) is a popular task, but we test both separately . a recent study found that LLMs are less accurate in numerical RQA than RQA .
Approach: We run 16 LLMs on QA and RQA with trivia questions/answers . they find question and answer types that lead to RQA errors and suggest improvements .
Outcome: The results show that LLMs are less accurate in RQA for numerical answers than RQA . RQA errors correlate with question difficulty and inversely correlate with answer frequencies .
What do Large Language Models Learn about Scripts? (2022.starsem-1)

Copied to clipboard

Challenge: Script Knowledge is important for language understanding but expensive to produce manually and difficult to induce from text due to reporting bias.
Approach: They propose a pipeline-based script induction framework which can generate good quality ESDs for unseen scenarios.
Outcome: The proposed framework produces good quality ESDs for unseen scenarios, but manual evaluation shows there is room for improvement.
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers (2026.acl-short)

Copied to clipboard

Challenge: Large language models (LLMs) give reasoning before answering, excelling in multiple-choice question answering (MCQA) . but, some studies find that LLMs sans reasoning fail in MCQA without using the question, i.e., choices-only.
Approach: They propose to use reasoning LLMs to separate problematic data from less problematic strategies by examining reasoning traces.
Outcome: The proposed models perform well in multiple-choice question answering without the question, but they fail to use the question.
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas (2025.acl-long)

Copied to clipboard

Challenge: a recent study shows that LLMs can't tailor outputs to users with uncommon preferences . despite the success of persona inference, we may need debiasing and abstention.
Approach: They propose to use preference data to infer needs and interests of users who prefer either output . they argue that training on preference data augmented with PI boosts personalization .
Outcome: The proposed method can be used to improve personalization with less privacy concerns.
Language Models Predict Empathy Gaps Between Social In-groups and Out-groups (2025.naacl-long)

Copied to clipboard

Challenge: Studies of human psychology have shown that people are more motivated to extend empathy to in-group members than out-group member.
Approach: They propose to use language models to study intergroup empathy gap . they use a short description of an experience to predict emotion intensity .
Outcome: The proposed model exhibited strongest intergroup bias among those tested.
On the Mutual Influence of Gender and Occupation in LLM Representations (2025.acl-long)

Copied to clipboard

Challenge: We examine LLM representations of gender for first names in various occupational contexts to study how occupations and the gender perception of first names influence each other mutually.
Approach: They examine LLM representations of gender for first names in various occupational contexts and examine how occupations and the gender perception of first names influence each other mutually.
Outcome: The representations shift with the occupational context and are influenced by stereotypically feminine or masculine occupations.
FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models (2023.findings-acl)

Copied to clipboard

Challenge: a recent study shows that commonsense knowledge is universally shared by most people . early efforts to schematize commonsensical knowledge as scripts provide examples of unintended biases .
Approach: They propose a set of questions for probing cultural biases and assumptions in commonsense reasoning systems . they test commonsensibleQA-style questions on food-related customs in the u.s.
Outcome: The proposed questions show that they are better at detecting biases in commonsense reasoning systems than on non-US cultures.
Neural Models of Factuality (N18-1)

Copied to clipboard

Challenge: A central function of natural language is to convey information about the properties of events.
Approach: They propose to use a FactBank, UW, and MEANTIME event factuality dataset to build two neural models that outperform previous models.
Outcome: The proposed models outperform previous models on FactBank, UW, and MEANTIME datasets.
Cross-lingual Decompositional Semantic Parsing (D18-1)

Copied to clipboard

Challenge: Renewed interest in semantic analysis has led to a surge of proposed new frameworks . many of these efforts are limited to the analysis of English, but with a number of exceptions e.g., recent efforts in Minimal Recursion Semantics (MRS) and multilingual FrameNet annotation and parsing.
Approach: They propose a cross-lingual decompositional semantic analysis task based on a target language . they propose 'end-to-end' model with an annotating mechanism that supports intra-sentential coreference .
Outcome: The proposed model outperforms baselines by at least 1.75 F1 score on an evaluation dataset.
Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs (2025.acl-long)

Copied to clipboard

Challenge: a misalignment or misunderstanding can disrupt communication, leading to confusion or conflict.
Approach: They study failures of grounding in Ubuntu IRC datasets to identify misalignments . they find disruptions in conversational flow are driven by a divergence in beliefs .
Outcome: The findings show that misalignment in common ground can disrupt communication . the study also shows that miscommunications can lead to confusion or conflict .
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning (2024.findings-emnlp)

Copied to clipboard

Challenge: Many commonsense reasoning questions require a hard selection of a single correct answer . ambiguity and semantic mismatches are common in many MCQs .
Approach: They collect plausibility judgments on 5 000 commonsense reasoning questions . they find that the answer rated most plausible does not match the benchmark gold answers .
Outcome: Experiments with LLMS reveal low accuracy and high variation in performance on the subset . high plausibility rating for the most plausible answer is highlighted in bold .
NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals (2025.naacl-long)

Copied to clipboard

Challenge: Decomposing text into atomic propositions allows for finergrained inspection of text.
Approach: They propose to decompose atomic propositions into atomic sub-problems that models must weigh when solving the overall problem.
Outcome: The proposed method measures the inferential consistency of models and the diversity of examples in benchmark datasets.
No Questions are Stupid, but some are Poorly Posed: Understanding Poorly-Posed Information-Seeking Questions (2025.acl-long)

Copied to clipboard

Challenge: When a question is poorly posed, answerers struggle to converge on dominant interpretations, while models attempt comprehensive coverage by addressing many interpretations simultaneously.
Approach: They propose a computational framework to study poorly-posedness of questions by generating spaces of potential interpretations and computing distributions based on interpretations chosen by answerers in the Reddit question thread.
Outcome: The proposed framework analyzes poorly-posed questions using a set of interpretations chosen by human answerers and large language models.
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)

Copied to clipboard

Challenge: Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing.
Approach: They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing .
Outcome: The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation.
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)

Copied to clipboard

Challenge: Standardized math assessments require expensive human pilot studies to establish the difficulty of test items.
Approach: They propose to use large language models to model difficulty of multiple-choice math questions for real-world students.
Outcome: The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show .
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020.emnlp-main)

Copied to clipboard

Challenge: Pre-trained language models perpetuate biases originating in their training corpus to downstream models.
Approach: They focus on the representations of given names in pre-trained language models and show that name perturbation can have an effect on downstream tasks.
Outcome: The proposed model can be used to model the representation of given names in pre-trained language models on reading comprehension probes where name perturbation changes the model answers.
Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? (2024.acl-short)

Copied to clipboard

Challenge: We study whether large language models exhibit race- and gender-based name discrimination in hiring decisions .
Approach: They propose templatic prompts to LLMs to write an email to a named job applicant informing them of a hiring decision.
Outcome: The proposed model generates an acceptance or rejection email based on the applicant's first name .
Causal Inference of Script Knowledge (2020.emnlp-main)

Copied to clipboard

Challenge: Prior work on script induction relied on correlation between instances of events in corpus . instead, we propose an approach based on causal effects between events .
Approach: They propose to use causal effects to induce scripts from text . they propose to compute a function that matches the intuition of what a script represents .
Outcome: The proposed method matches the intuition of what a script represents, the authors show .
Neural-Davidsonian Semantic Proto-role Labeling (D18-1)

Copied to clipboard

Challenge: Existing models for semantic proto-role labeling are based on a bidirectional LSTM encoding strategy.
Approach: They propose a neural model for semantic proto-role labeling using a bidirectional LSTM encoding strategy that is adapted for the task.
Outcome: The proposed model achieves state-of-the-art in a sentence with a LSTM encoder and a decoder.
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users (2025.emnlp-main)

Copied to clipboard

Challenge: We test alignment methods to ensure LLMs are helpful, but they train or evaluate on what users prefer .
Approach: They test alignment methods to ensure LLMs generate plans that help users . they get 4388 plan executions and 5584 comparisons to measure user preferences .
Outcome: The proposed approach can be applied to the problem of user preferences and helpfulness.
‘Rich Dad, Poor Lad’: How do Large Language Models Contextualize Socioeconomic Factors in College Admission ? (2025.emnlp-main)

Copied to clipboard

Challenge: Large Language Models are increasingly involved in high-stakes domains, yet how they reason about socially sensitive decisions remains underexplored.
Approach: They propose a dual-process audit framework to probe LLMs’ reasoning behaviors in sensitive applications using a synthetic dataset of 30,000 applicant profiles grounded in real-world correlations.
Outcome: The proposed framework exploits a synthetic dataset of 30,000 applicant profiles grounded in real-world correlations to probe LLMs' reasoning behaviors in sensitive applications.
What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions (2023.emnlp-main)

Copied to clipboard

Challenge: Existing systems that generate section-wise summaries of contracts can be tedious due to length and complexity of legalese.
Approach: They propose a task of party-specific extractive summarization for legal contracts . they train a pairwise importance ranker and propose incorporating domain-specific notions of importance .
Outcome: The proposed system generates a party-specific contract summary using a dataset of lease agreements and lease agreements.
SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Models (2023.eacl-main)

Copied to clipboard

Challenge: Existing diagnostic tests for detecting social biases in NLP models only detect stereotypic associations pre-specified by the designer.
Approach: They propose an approach for automatic social bias discovery in social commonsense question-answering by substituting names associated with different demographic groups and generating many distractor answers from a masked language model.
Outcome: The proposed approach uncovers model’s stereotypic associations between demographic groups and an open set of words.
Agent-Specific Deontic Modality Detection in Legal Language (2022.emnlp-main)

Copied to clipboard

Challenge: a contract is a legal document executed by two or more parties.
Approach: They propose a corpus of English contracts annotated with deontic modalities . they benchmark this corpus on agent-specific deontics and trigger span detection .
Outcome: The proposed dataset can detect red flags with high recall.
Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)

Copied to clipboard

Challenge: Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts .
Approach: They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone .
Outcome: The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain.
The Universal Decompositional Semantics Dataset and Decomp Toolkit (2020.lrec-1)

Copied to clipboard

Challenge: Decompositional semantics is a method of crowd-sourcing semantic annotations while retaining high interannotator agreement.
Approach: They present the Universal Decompositional Semantics dataset (v1.0) they propose a decomposition-aligned approach to semantic annotation that uses simple questions to answer .
Outcome: The dataset is bundled with the Decomp toolkit (v0.1) both datasets are publicly available at http://decomp.io.
Nichelle and Nancy: The Influence of Demographic Attributes and Tokenization Length on First Name Biases (2023.acl-short)

Copied to clipboard

Challenge: Prior research has shown that social commonsense reasoning models exhibit biases along dimensions of race, ethnicity, and gender.
Approach: They conduct first name substitution experiments to measure the influence of demographic attributes of a name and name tokenization length on models' disparate behavior.
Outcome: The results show that demographic attributes of a name and name tokenization length are factors that systematically affect the behavior of social commonsense reasoning models.
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks (2026.acl-long)

Copied to clipboard

Challenge: Multiple-choice question answering (MCQ) is standard in NLP, but benchmarks lack rigorous quality control.
Approach: They propose an education-inspired toolkit that uses LLM judges to flag flaws in MCQs . they validate the tool with annotations and run it to audit 12 benchmarks based on 19-rule education rubric .
Outcome: The proposed toolkit flags three common MCQ flaws based on a 19-rule education rubric . contaminated MCqs tend to inflate accuracy, while writing errors lower it and change rankings .

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations