Papers by Alan Ritter

Extracting a Knowledge Base of COVID-19 Events from Social Media (2022.coling-1)

Challenge:	despite progress toward data-driven conversational agents, dialogue models still suffer from issues surrounding safety and offensive language.
Approach:	They analyze reddit threads and reddits to determine the stance of offensive dialogue models . they find 42% of human responses agree with toxic comments, compared to 13% with safe comments .
Outcome:	The proposed model produces 29% fewer offensive replies than the baseline model.

Copied to clipboard

Shi Zong, Ashutosh Baheti, Wei Xu, Alan Ritter

How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation (2025.emnlp-main)

Challenge:	a flood of COVID-19 related information has appeared on social media since December 2019 . this includes reports on public figures who have tested positive/negative for the virus .
Approach:	They construct a corpus of 10,000 tweets with annotated public reports of five COVID-19 events, using slot-filling questions to fill in slots.
Outcome:	The proposed method can be quickly applied to develop knowledge bases for new domains in response to emerging crises, including natural disasters or future disease outbreaks.

Copied to clipboard

Ruohao Guo, Wei Xu, Alan Ritter

Responsible & Trustworthy NLP

Challenge:	Current studies evaluate LLMs on explicit false statements, overlooking how misinformation manifests subtly as unchallenged premises in real-world interactions.
Approach:	They propose to use EchoMist to analyze implicit misinformation from diverse sources . they also investigate two mitigation methods to enhance LLMs’ capability to counter implicit mis information.
Outcome:	The proposed model fails to detect false premises and generate counterfactual explanations.

Analyzing the Perceived Severity of Cybersecurity Threats Reported on Social Media (N19-1)

Copied to clipboard

Shi Zong, Alan Ritter, Graham Mueller, Evan Wright

Challenge:	6,000 tweets describe software vulnerabilities, which are shared across a range of websites and social media platforms.
Approach:	They propose a method to link software vulnerabilities reported in tweets to CVEs in the National Vulnerability Database (NVD) a Precision@50 of 0.86 is achieved when forecasting high severity vulnerabilities, they show .
Outcome:	The proposed method outperforms baseline methods based on tweet volume and the language used to describe them online.

Code and Named Entity Recognition in StackOverflow (2020.acl-main)

Copied to clipboard

Jeniya Tabassum, Mounica Maddela, Wei Xu, Alan Ritter

Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs (2025.acl-long)

Challenge:	StackOverflow has 15 million programming related questions written by 8.5 million users . however, there is a lack of fundamental NLP resources and techniques for identifying software-related named entities within natural language sentences.
Approach:	They propose a named entity recognition corpus for the computer programming domain with 15,372 sentences annotated with 20 fine-grained entity types.
Outcome:	The proposed model improves on 152 million sentences from StackOverflow and achieves 79.10 F-1 score for code and named entity recognition.

Copied to clipboard

Jungsoo Park, Junmo Kang, Gabriel Stanovsky, Alan Ritter

Reasoning

Challenge:	Recent surveys of literature highlight the overwhelming growth of Large Language Models (LLMs).
Approach:	They propose a semi-automated literature analysis approach that automates literature analysis using LLMs.
Outcome:	The proposed approach reduces paper surveying and data extraction by 93% compared to manual methods.

Measuring Forecasting Skill from Text (2020.acl-main)

Copied to clipboard

Shi Zong, Alan Ritter, Eduard Hovy

Sentiment Analysis

Challenge:	Prior studies have shown that some individuals can make accurate predictions with consistently better accuracy.
Approach:	They examine linguistic factors associated with people's predictions including uncertainty, readability, and emotion.
Outcome:	The proposed model can accurately predict forecasting skill using only language.

Granular Privacy Control for Geolocation with Vision Language Models (2024.emnlp-main)

Copied to clipboard

Ethan Mendes, Yang Chen, James Hays, Sauvik Das, Wei Xu, Alan Ritter

Improved Instruction Ordering in Recipe-Grounded Conversation (2023.acl-long)

Challenge:	Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions.
Approach:	They develop a benchmark to evaluate the ability of VLMs to moderate geolocation dialogues with users.
Outcome:	a new benchmark evaluates the ability of VLMs to moderate geolocation conversations with users.

Copied to clipboard

Duong Le, Ruohao Guo, Wei Xu, Alan Ritter

Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding (2024.acl-long)

Challenge:	In this paper, we explore the task of instructional dialogue and focus on the cooking domain.
Approach:	They propose to explore two auxiliary subtasks to support response generation with improved instruction grounding by incorporating user intent and instruction state information into the model.
Outcome:	The proposed model lacks understanding of user intent and inability to track instruction state (i.e., which step was last instructed) incorporating user intent information helps the response generation model mitigate the incorrect order issue.

Copied to clipboard

Ruohao Guo, Wei Xu, Alan Ritter

Process-Level Representation of Scientific Protocols with Interactive Annotation (2021.eacl-main)

Challenge:	Existing large language models struggle to capture some language styles without fine-tuning.
Approach:	They propose to meta-trained LLMs based on representative lexicons to recognize new styles they have not been fine-tuned on.
Outcome:	The proposed method improves zero-shot transfer across styles on 13 established and 63 novel tasks generated with LLMs.

Copied to clipboard

Ronen Tamari, Fan Bai, Alan Ritter, Gabriel Stanovsky

Structured Data in NLP

Challenge:	Existing efforts to automate wet lab workflows are focusing on graph-prediction models that capture both concrete, exact quantities ("30 minutes") and vague instructions ("swirl")
Approach:	They manually annotate PEGs in a corpus of complex lab protocols with a novel interactive textual simulator that keeps track of entity traits and semantic constraints during annotation.
Outcome:	The proposed graph-prediction models are good at entity identification and local relation extraction while addressing challenges such as cross-sentence relations and long-range coreference.

Self-Specialization: Uncovering Latent Expertise within Large Language Models (2024.findings-acl)

Copied to clipboard

Junmo Kang, Hongyin Luo, Yada Zhu, Jacob Hansen, James Glass, David Cox, Alan Ritter, Rogerio Feris, Leonid Karlinsky

Fluent Response Generation for Conversational Question Answering (2020.acl-main)

Challenge:	Recent studies have demonstrated the effectiveness of self-alignment in which a large language model is aligned to follow general instructions using instructional data generated from the model itself.
Approach:	They propose to use human-written seeds to align large language models to follow general instructions to achieve cross-task generalization.
Outcome:	The proposed model outperforms base models and models that are generally instruction-tuned or have been adapted to the target domain by a large margin.

Copied to clipboard

Ashutosh Baheti, Alan Ritter, Kevin Small

Natural Language Interfaces

Challenge:	Question answering (QA) is an important aspect of open-domain conversational agents, garnering specific research focus in the conversational QA subtask.
Approach:	They propose a method for situating QA responses within a SEQ2SEQ NLG approach to generate fluent grammatical answer responses while maintaining correctness.
Outcome:	The proposed model outperforms baseline CoQA and QuAC models in generating conversational responses.

Pre-train or Annotate? Domain Adaptation with a Constrained Budget (2021.emnlp-main)

Copied to clipboard

Fan Bai, Alan Ritter, Wei Xu

Schema-Driven Information Extraction from Heterogeneous Tables (2024.findings-emnlp)

Challenge:	Recent work shows that pre-training in-domain language models can boost performance when adapting to a new domain.
Approach:	They propose to combine annotation and pre-training to maximize performance under budget constraints.
Outcome:	The proposed approach is based on the annotation cost of three procedural text datasets and pre-training cost of 3 in-domain language models.

Copied to clipboard

Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Mark Dredze, Alan Ritter

Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (D18-1)

Challenge:	Existing work on information extraction from tables has focused on developing custom pipelines for each table collection.
Approach:	They propose a task that transforms tabular data into structured records following a human-authored schema.
Outcome:	The proposed task achieves F1 scores ranging from 74.2 to 96.1 while maintaining cost efficiency.

Copied to clipboard

Ashutosh Baheti, Alan Ritter, Jiwei Li, Bill Dolan

Auditing Language Model Unlearning via Information Decomposition (2026.eacl-long)

Challenge:	Neural conversation models tend to generate safe, generic responses for most inputs . this is due to the limitations of likelihood-based decoding objectives in generation tasks with diverse outputs, such as conversation.
Approach:	They propose a distributional constraint approach that incorporates side information into the generated responses.
Outcome:	The proposed approach generates responses that are less generic without sacrificing plausibility.

Copied to clipboard

Anmol Goel, Alan Ritter, Iryna Gurevych

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? (2023.emnlp-main)

Challenge:	Existing approaches to unlearning in language models do not account for information about forgotten data . despite the apparent success of unlearning, information about the forgotten data remains linearly decodable from internal representations.
Approach:	They propose an interpretable framework for auditing unlearning using Partial Information Decomposition . they propose a representation-based risk score that can guide abstention on sensitive inputs .
Outcome:	The proposed framework can guide abstention on sensitive inputs at inference time.

Copied to clipboard

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, Ming-Wei Chang

Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023? (2023.acl-long)

Challenge:	Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts.
Approach:	They analyze a visual question answering dataset tailored for info-seeking questions . they show that pre-trained visual and language models can use fine-grained knowledge .
Outcome:	The proposed dataset elicits models to use fine-grained knowledge learned during pre-training.

Copied to clipboard

Shuheng Liu, Alan Ritter

Information Extraction & Text Mining

Challenge:	NER models trained on 20-year-old test set may not perform well on modern data.
Approach:	They evaluate the generalization of over 20 different models trained on the CoNLL-2003 dataset . they find no evidence of performance degradation in pre-trained Transformers .
Outcome:	The proposed model generalizations show that some models generalize well on new data while others do not.

Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments (2023.acl-long)

Copied to clipboard

Ethan Mendes, Yang Chen, Wei Xu, Alan Ritter

Responsible & Trustworthy NLP

Challenge:	Existing evaluations of human-in-the-loop systems to combat misinformation are often set up automatically using datasets that were retrospectively constructed.
Approach:	They propose a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them.
Outcome:	The proposed framework is based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments.

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models (2023.acl-long)

Copied to clipboard

Junmo Kang, Wei Xu, Alan Ritter

NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms (2024.acl-long)

Challenge:	Increasing the size of pre-trained models can consistently improve performance on downstream tasks after fine-tuning, as seen in studies based on BERT, RoBERTa, T5 and empirical scaling laws.
Approach:	They propose to use knowledge distillation to build a compact model with a fixed budget instead of annotating data and manually labeling it.
Outcome:	The proposed approach reduces inference costs by reducing costs by hiring annotators and labelling data.

Copied to clipboard

Jonathan Zheng, Alan Ritter, Wei Xu

Robustness in NLP

Challenge:	Prior work on temporal language change observed degradation when finetuning on older text and evaluating on newer data and named entities.
Approach:	They construct a benchmark to evaluate LLMs’ ability to generalize to neologisms with various natural language understanding tasks and model perplexity.
Outcome:	The proposed model performs better in downstream tasks and with later knowledge cutoff dates than models with earlier knowledge cut off dates.

GeoRC: A Benchmark for Geolocation Reasoning Chains (2026.acl-long)

Copied to clipboard

Mohit Talreja, Joshua Diao, Jim James, Radu Casapu, Tejas Santanam, Ethan Mendes, Alan Ritter, Wei Xu, James Hays

SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? (2025.emnlp-main)

Challenge:	Vision Language Models (VLMs) are good at recognizing the global location of a photograph but are startlingly bad at explaining which image evidence led to their location prediction.
Approach:	They propose a benchmark for geolocation reasoning chains based on the global location prediction task in the popular GeoGuessr game.
Outcome:	The proposed benchmark compares LLM-as-a-judge and VLM-As-jumble strategies against human scoring.

Copied to clipboard

Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, Jianfeng Gao

Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning (2025.acl-long)

Challenge:	Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations.
Approach:	They propose to use large language models to simulate users for automatic assistant evaluation.
Outcome:	The proposed model outperforms human evaluations on two interactive tasks and achieves Spearman’s of 0.7 on both tasks.

Copied to clipboard

Mohit Raghavendra, Junmo Kang, Alan Ritter

ChatHF: Collecting Rich Human Feedback from Real-time Conversations (2024.emnlp-demo)

Challenge:	Results show that supervised fine-tuning and preference finetunation are the most efficient approaches for large language models.
Approach:	They propose to use Supervised Finetuning and Preference Finetunes to optimize training data budgets for Large Language Models.
Outcome:	The proposed approach improves performance on math tasks by 15% on the most expensive model, 1,000 examples.

Copied to clipboard

Andrew Li, Zhenduo Wang, Ethan Mendes, Duong Le, Wei Xu, Alan Ritter

Translation and Fusion Improves Cross-lingual Information Extraction (2025.acl-long)

Challenge:	We present an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface.
Approach:	They propose an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface.
Outcome:	The proposed framework supports fine-grained error detection and human evaluation at the same time.

Copied to clipboard

Yang Chen, Vedaant Shah, Alan Ritter

Multilinguality

Challenge:	Large language models (LLMs) have shown significant progress in information extraction tasks due to lack of labeled data for fine-tuning and unlabeled text for pre-training.
Approach:	They propose a framework in which large language models are fine-tuned to use English translations of low-resource language data.
Outcome:	The proposed model improves cross-lingual transfer over the base model on 12 multilingual IE datasets spanning 50 languages.

An Empirical Study of Pre-trained Transformers for Arabic Information Extraction (2020.emnlp-main)

Copied to clipboard

Wuwei Lan, Yang Chen, Wei Xu, Alan Ritter

Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts (2022.findings-emnlp)

Challenge:	Multilingual pre-trained Transformers have been shown to enable effective cross-lingual zero-shot transfer, but their performance on Arabic information extraction tasks is not well studied.
Approach:	They pre-train a bilingual BERT that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning.
Outcome:	The pre-trained model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both the supervised and zero-shot transfer settings.

Copied to clipboard

Nghia T. Le, Fan Bai, Alan Ritter

Low-Resource NLP

Challenge:	In-context learning has emerged as a promising approach to resolve anaphora, but there are challenges in applying it to scientific protocols.
Approach:	They propose a method which combines predictions of hundreds of in-context experts and combines them to yield a 30% increase in F1 over a competitive prompt retrieval baseline.
Outcome:	The proposed method yields 30% increase in F1 score over a competitive prompt retrieval baseline.

SynKB: Semantic Search for Synthetic Procedures (2022.emnlp-demos)

Copied to clipboard

Fan Bai, Alan Ritter, Peter Madrid, Dayne Freitag, John Niekrasz

Information Retrieval

Challenge:	SynKB is an open-source, automatically extracted knowledge base of chemical synthesis protocols.
Approach:	They propose to make SynKB available as an open-source tool for chemists . synKB supports more flexible queries about reaction conditions .
Outcome:	The proposed open-source tool has higher recall and high precision than proprietary chemistry databases.

Model Selection for Cross-lingual Transfer (2021.emnlp-main)

Copied to clipboard

Yang Chen, Alan Ritter

An Annotated Corpus for Machine Reading of Instructions in Wet Lab Protocols (N18-2)

Challenge:	Existing work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices.
Approach:	They propose a machine learning approach that uses the fine-tuned model’s internal representations to predict its cross-lingual capabilities.
Outcome:	The proposed model selects better than English validation data across twenty five languages, including eight low-resource languages, and often achieves comparable results to model selection using target language development data.

Copied to clipboard

Chaitanya Kulkarni, Wei Xu, Alan Ritter, Raghu Machiraju

Semantic Parsing

Challenge:	Existing efforts to annotate natural language instructions into machine-readable formats are limited.
Approach:	They propose to annotate a corpus of natural language instructions consisting of 622 wet lab protocols to facilitate automatic or semi-automatic conversion into a machine-readable format.
Outcome:	The proposed corpus can be used to facilitate automatic or semi-automatic conversion of protocols into a machine-readable format and benefit biological research.

CARE: Multilingual Human Preference Learning for Cultural Awareness (2025.emnlp-main)

Copied to clipboard

Geyang Guo, Tarek Naous, Hiromi Wakaki, Yukiko Nishimura, Yuki Mitsufuji, Alan Ritter, Wei Xu

What are Foundation Models Cooking in the Post-Soviet World? (2025.emnlp-main)

Challenge:	Language Models are tuned with human preferences to produce helpful responses, but the impact of preference tuning on the ability to handle culturally diverse queries remains understudied.
Approach:	They introduce a multilingual resource that contains culturally specific questions and 31.7k responses with human judgments.
Outcome:	The proposed model outperforms models with stronger initial cultural performance . the proposed model has gaps in the literature on culturally relevant data .

Copied to clipboard

Anton Lavrouk, Tarek Naous, Alan Ritter, Wei Xu

Structured Minimally Supervised Learning for Neural Relation Extraction (N19-1)

Challenge:	During the Soviet era, these identities were pressured through forced assimilation under the Russian language and culture.
Approach:	They construct a multi-modal dataset encompassing 1147 and 823 dishes in the Russian and Ukrainian languages, centered around the Post-Soviet region.
Outcome:	The results show that leading models struggle to correctly identify the origins of dishes from Post-Soviet nations in both text-only and multi-modal Question Answering (QA) the weak correlation between this task and QA suggests that QA alone may be insufficient as an evaluation of cultural understanding.

Copied to clipboard

Fan Bai, Alan Ritter

Information Extraction & Text Mining

Challenge:	Recent work shows that distant supervision can cause significant label noise when learning from large quantities of unlabeled text.
Approach:	They propose a method that combines the benefits of learning representations and structured learning to predict sentence-level relation mentions given only proposition-level supervision from a KB.
Outcome:	The proposed approach outperforms a number of baseline approaches while minimizing label noise.

Frustratingly Easy Label Projection for Cross-lingual Transfer (2023.findings-acl)

Copied to clipboard

Yang Chen, Chao Jiang, Alan Ritter, Wei Xu

Multilinguality

Challenge:	Existing approaches to improve cross-lingual transfer performance are based on word alignment, but no empirical studies have evaluated their effectiveness or limitations.
Approach:	They propose a mark-then-translate method that integrates translation and projection by inserting special markers around the labeled spans in the original sentence.
Outcome:	The proposed method outperforms word alignment-based methods in 57 languages and three tasks.

Reducing Privacy Risks in Online Self-Disclosures with Language Models (2024.acl-long)

Copied to clipboard

Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, Wei Xu