Papers by Chris Callison-Burch

I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons (2023.acl-long)

Challenge:	Existing datasets for learning translations of words are limited to a few high-resource languages and unrealistically easy settings.
Approach:	They propose a large-scale multilingual corpus of images labeled with the word they represent to facilitate translation research.
Outcome:	The proposed method improves on an unsupervised technique that has been limited to a few languages and unrealistic settings.

Copied to clipboard

Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris Callison-Burch, Yejin Choi, Prithviraj Ammanabrolu

Linguistics & Cognitive NLP

Challenge:	Existing dialogue agents, while able to produce human-like responses, often do not model goal-driven and grounded language interactions.
Approach:	They propose to decompose and model teacher-student natural language interactions into (1) the DM’s intent to guide players toward a given goal; (2) the dm’s guidance utterance to the players expressing this intent; (3) a theory-of-mind model that anticipates the players’ reaction to the guidance one turn into the future.
Outcome:	The proposed task is based on a goal-driven and grounded environment with a teacher-student interaction model and theory-of-mind model.

MiRAGeNews: Multimodal Realistic AI-Generated News Detection (2024.findings-emnlp)

Copied to clipboard

Runsheng Huang, Liam Dugan, Yue Yang, Chris Callison-Burch

Simplification Using Paraphrases and Context-Based Lexical Substitution (N18-1)

Challenge:	inflammatory “fake” news content is increasingly common, but it is also difficult to detect by humans.
Approach:	They propose a dataset of 12,500 high-quality real and AI-generated image-caption pairs from state-of-the-art generators to combat the spread of fake news.
Outcome:	The proposed dataset improves on image-caption pairs from out-of-domain image generators and news publishers.

Copied to clipboard

Reno Kriz, Eleni Miltsakaki, Marianna Apidianaki, Chris Callison-Burch

Paraphrasing

Challenge:	Lexical simplification involves identifying complex words or phrases that need to be simplified and suggesting simpler meaning-preserving substitutes.
Approach:	They propose a complex word identification model that exploits both lexical and contextual features and a word-embedding lexical substitution model to replace the detected complex words with simpler paraphrases.
Outcome:	The proposed model detects complex words with higher accuracy than other models and proposes good substitutes in context.

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims (2026.acl-long)

Copied to clipboard

Delip Rao, Weiqiu You, Eric Wong, Chris Callison-Burch

Information Extraction & Text Mining

Challenge:	NSF-SciFy contains 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines.
Approach:	They propose to use a dataset to extract scientific claims from National Science Foundation award abstracts and to use it to refine language models.
Outcome:	The proposed method improves non-technical abstract generation, claim extraction, and investigation proposal extraction tasks while maintaining high precision and lower recall.

BiSECT: Learning to Split and Rephrase Sentences with Bitexts (2021.emnlp-main)

Copied to clipboard

Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, Chris Callison-Burch

OpenPI2.0: An Improved Dataset for Entity Tracking in Texts (2024.eacl-long)

Challenge:	Several past efforts have created Split and Rephrase training sets, which consist of long, complex input sentences paired with multiple shorter sentences that preserve the meaning of the input sentence.
Approach:	They propose a new dataset and a model for this task by extracting 1-2 sentence alignments from bilingual parallel corpora and using machine translation to convert both sides of the corpus into the same language.
Outcome:	The proposed model can perform a wider variety of split operations and improve upon previous state-of-the-art approaches in automatic and human evaluations.

Copied to clipboard

Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, Niket Tandon

Challenge:	Recent work shows that explicit modeling entity states benefits LMs in procedural tasks.
Approach:	They propose a dataset where entities and attributes are fully canonicalized and additional entity salience annotations are added.
Outcome:	The proposed dataset outperforms existing models on question answering and classical planning tasks.

Human-in-the-loop Schema Induction (2023.acl-demo)

Copied to clipboard

Tianyi Zhang, Isaac Tham, Zhaoyi Hou, Jiaxuan Ren, Leon Zhou, Hainiu Xu, Li Zhang, Lara Martin, Rotem Dror, Sha Li, Heng Ji, Martha Palmer, Susan Windisch Brown, Reece Suchocki, Chris Callison-Burch

Structured Data in NLP

Challenge:	Existing approaches to event-centric natural language understanding (NLU) have been limited to linear and temporal ones.
Approach:	They propose a human-in-the-loop schema induction system powered by GPT-3 . they show that it transfers to new domains more easily than previous approaches .
Outcome:	The proposed system transfers to new domains more easily than previous approaches and reduces human curation.

Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases (2022.naacl-main)

Copied to clipboard

Qing Lyu, Zheng Hua, Daoxin Li, Li Zhang, Marianna Apidianaki, Chris Callison-Burch

Challenge:	Recursive noun phrases have interesting semantic properties, yet it is unknown whether language models have such knowledge.
Approach:	They propose a dataset of three textual inference tasks targeting recursive noun phrases . they show that such knowledge is learnable with appropriate data .
Outcome:	The proposed model achieves strong zero-shot performance on an extrinsic Harm Detection task.

Did that happen? Predicting Social Media Posts that are Indicative of what happened in a scene: A case study of a TV show (2022.lrec-1)

Copied to clipboard

Anietie Andy, Reno Kriz, Sharath Chandra Guntuku, Derry Tanti Wijaya, Chris Callison-Burch

Sentiment Analysis

Challenge:	Prior work identified and summarized scenes associated with a TV show by selecting a few representative social media posts (5 posts) that were published during the timeline of the scenes.
Approach:	They propose a method to predict social media posts associated with a TV show from those that are not-indicative.
Outcome:	The proposed method can predict posts indicative of what happened in a scene from those that are not-indicative based on high AUC's on social media posts associated with a popular TV show .

Visual Goal-Step Inference using wikiHow (2021.emnlp-main)

Copied to clipboard

Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, Chris Callison-Burch

Cultural and Geographical Influences on Image Translatability of Words across Languages (2021.naacl-main)

Challenge:	Past work in NLP examined the task of goal-step inference for textual goals . wikiHow dataset shows that goal-step inference is challenging for state-of-the-art models .
Approach:	They propose a task where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal.
Outcome:	The proposed task is challenging for state-of-the-art multimodal models and can be transferred to other datasets.

Copied to clipboard

Nikzad Khani, Isidora Tourni, Mohammad Sadegh Rasooli, Chris Callison-Burch, Derry Tanti Wijaya

Machine Translation

Challenge:	Neural machine translation models produce poor translations when there are few/no parallel sentences to train the models.
Approach:	They define image translatability as the translability of words as images associated with words in different languages that have a high degree of visual similarity.
Outcome:	The proposed model improves upon text-only models only marginally.

CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding (2023.findings-acl)

Copied to clipboard

Yijiang Dong, Lara Martin, Chris Callison-Burch

Intent Detection with WikiHow (2020.aacl-main)

Challenge:	Story generation and understanding has seen a surge in neurosymbolic work . symbolic methods are expensive and require a lot of time and expertise .
Approach:	They use Code-LLMs to bootstrap the use of symbolic methods for story understanding . they show that they can beat current LLM techniques on pre-existing stories with minimal hand engineering .
Outcome:	The proposed system beats state-of-the-art structured LLM techniques on pre-existing story understanding tasks with minimal hand engineering.

Copied to clipboard

Li Zhang, Qing Lyu, Chris Callison-Burch

Sentiment Analysis

Challenge:	Existing approaches to intent detection have limited data annotated for new domains or languages.
Approach:	They propose to train a set of pretraining intent detection models on wikiHow which can predict a broad range of intended goals from many actions.
Outcome:	The proposed models achieve state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets.

RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System (2021.naacl-demos)

Copied to clipboard

Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Heng Ji

Visual Data in NLP

Challenge:	We present a new information extraction system that can construct temporal event graphs from news documents.
Approach:	They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction .
Outcome:	The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities.

RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)

Copied to clipboard

Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji

Information Extraction & Text Mining

Challenge:	Existing methods for event prediction are incomplete and noisy.
Approach:	They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework .
Outcome:	The proposed framework can be applied to a wide variety of newsworthy scenarios.

ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems (2024.emnlp-demo)

Copied to clipboard

Andrew Zhu, Liam Dugan, Chris Callison-Burch

Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification (N19-1)

Challenge:	Recent studies show that large language models can be used to construct complex multi-agent systems.
Approach:	They propose a toolkit for recursive multi-agent systems that supports custom tool-use, delegation schemes, event-based logging, and interactive replay.
Outcome:	The proposed tool achieves significant performance gains on agentic benchmarks and identify potential areas of improvement through visualization and debugging tools.

Copied to clipboard

Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, Chris Callison-Burch

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows (2024.acl-long)

Challenge:	Recent research has applied sequence-to-sequence (Seq2Sequen) models to text simplification . generic models tend to copy directly from the original sentence, resulting in outputs that are long and complex.
Approach:	They propose to incorporate word complexities into the loss function during training and generate a large set of diverse candidate simplifications at test time.
Outcome:	The proposed model can perform competitively with state-of-the-art systems while generating simpler sentences.

Copied to clipboard

Ajay Patel, Colin Raffel, Chris Callison-Burch

Responsible & Trustworthy NLP

Challenge:	Large language models (LLMs) have become a dominant tool for NLP researchers in a wide range of tasks.
Approach:	They propose an open source Python library that allows researchers to write simple code to implement powerful LLM workflows.
Outcome:	The proposed library is open source and can be used to implement powerful LLM workflows.

ChatEval: A Tool for Chatbot Evaluation (N19-4)

Copied to clipboard

João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, Chris Callison-Burch

Natural Language Interfaces

Challenge:	open-domain dialog systems are difficult to evaluate due to lack of standardization and standardization in evaluation procedures.
Approach:	They propose a framework for human evaluation of chatbots that augments existing tools . researchers can submit their trained models to the ChatEval web interface . reproducibility and model assessment for opendomain dialog systems is challenging .
Outcome:	The proposed framework provides a web-based hub for researchers to compare their models with baselines and prior work.

PerspectroScope: A Window to the World of Diverse Perspectives (P19-3)

Copied to clipboard

Sihao Chen, Daniel Khashabi, Chris Callison-Burch, Dan Roth

Textual Inference

Challenge:	PerspectroScope is a web-based system that lets users query a discussion-worthy natural language claim .
Approach:	They propose a web-based system which lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim.
Outcome:	The proposed system lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim.

“Wikily” Supervised Neural Translation Tailored to Cross-Lingual Tasks (2021.emnlp-main)

Copied to clipboard

Mohammad Sadegh Rasooli, Chris Callison-Burch, Derry Tanti Wijaya

Multilinguality

Challenge:	Unsupervised neural machine translation models perform well in low-resource or distant languages.
Approach:	They propose a model that leverages Wikipedia for machine translation and cross-lingual tasks without supervision from external parallel data or supervised models in target language.
Outcome:	The proposed model outperforms supervised models in Arabic and English translation tasks.

The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank (2022.findings-naacl)

Copied to clipboard

Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch

Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction (2022.findings-emnlp)

Challenge:	a natural language generation system can be used to create text at the end of a passage . fill in the blank (FITB) is a task of inserting text into a specified position in a text .
Approach:	They evaluate the feasibility of using a single model to perform both tasks . they show that models pre-trained with a FitB-style objective are capable of both tasks.
Outcome:	The proposed model can perform both fill in the blank and continuation tasks.

Copied to clipboard

Yue Yang, Artemis Panagopoulou, Marianna Apidianaki, Mark Yatskar, Chris Callison-Burch

FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models (2024.acl-short)

Challenge:	Neural language models encode rich knowledge about entities and their relationships but common properties of nouns are difficult to extract because they are rarely explicitly stated in texts.
Approach:	They propose to extract perceptual properties from images and use them in an ensemble model to complement the information extracted from language models.
Outcome:	The proposed model improves noun property prediction compared to powerful text-based language models.

Copied to clipboard

Andrew Zhu, Alyssa Hwang, Liam Dugan, Chris Callison-Burch

Question Answering

Challenge:	Existing benchmarks for large language models focus on intradocument dependencies or dependencies between a small number of documents.
Approach:	They propose to use a dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base to evaluate models' reasoning.
Outcome:	The proposed dataset shows that models still have room to improve reasoning over inter-document dependencies in a long context.

Causal Reasoning of Entities and Events in Procedural Texts (2023.findings-eacl)

Copied to clipboard

Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, Chris Callison-Burch

Low-Resource NLP

Challenge:	Existing work on entity state tracking or event reasoning is limited to procedural texts.
Approach:	They propose a benchmark for causal reasoning of event plausibility and entity states . they represent entities as programming languages while prompting language models .
Outcome:	The proposed model outperforms existing models on human reasoning and event reasoning.

Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence (2022.emnlp-main)

Copied to clipboard

Chris Callison-Burch, Gaurav Singh Tomar, Lara Martin, Daphne Ippolito, Suma Bailis, David Reitter

Dialogue Systems & Conversational Agents

Challenge:	researchers have posited Dungeons and Dragons as a challenge problem to test systems on various language-related capabilities.
Approach:	They frame Dungeons and Dragons specifically as a dialogue system challenge . they train a large language model to generate the next game turn, conditioning it on different information.
Outcome:	The proposed game generates the next conversational turn and predicts the state of the game given the dialogue history.

Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation (2026.findings-eacl)

Copied to clipboard

Meiqing Jin, Liam Dugan, Chris Callison-Burch

Dialogue Systems & Conversational Agents

Challenge:	Practicing conversations with large language models is a promising alternative to traditional in-person language learning.
Approach:	They propose a new token-level evaluation metric, Token Miss Rate, that measures the proportion of incomprehensible tokens per utterance and correlates strongly with human judgments.
Outcome:	The proposed methods improve comprehensibility for beginner speakers from 39.4% to 83.3%, compared with prompting alone and a token-level evaluation metric, Token Miss Rate (TMR).

Comparison of Diverse Decoding Methods from Conditional Language Models (P19-1)

Copied to clipboard

Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, Chris Callison-Burch

Language Models

Challenge:	Conditional language models can generate a diverse set of outputs, but for open-ended tasks, beam search is ill-suited to generating a set of diverse sequences.
Approach:	They propose a method where we over-sample candidates and use clustering to remove similar sequences to achieve high diversity without sacrificing quality.
Outcome:	The proposed method over-samples candidates and removes similar sequences to achieve high diversity without sacrificing quality.

Explanation-based Finetuning Makes Models More Robust to Spurious Cues (2023.acl-long)

Copied to clipboard

Josh Magnus Ludan, Yixuan Meng, Tai Nguyen, Saurabh Shah, Qing Lyu, Marianna Apidianaki, Chris Callison-Burch

Responsible & Trustworthy NLP

Challenge:	Large Language Models (LLMs) learn correlations between labels and features that are irrelevant to the task, leading to poor generalization on out-of-distribution data.
Approach:	They propose an explanation-based approach to fine tune large language models to generate a free-text explanation supporting their answer.
Outcome:	The proposed model is more robust against spurious cues in terms of accuracy drop across four classification tasks: ComVE (+1.2), CREAK (+9.1), e-SNLI (+5.4), and SBIC (+6.5).

PDDLEGO: Iterative Planning in Textual Environments (2024.starsem-1)

Copied to clipboard

Li Zhang, Peter Jansen, Tianyi Zhang, Peter Clark, Chris Callison-Burch, Niket Tandon

Challenge:	Existing methods to plan in textual environments rely on a fully-observed environment where all entity states are known, but are not interpretable.
Approach:	They propose to use LLMs to generate a formal representation of the environment that can be solved by a symbolic planner.
Outcome:	The proposed model outperforms existing methods in the Coin Collector simulation and Cooking World simulations.

FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information (2023.acl-long)

Copied to clipboard

Andrew Zhu, Karmanya Aggarwal, Alexander Feng, Lara Martin, Chris Callison-Burch

Challenge:	Recent work shows that large language models that have access to state information can generate higher quality game turns than LLMs that use dialog history alone.
Approach:	They present a dataset of game play sessions from real D&D gameplay on Discord with true game state info.
Outcome:	The proposed model can generate executable Avrae commands, especially after fine tuning.

Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness (2025.findings-acl)

Copied to clipboard

Bryan Li, Fiona Luo, Samar Haider, Adwait Agashe, Siyu Li, Runqi Liu, Miranda Muqing Miao, Shriya Ramakrishnan, Yuan Yuan, Chris Callison-Burch

Multilinguality

Challenge:	Existing approaches to retrieval-augmented generated (RAG) can be useful in multilingual settings, but they also introduce biases in the retrieved documents.
Approach:	They propose a dataset of territorial disputes paired with retrieved Wikipedia documents in 49 languages to evaluate cross-lingual robustness.
Outcome:	The proposed paradigm helps mitigate hallucinations of large language models (LLMs).

Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D (2025.emnlp-main)

Copied to clipboard

Artemis Panagopoulou, Le Xue, Honglu Zhou, Silvio Savarese, Ran Xu, Caiming Xiong, Chris Callison-Burch, Mark Yatskar, Juan Carlos Niebles

Learning Scalar Adjective Intensity from Paraphrases (D18-1)

Challenge:	a recent study shows that multimodal models can reason across multiple modalities . a limited number of models are able to reason across a variety of inputs .
Approach:	They propose a dataset for contrastive cross-modal reasoning across four modalities . they use human annotations and a mixture-of-models round-trip-consistency filter .
Outcome:	a new model evaluates models on multiple modalities to determine which one best answers a natural language prompt . the model must select the one that best satisfies the query and then fine-tune it . state-of-the-art models still achieve only 56% accuracy overall and 42% in four-modal settings .

Copied to clipboard

Anne Cocos, Skyler Wharton, Ellie Pavlick, Marianna Apidianaki, Chris Callison-Burch

Paraphrasing

Challenge:	Existing lexical resources do not include the relative intensities of adjectives.
Approach:	They propose a method to automatically learn relative intensity relation between scalar adjectives . they use a paraphrase-based method that assumes that a pair of adjectives is "really hot" a similar method is used to infer the polarity of indirect answers to "yes/no" questions .
Outcome:	The proposed method improves the quality of systems for ordering sets of scalar adjectives and inferring the polarity of indirect answers to "yes/no" questions.

Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data (2022.acl-long)

Copied to clipboard

Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, Graham Neubig

PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale (2023.findings-emnlp)

Challenge:	Existing work has treated procedures as shallow structures without modeling the parent-child relation.
Approach:	They propose to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow . they link steps in an article to other articles with similar goals, recursively building the KB .
Outcome:	The proposed method significantly outperforms baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval.

Copied to clipboard

Bryan Li, Chris Callison-Burch

Multilinguality

Challenge:	Existing question answering systems rely on large, high-quality training data.
Approach:	They propose a synthetic data generation method which decomposes cross-lingual QA into two stages . they apply a question generation model to the English side and annotation projection to translate both questions and answers.
Outcome:	The proposed method outperforms existing methods on cross-lingual QA datasets.

Probabilistic Soundness Guarantees in LLM Reasoning Chains (2025.emnlp-main)

Copied to clipboard

Weiqiu You, Anton Xue, Shreya Havaldar, Delip Rao, Helen Jin, Chris Callison-Burch, Eric Wong

Reasoning

Challenge:	Existing methods for detecting propagated errors in reasoning chains are inadequate . author et al. (2017) show that initial errors propagate and undermine reliability of final conclusion .
Approach:	They propose a framework that evaluates each reasoning step based solely on previously-verified premises and provides certified statistical guarantees of its soundness.
Outcome:	ARES achieves state-of-the-art performance across four benchmarks and demonstrates superior robustness on very long synthetic reasoning chains.

Toward Better Storylines with Sentence-Level Language Models (2020.acl-main)

Copied to clipboard

Daphne Ippolito, David Grangier, Douglas Eck, Chris Callison-Burch

Learning Interpretable Style Embeddings via Prompting LLMs (2023.findings-emnlp)

Challenge:	Rather than modeling fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence.
Approach:	They propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives.
Outcome:	The proposed model can focus on longer range dependencies, crucial for multi-sentence coherence.

Copied to clipboard

Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKeown, Chris Callison-Burch

Choice-75: A Dataset on Decision Branching in Script Learning (2024.lrec-main)

Challenge:	Prior work has treated the style of a text as separable from the content.
Approach:	They use prompting to perform stylometry on a large number of texts to generate a synthetic stylometric dataset.
Outcome:	The proposed model trains human-interpretable representations on a large stylometric dataset and a linguistic model for style representation learning.

Copied to clipboard

Zhaoyi Hou, Li Zhang, Chris Callison-Burch

Challenge:	Existing studies only consider scripts as linear developments of events, ignoring the potential branches that arise due to people’s circumstantial choices.
Approach:	They propose a benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios.
Outcome:	The proposed benchmarks show that they perform well in hard scenarios, but there is still significant headroom in hard ones.

Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims (N19-1)

Copied to clipboard

Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, Dan Roth

Reasoning

Challenge:	a number of fact checking techniques are used to identify and eliminate biases in text data.
Approach:	They propose to use search engines to expand and diversify a dataset of claims, perspectives and evidence to address a selection bias.
Outcome:	The proposed approach outperforms existing methods in a language understanding task.

Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package (D18-2)

Copied to clipboard

Ajay Patel, Alexander Sands, Chris Callison-Burch, Marianna Apidianaki

Reasoning about Goals, Steps, and Temporal Ordering with WikiHow (2020.emnlp-main)

Challenge:	Magnitude is an open source Python package that performs common operations up to 6,000 times faster than Gensim.
Approach:	They present a Python tool for utilizing vector embeddings that performs common operations up to 6,000 times faster than Gensim.
Outcome:	The Magnitude package performs common operations up to 6,000 times faster than Gensim and introduces several novel features for improved robustness like out-of-vocabulary lookups.

Copied to clipboard

Li Zhang, Qing Lyu, Chris Callison-Burch

Reasoning

Challenge:	Existing datasets focus on relation between procedural events, but little attention has been paid to relation between events.
Approach:	They propose a set of reasoning tasks targeting goal-step relations and step-step temporal relations based on wikiHow articles . their automatically-generated training set allows models to transfer to out-of-domain tasks requiring knowledge of procedural events .
Outcome:	The proposed dataset improves on SWAG, Snips, and Story Cloze Test in zero- and few-shot settings.

Automated Paraphrase Lattice Creation for HyTER Machine Translation Evaluation (N18-2)

Copied to clipboard

Marianna Apidianaki, Guillaume Wisniewski, Anne Cocos, Chris Callison-Burch

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification (2023.acl-long)

Challenge:	Existing machine translation evaluation metrics use synonyms and paraphrases to reward meaning-equivalent but lexically divergent translations.
Approach:	They propose a machine translation evaluation metric which exploits reference translations enriched with meaning equivalent expressions.
Outcome:	The proposed metric achieves medium performance on large and noisier datasets . it is compared with the existing HyTER evaluation metric .

Copied to clipboard

Sha Li, Ruining Zhao, Manling Li, Heng Ji, Chris Callison-Burch, Jiawei Han

Low-Resource NLP

Challenge:	Recent methods for event schema induction use information extraction systems to construct event graph instances from documents . compared to the previous state-of-the-art closed-domain schema inducing model, human assessors were able to cover 10% more events when translating the schemas into coherent stories .
Approach:	They propose to treat event schemas as commonsense knowledge that can be derived from large language models.
Outcome:	The proposed method simplifies the schema induction process and improves readability.

RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text (2020.emnlp-demos)

Copied to clipboard

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Chris Callison-Burch

A Feasibility Study of Answer-Agnostic Question Generation for Education (2022.findings-acl)

Challenge:	Existing studies on how humans perceive machine-generated text are limited due to the prohibitive cost of running human evaluation studies.
Approach:	They propose a task to detect the boundary at which a text passage starts off human-written transitions to being machine-generated.
Outcome:	The proposed system evaluates machine-generated news articles on a wide range of domains.

Copied to clipboard

Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, DaHyeon Choi, Chuning Yuan, Chris Callison-Burch

RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors (2024.acl-long)

Challenge:	a feasibility study into the applicability of answer-agnostic question generation models to textbook passages is conducted . a significant portion of errors arise from asking irrelevant or un-interpretable questions, a study finds .
Approach:	They conduct a feasibility study into the applicability of answer-agnostic question generation models to textbook passages.
Outcome:	The proposed model reduces the time it takes to write questions that target salient concepts . the proposed model would help professors write quizzes faster and help students stay engaged .

Copied to clipboard

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, Chris Callison-Burch

Robustness in NLP

Challenge:	Existing methods for detecting machine-generated text are often insufficiently robust and lack benchmark datasets.
Approach:	They evaluate the out-of-domain and adversarial robustness of 8 open- and 4 closed-source detectors using RAID benchmark datasets.
Outcome:	The proposed detectors are fooled by adversarial attacks, repetition penalties, and unseen generative models.

This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes (2024.naacl-long)

Copied to clipboard

Bryan Li, Samar Haider, Chris Callison-Burch

Language Models

Challenge:	Pretrained large language models may answer differently in different languages . this contrasts with a multilingual human, who would likely answer consistently .
Approach:	They propose a dataset of territorial disputes which includes multiple-choice questions in 49 languages . they propose metrics to quantify bias and consistency in responses across different languages based on their data .
Outcome:	The proposed model recalls certain knowledge inconsistently when asked in different languages.

GooAQ: Open Question Answering with Diverse Answer Types (2021.findings-emnlp)

Copied to clipboard

Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, Chris Callison-Burch

Question Answering

Challenge:	Existing research on niche answer types, mainly short responses and, in a few cases, long responses, has failed to adequately address the answer diversity of questions.
Approach:	They propose to use Google's autocomplete feature to collect questions from a large-scale dataset with a variety of answer types to facilitate further research on improving QA with diverse response types.
Outcome:	The proposed model produces naturalistic questions that are short and expressed using simple language.

A Recipe for Arbitrary Text Style Transfer with Large Language Models (2022.acl-short)

Copied to clipboard

Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, Jason Wei

Low-Resource NLP

Challenge:	augmented zero-shot learning is a prompting method that allows large language models to perform zero-shoot text style transfer to arbitrary styles, without any model fine-tuning or exemplars in the target style.
Approach:	They propose a prompting method that frames style transfer as a sentence rewriting task and requires only a natural language instruction.
Outcome:	The proposed method is based on a large language model and is shown to perform on standard style transfer tasks and arbitrary transformations.

StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples (2025.naacl-long)

Copied to clipboard

Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, Chris Callison-Burch

Representation Learning

Challenge:	Existing methods for embedding text are limited by the imperfect nature of data acquired under such assumptions.
Approach:	They propose a new approach to training stronger content-independent style embeddings using a synthetic dataset of near-exact paraphrases with controlled style variations.
Outcome:	The proposed model outperforms existing methods in real-world benchmarks and outperformed leading style representations in downstream applications.

mStyleDistance: Multilingual Style Embeddings and their Evaluation (2025.findings-acl)

Copied to clipboard

Justin Qiu, Jiacheng Zhu, Ajay Patel, Marianna Apidianaki, Chris Callison-Burch

Representation Learning

Challenge:	Multilingual StyleDistance embeddings are useful for stylistic analysis and style transfer, but they only exist for English.
Approach:	They propose a method that can generate style embeddings in new languages using synthetic data and a contrastive loss.
Outcome:	The proposed method outperforms existing style embeddings on these benchmarks and generalizes well to unseen features and languages.

Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)

Copied to clipboard

Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark