Papers by Sean Welleck
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)
Copied to clipboard
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
| Challenge: | Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses. |
| Approach: | They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability. |
| Outcome: | The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. |
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)
Copied to clipboard
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate . |
| Approach: | They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs. |
| Outcome: | The proposed language model-based evaluation model is more powerful than its predecessor. |
Agentic-R1: Distilled Dual-Strategy Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current long chain-of-thought models rely on slow and error-prone natural language traces. |
| Approach: | They propose a framework that distills complementary reasoning strategies from multiple teachers into a unified student model. |
| Outcome: | The proposed framework improves accuracy on computation-intensive tasks and reduces inference latency on standard benchmarks. |
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)
Copied to clipboard
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi
| Challenge: | Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks. |
| Approach: | They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve. |
| Outcome: | The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy. |
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2020.acl-main)
Copied to clipboard
| Challenge: | Unlikelihood is a technique developed for removal of repetition in language model completions . it allows for a model to be generalized to solve a number of problems . |
| Approach: | They extend the unlikelihood objective to generate generations that contain repetitions . they show that such an objective can be used to improve logical consistency . |
| Outcome: | The proposed approach can be applied to a number of dialogue tasks. |
Generated Knowledge Prompting for Commonsense Reasoning (2022.acl-long)
Copied to clipboard
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi
| Challenge: | Existing methods for commonsense reasoning rely on high-quality knowledge, but they are often dominated by large-scale pretrained models that are fine-tuned on a target benchmark. |
| Approach: | They develop generated knowledge prompting which generates knowledge from a language model and provides it as additional input when answering a question. |
| Outcome: | The proposed method improves state-of-the-art models on four commonsense reasoning tasks. |
LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)
Copied to clipboard
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan
| Challenge: | Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks . |
| Approach: | They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs . |
| Outcome: | The proposed model improves on multi-tasking while the best performing model only achieves 60.40%. |
A Survey of Deep Learning for Mathematical Reasoning (2023.acl-long)
Copied to clipboard
| Challenge: | a survey of deep learning for mathematical reasoning examines the field . a comprehensive reading list is provided to assist readers interested in the field. |
| Approach: | They present a survey of deep learning for mathematical reasoning over the past decade . they outline directions for future research and highlight potential for further exploration . |
| Outcome: | The proposed framework is based on the results of a decade-long survey of deep learning for mathematical reasoning. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. |
| Approach: | They propose to use a recurrent language model to address inconsistency in decoding algorithms that are inconsistent despite the fact that recursive language models are trained to produce sequences of finite length. |
| Outcome: | The proposed methods prevent inconsistency in the proposed models. |
Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities. |
| Approach: | They propose a method for explicitly up-weighting rare but correct solutions to overcome rank bias in group relative policy optimization (GRPO) . |
| Outcome: | The proposed method mitigates rank bias and improves pass@N across a large range of N in both synthetic and real theorem proving settings. |
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022.emnlp-main)
Copied to clipboard
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi
| Challenge: | Pre-trained language models struggle with consistent reasoning, and prompting methods are often noisy and inconsistent. |
| Approach: | They propose a few-shot inference method inspired by the Socratic way of conversation that generates a tree of explanations that bear logical relations between each other and frames it as a satisfiability problem. |
| Outcome: | The proposed method achieves 20% better accuracy than state-of-the-art prompting methods and performs competitively with supervised models. |
STEER: Unified Style Transfer with Expert Reinforcement (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show unified style transfer models outperform the 175B instruction-tuned GPT-3 on overall style transfer quality. |
| Approach: | They propose a unified style transfer framework that can transfer to multiple target styles from an arbitrary source style. |
| Outcome: | The proposed method outperforms the 175B instruction-tuned GPT-3 on overall style transfer quality despite being 226 times smaller in size . |
Dialogue Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Consistency is a long standing issue faced by dialogue models. |
| Approach: | They propose to frame the consistency of dialogue agents as natural language inference and create a new natural language dataset called Dialogue NLI. |
| Outcome: | The proposed model can improve the consistency of a dialogue model with human evaluation and automatic metrics on a suite of evaluation sets designed to measure the model’s consistency. |
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that relevant knowledge can provide useful context for commonsense tasks. |
| Approach: | They propose a method that learns to generate contextually relevant knowledge in response to given questions. |
| Outcome: | The proposed method shows consistent gains over 9 commonsense benchmarks. |
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2022.naacl-main)
Copied to clipboard
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi
| Challenge: | Prior studies suggested pre-trained language models possess limited understanding of commonsense knowledge despite otherwise stellar performance on leaderboards. |
| Approach: | They propose a framework that uses larger models to teach smaller models by distilling knowledge symbolically as text in addition to the neural model. |
| Outcome: | The proposed framework is based on a general language model teacher's commonsense knowledge graphs and a neural commonsensing model surpassing the teacher model's in all three criteria. |
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2022.naacl-main)
Copied to clipboard
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, Yejin Choi
| Challenge: | Existing paradigms for text generation are left-to-right decoding from autoregressive language models. |
| Approach: | They propose a decoding algorithm that incorporates heuristic estimates of future cost that are efficient for large-scale language models. |
| Outcome: | The proposed method outperforms baselines on five generation tasks and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. |