Papers by Mrinmaya Sachan

A Diachronic Perspective on User Trust in AI under Uncertainty (2023.emnlp-main)

Challenge:	We hypothesize that questioning can enhance human performance and assist solvers .
Approach:	They propose to use large language models to generate sequential questions for math word problem-solving . they propose to apply these models to a variety of math word problems .
Outcome:	The proposed model improves the performance of a math word problem solver by generating more questions than other models.

Copied to clipboard

Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, Mrinmaya Sachan

Challenge:	Modern NLP systems are rarely calibrated and are often confidently incorrect about their predictions, which violates users’ mental model and erodes their trust.
Approach:	They propose to use a mental model to bet on the correctness of an NLP system and to study how trust is rebuilt as a function of time after these events.
Outcome:	The proposed model shows that even a few highly inaccurate confidence estimation instances damage users’ trust in the system and performance, which does not easily recover over time.

Enhancing Textbooks with Visuals from the Web for Improved Learning (2023.emnlp-main)

Copied to clipboard

Janvijay Singh, Vilém Zouhar, Mrinmaya Sachan

Visual Data in NLP

Challenge:	Textbooks lack visuals that support student learning, but many lack them . e-textbooks lack such visuals, and many lack these visuals .
Approach:	They propose to use vision-language models to automatically enhance textbooks with images from the web.
Outcome:	The proposed model improves textbooks with images from the web while allowing for better pedagogical value.

Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation (2026.acl-long)

Copied to clipboard

Minjing Shi, Junling Wang, Jingwei Ni, Sankalan Pal Chowdhury, Mrinmaya Sachan

World Models for Math Story Problems (2023.findings-acl)

Challenge:	Existing systems that detect logical fallacies in public discourse do not help people recognize them independently.
Approach:	They propose an intelligent tutoring system which uses large language models to help humans learn about logical fallacies.
Outcome:	The proposed system outperforms baseline LLMs lacking such pedagogical strategies.

Copied to clipboard

Andreas Opedal, Niklas Stoehr, Abulhair Saparov, Mrinmaya Sachan

A Formal Perspective on Byte-Pair Encoding (2023.findings-acl)

Challenge:	Recent efforts to solve math story problems have lacked accurate representations of mathematical concepts.
Approach:	They propose a graph-based semantic formalism for solving math story problems . they combine existing datasets and annotate a corpus of 1,019 problems with MathWorld .
Outcome:	The proposed model can be used to solve math story problems with pre-trained language models . the model can also be used for generating new problems by using the model as a design space .

Copied to clipboard

Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Tim Vieira, Mrinmaya Sachan, Ryan Cotterell

Challenge:	Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, but the underlying optimization problem that BPE seeks to solve has not yet been laid down.
Approach:	They propose an algorithm which is a 1/sigma*(1-e(-sigma))-approximation of an optimal merge sequence.
Outcome:	The proposed algorithm improves the runtime complexity from O(NM) to O(N log M) and the lower bound of the approximation is approx0.37.

Self-Training for Jointly Learning to Ask and Answer Questions (N18-1)

Copied to clipboard

Mrinmaya Sachan, Eric Xing

Question Answering

Challenge:	Existing methods for question answering and question generation are hard to obtain in many domains.
Approach:	They propose a method for jointly learning to ask and answer questions . they leverage unlabeled text along with labeled question answer pairs for learning .
Outcome:	The proposed method improves on four benchmark datasets on question answering and question generation tasks.

Generating Pedagogically Meaningful Visuals for Math Word Problems: A New Benchmark and Analysis of Text-to-Image Models (2025.findings-acl)

Copied to clipboard

Junling Wang, Anna Rutkiewicz, April Wang, Mrinmaya Sachan

Multimodality

Challenge:	Math word problems (MWPs) describe mathematical scenarios through text, requiring learners to interpret both linguistic and numerical information to derive mathematical expressions.
Approach:	They propose a framework for generating pedagogically meaningful visuals from MWP text descriptions using a pre-defined visual language and a design space grounded in interviews with math teachers.
Outcome:	The proposed framework illustrates the core mathematical relationships in math word problems.

Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)

Copied to clipboard

Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, Ryan Cotterell

Towards Aligning Language Models with Textual Feedback (2024.emnlp-main)

Challenge:	Several recent papers claim to have achieved human parity at sentence-level machine translation.
Approach:	They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures.
Outcome:	The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations .

Copied to clipboard

Saüc Lloret, Shehzaad Dhuliawala, Keerthiram Murugesan, Mrinmaya Sachan

Book2Dial: Generating Teacher Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots (2024.findings-acl)

Challenge:	Using textual feedback, language models can be trained to learn from textual inputs.
Approach:	They propose an approach that aligns language models with user preferences expressed in text.
Outcome:	The proposed approach outperforms PPO on toxicity reduction, summarization, and dialog response tasks while achieving the same performance with only 20% of the samples.

Copied to clipboard

Junling Wang, Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Mrinmaya Sachan

PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors (2026.findings-eacl)

Challenge:	Educational chatbots are a promising tool for assisting student learning, but high-quality data is difficult to obtain due to privacy concerns.
Approach:	They propose a framework for generating synthetic teacher-student interactions grounded in a set of textbooks and propose to open-source their results.
Outcome:	The proposed framework captures a key aspect of learning interactions where curious students with partial knowledge ask teachers questions about the material in the textbook.

Copied to clipboard

Donya Rooein, Sankalan Pal Chowdhury, Mariia Eremeeva, Yuan Qin, Debora Nozza, Mrinmaya Sachan, Dirk Hovy

AI-Assisted Human Evaluation of Machine Translation (2025.naacl-long)

Challenge:	pedagogical theories are not aligned with teaching strategies for educational tasks . quiet students may be disengaged or not thinking critically because they do not speak up .
Approach:	They propose a taxonomy that links pedagogical methods to personality profiles to map teaching strategies to student personality traits.
Outcome:	The proposed model improves the use of less common, high-impact strategies such as role-playing . the model also increases the use less common strategies such role-players .

Copied to clipboard

Vilém Zouhar, Tom Kocmi, Mrinmaya Sachan

Machine Translation

Challenge:	Annotation metrics are misaligned with the ideal measure of text quality and human evaluation remains the most accurate, reliable, and ultimate standard.
Approach:	They propose an annotation protocol that helps annotators mark erroneous parts of the translation and assign a final score.
Outcome:	The proposed protocol reduces the time per span annotation by half . the method reduces annotation budget by 25% with filtering of examples that the AI deems to be likely to be correct.

Autoregressive Structured Prediction with Language Models (2022.findings-emnlp)

Copied to clipboard

Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, Mrinmaya Sachan

GPT-4 as a Homework Tutor Can Improve Student Engagement and Learning Outcomes (2025.acl-long)

Challenge:	Recent years have seen a paradigm shift in NLP towards using pretrained language models for a wide range of tasks.
Approach:	They propose to model structures as sequences of actions in autoregressive manner with PLMs . their approach allows in-structure dependencies to be learned without any loss .
Outcome:	The proposed approach achieves state-of-the-art on all structured prediction tasks.

Copied to clipboard

Alessandro Vanzo, Sankalan Pal Chowdhury, Mrinmaya Sachan

ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning (2026.acl-demo)

Challenge:	a recent study has shown that homework is never graded or is done superficially.
Approach:	They propose a prompting strategy that enables GPT-4 to conduct interactive homework sessions for high school students learning English as a second language.
Outcome:	The proposed solution improves homework in high school students learning English as a second language with minimal effort in content preparation, one of the key challenges of alternative methods.

Copied to clipboard

Vladislav Smirnov, Quang-Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov

Early-Exit and Instant Confidence Translation Quality Estimation (2026.eacl-long)

Challenge:	Existing TTC scaling strategies and reasoning scorers are fragmented and evaluated under inconsistent protocols.
Approach:	They propose a framework for seamless test-time compute scaling of large language model reasoning . they use a modular Python library to implement state-of-the-art scaling strategy and scorer families .
Outcome:	The proposed framework evaluates performance and computational efficiency on mathematical and coding tasks.

Copied to clipboard

Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, Mrinmaya Sachan, Jan Niehues

From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning (2025.emnlp-main)

Challenge:	Quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines.
Approach:	They propose an uncertainty-aware quality estimation model that matches previous approaches at a fraction of their costs.
Outcome:	The proposed method reduces evaluation costs by 50% and improves reranking performance.

Copied to clipboard

David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi, Iryna Gurevych, Mrinmaya Sachan

Challenge:	Large language models (LLMs) are often optimized for direct question-answering, but their effectiveness is often undermined by strategically withholding answers.
Approach:	They propose an online reinforcement learning-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions.
Outcome:	The proposed model outperforms proprietary models like LearnLM and can be used to enhance interpretability and pedagogical quality.

CausalCite: A Causal Formulation of Paper Citations (2024.findings-acl)

Copied to clipboard

Ishan Agrawal, Zhijing Jin, Ehsan Mokhtarian, Siyuan Guo, Yuen Chen, Mrinmaya Sachan, Bernhard Schölkopf

Representation Learning

Challenge:	citation counts are often criticized for failing to accurately reflect the true impact of a paper.
Approach:	They propose a method to measure the impact of a paper on follow-up papers by comparing similar papers by cosine similarity.
Outcome:	The proposed method is based on a new causal inference method, TextMatch.

PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers in Overleaf (2026.acl-demo)

Copied to clipboard

Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner, Xuanqiang Angelo Huang, Vilém Zouhar, Dominik Glandorf, Isabel Dahlgren, Rishit Dagli, Yuen Chen, Felix Leeb, Van Q. Truong, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Mrinmaya Sachan, Bernhard Schölkopf, Mona T. Diab, Zhijing Jin

Challenge:	Emerging AI-powered writing assistants focus on grammar fixes or simulating peer review with final scores, yet they fall short of providing concrete, actionable suggestions that help students improve their papers during drafting.
Approach:	They propose a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments while leaving the actual writing entirely to human authors.
Outcome:	The proposed system outperforms a baseline with the skill library and provides actionable suggestions while leaving the actual writing to human authors.

Probing for Arithmetic Errors in Language Models (2025.emnlp-main)

Copied to clipboard

Yucheng Sun, Alessandro Stolfo, Mrinmaya Sachan

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators (2024.acl-long)

Challenge:	Using a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model’s output and the correct answer from hidden states.
Approach:	They extend their analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations.
Outcome:	The proposed probes can predict model correctness with over 90% accuracy on addition-only GSM8K problems and guide selective re-prompting of erroneous reasoning steps with minimal disruption to correct outputs.

Copied to clipboard

Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, Markus Leippold

Standardized Tests as benchmarks for Artificial Intelligence (D18-3)

Challenge:	generative AI is a counter-measure to misinformation, but factual claim detection suffers from inconsistency in definitions and high cost of manual annotation.
Approach:	They propose a framework that assists in the annotation of factual claims with the help of large language models.
Outcome:	The proposed framework can be used to annotate factual claims with the help of large language models and can work with or without expert supervision.

Copied to clipboard

Mrinmaya Sachan, Minjoon Seo, Hannaneh Hajishirzi, Eric Xing

Question Answering

Challenge:	Standardized tests have been proposed as replacements to the Turing test as a driver for progress in AI.
Approach:	et al. propose standardized tests as replacements to the Turing test as a driver for progress in AI.
Outcome:	a series of standardized tests have been proposed as replacements to the Turing test . the tutorial categorizes open domain and closed domain tests into two categories . open domain tests require the system to have significant domain knowledge and reasoning capabilities.

DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)

Copied to clipboard

Jingwei Ni, Tobias Schimanski, Meihong Lin, Mrinmaya Sachan, Elliott Ash, Markus Leippold

How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact (2021.findings-acl)

Challenge:	RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance.
Approach:	They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores.
Outcome:	Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs.

Copied to clipboard

Zhijing Jin, Geeticka Chauhan, Brian Tse, Mrinmaya Sachan, Rada Mihalcea

Challenge:	Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with many real-world applications.
Approach:	They propose a moral philosophy definition of social good and a framework to evaluate the direct and indirect real-world impact of NLP tasks.
Outcome:	The proposed framework evaluates the direct and indirect real-world impact of NLP tasks and adopts the methodology of global priorities research to identify priority causes for NLP research.

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (2026.eacl-long)

Copied to clipboard

Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Miserlis Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Elliott Ash

The ART of LLM Refinement: Ask, Refine, and Trust (2024.naacl-long)

Challenge:	Variation in human annotation (i.e., disagreements) is common in NLP, but it is unclear whether it is possible to model this variation in LLMs.
Approach:	They evaluate the influence of different reasoning settings on LLM disagreement modeling . RLVR-style reasoning degrades performance in disagreement modeling, they find .
Outcome:	The proposed reasoning settings improve LLM disagreement modeling, while RLVR-style reasoning degrades it.

Copied to clipboard

Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, Asli Celikyilmaz

Logical Fallacy Detection (2022.findings-emnlp)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve?
Approach:	They propose a reasoning with a refinement strategy called ART: Ask, Refine, and Trust that asks necessary questions to decide when an LLM should refine its output and uses it to affirm or deny trust.
Outcome:	The proposed reasoning with a refinement strategy achieves a performance gain of +5 points over baselines on two multistep reasoning tasks.

Copied to clipboard

Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, Bernhard Schoelkopf

Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure (2025.findings-acl)

Challenge:	Existing language models perform poorly on logical fallacy detection . fallacious arguments can lead to disagreements, conflicts, endless debates, and a lack of consensus .
Approach:	They propose a task of logical fallacy detection and propose LogicClimate to detect fallacies in text.
Outcome:	The proposed task outperforms the best language model on Logic and LogicClimate . human reasoning is marred by logical fallacies, and some exacerbate misinformation .

Copied to clipboard

Romain Puech, Jakub Macina, Julia Chatain, Mrinmaya Sachan, Manu Kapur

Scaling Within Document Coreference to Long Texts (2021.findings-acl)

Challenge:	Recent studies have shown that LLMs are not able to provide one-to-one tutoring solutions because of their high cost and efficiency.
Approach:	They propose an algorithm to optimize LLM prompts and steer it to follow a predefined multi-turn tutoring plan represented as a transition graph.
Outcome:	The proposed algorithm is able to optimize LLM prompts and steer it to follow a predefined multi-turn tutoring plan represented as a transition graph.

Copied to clipboard

Raghuveer Thirukovalluru, Nicholas Monath, Kumar Shridhar, Manzil Zaheer, Mrinmaya Sachan, Andrew McCallum

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)

Challenge:	Existing end-to-end coreference resolution models use expensive span representations and antecedent prediction mechanisms.
Approach:	They propose an approximation to end-to-end coreference resolution models which scales gracefully to documents of any length.
Outcome:	The proposed model reduces training and inference time and memory costs compared to current models with minimal loss in accuracy.

Copied to clipboard

Alessandro Stolfo, Yonatan Belinkov, Mrinmaya Sachan

Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP (2021.emnlp-main)

Challenge:	Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent.
Approach:	They propose to mechanize the processing of arithmetic queries by a causal mediation framework.
Outcome:	The proposed model improves the performance of arithmetic queries with a set of MLP modules.

Copied to clipboard

Zhijing Jin, Julius von Kügelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrinmaya Sachan, Bernhard Schoelkopf

Challenge:	a meta-analysis of published studies shows that the causal direction of data collection can explain some trends in NLP . semi-supervised learning and domain adaptation performance differ on a number of tasks .
Approach:	They argue that the causal direction of the data collection process has nontrivial implications . authors categorize common NLP tasks according to their causal direction . they also empirically assay the validity of the ICM principle for text data .
Outcome:	The proposed model can explain differences in semi-supervised learning and domain adaptation performance across settings.

A Transformer with Stack Attention (2024.findings-naacl)

Copied to clipboard

Jiaoda Li, Jennifer White, Mrinmaya Sachan, Ryan Cotterell

Adapters for Enhanced Modeling of Multilingual Knowledge and Text (2022.findings-emnlp)

Challenge:	Recent research suggests that transformer-based language models fail to learn basic algorithmic patterns.
Approach:	They propose to augment transformer-based language models with a differentiable stack-based attention mechanism that adds a level of interpretability to the model.
Outcome:	The proposed model can model some, but not all, deterministic context-freelanguages.

Copied to clipboard

Yifan Hou, Wenxiang Jiao, Meizhen Liu, Carl Allen, Zhaopeng Tu, Mrinmaya Sachan

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (2025.emnlp-main)

Challenge:	Large language models learn facts from text corpora, but knowledge graphs contain facts in an explicit triple format, restricting their research and application.
Approach:	They propose to enhance multilingual language models with knowledge from multilingual knowledge graphs . they propose to use cross-lingual entity alignment and facts from MLKGs to improve performance .
Outcome:	The proposed model improves MLLMs with cross-lingual entity alignment and facts from multilingual knowledge graphs for many languages while maintaining performance on other general language tasks.

Copied to clipboard

Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin

When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP (2023.acl-long)

Challenge:	Uncertainty quantification (UQ) is a framework for assessing the reliability of model outputs.
Approach:	They introduce pre-trained UQ heads for LLMs that are highly robust and generalized to languages they were not explicitly trained on.
Outcome:	The pre-trained heads significantly improve their ability to capture uncertainty compared to unsupervised methods.

Copied to clipboard

Jingwei Ni, Zhijing Jin, Qian Wang, Mrinmaya Sachan, Markus Leippold

Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach (2021.acl-long)

Challenge:	Multi-task learning (MTL) is a machine learning paradigm where multiple learning tasks are optimized simultaneously, exploiting commonalities and differences across them.
Approach:	They propose a parameter-efficient MTL architecture to improve task aggregation and to include loosely related skills from multiple datasets.
Outcome:	The proposed architecture outperforms single-task learning (STL) and is expected to outperformed it.

Copied to clipboard

Yifan Hou, Mrinmaya Sachan

Multimodality

Challenge:	Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how and to what extent do these representations encode a particular linguistic phenomenon.
Approach:	They propose a new information-theoretic probe, Bird’s Eye, which detects if and how representations encode the information in contextualized text representations.
Outcome:	The proposed method estimates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word representations.

How to Engage your Readers? Generating Guiding Questions to Promote Active Reading (2024.acl-long)

Copied to clipboard

Peng Cui, Vilém Zouhar, Xiaoyu Zhang, Mrinmaya Sachan

Challenge:	Using questions in written text is an effective strategy to enhance readability, but what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied.
Approach:	They present a dataset of 10K in-text questions from textbooks and scientific articles and explore various approaches to generate such questions using language models.
Outcome:	The generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers’ memorization and comprehension.

Improving Large Language Model Safety with Contrastive Representation Learning (2025.emnlp-main)

Copied to clipboard

Samuel Simko, Mrinmaya Sachan, Bernhard Schölkopf, Zhijing Jin

Test of Time: Rethinking Temporal Signal of Benchmark Contamination (2026.acl-long)

Challenge:	Existing defenses against large language models (LLMs) are limited by their ability to generate responses to diverse inputs.
Approach:	They propose a model defense framework that finetunes a large-scale model using a triplet-based loss combined with adversarial hard negative mining to encourage separation between benign and harmful representations.
Outcome:	The proposed model defense outperforms previous representation engineering-based defenses while improving robustness against input-level and embedding-space attacks.

Copied to clipboard

Terry Jingchen Zhang, Gopal Dev, Ning Wang, Max Obreiter, Punya Syon Pandey, Keenan Samway, Wenyuan Jiang, Yinya Huang, Bernhard Schölkopf, Mrinmaya Sachan, Zhijing Jin

Challenge:	Existing work on benchmarks containing publicly available information has been interpreted as a temporal signal for benchmark contamination.
Approach:	They show that LLM-transformed questions can produce remarkably different temporal patterns compared to fill-in-the-blank questions directly retrieved from the very same documents.
Outcome:	The proposed model can produce different temporal patterns compared to fill-in-the-blank questions retrieved from the same documents.

Knowledge Graph Embedding Compression (2020.acl-main)

Copied to clipboard

Mrinmaya Sachan

Strategize Before Teaching: A Conversational Tutoring System with Pedagogy Self-Distillation (2023.findings-eacl)

Challenge:	Knowledge graph (KG) embedding techniques that learn continuous embedds of entities and relations consume a large amount of storage and memory.
Approach:	They propose a method that compresses the KG embedding layer by representing each entity in the KA as a vector of discrete codes and then composes the embeddables from these codes.
Outcome:	The proposed approach achieves 50-1000x compression of embeddings with a minor loss in performance on standard KG embeddable evaluations and retains the ability to perform reasoning tasks such as KG inference.

Copied to clipboard

Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, Kam-Fai Wong

Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang (2022.acl-long)

Challenge:	Conversational tutoring systems (CTSs) aim to help students master educational material with natural language interaction in the form of a dialog.
Approach:	They propose to jointly predict teaching strategies and generate tutor responses accordingly to help students master educational material through dialog.
Outcome:	The proposed framework is based on three dialog tutoring datasets and is more realistic than previous models that generate responses given the strategies as input.

Copied to clipboard

Daphna Keidar, Andreas Opedal, Zhijing Jin, Mrinmaya Sachan

Challenge:	a recent study suggests that language evolution is a diachronic process, but no causal analysis is performed to verify these claims.
Approach:	They analyze the semantic change and frequency shift of slang words and compare them to those of standard, nonslang terms.
Outcome:	The proposed model shows that slang has smaller semantic change but larger frequency shifts over time.

Longtonotes: OntoNotes with Longer Coreference Chains (2023.findings-eacl)

Copied to clipboard

Kumar Shridhar, Nicholas Monath, Raghuveer Thirukovalluru, Alessandro Stolfo, Manzil Zaheer, Andrew McCallum, Mrinmaya Sachan

MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors (2025.emnlp-main)

Challenge:	Using Ontonotes, documents in certain genres were split into smaller parts for ease of annotation.
Approach:	They propose to merge annotations from documents split into smaller parts in Ontonotes for ease of annotation.
Outcome:	The proposed corpus restores documents to their original form, revealing dramatic increases in length in certain genres.

Copied to clipboard

Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

Natural Language Interfaces

Challenge:	Evaluating the pedagogical capabilities of AI-based tutoring models is critical for guided progress in the field.
Approach:	They propose an open-source benchmark for holistic tutoring model evaluation.
Outcome:	The proposed model can discriminate between expert and novice teachers with high accuracy.

MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems (2023.findings-emnlp)

Copied to clipboard

Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

Distilling Reasoning Capabilities into Smaller Language Models (2023.findings-acl)

Challenge:	Existing models for automatic dialogue tutoring fail to provide accurate feedback or reveal solutions to students too early.
Approach:	They propose a framework to generate one-to-one teacher-student tutoring dialogues by pairing human teachers with a Large Language Model (LLM) they use scaffolding questions and annotations to fine-tune models to be more effective tutors .
Outcome:	The proposed framework can generate 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems.

Copied to clipboard

Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan

Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks (2022.findings-naacl)

Challenge:	a step-by-step reasoning approach like chain of thought has proved to be effective in eliciting reasoning abilities in large language models.
Approach:	They propose a knowledge distillation approach that leverages CoT reasoning capabilities of larger models and distills them into smaller models.
Outcome:	The proposed scheme boosts the performance of smaller models over 70% on multiple reasoning datasets.

Copied to clipboard

Zhao Meng, Yihan Dong, Mrinmaya Sachan, Roger Wattenhofer

Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective (2023.emnlp-main)

Challenge:	Existing methods to improve model robustness against word substitution-based adversarial attacks are too slow to generate adversarials on the fly.
Approach:	They propose an approach to improve the robustness of BERT models against word substitution-based adversarial attacks by leveraging adversarials for self-supervised contrastive learning.
Outcome:	The proposed method improves robustness of BERT models against word substitution-based adversarial attacks without using any labeled data.

Copied to clipboard

Tianyu Liu, Afra Amini, Mrinmaya Sachan, Ryan Cotterell

A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models (2023.acl-long)

Challenge:	a new framework for structured prediction is developed for natural language processing . a systematic approach to structured prediction requires exhaustive pair-wise comparisons of tokens .
Approach:	They propose a method that models the relationship between pairs of tokens in a string . they use a parallel method that predicts real numbers for each token in .
Outcome:	The proposed method doubles the speed of graph-based dependency parsers and brings 10-times speed-up over graph-driven dependency parses.

Copied to clipboard

Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, Mrinmaya Sachan

Robustness in NLP

Challenge:	Recent work shows that language models can rely on shallow patterns in problem description when generating a solution.
Approach:	They propose a framework which pins down the causal effect of various factors on the output solution.
Outcome:	The proposed framework improves robustness and sensitivity to direct interventions on a test bed of math word problems.

PWESuite: Phonetic Word Embeddings and Tasks They Facilitate (2024.lrec-main)

Copied to clipboard

Vilém Zouhar, Kalvin Chang, Chenxuan Cui, Nate B. Carlson, Nathaniel Romney Robinson, Mrinmaya Sachan, David R. Mortensen

Syntactic Text Processing

Challenge:	Existing word embedding methods overlook phonetic information that is crucial for many tasks.
Approach:	They propose three methods that use articulatory features to build phonetically informed word embeddings.
Outcome:	The proposed methods improve word retrieval and correlation with sound similarity and on rhyme and cognate detection tasks.

Differentially Private Language Models for Secure Data Sharing (2022.emnlp-main)

Copied to clipboard

Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, Mrinmaya Sachan

The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025.emnlp-main)

Challenge:	a variety of deanonymization attacks allow the re-identification of individuals from tabular data.
Approach:	They propose to train a language model in a differentially private manner and sample data from it . they find that the model generates fluent textual datasets with privacy guarantees .
Outcome:	The proposed methods outperform direct classifiers with DP-SGD in the real-world.

Copied to clipboard

Yu Fan, Yang Tian, Shauli Ravfogel, Mrinmaya Sachan, Elliott Ash, Alexander Miserlis Hoyle

Representation Learning

Challenge:	Embedding-based similarity metrics can be influenced by content dimensions and spurious attributes like the text’s source or language.
Approach:	They propose a debiasing algorithm that removes observed confounders from encoder representations and removes them from the encoder.
Outcome:	The proposed method improves on out-of-distribution benchmarks and on benchmarks, but performance is not affected.

Grammar Control in Dialogue Response Generation for Language Learning Chatbots (2025.naacl-long)

Copied to clipboard

Dominik Glandorf, Peng Cui, Detmar Meurers, Mrinmaya Sachan

Contextual Parameter Generation for Universal Neural Machine Translation (D18-1)

Challenge:	Existing language learning chatbots and research on second language acquisition benefit from these affordances.
Approach:	They ground a dialogue response generation model in a pedagogical repository of grammar skills and evaluate prompting, fine-tuning, and decoding strategies for grammar-controlled dialogue response generators.
Outcome:	The proposed model outperforms GPT-3.5 when tolerating minor response quality losses and predicts grammar-controlled responses to support grammar acquisition adapted to learner proficiency.

Copied to clipboard

Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, Tom Mitchell

Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification (2025.emnlp-demos)

Challenge:	Existing approaches to multilingual neural machine translation lack language-specific parameterization.
Approach:	They propose a modification to existing neural machine translation models that allows for language specific parameterization and domain adaptation.
Outcome:	The proposed model surpasses state-of-the-art for both the IWSLT-15 and IWSTL-17 datasets and can perform zero-shot translation.

Copied to clipboard

Chenfei Xiong, Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Zhijing Jin, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Mennatallah El-Assady, Elliott Ash

SIKeD: Self-guided Iterative Knowledge Distillation for Mathematical Reasoning (2025.findings-acl)

Challenge:	Social scientists often need to develop codebooks that can be reliable but require significant human effort.
Approach:	They propose a mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models.
Outcome:	The proposed framework integrates human expertise with automatic annotation guided by large language models.

Copied to clipboard

Shivam Adarsh, Kumar Shridhar, Caglar Gulcehre, Nicholas Monath, Mrinmaya Sachan

Opportunities and Challenges in Neural Dialog Tutoring (2023.eacl-main)

Challenge:	Large language models (LLMs) can generate intermediate reasoning process for multistep reasoning tasks.
Approach:	They propose a distillation method that teaches the model to approach a task using different strategies and the model uses its self-generated on-policy outputs to choose the most suitable strategy.
Outcome:	The proposed method significantly outperforms distillation techniques on large models of different sizes.

Copied to clipboard

Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

A Simple Unsupervised Approach for Coreference Resolution using Rule-based Weak Supervision (2022.starsem-1)

Challenge:	Existing approaches to designing dialog tutors have been challenging . current approaches perform poorly in constrained learning scenarios, authors find .
Approach:	They analyze dialog tutoring models using automatic and human evaluations to understand the new opportunities brought by dialog tutors.
Outcome:	The proposed models perform poorly in less constrained learning scenarios, the authors show . they find large number of model reasoning errors in 45% of conversations .

Copied to clipboard

Alessandro Stolfo, Chris Tanner, Vikram Gupta, Mrinmaya Sachan

Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors (2024.emnlp-main)

Challenge:	state-of-the-art coreference models rely on labeled data, but an end-to-end model is needed to solve this problem.
Approach:	They propose an approach that leverages an end-to-end neural model in settings where labeled data is unavailable.
Outcome:	The proposed approach outperforms the previous best unsupervised model and outperformed the rule-based model on English OntoNotes corpus.

Copied to clipboard

Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation (2025.naacl-long)

Challenge:	Existing models for dialog tutoring fail to detect student errors and tailor their feedback to them.
Approach:	They propose to build dialog tutoring models to scaffold students' problem-solving and verify student solutions by using automatic and human evaluation.
Outcome:	The proposed model improves the quality of the tutor response generation by detecting student errors and adjusting the feedback to the errors.

Copied to clipboard

Tianyu Liu, Jirui Qi, Paul He, Arianna Bisazza, Mrinmaya Sachan, Ryan Cotterell

Efficiently Computing Susceptibility to Context in Language Models (2024.findings-emnlp)

Challenge:	Existing methods to improve language models' performance do not exploit this phenomenon .
Approach:	They propose to use contextual information to select and construct prompts that improve model performance.
Outcome:	The proposed methods show that the mutual information between a context and a question is an effective gauge for language model performance.

Copied to clipboard

Tianyu Liu, Kevin Du, Mrinmaya Sachan, Ryan Cotterell

Let’s Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models (2023.findings-emnlp)

Challenge:	a current language model is able to incorporate information from a user-input context when answering queries, but it is not equally sensitive to subtle changes to that context.
Approach:	They propose a metric to quantify the degree to which contexts can influence a model’s response to a query at a distributional level.
Outcome:	The proposed method is comparable to Monte Carlo's estimated susceptibility across a diverse set of query domains despite being 70 faster.

Copied to clipboard

Ruida Wang, Wangchunshu Zhou, Mrinmaya Sachan

BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2022.naacl-main)

Challenge:	Data Synthesis is a promising way to train a small model with very little labeled data.
Approach:	They propose a framework that iteratively extrapolates the errors of a small model trained on a real-world validation dataset using a large language model.
Outcome:	The proposed framework reduces the gap between the synthesized dataset and the real data . it improves on multiple NLP tasks and on large models with human-annotated data.

Copied to clipboard

Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, Ming Zhou

A Structured Span Selector (2022.naacl-main)

Challenge:	Standard evaluation metrics, e.g., BLEU, TER and METEOR, focus on the quality of translations at the sentence level and do not consider discourse-level features.
Approach:	They propose to use a metric to take discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans.
Outcome:	The proposed metric possesses better selectivity and interpretability at the document-level, and is more sensitive to document- level nuances.

Copied to clipboard

Tianyu Liu, Yuchen Jiang, Ryan Cotterell, Mrinmaya Sachan

What Has Been Enhanced in my Knowledge-Enhanced Language Model? (2022.findings-emnlp)

Challenge:	a typical approach to natural language processing tasks involves selecting text spans and making decisions about them.
Approach:	They propose a grammar-based structured span selection model which learns to make use of partial span annotations.
Outcome:	The proposed model improves on two popular span prediction tasks.

Copied to clipboard

Yifan Hou, Guoji Fu, Mrinmaya Sachan

Differentiable Subset Pruning of Transformer Heads (2021.tacl-1)

Challenge:	Existing knowledge integration methods such as linear probes and prompts have key limitations in answering these questions.
Approach:	They propose a new probe model which integrates external knowledge from knowledge graphs into pretrained language models (LMs) ERNIE and K-Adapter are proposed as KI methods .
Outcome:	The proposed model interprets two well-known KELMs using graph attention on the corresponding knowledge graph for interpretation.

Copied to clipboard

Jiaoda Li, Ryan Cotterell, Mrinmaya Sachan

Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023.findings-acl)

Challenge:	Recent work shows that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model.
Approach:	They propose a method that prunes a Transformer's multi-head attention mechanism away without significantly harming its performance.
Outcome:	The proposed method improves on natural language inference and machine translation tasks while offering precise control of sparsity level.

Copied to clipboard

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, Taylor Berg-Kirkpatrick

Robustness in NLP

Challenge:	Existing membership inference attacks aim to predict whether a data sample was present in training data of a machine learning model.
Approach:	They propose to compare model scores to neighbour texts to eliminate access to training data by comparing model scores with a given sample.
Outcome:	The proposed attacks outperform reference-based attacks with perfect knowledge of the training data distribution and outperformed reference-free attacks with imperfect knowledge.

Beyond Good Intentions: Reporting the Research Landscape of NLP for Social Good (2023.findings-emnlp)

Copied to clipboard

Fernando Adauto, Zhijing Jin, Bernhard Schölkopf, Tom Hope, Mrinmaya Sachan, Rada Mihalcea

Challenge:	Recent advances in natural language processing (NLP) have created a vast number of applications that are aimed at social good applications.
Approach:	They propose a dataset with three tasks that can help identify NLP4SG papers and characterize the NLP landscape by: (1) identifying the papers that address a social problem, (2) mapping them to the corresponding UN Sustainable Development Goals, and (3) identifying their methods.
Outcome:	The proposed dataset can help identify NLP4SG papers and characterize the NLP landscape by: (1) identifying the papers that address a social problem, (2) mapping them to the corresponding UN Sustainable Development Goals (SDGs), and (3) identifying their methods.

Implicit Personalization in Language Models: A Systematic Study (2024.findings-emnlp)

Copied to clipboard

Zhijing Jin, Nils Heil, Jiarui Liu, Shehzaad Dhuliawala, Yahang Qi, Bernhard Schölkopf, Rada Mihalcea, Mrinmaya Sachan

XDailyDialog: A Multilingual Parallel Dialogue Corpus (2023.acl-long)

Challenge:	Existing studies have focused on the implicit personalization problem, but no unified framework exists to study it.
Approach:	They propose a mathematical formulation and a moral reasoning framework to study the phenomenon of Implicit Personalization (IP) they propose 'direct intervention' to estimate causal effect of mediator variable that cannot be directly intervened upon.
Outcome:	The proposed method estimates the causal effect of a mediator variable that cannot be directly intervened upon.

Copied to clipboard

Zeming Liu, Ping Nie, Jie Cai, Haifeng Wang, Zheng-Yu Niu, Peng Zhang, Mrinmaya Sachan, Kaiping Peng

Natural Language Interfaces

Challenge:	Existing datasets for open-domain dialogue modeling limited to a single language . absence of multilingual datasets hinders development of robust open- domain dialog systems .
Approach:	They propose a multilingual parallel open-domain dialog dataset to explore multilingual and cross-lingual open- domain dialog.
Outcome:	The proposed model can be used to explore multilingual and cross-lingual open-domain dialogs in other languages.

Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation (2026.findings-acl)

Copied to clipboard

Ekaterina Fadeeva, Aleksandr Rubashevskii, Dzianis Piatrashyn, Roman Vashurin, Shehzaad Dhuliawala, Artem Shelmanov, Timothy Baldwin, Preslav Nakov, Mrinmaya Sachan, Maxim Panov

Tokenization and the Noiseless Channel (2023.acl-long)

Challenge:	Existing approaches to mitigating hallucinations conflate factuality with faithfulness to the retrieved evidence, incorrectly labeling factually correct statements as hallucinos . Existing methods to mitigate hallucinics rely on a lack of training data coverage, input ambiguity, and architectural constraints.
Approach:	They propose a method for hallucination detection in Large Language Models enhanced with knowledge retrieval based on faithfulness to the retrieved context.
Outcome:	The proposed method outperforms unsupervised UQ baselines, RAG-specific methods, and supervised classifiers across multiple tasks and LLMs.

Copied to clipboard

Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, Ryan Cotterell

Elastic Weight Removal for Faithful and Abstractive Dialogue Generation (2024.naacl-long)

Challenge:	Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance.
Approach:	They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords .
Outcome:	The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task .

Copied to clipboard

Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, Edoardo Ponti

Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification (2026.eacl-long)

Challenge:	Current-day large language models generate coherent, grammatical, and seemingly meaningful text, but are prone to hallucinating incorrect information.
Approach:	They propose to ‘subtract’ parameters of a model trained to hallucinate from a dialogue response generation model to ‘negate’ the contribution of such hallucinatedexamples from it.
Outcome:	The proposed method reduces hallucinations and discourages extractive responses, which are often a consequence of reducing hallucines by encouraging copy-pasting of document spans.

Copied to clipboard

Paul He, Yinya Huang, Mrinmaya Sachan, Zhijing Jin

“Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding (2021.findings-emnlp)

Challenge:	Large language models (LLMs) are increasingly being applied to causal reasoning tasks.
Approach:	They propose a symbolic verification framework that checks whether LLM-generated causal expressions are derivable from a given causal graph using do-calculus and probability theory.
Outcome:	The proposed framework can recover correct answers that would otherwise be marked incorrect due to superficial differences.

Copied to clipboard

Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, Snigdha Chaturvedi

Challenge:	Existing studies on character-centric understanding of narratives focus on understanding the characters in the narrative, but these studies are limited to understanding only certain aspects of characters.
Approach:	They propose a dataset of literary pieces and their summaries paired with descriptions of characters that appear in them that are used to facilitate character-centric narrative understanding.
Outcome:	The proposed dataset includes literary pieces and their summaries paired with descriptions of characters that appear in them.

What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)

Copied to clipboard

Jiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan Cotterell

Can Vision-Language Models Solve Visual Math Equations? (2025.emnlp-main)

Challenge:	Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine.
Approach:	They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context.
Outcome:	The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt.

Copied to clipboard

Monjoy Narayan Choudhury, Junling Wang, Yifan Hou, Mrinmaya Sachan

Visual Data in NLP

Challenge:	Vision-Language Models (VLMs) perform well on textual equations, but fail on visually grounded counterparts.
Approach:	They propose to decompose visual equation solving into symbolic equation solving and visual recognition into two core components to understand this gap.
Outcome:	The proposed models perform well on textual equations, but fail on visual grounded ones.

Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023.eacl-main)

Copied to clipboard

Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan

Machine Translation

Challenge:	State-of-the-art machine translation quality estimation systems have been achieving remarkable correlations with human judgements yet they require human annotations, which are expensive and computationally heavy.
Approach:	They propose a problem where one predicts automated metric scores without the reference.
Outcome:	The proposed model can estimate automated metrics at the sentence-level without the reference.

Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models (2026.acl-long)

Copied to clipboard

Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan

Efficient Text-based Reinforcement Learning by Jointly Leveraging State and Commonsense Graph Representations (2021.acl-short)

Challenge:	Existing verification approaches, such as Process Reward Models, are computationally expensive and limited to specific domains.
Approach:	They propose a transformer-based probe that uses internal states of frozen LLMs to estimate credibility of reasoning steps during generation.
Outcome:	The proposed probes match or exceed PRMs that are up to 810 larger.

Copied to clipboard

Keerthiram Murugesan, Mattia Atzeni, Pavan Kapanipathi, Kartik Talamadupula, Mrinmaya Sachan, Murray Campbell

Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)

Challenge:	Text-based games (TBGs) are useful benchmarks for evaluating progress in grounded language understanding and reinforcement learning (RL).
Approach:	They propose an agent that induces a graph representation of the game state and jointly grounds it with a commonsense knowledge from ConceptNet.
Outcome:	The proposed agent outperforms baseline agents in the proposed game .

Copied to clipboard

Dominik Stammbach, Vilém Zouhar, Alexander Hoyle, Mrinmaya Sachan, Elliott Ash

Probing via Prompting (2022.naacl-main)

Challenge:	Topic models are an unsupervised dimensionality reduction technique that help organize large text collections.
Approach:	They propose to use large language models to evaluate document output and determine optimal number of topics.
Outcome:	The proposed model performs better on coherence ratings of word sets than on intrustion detection.

Copied to clipboard

Jiaoda Li, Ryan Cotterell, Mrinmaya Sachan

Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance (2022.naacl-main)

Challenge:	Pre-trained language models have increased the performance of data-driven natural language processing (NLP) models on a wide variety of tasks.
Approach:	They propose a model-free approach to probing via prompting which formulates probing as a prompting task and combine pruning to analyze where the model stores the linguistic information in its architecture.
Outcome:	The proposed approach extracts information from pre-trained models while learning much less on its own.

Copied to clipboard

Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, Bernhard Schölkopf

Machine Translation

Challenge:	Existing work on translationese neglects important factors and conclusions are mostly correlational but not causal.
Approach:	They use a dataset where MT training data are also labeled with human translation directions to examine the impact of translationese on machine translation evaluation.
Outcome:	The proposed model learns in the same direction as human translation directions.

Adaptive and Personalized Exercise Generation for Online Language Learning (2023.acl-long)

Copied to clipboard

Peng Cui, Mrinmaya Sachan

Text Generation

Challenge:	Empirical studies have shown various benefits of adaptive learning, such as improved student learning outcomes, lower dropout rates, and increased instructor satisfaction.
Approach:	They propose to combine a knowledge tracing model that estimates each student’s evolving knowledge states from their learning history with a controlled text generation model that generates exercise sentences based on the student’ s current estimated knowledge state and instructor requirements of desired properties.
Outcome:	The proposed model can generate superior exercises based on student state and instructor requirements . Empirical studies have shown that adaptive learning improves student learning outcomes, lower dropout rates, and increased instructor satisfaction.

Calibration of Machine Reading Systems at Scale (2022.findings-acl)

Copied to clipboard

Shehzaad Dhuliawala, Leonard Adolphs, Rajarshi Das, Mrinmaya Sachan

Question Answering

Challenge:	Existing methods to calibrate open setting machine reading systems fail to scale to these settings due to various scale limitations in practical settings.
Approach:	They propose to extend existing calibration approaches to calibrate open-domain question answering and claim verification systems to these settings.
Outcome:	The proposed calibration methods can selectively predict answers when question answering systems are posed with unanswerable or out-of-the-training distribution questions.

Investigating the Zone of Proximal Development of Language Models for In-Context Learning (2025.findings-naacl)

Copied to clipboard

Peng Cui, Mrinmaya Sachan

Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysis (2024.findings-emnlp)

Challenge:	In-context learning is a dynamic and progressive process where learners integrate new information into their knowledge base through interactions with the environment.
Approach:	They propose a learning analytics framework to analyze the in-context learning behavior of large language models (LLMs) through the lens of the Zone of Proximal Development (ZPD), an established theory in educational psychology.
Outcome:	The proposed framework improves inference and fine-tuning scenarios by selectively applying it to queries that are most likely to benefit from demonstrations.

Copied to clipboard

Zhiheng Lyu, Zhijing Jin, Fernando Gonzalez Adauto, Rada Mihalcea, Bernhard Schölkopf, Mrinmaya Sachan

Sentiment Analysis

Challenge:	Sentiment analysis aims to identify the sentiment expressed in a piece of text, often in the form of a review.
Approach:	They propose a causal discovery task that distinguishes whether a review "primes" the sentiment and a traditional prediction task to model the sentiment using the review as input.
Outcome:	The proposed model improves by 32.13 F1 points on a zero-shot five-class SA.

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)

Copied to clipboard

Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan

Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations (2022.emnlp-main)

Challenge:	Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities.
Approach:	They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns.
Outcome:	The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples.

Copied to clipboard

Yu Fei, Zhao Meng, Ping Nie, Roger Wattenhofer, Mrinmaya Sachan