Papers by Chitta Baral

FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments (2026.findings-acl)

Challenge:	Recent advances in reasoning and planning capabilities of large language models have enabled their potential as autonomous agents capable of tool use in dynamic environments.
Approach:	They propose an input-reformulation multi-agent framework that reformulates user queries .
Outcome:	The proposed framework outperforms ReAct, Function Calling, and Self-Reflection in overall pass5 scores.

Copied to clipboard

Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral

Dialogue Systems & Conversational Agents

Challenge:	Large Language Models are being increasingly deployed as decision-making core of autonomous agents . however, in conversational benchmarks, these agents fail due to the cascading effects of incorrect decision- making .
Approach:	They propose a framework that analyzes failure trajectories from baseline agents to identify most prevalent errors.
Outcome:	Experiments show that the framework improves performance over open-source LLMs . the framework can be used to build reliable, multi-turn tool-use agents .

Self-Supervised Test-Time Learning for Reading Comprehension (2021.naacl-main)

Copied to clipboard

Pratyay Banerjee, Tejas Gokhale, Chitta Baral

Low-Resource NLP

Challenge:	Recent work on unsupervised question answering shows that models can be trained with procedurally generated question-answer pairs and achieve performance competitive with supervised methods.
Approach:	They propose a method that performs "test-time learning" on a given context . they use self-supervision to train models on synthetically generated question-answer pairs .
Outcome:	The proposed method outperforms current unsupervised methods and outperformed supervised methods.

Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020.emnlp-main)

Copied to clipboard

Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou Yang

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022.findings-naacl)

Challenge:	Observable changes in the scene are reflected in captions, but actions are also linked to social aspects such as intentions, effects, and attributes that describe the agent.
Approach:	They propose to generate captions from videos that describe latent aspects of the human agent's actions.
Outcome:	The proposed model can be used to describe latent aspects of human actions in video clips and answer questions about videos.

Copied to clipboard

Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, Chitta Baral

Lexi: Self-Supervised Learning of the UI Language (2022.findings-emnlp)

Challenge:	Experimental results show that the proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average.
Approach:	They propose a unified model that can learn all 32 instruction tasks of the BoX without any task-specific modules.
Outcome:	The proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average.

Copied to clipboard

Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, Oriana Riva

MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering (2020.emnlp-main)

Challenge:	Existing models rely on UI metadata, which is often missing or not accessible.
Approach:	They propose a vision and language model that can learn generic visio-linguistic representations of UIs . they use a dataset to train Lexi, which is based on UI metadata .
Outcome:	The proposed model can handle unique features of UI screens, including text richness and context sensitivity.

Copied to clipboard

Tejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou Yang

Is a Question Decomposition Unit All We Need? (2022.emnlp-main)

Challenge:	Availability of large-scale datasets has enabled statistical machine learning in vision and language understanding.
Approach:	They propose a training paradigm that exposes models to perceptually similar mutations of input . they show a 10.57% improvement in the VQA-CP challenge .
Outcome:	The proposed training paradigm improves on the visual question answering challenge with 10.57% accuracy.

Copied to clipboard

Pruthvi Patel, Swaroop Mishra, Mihir Parmar, Chitta Baral

Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge (P19-1)

Challenge:	Large Language Models (LMs) have achieved state-of-the-art performance on many NLP benchmarks.
Approach:	They propose to decompose a hard question into simpler questions that are easier for models to answer.
Outcome:	The proposed approach significantly improves model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) by decomposing a hard question into simpler questions that are easier for models to answer.

Copied to clipboard

Ashok Prakash, Arpit Sharma, Arindam Mitra, Chitta Baral

GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning (2025.acl-long)

Challenge:	Existing methods to solve Winograd Schema Challenge use only knowledge embedded in text . this limits the performance of such models on the WSC problems.
Approach:	They propose to augment existing language models with a commonsense knowledge hunting module and an explicit reasoning module to extract the needed knowledge from text.
Outcome:	The proposed system improves on the language model based methods by 5.53% and 7.7% on the dataset.

Copied to clipboard

Shikhhar Siingh, Abhinav Rawat, Chitta Baral, Vivek Gupta

Information Extraction & Text Mining

Challenge:	Existing methods for analyzing images of events fail to accurately extract contextual meaning from images.
Approach:	They propose a framework to infer global event, temporal, and geospatial information from images . they also introduce a new metric GREAT for a reasoning-weighted evaluation .
Outcome:	The proposed framework enables better understanding of event significance from images . it also shows that meaningful information can be inferred from images, allowing them to be effectively linked to their corresponding events and contextual background.

Less is More: Summary of Long Instructions is Better for Program Synthesis (2022.emnlp-main)

Copied to clipboard

Kirby Kuznia, Swaroop Mishra, Mihir Parmar, Chitta Baral

Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA (2023.acl-long)

Challenge:	Despite the success of large pre-trained language models, they show below-par performance on the larger and more complicated programming related questions.
Approach:	They propose to use a human-generated summary of programming questions to improve LMs' performance.
Outcome:	The proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on an average in terms of strict accuracy.

Copied to clipboard

Neeraj Varshney, Chitta Baral

Question Answering

Challenge:	Despite remarkable progress made in natural language processing, even the state-of-the-art systems often make incorrect predictions.
Approach:	They propose to use selective prediction to enable models to abstain from answering when their predictions are likely to be incorrect.
Outcome:	The proposed method improves performance on 11 QA datasets and in- and out-of-domain settings.

How Many Data Samples is an Additional Instruction Worth? (2023.findings-eacl)

Copied to clipboard

Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar, Chitta Baral

Challenge:	Recent introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language.
Approach:	They propose to define a task in natural language without creating task-specific datasets or building models.
Outcome:	The proposed model outperforms multitask learning models but is far from state-of-the-art task-specific models.

MMTabReal: Real-World Benchmark for Multimodal Table Understanding (2026.findings-acl)

Copied to clipboard

Prasham Yatinkumar Titiya, Jainil Trivedi, Chitta Baral, Vivek Gupta

Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model (2021.findings-emnlp)

Challenge:	Multimodal tables are ubiquitous in real applications but are difficult to evaluate in multimodal large language models.
Approach:	They propose a multimodal table benchmark that compares 500 real-world tables with 4021 question–answer pairs.
Outcome:	MMtabReal spans four question types, five reasoning categories, and eight structural archetypes.

Copied to clipboard

Kuntal Kumar Pal, Chitta Baral

Chaos with Keywords: Exposing Large Language Models Sycophancy to Misleading Keywords and Evaluating Defense Strategies (2024.findings-acl)

Challenge:	Existing pre-trained models fail to capture a better understanding of numbers.
Approach:	They propose to use a text-to-text transfer learning model (T5) to learn numeracy in four numeration tasks.
Outcome:	The model outperforms its predecessors in four numeracy tasks, but struggle in extrapolation setting.

Copied to clipboard

Aswin Rrv, Nemika Tyagi, Md Nayem Uddin, Neeraj Varshney, Chitta Baral

ILDAE: Instance-Level Difficulty Analysis of Evaluation Data (2022.acl-long)

Challenge:	sycophancy is a type of hallucination in Large Language Models, which can lead to false information being presented.
Approach:	They explore the sycophantic tendencies of Large Language Models where models provide accurate answers even if they are not entirely correct.
Outcome:	The proposed models generate factually correct statements even when they are not completely correct.

Copied to clipboard

Neeraj Varshney, Swaroop Mishra, Chitta Baral

Text Complexity

Challenge:	Instance-level difficulty analysis of evaluation data is a new field of research that focuses on leveraging instance difficulty in natural language processing.
Approach:	They conduct Instance-Level Difficulty Analysis of Evaluation data in a large-scale setup of 23 datasets and demonstrate its five novel applications.
Outcome:	The proposed model improves efficiency and accuracy, improves quality and improves Out-of-Domain performance.

Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions (2023.eacl-main)

Copied to clipboard

Mihir Parmar, Swaroop Mishra, Mor Geva, Chitta Baral

Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)

Challenge:	Recent studies have shown that data collected through crowdsourcing often exhibit various biases that lead to overestimation of model performance.
Approach:	They propose to model instruction bias in 14 recent NLU benchmarks by analyzing crowdsourcing instructions and analyzing their results.
Outcome:	The proposed model can be over-represented in datasets with a large number of examples, and the results are consistent with previous studies.

Copied to clipboard

Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral

PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving (2025.emnlp-main)

Challenge:	Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules.
Approach:	They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking.
Outcome:	The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths.

Copied to clipboard

Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, Chitta Baral, Chen-Yu Lee, Tomas Pfister, Hamid Palangi

Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task (2022.findings-emnlp)

Challenge:	Existing methods for natural planning lack constraint-guided iterative verification and adaptive selection . a recent study found that LLMs are not good at such planning.
Approach:	They propose a model-agnostic and easily scalable agent framework with three key components: constraint, verification, and selection agents.
Outcome:	The proposed framework improves inference-time algorithms on NATURAL PLAN and OlympiadBench benchmarks.

Copied to clipboard

Shailaja Keyur Sampat, Pratyay Banerjee, Yezhou Yang, Chitta Baral

InstructExcel: A Benchmark for Natural Language Instruction in Excel (2023.findings-emnlp)

Challenge:	‘Actions’ play a vital role in how humans interact with the world and therefore require the capability to perform ‘Reasoning about Actions & Change’ (RAC).
Approach:	They propose an encoder-decoder architecture to learn the representation of actions as vectors and combine it with existing modality parsers and a scene graph question answering model to evaluate the proposed system.
Outcome:	The proposed system performs better than baselines on the CLEVR_HYP dataset and compares with existing parsers and a scene graph question answering model.

Copied to clipboard

Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri

End-to-end Knowledge Retrieval with Multi-modal Queries (2023.acl-long)

Challenge:	Large Language Models (LLMs) can solve increasingly complex NLP tasks such as Excel specific tasks.
Approach:	They propose a large-scale benchmark to test whether Large Language Models can generate code that solves Excel specific tasks provided via natural language user instructions.
Outcome:	The proposed model outperforms existing models and provides a hard benchmark for state of the art models like GPT-4.

Copied to clipboard

Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, Chitta Baral

Investigating Acceleration of LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with ‘LITE’ (2024.findings-naacl)

Challenge:	a new task is proposed to learn knowledge retrieval with multimodal queries . a vision-language model can retrieve knowledge using images and text inputs .
Approach:	They propose a task for vision-language models to retrieve knowledge with multi-modal queries . they propose reViz, a model that integrates content from both text and image queries based on a multimodal query task .
Outcome:	The proposed task performs better under zero-shot settings than previous work on cross-modal retrieval.

Copied to clipboard

Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, Chitta Baral

Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks (2025.acl-srw)

Challenge:	Large Language Models (LLMs) have remarkable performance across a wide variety of tasks, however, their large size makes their inference slow and computationally expensive.
Approach:	They propose to perform 'dynamic confidence-based early exiting' at token level from the intermediate layers which improves the computational efficiency of text generation without sacrificing the quality of the generation.
Outcome:	The proposed model achieves significant cost and quality improvements while maintaining the quality of the generation.

Copied to clipboard

Amir Saeidi, Shivanshu Verma, Md Nayem Uddin, Chitta Baral

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021.emnlp-main)

Challenge:	Large Language Models (LLMs) excel in math reasoning problemsolving, text generation, summarization, creative writing, among other tasks.
Approach:	They evaluate Direct Preference Optimization and its variants for aligning Large Language Models with human preferences.
Outcome:	The proposed alignment methods achieve near-optimal performance even with smaller subsets of training data.

Copied to clipboard

Man Luo, Yankai Zeng, Pratyay Banerjee, Chitta Baral

Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering (2020.emnlp-main)

Challenge:	Existing knowledge-based visual question answering systems rely on Concept-Net and Wikipedia to obtain external knowledge.
Approach:	They propose a visual retriever-reader pipeline that uses a natural language knowledge base and a Visual retriever to retrieve relevant knowledge.
Outcome:	The proposed method significantly improves the visual retriever-reader pipeline on the OK-VQA benchmark.

Copied to clipboard

Pratyay Banerjee, Chitta Baral

UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs’ Memorization (2025.acl-long)

Challenge:	Current supervised Question Answering methods rely on expensive data annotations and can introduce unintended annotator bias.
Approach:	They propose a self-supervised task over knowledge graphs that can be supervised by a data annotation tool.
Outcome:	The proposed task performs better than pre-trained language models on a large dataset.

Copied to clipboard

Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven Corman, Chitta Baral

Question Answering

Challenge:	UnSeenTimeQA is a data contamination-free time-sensitive question-answering benchmark.
Approach:	They propose a data contamination-free time-sensitive question-answering benchmark that avoids web-searchable queries grounded in the real world.
Outcome:	The proposed benchmark avoids web-searchable queries grounded in the real world and enables on-demand generation of new samples, mitigating the risk of data leakage.

The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness (2024.findings-acl)

Copied to clipboard

Neeraj Varshney, Pavel Dolin, Agastya Seth, Chitta Baral

ThinkTuning: Instilling Cognitive Reflections without Distillation (2025.emnlp-main)

Challenge:	Recent work on Large Language Models (LLMs) has identified a number of approaches to protect against their vulnerabilities and safety.
Approach:	They propose to use a large collection of safe and unsafe prompts to evaluate various LLM defense strategies over both ‘safety’ and ‘over-defensiveness’.
Outcome:	The proposed defense strategies are compared on multiple state-of-the-art LLMs and show that they are effective against both ‘safety’ and ‘over-defensiveness’.

Copied to clipboard

Aswin Rrv, Jacob Dineen, Divij Handa, Md Nayem Uddin, Mihir Parmar, Chitta Baral, Ben Zhou

EDM3: Event Detection as Multi-task Text Generation (2024.starsem-1)

Challenge:	Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning.
Approach:	They propose a GRPO-based interactive training approach that augments the rollouts of a student model with the guidance of . a teacher poses a problem, lets the student try an answer, then gives corrective feedback–enough to point the mind in the right direction and then show the correct solution.
Outcome:	The proposed method shows 3.69% improvement over zero-shot baselines and 2.08% and 3.99% improvement over the vanilla-GRPO baselines.

Copied to clipboard

Ujjwala Anantheswaran, Himanshu Gupta, Mihir Parmar, Kuntal Pal, Chitta Baral

PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving (2025.emnlp-main)

Challenge:	Existing methods for Event Detection (ED) cannot easily leverage pre-trained semantic knowledge.
Approach:	They propose to decompose and reformulate ED and fine-tune over its atomic subtasks to enhance knowledge transfer while mitigating prediction error propagation inherent in pipelined approaches.
Outcome:	The proposed method achieves state-of-the-art performance on RAMS, MAVEN, and MLEE, while achieving 90% accuracy over rare event types.

Copied to clipboard

Mihir Parmar, Palash Goyal, Xin Liu, Yiwen Song, Mingyang Ling, Chitta Baral, Hamid Palangi, Tomas Pfister

Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow (2023.eacl-main)

Challenge:	Recent studies have shown that decomposing complex problems into simple subtasks has significantly boosted the performance of large language models (LLMs).
Approach:	They propose a unified post-training framework that distills synthetic task decompositions and fine-tunes smaller LLMs via supervised and reinforcement-learning objectives to improve complex reasoning.
Outcome:	The proposed framework outperforms strong baselines on GSM8k and MATH benchmarks and shows that it can improve generalization capabilities on out-of-domain datasets.

Copied to clipboard

Anjana Arunkumar, Swaroop Mishra, Bhavdeep Singh Sachdeva, Chitta Baral, Chris Bryan

Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction (2021.acl-short)

Challenge:	Recent research has shown that language models exploit ‘artifacts’ in benchmarks to solve tasks, rather than learning them, leading to inflated model performance.
Approach:	They propose a benchmark creation paradigm for NLP that focuses on guiding crowdworkers and provides realtime visual feedback to improve sample quality.
Outcome:	The proposed paradigm decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts, while increasing the performance of both user groups.

Copied to clipboard

Ming Shen, Pratyay Banerjee, Chitta Baral

Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts (2024.naacl-short)

Challenge:	Masked Noun-Phrase Prediction (MNPP) is a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting.
Approach:	They propose a pre-training strategy to tackle pronoun resolution in an unsupervised setting by fine-tuning a large pre-trained model on a human-labeled dataset and then transferring to a smaller dataset such as Winograd Schema Challenge (WSC).
Outcome:	The proposed method outperforms all previous unsupervised methods on all datasets by large margins.

Copied to clipboard

Michael Saxon, Yiran Luo, Sharon Levy, Chitta Baral, Yezhou Yang, William Yang Wang

Multilinguality

Challenge:	With growth in the popularity of text-to-image models has come interest in assessing their multilingual capabilities, including multilingual accessibility.
Approach:	They propose to correct translation errors in a concept list translated to seven languages and compare the outputs of the benchmark to those conditioned on the old.
Outcome:	The proposed benchmark contains translation errors in Spanish, Japanese, and Chinese.

Semantically Distributed Robust Optimization for Vision-and-Language Inference (2022.findings-acl)

Copied to clipboard

Tejas Gokhale, Abhishek Chaudhary, Pratyay Banerjee, Chitta Baral, Yezhou Yang

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)

Challenge:	Existing methods to integrate linguistic knowledge into training pipelines are under-explored.
Approach:	They propose a model-agnostic method that leverages linguistic transformations to infer a set of linguistic phenomena.
Outcome:	The proposed method improves on benchmark datasets with images and video and is generalizable to other V&L tasks.

Copied to clipboard

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral

Visuo-Linguistic Question Answering (VLQA) Challenge (2020.findings-emnlp)

Challenge:	Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics.
Approach:	They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models.
Outcome:	The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting.

Copied to clipboard

Shailaja Keyur Sampat, Yezhou Yang, Chitta Baral

A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution (2023.findings-acl)

Challenge:	Understanding images and text together is an important aspect of cognition and building advanced AI systems.
Approach:	They propose to derive joint inference about a given image-text modality and compile a question-answering corpus using an image and a reading passage.
Outcome:	The proposed method has better baseline performance but is still far behind human performance.

Copied to clipboard

Neeraj Varshney, Himanshu Gupta, Eric Robertson, Bing Liu, Chitta Baral

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022.acl-long)

Challenge:	State-of-the-art natural language processing models have been shown to achieve remarkable performance in ‘closed-world’ settings where all the labels in the evaluation set are known at training time.
Approach:	They propose a multi-stage task to evaluate a system's performance on pipelined novelty ‘detection’ and ‘accommodation’ tasks.
Outcome:	The proposed model performs poorly in ‘closed-world’ settings where all the labels in the evaluation set are known at training time.

Copied to clipboard

Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi

Challenge:	Despite the success of supervised learning, models often struggle with generalization across tasks.
Approach:	They propose to use crowdsourcing instructions to build a model that learns a new task by understanding the human-readable instructions that define it.
Outcome:	The proposed model can learn from seen tasks and generalize to unseen tasks given its natural crowdsourcing instructions.

Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness (2022.findings-acl)

Copied to clipboard

Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Sachdeva, Chitta Baral

LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)

Challenge:	Data modification has been proposed as an effective solution for generalizing to out-of-domain (OOD) inputs.
Approach:	They propose to use data modification to generalize to out-of-domain inputs . they also analyze their adversarial robustness using a synthetic dataset .
Outcome:	The proposed data modification strategies improve OOD accuracy and AR, but data filtering hurts OOD on other tasks.

Copied to clipboard

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan

Map&Make: Schema Guided Text to Table Generation (2025.acl-long)

Challenge:	Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks .
Approach:	They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs .
Outcome:	The proposed model improves on multi-tasking while the best performing model only achieves 60.40%.

Copied to clipboard

Naman Ahuja, Fenil Bardoliya, Chitta Baral, Vivek Gupta

Information Extraction & Text Mining

Challenge:	Existing methods for text-to-table generation overlook what complex information to extract and how to infer it from text.
Approach:	They propose a method that decomposes text into atomic propositions to infer latent schemas.
Outcome:	The proposed method shows significant gains in accuracy and interpretability on three datasets.

ToW: Thoughts of Words Improve Reasoning in Large Language Models (2025.naacl-long)

Copied to clipboard

Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou

Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents (2025.findings-naacl)

Challenge:	Unlike other data augmentation methods, thoughts of words (TOW) views next-word prediction as a core reasoning task and injects fine-grained thoughts into pre-training texts.
Approach:	They propose a training-time data-augmentation method called thoughts of words (TOW) that injects fine-grained thoughts directly into a next-word prediction task and teaches the model to understand how the observed next word is related to previous contexts.
Outcome:	The proposed method reduces model hallucination by 10% and improves reasoning performance by 7% to 9% on average.

Copied to clipboard

Shrinidhi Kumbhar, Venkatesh Mishra, Kevin Coutinho, Divij Handa, Ashif Iquebal, Chitta Baral

‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks (2021.eacl-main)

Challenge:	Recent research has leveraged Large Language Models to accelerate materials discovery and design.
Approach:	They propose a dataset that features goals, constraints, and methods for designing real-world applications and a method that emulates the process a materials scientist would use to evaluate a hypothesis critically.
Outcome:	The proposed method emulates the process a materials scientist would use to evaluate a hypothesis critically.

Copied to clipboard

Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, Chitta Baral

Constructing Flow Graphs from Procedural Cybersecurity Texts (2021.findings-acl)

Challenge:	Existing visual question answering datasets assume only one ground truth answer for each question.
Approach:	They propose alternative answer sets (AAS) of ground-truth answers to address this limitation . they modify top VQA solvers to support multiple plausible answers for a question .
Outcome:	The proposed approach improves on the GQA dataset and shows that it is more efficient than previous approaches.

Copied to clipboard

Kuntal Kumar Pal, Kazuaki Kashihara, Pratyay Banerjee, Swaroop Mishra, Ruoyu Wang, Chitta Baral

Reframing Instructional Prompts to GPTk’s Language (2022.findings-acl)

Challenge:	Graph Convolution Network with BERT sentence embeddings outperforms BERT in all three domains.
Approach:	They propose to identify relevant information from procedural texts and generate information flows between sentences from them.
Outcome:	The proposed method outperforms BERT in two domains of cybersecurity.

Copied to clipboard

Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, Hannaneh Hajishirzi

“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility (2023.eacl-main)

Challenge:	Using reframing techniques, we find that instructional prompts are easier to follow for Language Models (LMs)
Approach:	They propose reframing techniques for manual reformulation of prompts into more effective ones . they compare performance of LMs prompted with reframed instructions on 12 NLP tasks .
Outcome:	The reframing techniques used for prompt reformulation improve performance on 12 tasks . the techniques boost performance on LMs with different sizes compared with original prompts .

Copied to clipboard

Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, Chitta Baral

A Study on the Efficiency and Generalization of Light Hybrid Retrievers (2023.acl-short)

Challenge:	Recent work has found that large-scale language models lack commonsense reasoning ability . a dataset evaluating large-level language models is needed to evaluate their understanding of feasibility .
Approach:	They propose a question-answering dataset that tests understanding of feasibility . they propose to use commonsense reasoning to reason about when an action is feasible .
Outcome:	The proposed dataset shows that state-of-the-art models struggle to answer feasibility questions correctly.

Copied to clipboard

Man Luo, Shashank Jain, Anchit Gupta, Arash Einolghozati, Barlas Oguz, Debojeet Chatterjee, Xilun Chen, Chitta Baral, Peyman Heidari

Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning (2025.findings-naacl)

Challenge:	Recent research focuses on building neural retrievers which learn dense embeddings of query and document into a semantic space.
Approach:	They propose to use an indexing-efficient dense retriever to reduce hybrid retrievers' memory by using the state-based indexing algorithm.
Outcome:	The proposed hybrid retriever saves 13 memory while maintaining 98.0% performance on out-of-domain datasets and adversarial attacks datasets.

Copied to clipboard

Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral

CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering (2022.emnlp-main)

Challenge:	Reasoning abilities of LLMs have been a key focus in recent years.
Approach:	They propose to use a college-level Multiple Choice Question-Answering task to identify LLM errors and evaluate their performance.
Outcome:	The proposed framework can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks.

Copied to clipboard

Maitreya Patel, Tejas Gokhale, Chitta Baral, Yezhou Yang

LogicAttack: Adversarial Attacks for Evaluating Logical Consistency of Natural Language Inference (2023.findings-emnlp)

Challenge:	Videos often capture objects, their visible properties, their motion, and the interactions between different objects.
Approach:	They propose a video question answering dataset for reasoning about the implicit physical properties of objects in a scene.
Outcome:	The proposed dataset enables evaluation under several out-of-distribution settings – videos with objects with masses, coefficients of friction, and initial velocities that are not observed in the training distribution.

Copied to clipboard

Mutsumi Nakamura, Santosh Mashetty, Mihir Parmar, Neeraj Varshney, Chitta Baral

Robustness in NLP

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated impressive performance on Natural Language Inference (NLI) tasks.
Approach:	They propose a method to attack NLI models using diverse logical forms of premise and hypothesis using propositional logic to generate effective adversarial attacks.
Outcome:	The proposed method achieves an average 53% Attack Success Rate (ASR) across multiple logic-based attacks.

AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models (2025.emnlp-main)

Copied to clipboard

Vatsal Malaviya, Agneet Chatterjee, Maitreya Patel, Yezhou Yang, Chitta Baral

WeaQA: Weak Supervision via Captions for Visual Question Answering (2021.findings-acl)

Challenge:	Text-to-Image (T2I) models have been successful in generating images from textual descriptions, but they struggle to capture nuanced and implicit attributes inherent in action depiction.
Approach:	They propose a benchmark to evaluate the performance of T2I models in generating images from action-centric prompts.
Outcome:	The proposed model achieves an increase of 72% on AcT2I.

Copied to clipboard

Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, Chitta Baral

Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems (2022.emnlp-main)

Challenge:	Existing methods for training visual question answering models rely on datasets with human-annotated image-quest-answer triplets.
Approach:	They propose a method to train models with synthetic Q-A pairs generated procedurally from captions.
Outcome:	The proposed method trains models with synthetic Q-A pairs generated from captions on three VQA benchmarks.

Copied to clipboard

Neeraj Varshney, Chitta Baral

Green & Sustainable NLP

Challenge:	Pre-trained language models such as RoBERTa, ELECTRA, and T5 have a large number of parameters which makes them slow and computationally expensive.
Approach:	They propose a technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions.
Outcome:	The proposed technique saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%.

Careful Selection of Knowledge to Solve Open Book Question Answering (P19-1)

Copied to clipboard

Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, Chitta Baral

QA‐LIGN: Aligning LLMs through Constitutionally Decomposed QA (2025.findings-emnlp)

Challenge:	Open book question answering requires deeper reasoning involving linguistic understanding and common knowledge.
Approach:	They propose a dataset that mimics open book question answering to achieve 72.0% accuracy.
Outcome:	The proposed dataset achieves 72.0% accuracy, an 11.6% improvement over the current state of the art.

Copied to clipboard

Jacob Dineen, Aswin Rrv, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, Ben Zhou

Explainability & Interpretability in NLP

Challenge:	QA-LIGN decomposes monolithic rewards into interpretable principle-specific evaluations . scalar rewards obscure which objectives drive the training signal .
Approach:	a new method decomposes monolithic rewards into interpretable principle-specific evaluations . QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate .
Outcome:	QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate . the results outperform DPO and GRPO with state-of-the-art reward models given equivalent training .

NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)

Copied to clipboard

Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan

A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Context (2022.naacl-srw)

Challenge:	Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner.
Approach:	They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding.
Outcome:	The proposed benchmark compares the performance of AI systems on eight different tasks.

Copied to clipboard

Man Luo, Shuguang Chen, Chitta Baral

Information Retrieval

Challenge:	Existing frameworks for OBQA use separate models to select relevant passages and sentences.
Approach:	They propose a framework to jointly rank passages and select sentences to improve correlation between them.
Outcome:	The proposed framework outperforms baseline systems in terms of matching of relevant sentences on the hotpotQA dataset by 28%.

InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis (2024.naacl-short)

Copied to clipboard

Kevin Scaria, Himanshu Gupta, Siddharth Goyal, Saurabh Sawant, Swaroop Mishra, Chitta Baral

Sentiment Analysis

Challenge:	Experimental results on the Sem Eval 2014, 15, and 16 datasets demonstrate that InstructABSA outperforms the previous state-of-the-art (SOTA) approaches on Term Extraction (ATE), Sentiment Classification(ATSC) and Sentimence Pair Extraction(ASPE) subtasks.
Approach:	They introduce positive, negative, and neutral examples to each training sample, and instruction tune the model (Tk-Instruct) for ABSA subtasks.
Outcome:	The proposed model outperforms the state-of-the-art (SOTA) on Term Extraction (ATE), Sentiment Classification (ATSC) and Sentimence Pair Extractions (ASPE) subtasks.

Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)

Copied to clipboard

Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images (2021.naacl-main)

Challenge:	Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them.
Approach:	They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains.
Outcome:	The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans.

Copied to clipboard

Shailaja Keyur Sampat, Akshay Kumar, Yezhou Yang, Chitta Baral

Unsupervised Natural Language Inference Using PHL Triplet Generation (2022.findings-acl)

Challenge:	Existing research on visual question answering is limited to information explicitly present in an image or a video.
Approach:	They propose a vision-language question answering task based on a CLEVR dataset . they modify existing methods and propose baseline solvers for this task .
Outcome:	The proposed model motivates the development of better vision-language models . it provides insights about the capability of diverse architectures to perform joint reasoning over image-text modality.

Copied to clipboard

Neeraj Varshney, Pratyay Banerjee, Tejas Gokhale, Chitta Baral

Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings (2022.findings-acl)

Challenge:	In some cases, training samples may not be available or collecting them could be time-consuming and resource-intensive.
Approach:	They propose a procedural approach that leverages sentence transformations to collect PHL triplets for training NLI models.
Outcome:	The proposed model outperforms existing models on several NLI benchmarks with a set of sentence transformations.

Copied to clipboard

Neeraj Varshney, Swaroop Mishra, Chitta Baral

Robustness in NLP

Challenge:	Existing approaches to selective prediction are not consistently outperform the simplest baseline MaxProb in all three settings.
Approach:	They propose to use a large-scale setup of 17 datasets to study selective prediction in NLP tasks using in-domain, out-of-domain and adversarial settings.
Outcome:	The proposed approaches outperform the simplest baseline MaxProb in in-domain, out-of-domain and adversarial settings, but none consistently outperformed in all three settings.

To Find Waldo You Need Contextual Cues: Debiasing Who’s Waldo (2022.acl-short)

Copied to clipboard

Yiran Luo, Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, Chitta Baral