Papers by Siva Reddy

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations (2023.emnlp-main)

Challenge:	Existing work on editing LLMs neglects the dependency between a fact and its logical implications.
Approach:	They propose an evaluation protocol that assesses the editing process using StandUp . they aim to ensure that the editing respects internal logical constraints .
Outcome:	The proposed evaluation protocol assesses the editing process using a standup question-answering dataset.

Copied to clipboard

Arkil Patel, Satwik Bhattamishra, Siva Reddy, Dzmitry Bahdanau

Visually Grounded Reasoning across Languages and Cultures (2021.emnlp-main)

Challenge:	Large Language Models (LLMs) have a knowledge cutoff and are costly to finetune repeatedly.
Approach:	They introduce a language evaluation suite that incorporates diverse tokens and prompt settings to simulate real-world complexity.
Outcome:	The proposed evaluation suite incorporates diverse tokens and prompt settings to simulate real-world complexity.

Copied to clipboard

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, Desmond Elliott

FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (2022.tacl-1)

Challenge:	a new protocol allows for a multilingual hierarchy of concepts and images based on native speakers . the results suggest that the current models are not robust enough to handle multilingual data .
Approach:	They propose a protocol to construct an ImageNet-style hierarchy representative of more languages and cultures.
Outcome:	The proposed protocol lets the selection of concepts and images be entirely driven by native speakers, rather than scraping them automatically.

Copied to clipboard

Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, Siva Reddy

Dialogue Systems & Conversational Agents

Challenge:	a new benchmark for hallucination-free dialogues is based on knowledge-based conversational models that generate unsupported utterances . a recent study shows that models that are trustworthy generate unverifiable or factually incorrect statements .
Approach:	They propose a data-centric solution to edit hallucinated responses in the Wizard of Wikipedia benchmark.
Outcome:	The proposed model improves on the Wizard of Wikipedia benchmark while maintaining engaging conversations.

Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval (2021.emnlp-main)

Copied to clipboard

Devang Kulshreshtha, Robert Belfer, Iulian Vlad Serban, Siva Reddy

Low-Resource NLP

Challenge:	Using self-training to train unsupervised domains can be expensive, resulting in poor generalization due to distributional shift.
Approach:	They propose to use unaligned data to train unsupervised domain adaptation models using cheap synthetically generated labeled data.
Outcome:	The proposed method significantly outperforms self-training on question generation and passage retrieval domains and on MLQuestions and PubMedQA.

Combining Parameter-efficient Modules for Task-level Generalisation (2023.eacl-main)

Copied to clipboard

Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, Siva Reddy

Responsible & Trustworthy NLP

Challenge:	A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to new tasks.
Approach:	They propose a modular neural network where a subset of latent skills is associated with a parameter-efficient model adapter.
Outcome:	The proposed model improves sample efficiency and few-shot generalisation in supervised learning compared to baselines.

REARANK: Reasoning Re-ranking Agent via Reinforcement Learning (2025.emnlp-main)

Copied to clipboard

Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal

StereoSet: Measuring stereotypical bias in pretrained language models (2021.acl-long)

Challenge:	REARANK is a large language model-based listwise reasoning reranking agent . it explicitly reasons be- fore reranked results, significantly improving performance and interpretability.
Approach:	They propose a large language model-based listwise reasoning reranking agent that explicitly reasons be- fore reranked lists.
Outcome:	The proposed agent outperforms GPT-4 on reasoning-intensive benchmarks and surpasses GPL-4 on BRIGHT benchmarks.

Copied to clipboard

Moin Nadeem, Anna Bethke, Siva Reddy

Benchmarking Vision Language Models for Cultural Understanding (2024.emnlp-main)

Challenge:	Existing literature on stereotypical biases in language models is limited . current evaluations focus on measuring bias without considering language modeling ability .
Approach:	They propose to measure stereotypical biases in four domains: gender, profession, race, and religion . they compare stereotypical and language modeling ability of popular models like BERT, GPT-2, RoBERTa and XLnet .
Outcome:	The proposed model shows strong stereotypical biases in gender, profession, race, and religion domains.

Copied to clipboard

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Steenkiste, Lisa Hendricks, Karolina Stanczak, Aishwarya Agrawal

Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2023.findings-emnlp)

Challenge:	Recent multimodal vision-language models have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning.
Approach:	They propose a visual question-answering benchmark to assess VLMs' cultural understanding of various facets of culture from 11 countries across 5 continents.
Outcome:	The visual question-answering benchmark aims to assess VLMs' cultural understanding across regions.

Copied to clipboard

Parishad BehnamGhader, Santiago Miret, Siva Reddy

Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback (2020.findings-emnlp)

Challenge:	kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 are popular retriever-augmented language models for a variety of tasks.
Approach:	They evaluate the strengths and weaknesses of kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 in reasoning over retrieved statements across different tasks.
Outcome:	The proposed models do not exhibit strong reasoning even when provided with only the required statements.

Copied to clipboard

Makesh Narsimhan Sreedhar, Kun Ni, Siva Reddy

Robustness in NLP

Challenge:	Currently, user feedback contains extraneous sequences hindering their usefulness as a training sample.
Approach:	They propose a generative adversarial model that converts noisy feedback into a plausible natural response in a conversation and fools the discriminator which distinguishes feedback from natural responses.
Outcome:	The proposed model improves the original chatbot performance from 69.94%to 75.96% in ranking correct responses on the PERSONACHATdataset.

Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)

Copied to clipboard

Jacob Mitchell Springer, Vaibhav Adlakha, Siva Reddy, Aditi Raghunathan, Marius Mosbach

Syntactic Substitutability as Unsupervised Dependency Syntax (2023.emnlp-main)

Challenge:	Recent advances in general purpose text embedders have been driven by training on synthetic training data.
Approach:	They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks.
Outcome:	The proposed dataset is high quality and leads to consistent improvements in performance.

Copied to clipboard

Jasper Jian, Siva Reddy

Responsible & Trustworthy NLP

Challenge:	Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language.
Approach:	They propose a method to induce syntactic dependencies theory-agnostically by substituting words from the same category for words at either end of a dependency.
Outcome:	The proposed method achieves 79.5% recall on long-distance subject-verb agreement constructions compared to 8.9% using a previous method.

Language Models Largely Exhibit Human-like Constituent Ordering Preferences (2025.naacl-long)

Copied to clipboard

Ada Tur, Gaurav Kamath, Siva Reddy

TopiOCQA: Open-domain Conversational Question Answering with Topic Switching (2022.tacl-1)

Challenge:	English sentences are typically inflexible vis-à-vis word order, but constituents show far more variability in ordering.
Approach:	They compare LLMs with four types of constituent movement to evaluate their performance on heavy NP shift, particle movement, dative alternation, and multiple PPs.
Outcome:	The proposed model performs well on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs.

Copied to clipboard

Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, Siva Reddy

Natural Language Interfaces

Challenge:	Current datasets for conversational question answering do not contain topic switches . people often engage in information-seeking conversations to discover new knowledge .
Approach:	They propose an open-domain conversational dataset with topic switches based on Wikipedia.
Outcome:	The proposed dataset achieves an F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of the dataset.

The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)

Copied to clipboard

Nathan Schucher, Siva Reddy, Harm de Vries

An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2022.acl-long)

Challenge:	Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks.
Approach:	They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations.
Outcome:	The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations.

Copied to clipboard

Nicholas Meade, Elinor Poole-Dayan, Siva Reddy

Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions (2021.emnlp-main)

Challenge:	Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on.
Approach:	They propose to use Counterfactual Data Augmentation, Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia as bias mitigation techniques to quantify their effectiveness.
Outcome:	The proposed techniques are Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia.

Copied to clipboard

Arjun Akula, Spandana Gella, Keze Wang, Song-Chun Zhu, Siva Reddy

Visual Data in NLP

Challenge:	Prior implementations of NMN use pre-defined and fixed textual inputs in their module instantiation.
Approach:	They propose to parameterize the module arguments to reduce the number of modules in NMN by up to 75% without any loss in performance.
Outcome:	The proposed model outperforms the state-of-the-art model on CLEVR-Ref+ dataset with +8.1% improvement in accuracy and +4.3% on full test set.

Using In-Context Learning to Improve Dialogue Safety (2023.findings-emnlp)

Copied to clipboard

Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, Dilek Hakkani-Tur

Dialogue Systems & Conversational Agents

Challenge:	Recent work has highlighted safety issues with large neural-based conversational models.
Approach:	They propose a retrieval-based approach for reducing bias and toxicity in chatbot responses . they retrieve demonstrations of safe responses to similar dialogue contexts to generate a response .
Outcome:	The proposed method reduces bias and toxicity in three chatbot models . it can be used in compliment to existing dialogue safety approaches, such as RLHF.

Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020.acl-main)

Copied to clipboard

Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, Siva Reddy

Responsible & Trustworthy NLP

Challenge:	Visual referring expression recognition is a task that requires natural language understanding in the context of an image.
Approach:	They propose to use contrastive learning and multi-task learning to increase the robustness of ViLBERT, the current state-of-the-art model for this task.
Outcome:	The proposed methods are 12% to 23% lower in performance than the established progress for this task.

The Curious Case of Absolute Position Embeddings (2022.findings-emnlp)

Copied to clipboard

Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams

On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? (2022.naacl-main)

Challenge:	In natural language, it is not absolute position that matters, but relative position . et al., 2017) language models incorporate positional encodings that encode absolute (linear) word order.
Approach:	They find that Transformer language models encode word order using positional information . they also find that models that use absolute position embeddings over-rely on positional data .
Outcome:	The results raise questions about the efficacy of APEs to model the relativity of position information.

Copied to clipboard

Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, Siva Reddy

Natural Language Interfaces

Challenge:	Existing knowledge-grounded conversational benchmarks produce factually invalid statements, a phenomenon commonly called hallucination.
Approach:	They conduct a human study on knowledge-grounded conversational benchmarks and state-of-the-art models.
Outcome:	The findings raise important questions on the quality of existing datasets and models.

Compositional Generalization in Dependency Parsing (2022.acl-long)

Copied to clipboard

Emily Goodwin, Siva Reddy, Timothy O’Donnell, Dzmitry Bahdanau

Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment (2022.findings-acl)

Challenge:	Compositionality is the ability to combine familiar units like words into novel phrases and sentences.
Approach:	They introduce a set of dependency parses for Compositional Freebase Queries (CFQ) they analyze the behaviour of a state-of-the-art dependency parser on the CFQ dataset .
Outcome:	The proposed dependency parser performs lower on the most challenging splits with the highest compound divergence.

Copied to clipboard

Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Cheung, Siva Reddy

Natural Language Interfaces

Challenge:	Existing work on question answering focuses on the pre-deployment stage; building an accurate model for deployment.
Approach:	They collect feedback from users and train a neural model with the feedback data.
Outcome:	The proposed model can explain the correctness or incorrectness of an answer.

Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining (2022.findings-emnlp)

Copied to clipboard

Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, Siva Reddy

The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents (2023.eacl-main)

Challenge:	To explain NLP models, importance measures are often used to inform input tokens are important for making a prediction.
Approach:	They propose a faithfulness metric that masks allegedly important tokens and retrains the model.
Outcome:	The proposed metric is based on LSTM-attention models and RoBERTa models.

Copied to clipboard

Xing Han Lu, Siva Reddy, Harm de Vries

Understanding by Understanding Not: Modeling Negation in Language Models (2021.naacl-main)

Challenge:	StatCan Dialogue Dataset consists of 19,379 conversation turns between agents and online users . researchers propose two tasks to help knowledge workers find relevant tables for live chat users based on real-world intents .
Approach:	They propose two tasks based on 19,379 conversation turns between agents and online users . they investigate the difficulty of each task by establishing strong baselines .
Outcome:	The proposed task is based on a dataset of 19,379 conversation turns . the researchers show that the models struggle to generalize to future conversations .

Copied to clipboard

Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, Aaron Courville

WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation (2025.emnlp-main)

Challenge:	Negation is a core construction in natural language, but state-of-the-art pre-trained language models often handle it incorrectly.
Approach:	They propose to augment language modeling objective with unlikelihood objective based on negated generic sentences from a raw text corpus.
Outcome:	The proposed approach reduces the top 1 error rate to 4% on negated LAMA dataset and improves on negating NLI benchmarks.

Copied to clipboard

Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar

Programming Languages in NLP

Challenge:	Existing benchmarks focus on specific aspects of web tasks but lack comprehensive coverage.
Approach:	They propose a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing involving HTML/CSS/JavaScript, and (3) mockup-to-code generation.
Outcome:	The proposed model performs well on basic information extraction, but struggles with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content.

Are self-explanations from Large Language Models faithful? (2024.findings-acl)

Copied to clipboard

Andreas Madsen, Sarath Chandar, Siva Reddy

Explainability & Interpretability in NLP

Challenge:	Instruction-tuned Large Language Models excel at many tasks and will explain their reasoning, so-called self-explanations.
Approach:	They propose to employ self-consistency checks to measure faithfulness to LLMs to determine if they are model-dependent and if their reasoning is convincing and wrong.
Outcome:	The proposed measures show that self-explanations are explanation, model, and task-dependent and should not be trusted in general.

Minimax and Neyman–Pearson Meta-Learning for Outlier Languages (2021.findings-acl)

Copied to clipboard

Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy, Anders Søgaard

Robustness in NLP

Challenge:	Model-agnostic meta-learning (MAML) is a strategy to learn resource-poor languages in a sample-efficient fashion.
Approach:	They propose a model-agnostic meta-learning strategy that minimizes the expected risk across languages with a uniform prior . they propose 'minimax' and 'neyman-pearson' models that constrain the risk in each language to a maximum threshold.
Outcome:	The proposed model reduces the maximum risk across languages while constraining the risk in each language to a maximum threshold.

Image Retrieval from Contextual Descriptions (2022.acl-long)

Copied to clipboard

Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, Siva Reddy