Papers by Sunayana Sitaram

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data (P18-1)

Challenge:	Zero-shot cross-lingual transfer has been shown to be sub-optimal across low-resource languages due to the skew in resource distribution in languages.
Approach:	They propose to jointly reduce feature incongruity between the source and target language and increase generalization capabilities of pre-trained multilingual transformers.
Outcome:	Empirical results show that the proposed approach outperforms the standard zero-shot fine-tuning method on multiple datasets across all languages using only unlabeled instances in the target language.

Copied to clipboard

Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, Kalika Bali

A Survey of Multilingual Models for Automatic Speech Recognition (2022.lrec-1)

Challenge:	Code-mixed (CM) language training is a difficult problem because of lack of data and the increased confusability due to the presence of more than one language.
Approach:	They propose a computational technique for creating grammatically valid artificial CM data based on the Equivalence Constraint Theory.
Outcome:	The proposed method reduces the perplexity of the model and does not reduce the perceptibility of the models.

Copied to clipboard

Hemant Yadav, Sunayana Sitaram

Speech & Audio in NLP

Challenge:	Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, but the majority of the world’s languages do not have usable systems due to the lack of large speech datasets to train these models.
Approach:	They propose to use unlabeled speech data to build multilingual ASR models that can be used for improved performance on low-resource languages.
Outcome:	The proposed models can be used to improve performance on low-resource languages by using unlabeled speech data.

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach (L18-1)

Copied to clipboard

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

Multimodality

Challenge:	Automated Speech Recognition systems degrade in performance when recognizing accents that are different from the ones in training data.
Approach:	They propose to adapt Acoustic Models that are trained on one accent to a target accent by using a small amount of speech data in the target accent.
Outcome:	The proposed model can be used to identify accents in Indian English and other languages.

Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers (2020.lrec-1)

Copied to clipboard

Basil Abraham, Danish Goel, Divya Siddarth, Kalika Bali, Manu Chopra, Monojit Choudhury, Pratik Joshi, Preethi Jyoti, Sunayana Sitaram, Vivek Seshadri

Multimodality

Challenge:	Existing platforms collect labelled speech data from urban speakers whose dialects are often very different from low-income users.
Approach:	They propose to collect labelled speech data directly from low-income workers . they collect 109 hours of data from 36 participants in the Marathi language .
Outcome:	The proposed approach can provide valuable supplemental earning opportunities to low-income rural and urban workers.

UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages (2026.acl-long)

Copied to clipboard

Pranjal A Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram

M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks (2024.findings-emnlp)

Challenge:	Developing culturally grounded multilingual AI systems is challenging for low-resource languages . synthetic data is underexplored, but its effectiveness in multilingual and multicultural contexts is understudied .
Approach:	They propose a top-up synthetic data generation framework grounded in Wikipedia content . they use 9.5M data points across 13 Indian languages and English to generate a high-quality dataset .
Outcome:	The proposed model improves on NLG tasks and narrows performance gaps with high-resource languages.

Copied to clipboard

Florian Schneider, Sunayana Sitaram

Multimodality

Challenge:	Recent advances in Large Language Models and their multimodal counterparts have shown significant performance disparities across different languages and cultural contexts.
Approach:	They propose to evaluate LLMs on diverse vision-language tasks within a multilingual and multicultural context using M5 benchmark.
Outcome:	The proposed benchmarks highlight task-agnostic performance disparities between languages and cultural contexts.

On the Calibration of Massively Multilingual Language Models (2022.emnlp-main)

Copied to clipboard

Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, Monojit Choudhury

A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies (2021.acl-long)

Challenge:	Massively Multilingual Language Models (MMLMs) have gained popularity due to their effectiveness in cross-lingual transfer.
Approach:	They investigate how well calibrated MMLMs are with respect to confidence . they find that calibration methods like temperature scaling and label smoothing improve calibration .
Outcome:	The proposed models are able to generalize in languages unseen during fine-tuning, but they are not reliable across languages.

Copied to clipboard

A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, Almeida Jacqueline Toribio

A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications (2025.emnlp-main)

Challenge:	linguistic and social aspects of code-switching are not discussed in the literature in linguistics.
Approach:	They propose to examine linguistic and social aspects of code-switching across a wide range of languages in a survey of the literature in linguistics and language technologies.
Outcome:	The proposed framework aims to increase the clarity and depth of computational investigations of C-S and bridge the fields so that they might be mutually reinforcing.

Copied to clipboard

Sunayana Sitaram, Adrian de Wynter, Isobel McCrum, Qilong Gu, Si-Qing Chen

METAL: Towards Multilingual Meta-Evaluation (2024.findings-naacl)

Challenge:	Misgendering is the act of referring to someone by using words that do not match their chosen identity.
Approach:	They propose to use a participatory-design approach to assess and mitigate misgendering across 42 languages and dialects using a human-in-the-loop approach.
Outcome:	The proposed guardrails reduce misgendering rates across all languages and dialects without loss of quality and without loss in quality.

Copied to clipboard

Rishav Hada, Varun Gumma, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)

Challenge:	Recent studies show that Large Language Models excel on many standard NLP benchmarks.
Approach:	They propose a framework for end-to-end evaluation of Large Language Models as evaluators in multilingual scenarios.
Outcome:	The proposed framework evaluates LLMs as evaluators in multilingual scenarios.

Copied to clipboard

Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram

Multilingual CheckList: Generation and Evaluation (2022.findings-aacl)

Challenge:	Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations.
Approach:	They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.
Outcome:	The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.

Copied to clipboard

Karthikeyan K, Shaily Bhatt, Pankaj Singh, Somak Aditya, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

MAFIA: Multi-Adapter Fused Inclusive Language Models (2024.eacl-long)

Challenge:	Multilingual evaluation benchmarks usually contain limited high-resource languages and do not test models for specific linguistic capabilities.
Approach:	They propose an algorithm for automatically extracting target language CheckList templates from machine translated instances of a source language templates.
Outcome:	The proposed algorithm compares with CheckLists created with human verification in Hindi and 9 other languages.

Copied to clipboard

Prachi Jain, Ashutosh Sathe, Varun Gumma, Kabir Ahuja, Sunayana Sitaram

Responsible & Trustworthy NLP

Challenge:	Pretrained Language Models (PLMs) are widely used in NLP for various tasks.
Approach:	They propose to modularly debias a pre-trained language model across multiple bias dimensions using structured knowledge and a large generative model.
Outcome:	The proposed model is able to debias a pre-trained language model across multiple bias dimensions in a semi-automated way.

MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)

Copied to clipboard

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

GLUECoS: An Evaluation Benchmark for Code-Switched NLP (2020.acl-main)

Challenge:	Several new LLMs have been introduced necessitating their evaluation on non-English languages.
Approach:	They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets.
Outcome:	The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages.

Copied to clipboard

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)

Challenge:	Recent studies show multilingual contextual embedding models perform better on cross-lingual and multilingual tasks.
Approach:	They propose to evaluate multilingual contextual embedding models on multilingual data . they use language identification from text, POS tagging, Named Entity Recognition and Question Answering .
Outcome:	The proposed benchmark evaluates models on language identification from text, POS tagging, Named Entity Recognition, Question Answering and a new task for code-switching, Natural Language Inference.

Copied to clipboard

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)

Challenge:	Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation.
Approach:	They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.
Outcome:	The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models.

Copied to clipboard

Divyanshu Aggarwal, Ashutosh Sathe, Ishaan Watts, Sunayana Sitaram

Responsible & Trustworthy NLP

Challenge:	Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages.
Approach:	They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages.
Outcome:	The proposed model can improve on six multilingual tasks while degrading on high-resource languages.

On Evaluating and Mitigating Gender Biases in Multilingual Settings (2023.findings-acl)

Copied to clipboard

Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram

Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs (2025.coling-main)

Challenge:	Existing benchmarks and resources for evaluating gender biases in multilingual settings are limited.
Approach:	They propose to extend DisCo to different Indian languages using human annotations to evaluate gender biases in multilingual models.
Outcome:	The proposed benchmarks and mitigation techniques are extended beyond English to evaluate gender biases in multilingual models.

Copied to clipboard

Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Sunayana Sitaram, Kalika Bali, Tanuja Ganu, Akshay Nambi

Improving Consistency in LLM Inference using Probabilistic Tokenization (2025.findings-naacl)

Challenge:	Large language models (LLMs) excel in diverse applications but still struggle with non-Latin scripts and low-resource languages.
Approach:	They propose a dynamic learning approach that optimizes prompt strategy, embedding model, and LLM per query at runtime.
Outcome:	The proposed approach achieves 10-15% improvements in multilingual performance over pre-trained models and 4x gains compared to fine-tuned, language-specific models.

Copied to clipboard

Ashutosh Sathe, Divyanshu Aggarwal, Sunayana Sitaram

Syntactic Text Processing

Challenge:	Prior work has shown that probabilistic tokenizations can generate multiple tokenization of the same input string.
Approach:	They propose a method to leverage the multiple tokenization capabilities of modern LLM tokenizers.
Outcome:	The proposed method improves the self-consistency of large language models by generating multiple tokenizations.

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures (2024.lrec-main)

Copied to clipboard

Agrima Seth, Sanchit Ahuja, Kalika Bali, Sunayana Sitaram

Reasoning

Challenge:	Large Language Models (LLMs) are increasingly being integrated with social applications . large data sets are limited in their representation of information and do not capture knowledge from the Web .
Approach:	They propose a gamified framework that uses collective sensemaking to collect artifacts from 19 different Indian geographic subcultures and benchmark four popular LLMs.
Outcome:	The proposed framework is based on 260 participants from 19 different Indian geographic subcultures and shows that it can be used across regional sub-cultures.

PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data (2024.emnlp-main)

Copied to clipboard

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, Sunayana Sitaram

Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting (2024.emnlp-main)

Challenge:	Evaluation of multilingual Large Language Models is challenging due to a variety of factors including the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and lack of local, cultural nuances in translated benchmarks.
Approach:	They evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations.
Outcome:	The proposed models perform best in most Indic languages, while the agreement drops for direct assessment especially for Bengali and Odia.

Copied to clipboard

Sagnik Mukherjee, Muhammad Adilazuarda, Sunayana Sitaram, Kalika Bali, Alham Aji, Monojit Choudhury

Responsible & Trustworthy NLP

Challenge:	Socio-demographic prompting is a commonly employed approach to study cultural biases in LLMs as well as for aligning models to certain cultures.
Approach:	They propose to use socio-demographic prompting to probe four LLMs with culturally sensitive and non-sensitive cues on datasets that are supposed to be culturally neutral or sensitive.
Outcome:	The proposed model shows significant differences in responses on both kinds of datasets, casting doubt on its robustness.

Exploring Two-Phase Continual Instruction Fine-tuning for Multilingual Adaptation in Large Language Models (2026.findings-acl)

Copied to clipboard

Divyanshu Aggarwal, Sankarshan Damle, Navin Goyal, Satya Lokam, Sunayana Sitaram

Teaching LLMs to Abstain across Languages via Multilingual Feedback (2024.emnlp-main)

Challenge:	A key challenge for Large Language Models (LLMs) is improving their Multilingual instruction-following ability over time without deteriorating their ability in languages they already excel at, typically English.
Approach:	They propose a two-phase Continual Fine-tuning setup to improve a model's Multilingual adaptability by comparing an English-only LLM with a multilingual instruction dataset.
Outcome:	The proposed model improves on two-phase Continual Fine-tuning (CFT) setups on a multilingual instruction dataset.

Copied to clipboard

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Orevaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, Yulia Tsvetkov

A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models (2024.findings-emnlp)

Challenge:	Existing studies on LLM abstention focus on English, but they show that it can reduce the accuracy of the model by 20.5% .
Approach:	They propose to teach LLMs to abstain in the face of knowledge gaps by generating multiple feedback items in related languages.
Outcome:	Extensive experiments show that the proposed approach outperforms baselines and achieves 9.2% improvement for low-resource languages.

Copied to clipboard

Ashutosh Sathe, Prachi Jain, Sunayana Sitaram

Fairness in Language Models Beyond English: Gaps and Challenges (2023.findings-eacl)

Challenge:	Existing studies have highlighted the existence of social biases within large vision and language models.
Approach:	They propose a framework for systematically evaluating gender, race, and age biases in vision-language models with respect to professions.
Outcome:	The proposed framework covers all supported inference modes of the recent vision-language models, including image-to-text, text-to image, and image- to-image.

Copied to clipboard

Krithika Ramesh, Sunayana Sitaram, Monojit Choudhury

Everything you need to know about Multilingual LLMs: Towards fair, performant and reliable models for languages of the world (2023.acl-tutorials)

Challenge:	Language models are inequitable at encoding and re-presentation, but there is much to be studied and criticism for the existing research that remains to be addressed.
Approach:	They propose to survey fairness in multilingual and non-English contexts . they argue that it is infeasible to achieve comprehensive coverage in terms of fairness datasets based on English .
Outcome:	The proposed methods are infeasible to scale across languages and cultures, the authors argue . they argue that the current methods are too narrowly focused on specific dimensions and types of biases and cannot scale across cultures.

Copied to clipboard

Sunayana Sitaram, Monojit Choudhury, Barun Patra, Vishrav Chaudhary, Kabir Ahuja, Kalika Bali

GCM: A Toolkit for Generating Synthetic Code-mixed Text (2021.eacl-demos)

Challenge:	Responsible AI issues such as fairness, bias and toxicity will be discussed in this tutorial .
Approach:	This tutorial will describe various aspects of scaling up language technologies to many of the world’s languages by describing the latest research in Massively Multilingual Language Models (MMLMs).
Outcome:	This tutorial will cover various aspects of scaling up language technologies to many of the world's languages by describing the latest research in multilingual models.

Copied to clipboard

Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, Sunayana Sitaram

Linguistic Theories

Challenge:	Code-mixing is a spoken language phenomenon and is difficult to train in multilingual communities.
Approach:	They propose a tool that can automatically generate code-mixed data given parallel data in two languages.
Outcome:	The proposed tool can generate code-mixed data in two languages using two linguistic theories.

Improving Cross Lingual Transfer by Pretraining with Active Forgetting (2025.emnlp-main)

Copied to clipboard

Divyanshu Aggarwal, Ashutosh Sathe, Sunayana Sitaram

Performance and Risk Trade-offs for Multi-word Text Prediction at Scale (2023.findings-eacl)

Challenge:	Prior work has shown that encoder-only LLMs show impressive cross lingual transfer of their capabilities from English to other languages.
Approach:	They propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs.
Outcome:	The proposed model improves cross lingual transfer capabilities on non-English languages despite being trained on English data.

Copied to clipboard

Aniket Vashishtha, S Sai Prasad, Payal Bajaj, Vishrav Chaudhary, Kate Cook, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Challenge:	Large Language Models (LLMs) generate ethically inappropriate texts even for seemingly innocuous contexts.
Approach:	They propose to use large language models to detect and filter toxic content in text prediction tasks by evaluating their toxicity detection approaches against a manually crafted CheckList of harms.
Outcome:	The proposed methods are compared against a checklist of harms targeted at different groups and different levels of severity in English.

Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation (2023.findings-emnlp)

Copied to clipboard

A. Seza Doğruöz, Sunayana Sitaram, Zheng Xin Yong