Papers by Harshita Diddee
Chasing Random: Instruction Selection Strategies Fail to Generalize (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. |
| Approach: | They analyze popular selection strategies across different datasets and benchmarks to find out whether they generalize poorly. |
| Outcome: | The proposed methods outperform random baselines and cost-performance trade-offs on the full dataset and a random subset. |
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)
Copied to clipboard
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. |
| Approach: | They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
| Outcome: | The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2022.tacl-1)
Copied to clipboard
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh Shantadevi Khapra
| Challenge: | We present Samanantar, the largest publicly available parallel corpora collection for Indic languages . based on existing corporative, there has been limited benefit for resource-poor languages despite the lack of parallel corporals and monolingual corporata. |
| Approach: | They compile 12.4 million sentence pairs from existing corpora and mine 37.4 million from the Web. |
| Outcome: | The proposed model outperforms existing models and benchmarks on public datasets. |
MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)
Copied to clipboard
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation. |
| Approach: | They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field. |
| Outcome: | The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models. |
INMT-Lite: Accelerating Low-Resource Language Data Collection via Offline Interactive Neural Machine Translation (2024.lrec-main)
Copied to clipboard
Harshita Diddee, Anurag Shukla, Tanuja Ganu, Vivek Seshadri, Sandipan Dandapat, Monojit Choudhury, Kalika Bali
| Challenge: | Interactive Neural Machine Translation (INMT) systems can be used to promote data collection in several under-resourced languages, but are often not adapted to the deployment constraints native language speakers operate in. |
| Approach: | They propose to use interactive neural machine translation systems to promote data collection in several under-resourced languages by integrating three different modes of Internet-independent deployment and four assistive interfaces suitable for data-sparse languages. |
| Outcome: | The proposed model improves the data generation experience of community members along multiple axes without compromising on the quality of the generated translations. |
CIE: Controlling Language Model Text Generations Using Continuous Signals (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to control language models with intent are brittle and hard to scale. |
| Approach: | They propose to use a set of LMs to fine-tune to expect a control vector that is interpolated between a "low" and a 'high' token embedding. |
| Outcome: | The proposed method can be finetuned to expect a control vector that is interpolated between a “low” and a ‘high” token embedding. |
“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work treats gender bias as a binary classification task, but a comparative annotation framework can be used to assess the impact of biases. |
| Approach: | They propose to generate a dataset with normative ratings of gender bias in English text with a comparative annotation framework. |
| Outcome: | The first dataset of GPT-generated English text with normative ratings of gender bias is analyzed using Best–Worst Scaling . |