Papers by Varun Gumma
UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages (2026.acl-long)
Copied to clipboard
Pranjal A Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram
| Challenge: | Developing culturally grounded multilingual AI systems is challenging for low-resource languages . synthetic data is underexplored, but its effectiveness in multilingual and multicultural contexts is understudied . |
| Approach: | They propose a top-up synthetic data generation framework grounded in Wikipedia content . they use 9.5M data points across 13 Indian languages and English to generate a high-quality dataset . |
| Outcome: | The proposed model improves on NLG tasks and narrows performance gaps with high-resource languages. |
METAL: Towards Multilingual Meta-Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models excel on many standard NLP benchmarks. |
| Approach: | They propose a framework for end-to-end evaluation of Large Language Models as evaluators in multilingual scenarios. |
| Outcome: | The proposed framework evaluates LLMs as evaluators in multilingual scenarios. |
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)
Copied to clipboard
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. |
| Approach: | They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
| Outcome: | The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
MAFIA: Multi-Adapter Fused Inclusive Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) are widely used in NLP for various tasks. |
| Approach: | They propose to modularly debias a pre-trained language model across multiple bias dimensions using structured knowledge and a large generative model. |
| Outcome: | The proposed model is able to debias a pre-trained language model across multiple bias dimensions in a semi-automated way. |
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)
Copied to clipboard
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Several new LLMs have been introduced necessitating their evaluation on non-English languages. |
| Approach: | They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets. |
| Outcome: | The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages. |
Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models traditionally use Sinusoidal Positional Embeddings . retraining with newer methods like ROPE or ALIBI is computationally expensive . |
| Approach: | They propose to transition NMT models from Sinusoidal to Relative PEs without compromising performance. |
| Outcome: | The proposed approach outperforms models trained with Sinusoidal PEs on document-level benchmarks . the results show that parameter-efficient fine-tuning can facilitate the transition . |
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation of multilingual Large Language Models is challenging due to a variety of factors including the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and lack of local, cultural nuances in translated benchmarks. |
| Approach: | They evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations. |
| Outcome: | The proposed models perform best in most Indic languages, while the agreement drops for direct assessment especially for Bengali and Odia. |