Papers by Simran Khanuja
Towards Automatic Evaluation for Image Transcreation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating image transcreation have relied on human evaluation. |
| Approach: | They propose a suite of automatic evaluation metrics inspired by machine translation metrics . they identify cultural relevance, semantic equivalence and visual similarity as critical dimensions of image transcreation . |
| Outcome: | The proposed evaluation metrics agree with human ratings across 7 countries. |
Evaluating the Diversity, Equity, and Inclusion of NLP Technology: A Case Study for Indian Languages (2023.findings-eacl)
Copied to clipboard
| Challenge: | In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world’s languages, be equitable, not unduly biased towards any particular language, and be inclusive of all users. |
| Approach: | They propose to use Gini coefficient to assess NLP across all three dimensions to assess diversity, equity, and inclusion across all languages. |
| Outcome: | The proposed evaluation paradigm assesses NLP technologies across all three dimensions and identifies the need for regional-specific choices in model building and dataset creation. |
Multi-lingual and Multi-cultural Figurative Language Understanding (2023.findings-acl)
Copied to clipboard
Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, Graham Neubig
| Challenge: | Figures permeate human communication, but are understudied in NLP. |
| Approach: | They create a figurative language inference dataset for seven languages associated with a variety of cultures, using cultural and regional concepts for figurativ expressions. |
| Outcome: | The results show that the most common figurative expressions are found in Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. |
GlobalBench: A Benchmark for Global Progress in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, Graham Neubig
| Challenge: | despite advances in NLP, significant disparities in performance across languages still exist . prior benchmarks focused on a limited number of tasks and languages, but now GlobalBench tracks progress on all languages. |
| Approach: | They propose to use global benchmarks to track progress on all NLP datasets in all languages. |
| Outcome: | a new tool tracks progress on all NLP datasets in all languages and tracks per-speaker utility and equity . globalbench is designed to identify the most under-served languages and reward research efforts . a globalbech is available at https://github.com/neulab/globalbench. |
DeMuX: Data-efficient Multilingual Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained multilingual models have enabled deployment of NLP technologies for multiple languages, but their performance under an annotation budget remains an open question. |
| Approach: | They propose a framework that prescribes the exact data-points to label from vast amounts of unlabelled multilingual data, having unknown degrees of overlap with the target set. |
| Outcome: | The proposed framework outperforms strong baselines in 84% of the test cases in the zero-shot setting of disjoint source and target language sets. |
GLUECoS: An Evaluation Benchmark for Code-Switched NLP (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show multilingual contextual embedding models perform better on cross-lingual and multilingual tasks. |
| Approach: | They propose to evaluate multilingual contextual embedding models on multilingual data . they use language identification from text, POS tagging, Named Entity Recognition and Question Answering . |
| Outcome: | The proposed benchmark evaluates models on language identification from text, POS tagging, Named Entity Recognition, Question Answering and a new task for code-switching, Natural Language Inference. |
What Is Missing in Multilingual Visual Reasoning and How to Fix It (2025.findings-naacl)
Copied to clipboard
| Challenge: | NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users. |
| Approach: | They propose a translation-test approach to tackle multilinguality, visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality. |
| Outcome: | The proposed interventions boost open models LLaVA-v1.5-13B by 13.4%, LLva-v1.6-34B by 20.3%, and Qwen-VL by 16.7% while minorly improving GPT-4V’s performance. |
MergeDistill: Merging Language Models using Pre-trained Distillation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained multilingual language models often lack capacity and skewed data . this leads to inequitable representation of languages due to limited capacity and sub-optimal vocabularies. |
| Approach: | They propose a framework to merge pre-trained multilingual language models to maximize their assets with minimal dependencies. |
| Outcome: | The proposed framework outperforms teacher-trained models on multiple datasets and with a fixed model capacity. |
CAIRE: Cultural Attribution of Images with Retrieval (2026.eacl-long)
Copied to clipboard
| Challenge: | Current text-to-image models produce homogeneous outputs given under-specified prompts and their outputs are disproportionately biased toward Western cultures. |
| Approach: | They propose a framework that assesses the degree of cultural relevance of an image, given a user-defined set of labels. |
| Outcome: | The proposed evaluation metric surpasses baselines on a manually curated dataset of culturally salient but rare items built using language models by 22% F1 points. |
MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking (2026.tacl-1)
Copied to clipboard
Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja, Zaid Sheikh, Shan Jie, Ann Chia, Shearman Chua, Graham Neubig
| Challenge: | Existing methods for multilingual entity linking are limited by textual contexts and limited resources. |
| Approach: | They propose a testbed system for multilingual multimodal entity linking using BBC news articles paired with corresponding images in five languages. |
| Outcome: | The proposed system improves accuracy for entities with ambiguous textual contexts and models with weak multilingual abilities. |
Grounding Multilingual Multimodal LLMs With Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new data-centric approach could address cultural gaps in multimodal large language models . despite being trained on billions of image-text pairs, today's models are biased towards English and Western data. |
| Approach: | They propose a data-centric approach that directly grounds MLLMs in cultural knowledge. |
| Outcome: | The proposed approach outperforms open-source models on cultural-focused benchmarks without degrading results on mainstream vision–language tasks. |
An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new task is to translate images to make them culturally relevant . currently, translation systems focus on translating words and images . |
| Approach: | They propose a task of translating images to make them culturally relevant . they build pipelines comprising state-of-the-art generative models to do the task . |
| Outcome: | The proposed pipelines can translate only 5% of translated images for some countries and no translation is successful for others. |