Papers by Himanshu Singh
Commentator: A Code-mixed Multilingual Text Annotation Framework (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing annotation tools fail to address multilingual datasets efficiently. |
| Approach: | They introduce a code-mixed multilingual text annotation framework, COMMENTATOR . they perform robust qualitative human-based evaluations to showcase its effectiveness . |
| Outcome: | The proposed framework performs faster than baseline annotations in Hinglish and Hindi. |
COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances across five core NLP tasks are annotating by three bilingual annotators . |
| Approach: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances are annotating by three bilingual annotators . |
| Outcome: | The dataset covers five core NLP tasks, including Token-level Language Identification, Matrix Language Identification and Named Entity Recognition. |
Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to effectively retain and reason about temporal information remains limited. |
| Approach: | They propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. |
| Outcome: | The proposed methods improve performance and reduce incorrect outputs. |
Cross-lingual Editing in Multilingual Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models editing techniques (METs) can efficiently update outdated LLMs without retraining. |
| Approach: | They propose a cross-lingual model editing paradigm where a fact is edited in one language and the subsequent update propagation is observed across other languages. |
| Outcome: | The proposed techniques perform well in multilingual models with knowledge stored in multiple languages. |
PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | *HumanEval* and *MBPP* are two popular benchmarks for Python code generation. |
| Approach: | They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels. |
| Outcome: | The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. |
Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past decade. |
| Approach: | They propose a benchmark dataset for evaluating the problem solving abilities of large language models (LLMs) they curate 515 challenging problems from the highly competitive IIT JEE-Advanced exam. |
| Outcome: | The proposed model performs better on open-source and proprietary models than the current model, but with techniques like self-consistency, self-refinement and chain-of-thought prompting. |
Beyond Monolingual Assumptions: A Survey on Code-Switched NLP in the Era of Large Language Models across Modalities (2026.acl-long)
Copied to clipboard
| Challenge: | Amidst the rapid advances of large language models, most LLMs struggle with mixed-language inputs, limited Code-switching datasets, and evaluation biases. |
| Approach: | They propose a roadmap for inclusive datasets, fair evaluation, and linguistically grounded models to achieve truly multilingual intelligence. |
| Outcome: | The proposed frameworks are based on 327 studies spanning five research areas, 15+ NLP tasks, 30+ datasets, and 80+ languages. |
Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialogues (2022.lrec-1)
Copied to clipboard
Sandhya Singh, Prapti Roy, Nihar Sahoo, Niteesh Mallela, Himanshu Gupta, Pushpak Bhattacharyya, Milind Savagaonkar, Nidhi Sultan, Roshni Ramnani, Anutosh Maitra, Shubhashis Sengupta
| Challenge: | Movies reflect society and also hold power to transform opinions. |
| Approach: | They propose to annotate movie scripts for identity bias using a dataset that is annotated for gender, race/ethnicity, religion, age, occupation, LGBTQ, and other . |
| Outcome: | The proposed dataset contains dialogue turns annotated for gender, race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains biases like body shaming, personality bias, etc. |
UnityAI Guard: Pioneering Toxicity Detection Across Low-Resource Indian Languages (2025.emnlp-demos)
Copied to clipboard
Himanshu Beniwal, Reddybathuni Venkat, Rohit Kumar, Birudugadda Srivibhav, Daksh Jain, Pavan Deekshith Doddi, Eshwar Dhande, Adithya Ananth, null Kuldeep, Mayank Singh
| Challenge: | Existing systems target high-resource languages, but UnityAI-Guard addresses this gap by developing state-of-the-art models for binary toxicity classification targeting low-resourced Indian languages. |
| Approach: | They propose a framework for binary toxicity classification targeting low-resource Indian languages. |
| Outcome: | The proposed framework achieves an impressive average F1-score of 84.23% across seven languages, leveraging a dataset of 567k training instances and 30k manually verified test instances. |