Papers by Mayank Singh
The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work does not critically analyze the scientific language models to the best of our knowledge. |
| Approach: | They evaluate scientific language models in handling short-query texts and textual neighbors by leveraging perturbations to generate textual neighbor classes. |
| Outcome: | The proposed model is ineffective for retrieving documents for short-query texts under the most relaxed conditions. |
CL Scholar: The ACL Anthology Knowledge Graph Miner (N18-5)
Copied to clipboard
Mayank Singh, Pradeep Dogga, Sohan Patro, Dhiraj Barnwal, Ritam Dutt, Rajarshi Haldar, Pawan Goyal, Animesh Mukherjee
| Challenge: | ACL Anthology is a repository for papers related to computational linguistics and natural language processing. |
| Approach: | They propose to automate periodic crawling, indexing and processing of new articles . they propose to use CL Scholar to support more than 1200 natural language queries . |
| Outcome: | The proposed system can answer three different types of natural language queries. |
Commentator: A Code-mixed Multilingual Text Annotation Framework (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing annotation tools fail to address multilingual datasets efficiently. |
| Approach: | They introduce a code-mixed multilingual text annotation framework, COMMENTATOR . they perform robust qualitative human-based evaluations to showcase its effectiveness . |
| Outcome: | The proposed framework performs faster than baseline annotations in Hinglish and Hindi. |
The Bull and the Bear: Summarizing Stock Market Discussions (2022.lrec-1)
Copied to clipboard
| Challenge: | a dataset of 7888 reddit posts and 400 posts is used to summarize stock market topics. |
| Approach: | They curate discussions on social media platforms and construct an abstractive summarization dataset. |
| Outcome: | The proposed dataset consists of 7888 Reddit posts and summaries for 400 posts . it is robustly evaluated and will be made publicly available . |
How Robust Are the QA Models for Hybrid Scientific Tabular Data? A Study Using Customized Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing tabular QA models are lacking in understanding their robustness on scientific information. |
| Approach: | They propose a dataset to assess the robustness of tabular QA models on scientific hybrid tabular data. |
| Outcome: | The proposed model performs well on scientific tables and text, while the best score is 0.462. |
COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances across five core NLP tasks are annotating by three bilingual annotators . |
| Approach: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances are annotating by three bilingual annotators . |
| Outcome: | The dataset covers five core NLP tasks, including Token-level Language Identification, Matrix Language Identification and Named Entity Recognition. |
Remember This Event That Year? Assessing Temporal Information and Understanding in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to effectively retain and reason about temporal information remains limited. |
| Approach: | They propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. |
| Outcome: | The proposed methods improve performance and reduce incorrect outputs. |
Cross-lingual Editing in Multilingual Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models editing techniques (METs) can efficiently update outdated LLMs without retraining. |
| Approach: | They propose a cross-lingual model editing paradigm where a fact is edited in one language and the subsequent update propagation is observed across other languages. |
| Outcome: | The proposed techniques perform well in multilingual models with knowledge stored in multiple languages. |
PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | *HumanEval* and *MBPP* are two popular benchmarks for Python code generation. |
| Approach: | They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels. |
| Outcome: | The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. |
MUTANT: A Multi-sentential Code-mixed Hinglish Dataset (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to identify code-mixed text are difficult to scale effectively and efficiently on multi-sentential data. |
| Approach: | They propose to identify multi-sentential code-mixed text (MCT) from multilingual articles using a token-level language-aware pipeline. |
| Outcome: | The proposed dataset includes 67k articles with 85k identified Hinglish MCTs. |
Learning to Generate Rules for Realistic Few-Shot Relation Classification: An Encoder-Decoder Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to relation classification is proposed to use data-driven approaches to perform fewshot tasks with limited training data. |
| Approach: | They propose a neuro-symbolic approach for realistic few-shot relation classification via rules . they propose to generate rules that can be used to extract relations using custom T5-style models . |
| Outcome: | The proposed approach is interpretable and pliable and outperforms the state-of-the-art on TACRED and NYT29 benchmarks while maintaining pliability. |
Grammar Search for Multi-Agent Systems (2026.acl-long)
Copied to clipboard
Mayank Singh, Vikas Yadav, Shiva Krishna Reddy Malay, Shravan Nayak, Sai Rajeswar, Sathwik Tejaswi Madhusudhan, Eduardo Blanco
| Challenge: | Several prior approaches have relied on LLM-based free-form search over the code space. |
| Approach: | They propose a more structured framework that explores the same space through a fixed set of composable components. |
| Outcome: | The proposed framework outperforms existing approaches on most benchmarks across two backbone LLMs and two domains: mathematics and question answering. |
LEGOBench: Scientific Leaderboard Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of papers make it difficult to stay informed about the latest state-of-the-art research. |
| Approach: | They propose a benchmark to evaluate systems that generate scientific leaderboards . they use 22 years of submission data on arXiv and 11k machine learning leaderboard data on paperswithcode . |
| Outcome: | The proposed model shows significant performance gaps in the LEGOBench model . the model is based on a language model and four graph-based leaderboard generation task configuration . |
TweeNLP: A Twitter Exploration Portal for Natural Language Processing (2021.acl-demo)
Copied to clipboard
| Challenge: | Currently, Twitter curates 19,395 tweets from various NLP conferences and general NLP discussions. |
| Approach: | They propose to integrate tweets pertaining to research papers with the NLPExplorer scientific literature search engine to organize Twitter's natural language processing data. |
| Outcome: | The proposed system curates 19,395 tweets from various NLP conferences and general discussions. |
Beyond Monolingual Assumptions: A Survey on Code-Switched NLP in the Era of Large Language Models across Modalities (2026.acl-long)
Copied to clipboard
| Challenge: | Amidst the rapid advances of large language models, most LLMs struggle with mixed-language inputs, limited Code-switching datasets, and evaluation biases. |
| Approach: | They propose a roadmap for inclusive datasets, fair evaluation, and linguistically grounded models to achieve truly multilingual intelligence. |
| Outcome: | The proposed frameworks are based on 327 studies spanning five research areas, 15+ NLP tasks, 30+ datasets, and 80+ languages. |
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-annotator datasets can improve performance when they expand from a single annotation per instance to multiple annotations. |
| Approach: | They propose a multi-annotator simulation process to generate datasets with varying annotation budgets and compare them to a single annotation per instance. |
| Outcome: | The proposed model can generate datasets with varying annotation budgets and show that similar datasets can lead to varying performance gains. |
UnityAI Guard: Pioneering Toxicity Detection Across Low-Resource Indian Languages (2025.emnlp-demos)
Copied to clipboard
Himanshu Beniwal, Reddybathuni Venkat, Rohit Kumar, Birudugadda Srivibhav, Daksh Jain, Pavan Deekshith Doddi, Eshwar Dhande, Adithya Ananth, null Kuldeep, Mayank Singh
| Challenge: | Existing systems target high-resource languages, but UnityAI-Guard addresses this gap by developing state-of-the-art models for binary toxicity classification targeting low-resourced Indian languages. |
| Approach: | They propose a framework for binary toxicity classification targeting low-resource Indian languages. |
| Outcome: | The proposed framework achieves an impressive average F1-score of 84.23% across seven languages, leveraging a dataset of 567k training instances and 30k manually verified test instances. |