| Challenge: | Existing methods to identify code-mixed text are difficult to scale effectively and efficiently on multi-sentential data. |
| Approach: | They propose to identify multi-sentential code-mixed text (MCT) from multilingual articles using a token-level language-aware pipeline. |
| Outcome: | The proposed dataset includes 67k articles with 85k identified Hinglish MCTs. |
Similar Papers
MUTANT: A Recipe for Multilingual Tokenizer Design (2026.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization schemes such as Byte Pair Encoding (BPE) are widely adopted, but their effectiveness in multilingual settings remains understudied. |
| Approach: | They propose a multilingual tokenizer that produces linguistically coherent tokens for multilingual LLMs. |
| Outcome: | The proposed tokenizer improves fertility score by 39.5% over LLaMA4 and 18% over Sutra. |
MHE: Code-Mixed Corpora for Similar Language Identification (2022.lrec-1)
Copied to clipboard
| Challenge: | a new corpus of code-mixed data-sets is presented for similar language identification . the data-settings are based on a more-resourced minority language, Magahi . |
| Approach: | They propose a Magahi-Hindi-English code-mixed corpus for similar language identification . they discuss the complexity of the data-set and provide a few baselines . |
| Outcome: | The proposed corpus provides a language id at two levels: word and sentence. |
Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | a number of languages are used in online conversations, resulting in code-mixing . the problem is largely unexplored due to the lack of annotated data and noise . |
| Approach: | They propose a robust perturbation-based joint-training model that learns to handle noise in code-mixed text by parameter sharing across clean and noisy words. |
| Outcome: | The proposed model learns to handle noise in the real-world code-mixed text by parameter sharing across clean and noisy words. |
Processing and Understanding Mixed Language Data (D19-2)
Copied to clipboard
| Challenge: | Multilingual communities exhibit code-mixing, mixing of two or more languages in a single conversation . social media and other informal interactive platforms are allowing code-switching in user-generated text . |
| Approach: | a tutorial aims to provide a foundation for researchers to study code-mixing in multilingual communities. |
| Outcome: | a tutorial aims to provide new researchers with a foundation in linguistics and computational aspects of code-mixing. |
MaCmS: Magahi Code-mixed Dataset for Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Sociolinguists and psychologists have been studying these variations in the lexicons and the language from the 50's . code-mixing is a popular method for understanding people's emotions and attitudes towards various subjects, but low-resourced languages often have a mix of scripts and languages. |
| Approach: | They introduce a new sentiment data, MaCMS, for Magahi-Hindi-English code-mixed language, where Magai is a less-resourced minority language. |
| Outcome: | The proposed dataset is the first Magahi-Hindi-English code-mixed dataset for sentiment analysis tasks. |
COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances across five core NLP tasks are annotating by three bilingual annotators . |
| Approach: | COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset . 125K+ high-quality instances are annotating by three bilingual annotators . |
| Outcome: | The dataset covers five core NLP tasks, including Token-level Language Identification, Matrix Language Identification and Named Entity Recognition. |
Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien (2022.findings-emnlp)
Copied to clipboard
| Challenge: | CM is a challenging task when mixed languages include dialects. |
| Approach: | They propose to construct a Hokkien-Mandarin CM dataset to overcome the limitation . they propose to use a linguistics-based toolkit to train the model for translation tasks . |
| Outcome: | The proposed model achieves good results on CM data translation while maintaining monolingual translation quality. |
A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to train neural network-based models for code-mixing are limited due to language specificity of code-mixed text. |
| Approach: | They propose a deep learning approach to generate code-mixed text from English to multiple languages without any parallel data. |
| Outcome: | The proposed approach generates a code-mixed text from English to multiple languages without any parallel data. |
SyMCoM - Syntactic Measure of Code Mixing A Study Of English-Hindi Code-Mixing (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work on code mixing in computational settings has leveraged social media code mixed texts to train NLP models. |
| Approach: | They propose to use language ID tags to measure syntactic variety in code-mixed text and their relationship with computational model performance. |
| Outcome: | The proposed measure can be applied to English(en)-hindi(hi) code-mixed datasets and compares them with other measures. |
Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues (2024.findings-eacl)
Copied to clipboard
| Challenge: | blending multiple languages within a single conversation presents a formidable challenge, given the wide-ranging variations influenced by individual speaking styles and cultural backgrounds. |
| Approach: | They propose a novel approach to harness the Big Five personality traits acquired in an unsupervised manner from code-mixed conversations to bolster the performance of response generation. |
| Outcome: | The proposed approach enhances contextual relevance and performance of the proposed model by combining personality traits with dialogue context. |