Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data (P18-1)
Copied to clipboard
| Challenge: | Code-mixed (CM) language training is a difficult problem because of lack of data and the increased confusability due to the presence of more than one language. |
| Approach: | They propose a computational technique for creating grammatically valid artificial CM data based on the Equivalence Constraint Theory. |
| Outcome: | The proposed method reduces the perplexity of the model and does not reduce the perceptibility of the models. |
Similar Papers
GCM: A Toolkit for Generating Synthetic Code-mixed Text (2021.eacl-demos)
Copied to clipboard
| Challenge: | Code-mixing is a spoken language phenomenon and is difficult to train in multilingual communities. |
| Approach: | They propose a tool that can automatically generate code-mixed data given parallel data in two languages. |
| Outcome: | The proposed tool can generate code-mixed data in two languages using two linguistic theories. |
A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to train neural network-based models for code-mixing are limited due to language specificity of code-mixed text. |
| Approach: | They propose a deep learning approach to generate code-mixed text from English to multiple languages without any parallel data. |
| Outcome: | The proposed approach generates a code-mixed text from English to multiple languages without any parallel data. |
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (D18-1)
Copied to clipboard
| Challenge: | Using recurrent neural networks to build language models for code-switched text is an important problem with implications to downstream applications such as speech recognition and machine translation. |
| Approach: | They propose a novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately and a generative model estimated using the training data. |
| Outcome: | The proposed techniques yield significant reductions in perplexity on Mandarin-English task and improve on baseline models. |
Minority Positive Sampling for Switching Points - an Anecdote for the Code-Mixing Language Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Multilingual people code-mix using English phonetic typing and insertion of anglicisms in their native language. |
| Approach: | They propose to use minority positive sampling to selectively induce more sample to achieve better performance. |
| Outcome: | The proposed model performs better than other models, but switching points are the main challenge . |
Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien (2022.findings-emnlp)
Copied to clipboard
| Challenge: | CM is a challenging task when mixed languages include dialects. |
| Approach: | They propose to construct a Hokkien-Mandarin CM dataset to overcome the limitation . they propose to use a linguistics-based toolkit to train the model for translation tasks . |
| Outcome: | The proposed model achieves good results on CM data translation while maintaining monolingual translation quality. |
Processing and Understanding Mixed Language Data (D19-2)
Copied to clipboard
| Challenge: | Multilingual communities exhibit code-mixing, mixing of two or more languages in a single conversation . social media and other informal interactive platforms are allowing code-switching in user-generated text . |
| Approach: | a tutorial aims to provide a foundation for researchers to study code-mixing in multilingual communities. |
| Outcome: | a tutorial aims to provide new researchers with a foundation in linguistics and computational aspects of code-mixing. |
The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study examines the effects of training language models on synthetic data generated by their predecessors. |
| Approach: | They propose to use recursive finetuning techniques to assess linguistic diversity of models. |
| Outcome: | The proposed metrics show a decrease in diversity of model outputs through successive iterations, especially for tasks demanding high levels of creativity. |
Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for data mixture improve the generalization capability of large language models (LLMs) on downstream tasks. |
| Approach: | They propose a fine-grained categorization of existing methods and propose three subtypes of offline and online methods. |
| Outcome: | The proposed methods extend beyond offline and online classifications and highlight key challenges in the field of data mixture. |
Scaling Data-Constrained Language Models with Synthetic Data (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) improve with more training data, but practical limitations on data collection constrain further scaling. |
| Approach: | They compare three strategies to generate Japanese text, repeat the limited Japanese Web text, and use English Web text to fill the data shortfall. |
| Outcome: | The proposed model outperforms baselines and achieves the performance achieved when the entire token budget is filled with additional organic Japanese Web text. |
Synthetic Data in the Era of Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | 'synthetic data' is a data generated with the assistance of large language models to make dataset construction faster and cheaper. |
| Approach: | This tutorial seeks to build a shared understanding of recent progress in synthetic data generation from NLP and related fields by grouping and describing major methods, applications, and open problems. |
| Outcome: | This tutorial will describe methods, applications, and open problems that have been developed and are being used to improve the quality and efficiency of synthetic data generation. |