Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text normalization is a range of tasks that consist in replacing non-standard spellings with their standard equivalents. |
| Approach: | They introduce dialect-to-standard normalization as a sentence-level character transduction task and provide a large-scale analysis of these methods. |
| Outcome: | The proposed model performs best for Finnish, Swiss German and Slovene while the pre-trained model using full sentences performs the best for Norwegian. |
Similar Papers
A Large-Scale Comparison of Historical Text Normalization Systems (N19-1)
Copied to clipboard
| Challenge: | a large study of historical text normalization is done on eight languages . there is no consensus on the state-of-the-art approach to normalization . |
| Approach: | They present a large study of historical text normalization done on eight languages . they evaluate four different systems based on supervised learning on datasets from eight different languages based in the literature . |
| Outcome: | The proposed methods are based on supervised learning and are available online. |
Dialect Normalization using Large Language Models and Morphological Rules (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural language understanding systems struggle with low-resource languages, including many dialects of high-resourced ones. |
| Approach: | They propose a method that combines rule-based linguistically informed transformations and large language models with targeted few-shot prompting without any parallel data. |
| Outcome: | The proposed method is able to transform dialectal text into a standard variety while maintaining as much of the original meaning as possible. |
Text Normalization Infrastructure that Scales to Hundreds of Language Varieties (L18-1)
Copied to clipboard
| Challenge: | a multi-language text normalization infrastructure is used to train language models for keyboards and speech recognition systems. |
| Approach: | They describe a multi-language text normalization infrastructure that prepares textual data to train language models used in Google's keyboards and speech recognition systems. |
| Outcome: | The proposed system can normalize training data across hundreds of languages . it can detect errors in training data and detect corruption issues . |
Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects (2026.acl-long)
Copied to clipboard
| Challenge: | Research on cross-dialectal transfer from a standard to a non-standard dialect variety has typically focused on text data. |
| Approach: | They compare standard-to-dialect transfer in three settings: text models, speech models, and cascaded systems where speech first gets automatically transcribed and then further processed by a text model. |
| Outcome: | The proposed model performs best on German dialect data while the text-only model perform best on the standard data. |
Proteno: Text Normalization with Limited Data for Fast Deployment in Text to Speech Systems (2021.naacl-industry)
Copied to clipboard
| Challenge: | Developing Text Normalization systems for Text-to-Speech (TTS) on new languages is hard. |
| Approach: | They propose a novel architecture to facilitate Text Normalization systems for TTS on new languages . they use a granular tokenization mechanism that enables the system to learn majority of classes . |
| Outcome: | The proposed architecture performs comparable with the state-of-the-art systems on English . the proposed system learns most classes from training data and precodes them for other classes . |
MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool (P19-3)
Copied to clipboard
| Challenge: | In this paper, we demonstrate the online demo and command line interface of a lexical normalization system (MoNoise) for a variety of languages. |
| Approach: | They propose to bundle seven datasets in six languages to form a new benchmark and a novel evaluation metric which is particularly suitable for cross-dataset comparisons. |
| Outcome: | The proposed model is based on the original word and features from the original language for each normalization candidate. |
Lexical Normalization for Code-switched Data and its Effect on POS Tagging (2021.eacl-main)
Copied to clipboard
| Challenge: | Social media data can be used to improve natural language processing performance, but it is often overlooked by lexical normalization systems. |
| Approach: | They propose three lexical normalization models specifically designed to handle code-switched data and evaluate their performance on POS tags. |
| Outcome: | The proposed models outperform monolingual models and lead to 5.4% performance increase for POS tagging compared to unnormalized input. |
A Taxonomy for In-depth Evaluation of Normalization for User Generated Content (L18-1)
Copied to clipboard
| Challenge: | Existing taxonomies for lexical normalization are not suitable for the task of normalization since the categories are substantially different. |
| Approach: | They propose a taxonomy of error categories for lexical normalization . they annotate a recent normalization dataset and read a near-perfect agreement . |
| Outcome: | The proposed taxonomy is based on a recent normalization dataset and it performs well. |
Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text. |
| Approach: | They focus on Japanese normalization and developing methods based on state-of-the-art pre-trained models . |
| Outcome: | The proposed methods achieve high accuracy and efficiency across multiple evaluation perspectives. |
Normalizing Non-canonical Turkish Texts Using Machine Translation Approaches (P19-2)
Copied to clipboard
| Challenge: | a study using non-canonical text normalization shows that it can surpass the current best performing system by a large margin. |
| Approach: | They propose a fully automated, context-aware machine translation approach with fewer stages of processing. |
| Outcome: | The proposed approach surpasses the current best-performing system by a large margin . the proposed method is more data-hungry and more data sensitive than other methods . |