Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for lexical normalization of social media data are designed with canonical texts in mind, and this makes it difficult to process data in multiple languages. |
| Approach: | They propose to create a lexical normalization dataset for Italian and analyze the inter-annotator agreement for this task. |
| Outcome: | The proposed model improves the parsing of social media data in Italian and shows that it can be used to translate non-standard social media content to canonical language. |
Similar Papers
An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media (D19-55)
Copied to clipboard
| Challenge: | Existing natural language processing tools are focused on standard texts, but performance drops when used on a different domain. |
| Approach: | They analyze the effect of manual and automatic lexical normalization for dependency parsing . they conclude that automatic normalization scores close to manually annotated normalization . |
| Outcome: | The proposed approach improves performance on social media data for many tasks . it is unclear which replacements have the most impact and what weaknesses exist in the system . |
Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data? (2020.lrec-1)
Copied to clipboard
| Challenge: | Social media data is a valuable data resource for natural language processing tasks. |
| Approach: | They propose to adapt input text to a more standard form, a task also referred to as normalization. |
| Outcome: | The proposed system scores 94.29 accuracy on the test data compared to 95.22 when trained on human-annotated data. |
MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool (P19-3)
Copied to clipboard
| Challenge: | In this paper, we demonstrate the online demo and command line interface of a lexical normalization system (MoNoise) for a variety of languages. |
| Approach: | They propose to bundle seven datasets in six languages to form a new benchmark and a novel evaluation metric which is particularly suitable for cross-dataset comparisons. |
| Outcome: | The proposed model is based on the original word and features from the original language for each normalization candidate. |
A Taxonomy for In-depth Evaluation of Normalization for User Generated Content (L18-1)
Copied to clipboard
| Challenge: | Existing taxonomies for lexical normalization are not suitable for the task of normalization since the categories are substantially different. |
| Approach: | They propose a taxonomy of error categories for lexical normalization . they annotate a recent normalization dataset and read a near-perfect agreement . |
| Outcome: | The proposed taxonomy is based on a recent normalization dataset and it performs well. |
Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text normalization is a range of tasks that consist in replacing non-standard spellings with their standard equivalents. |
| Approach: | They introduce dialect-to-standard normalization as a sentence-level character transduction task and provide a large-scale analysis of these methods. |
| Outcome: | The proposed model performs best for Finnish, Swiss German and Slovene while the pre-trained model using full sentences performs the best for Norwegian. |
Normalizing Non-canonical Turkish Texts Using Machine Translation Approaches (P19-2)
Copied to clipboard
| Challenge: | a study using non-canonical text normalization shows that it can surpass the current best performing system by a large margin. |
| Approach: | They propose a fully automated, context-aware machine translation approach with fewer stages of processing. |
| Outcome: | The proposed approach surpasses the current best-performing system by a large margin . the proposed method is more data-hungry and more data sensitive than other methods . |
Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text. |
| Approach: | They focus on Japanese normalization and developing methods based on state-of-the-art pre-trained models . |
| Outcome: | The proposed methods achieve high accuracy and efficiency across multiple evaluation perspectives. |
Lexical Normalization for Code-switched Data and its Effect on POS Tagging (2021.eacl-main)
Copied to clipboard
| Challenge: | Social media data can be used to improve natural language processing performance, but it is often overlooked by lexical normalization systems. |
| Approach: | They propose three lexical normalization models specifically designed to handle code-switched data and evaluate their performance on POS tags. |
| Outcome: | The proposed models outperform monolingual models and lead to 5.4% performance increase for POS tagging compared to unnormalized input. |
Normalization of Indonesian-English Code-Mixed Twitter Data (D19-55)
Copied to clipboard
| Challenge: | Twitter is an excellent source of textual data for NLP researches, but it is noisy and often contains typos, slang terms, and non-standard abbreviations. |
| Approach: | They propose a standardization system for Indonesian-English code-mixed Twitter data that includes tokenization, language identification, lexical normalization, and translation. |
| Outcome: | The proposed standardization system is based on four modules for tokenization, language identification, lexical normalization, and translation. |
ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text (2024.eacl-long)
Copied to clipboard
| Challenge: | Lexical normalization is a fundamental task in Natural Language Processing (NLP) it involves the transformation of words into their canonical forms. |
| Approach: | They present a corpus of Vietnamese words annotated by human annotators for the Vietnamese lexical normalization task. |
| Outcome: | The best-performing system achieved 57.74% using the Error Reduction Rate (ERR) metric with the Leave-As-Is (LAI) baseline. |