| Challenge: | uroman converts text in Chinese, Arabic and Cyrillic into a common Latin-script representation . the tool uses string similarity metrics to compare text from different scripts . |
| Approach: | They propose a tool that converts text in Chinese, Arabic and Cyrillic into a common Latin-script representation. |
| Outcome: | uroman converts text in Chinese, Arabic and Cyrillic into a common Latin-script representation . the tool is available as a Perl script and as an interactive demo web page . |
Similar Papers
Romanization-based Large-scale Adaptation of Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models are limited by their vocabulary size and parameter budget. |
| Approach: | They explore the potential of leveraging transliteration on a massive scale to improve performance for multilingual pretrained language models. |
| Outcome: | The proposed transliteration tool outperforms other methods on low-resource languages. |
MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool (P19-3)
Copied to clipboard
| Challenge: | In this paper, we demonstrate the online demo and command line interface of a lexical normalization system (MoNoise) for a variety of languages. |
| Approach: | They propose to bundle seven datasets in six languages to form a new benchmark and a novel evaluation metric which is particularly suitable for cross-dataset comparisons. |
| Outcome: | The proposed model is based on the original word and features from the original language for each normalization candidate. |
Unicode Normalization and Grapheme Parsing of Indic Languages (2024.lrec-main)
Copied to clipboard
Nazmuddoha Ansary, Quazi Adibur Rahman Adib, Tahsin Reasat, Asif Shahriyar Sushmit, Ahmed Imtiaz Humayun, Sazia Mehnaz, Kanij Fatema, Mohammad Mamun Or Rashid, Farig Sadeque
| Challenge: | Indic writing systems encode words as linear sequences of Unicode characters . authors propose a grapheme parser for Abugida text to normalize inconsistencies . |
| Approach: | They propose a normalizer for normalizing inconsistencies caused by Unicode encoding schemes . grapheme parser for Abugida deconstructs words into visually distinct orthographic syllables . |
| Outcome: | The proposed library is more efficient than the previously used IndicNLP normalizer . it deconstructs words into visually distinct orthographic syllables or complex graphemes . |
Finite-state script normalization and processing utilities: The Nisaba Brahmic library (2021.eacl-demos)
Copied to clipboard
| Challenge: | a library for low-level processing of brahmic scripts is available for free. |
| Approach: | They propose an open-source library for efficient low-level processing of ten major South Asian Brahmic scripts. |
| Outcome: | The proposed library supports low-level processing of ten major south Asian Brahmic scripts. |
From FreEM to D’AlemBERT: a Large Corpus and a Language Model for Early Modern French (2022.lrec-1)
Copied to clipboard
Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, Benoît Sagot
| Challenge: | Anguage models for historical states of language are becoming more complex to process and more scarce in the corpora available. |
| Approach: | They propose to use a contextualised language model to analyse historical states of language in French. |
| Outcome: | The proposed model is based on a corpus of historical texts and is evaluated with an NLP task. |
Criteria for Useful Automatic Romanization in South Asian Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | a number of possible criteria for systems that transliterate South Asian languages are considered . romanization is the special case where the target script is the Latin script. |
| Approach: | They propose a set of criteria for systems that transliterate South Asian languages . criteria include fidelity to human linguistic behavior, processing utility for people, invertibility . they then propose several algorithms that address different criteria . |
| Outcome: | The proposed algorithms address linguistic considerations in the context of Brahmic scripts and languages that use them, such as Hindi and Malayalam. |
Scripts Through Time: A Survey of the Evolving Role of Transliteration in NLP (2026.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual transfer is often hindered by the "script barrier" where differences in writing systems inhibit transfer learning . transliteration is a powerful technique to bridge this gap by increasing lexical overlap . authors present a taxonomy of key motivations to utilize transliterations in language models . |
| Approach: | They propose a taxonomy of key motivations to utilize transliterations in NLP . they analyze the evolution and effectiveness of these methods and discuss trade-offs . |
| Outcome: | The proposed transliteration technique is effective in cross-lingual NLP, the authors argue . the proposed translliteration method is a powerful tool to overcome the "script barrier" |
Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities (2022.lrec-1)
Copied to clipboard
| Challenge: | a brahmic script is used to record endangered languages such as Dogri and Bengali for low-resource languages such that do not require visual normalization. |
| Approach: | They propose to extend Brahmic script functionality within the Nisaba library of finite-state script normalization and processing utilities. |
| Outcome: | The proposed extensions extend coverage from the original ten scripts to an additional ten of South Asia and beyond, including some used to record endangered languages such as Dogri. |
A Romanization System and WebMAUS Aligner for Arabic Varieties (2022.lrec-1)
Copied to clipboard
Jalal Al-Tamimi, Florian Schiel, Ghada Khattab, Navdeep Sokhey, Djegdjiga Amazouz, Abdulrahman Dallak, Hajar Moussa
| Challenge: | The WebMAUS 1 is a suite of webservices that is free for academic users that processes 42 languages and language varieties. |
| Approach: | They propose to develop an Arabic variety-independent romanization system that aims to homogenize and simplify the romanization of the Arabic script. |
| Outcome: | The proposed system is based on the existing Arabic variety-independent WebMAUS services. |
Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities (2023.acl-long)
Copied to clipboard
| Challenge: | linguistically under-represented communities have an extraordinary opportunity to create content in their native languages. |
| Approach: | They propose to solve the problem of script normalization for languages written in a Perso-Arabic script and use a transformer-based model to analyze the noise levels. |
| Outcome: | The proposed model can normalize a language written in a Perso-Arabic script and improve machine translation and language identification tasks. |