| Challenge: | morphological complexity is still a major challenge for NLP and the study of language. |
| Approach: | They perform a cross-linguistic comparison following incremental merges of BPE for 47 diverse languages. |
| Outcome: | The results show that language distributions are similar under specific levels of tokenization. |
Similar Papers
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Multilingual modelling can improve machine translation for low-resource languages, partly through shared subword representations. |
| Approach: | They propose to use subword regularisation to promote synergy and BPE to facilitate cross-lingual transfer. |
| Outcome: | The proposed methods promote synergy and prevent interference across different linguistic typologies. |
X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs (2024.naacl-long)
Copied to clipboard
| Challenge: | X-parade is the first cross-lingual dataset of paragraph-level information divergences and entailments . ability to recognize differences in meaning underlies many NLP tasks . |
| Approach: | They propose a cross-lingual dataset of paragraph-level information divergences . they use a dictionary to identify new or inferred information in Wikipedia pages . |
| Outcome: | The proposed dataset shows that the proposed methods fail to handle inferable information . the dataset contains fine-grained span-level annotations for content in different languages . |
Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies using LLMs on psycholinguistic data have gone unverified . a growing body of research is using word-level prediction as a computational proxy . |
| Approach: | They compare morphological, morphologic, and BPE tokenization estimates with reading time data. |
| Outcome: | The proposed method could be used to evaluate morphological prediction. |
Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are powerful representations that form the foundation of many natural language processing architectures. |
| Approach: | They explore word embedding stability in a wide range of languages to gain insight into their stability. |
| Outcome: | The proposed results provide insights into word embedding stability in English and other languages. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown that contextualized word representations are a viable alternative to simple word prediction tasks. |
| Approach: | They propose to use subword units and characters to model morphology, syntax, and semantics instead of word embeddings. |
| Outcome: | The proposed representations are better for modeling syntax and more robust to noisy input. |
An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers (2021.acl-short)
Copied to clipboard
| Challenge: | Existing word-level quality estimation models require labelled data for each language pair and expensive maintenance. |
| Approach: | They propose to use multilingual QE models to generalise across languages . they propose to train models on other language pairs to predict word-level quality . |
| Outcome: | The proposed models generalise well across languages, making them more useful in real-world scenarios. |
Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that token overlap is a strong predictor of multilinguality and cross-lingual knowledge transfer between languages with different scripts. |
| Approach: | They propose a subword token alignability metric to understand the impact and quality of multilingual tokenisation. |
| Outcome: | The proposed metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. |
Investigating the Effectiveness of BPE: The Power of Shorter Sequences (D19-1)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, but its reasons for its effectiveness are not well understood. |
| Approach: | They link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family. |
| Outcome: | The proposed method is compared with dictionary-based compression algorithms and improves on a fixed vocabulary size budget. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |