Bits and Pieces: Investigating the Effects of Subwords in Multi-task Parsing across Languages and Domains (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural parsing is dependent on the underlying language model, but little is known about how choices affect parser performance. |
| Approach: | They examine how subword sharing is responsible for gains or negative transfer in multi-task learning . they find a preference for averaged or last subwords across languages and domains . |
| Outcome: | The proposed model favors averaged or last subwords across languages and domains . specific POS tags may require different subword, and distribution overlap is more important than discrepancies in the data sizes. |
Similar Papers
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
Subword Pooling Makes a Difference (2021.eacl-main)
Copied to clipboard
| Challenge: | Contextual word-representations use subword tokenization to handle large vocabularies and unknown words. |
| Approach: | They propose to use the first subword for morphological probing, POS tagging and NER to pool multiple subwords that correspond to a single word in contextual language models. |
| Outcome: | The proposed model outperforms two multilingual models on morphological probing, POS tagging and NER tasks in 9 languages. |
Do Text-to-Text Multi-Task Learners Suffer from Task Conflict? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-task learning architectures learn a single model across multiple tasks through a shared encoder followed by task-specific decoders. |
| Approach: | They propose to use a shared encoder and language model decoder to learn a single model across multiple tasks. |
| Outcome: | The proposed architecture does surprisingly well across a range of diverse tasks. |
Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior work on cross-lingual transfer often focuses on a small set of languages from a few language families and/or a single task. |
| Approach: | They analyze cross-lingual transfer for 263 languages from a wide variety of language families . they include three popular NLP tasks: POS tagging, dependency parsing, topic classification . |
| Outcome: | The proposed approach is based on linguistic similarity measures for 263 languages . the results show that the effect of linguistic similarities on transfer performance depends on a range of factors . |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Subword segmentation algorithms can produce sub-optimal segmentation when the target language is rich in morphological changes or there is not enough data for learning compact composition rules. |
| Approach: | They compare character-based and subword-based neural machine translation systems . they find character-driven models are better at handling morphological phenomena . |
| Outcome: | The character-based models are better at handling morphological phenomena, generating rare and unknown words, and more suitable for transferring to unseen domains. |
Subword models struggle with word learning, but surprisal hides it (2025.acl-short)
Copied to clipboard
| Challenge: | Subword LMs struggle to discern words and non-words with high accuracy, character LM models do this easily and consistently. |
| Approach: | They propose to model word learning in subword and character language models with the psycholinguistic lexical decision task. |
| Outcome: | The results suggest that word learning and syntactic learning are separable in character LMs. |
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |
Reusing Weights in Subword-Aware Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | a statistical language model assigns a probability to a sequence of words . data sparsity is a major problem in building traditional n-gram language models . |
| Approach: | They propose several ways to reuse subword embeddings and other weights in subword-aware neural language models. |
| Outcome: | The proposed techniques do not benefit a competitive character-aware model . but they show significant reductions in model sizes and performance. |
Subword-Delimited Downsampling for Better Character-Level Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword-level models are expensive in terms of time and computation, but character-level model with downsampling component can be used for machine translation. |
| Approach: | They propose a character-level downsampling method which is informed by subwords to improve model performance. |
| Outcome: | The proposed method outperforms existing methods and shows that it can be done without sacrificing quality. |