| Challenge: | Existing research on word vectors for English focuses on decomposing words into subword units and using subwords to improve performance. |
| Approach: | They propose to decompose Korean words into the jamo-level, beyond the character-level . they develop Korean test sets for word similarity and analogy and make them publicly available . |
| Outcome: | The proposed method outperforms word2vec and character-level skip-grams on similarity and analogy tasks and contributes positively toward downstream NLP tasks such as sentiment analysis. |
Similar Papers
Don’t Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja (D19-1)
Copied to clipboard
| Challenge: | Existing knowledge of Korean and Chinese is based on cultural and historical reasons. |
| Approach: | They propose a method for improving Korean word representations using additional linguistic annotation by leveraging the fact that Hanja is closely related to Chinese. |
| Outcome: | The proposed approach improves representations on a novel Korean news headline generation task. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
Parameter-Efficient Korean Character-Level Language Modeling (2023.eacl-main)
Copied to clipboard
| Challenge: | Character-level language modeling has been shown to perform well on highly agglutinative or morphologically rich languages while using only a small fraction of the parameters required by (sub)word models. |
| Approach: | They propose a “three-hot” embedding and decoding scheme that exploits the decomposability of Korean characters to model at the syllable level but using only jamo-level representations. |
| Outcome: | The proposed model reduces the embedding parameters by 99.6% and does not lose translation quality compared to the baseline model. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Learning Word Vectors for 157 Languages (L18-1)
Copied to clipboard
| Challenge: | Distributed word representations, or word vectors, have been used in natural language processing for many tasks. |
| Approach: | They propose to use the encyclopedia Wikipedia and the common crawl corpus to train distributed word representations on large corpora and use them in downstream tasks. |
| Outcome: | The proposed model performs very well on 10 languages for which evaluation dataset exists. |
Generalizing Word Embeddings using Bag of Subwords (D18-1)
Copied to clipboard
| Challenge: | Existing word embeddings techniques have a fixed vocabulary, i.e., they can only provide vectors over a finite set of common words that appear frequently in a given corpus. |
| Approach: | They propose a subword-level word vector generation model that views words as bags of character n-grams and provides good vectors for rare or unseen words. |
| Outcome: | The proposed model performs state-of-the-art in English word similarity task and in joint prediction of part-of speech tag and morphosyntactic attributes in 23 languages. |
Rich Character-Level Information for Korean Morphological Analysis and Part-of-Speech Tagging (C18-1)
Copied to clipboard
| Challenge: | Korean is a highly agglutinative, character-rich language, requiring dictionary-less morphological analysis . a novel model can perform morphology and part-of-speech tagging without prior knowledge . |
| Approach: | They propose a multi-stage action-based model that performs morphological transformation and part-of-speech tagging using arbitrary units of input. |
| Outcome: | The proposed model achieves state-of-the-art word and sentence-level tagging accuracy with Korean corpus. |
Semantic Specialization of Distributional Word Vectors (D19-2)
Copied to clipboard
| Challenge: | Distributional word vectors conflate various paradigmatic and syntagmatic lexico-semantic relations. |
| Approach: | This tutorial provides an overview of specialization methods for distributional word vectors . a common solution is to include external lexico-semantic knowledge in a reshaped vector space . |
| Outcome: | This paper provides an overview of specialization methods for distributional word vectors . the most recent developments include a new method for asymmetric relations in Euclidean . |
Gating Mechanisms for Combining Character and Word-level Word Representations: an Empirical Study (N19-3)
Copied to clipboard
| Challenge: | Existing studies show that combining character and word-level representations improves word and sentence representations . however, word-based embeddings do not account for derivational processes resulting in syntactically-similar words with different meanings. |
| Approach: | They propose to combine character and word-level representations to improve word and sentence representations. |
| Outcome: | The proposed method performed well in several word similarity datasets. |
Jamo Pair Encoding: Subcharacter Representation-based Extreme Korean Vocabulary Compression for Efficient Subword Tokenization (2020.lrec-1)
Copied to clipboard
| Challenge: | a vocabulary size problem is unsolved in multilingual pre-training tasks. |
| Approach: | They propose two algorithms that increase the elasticity of budget for building the vocabulary in Byte-Pair Encoding inspired tokenizers, and reduce the cost of supporting Korean in a multilingual model. |
| Outcome: | The proposed method reduces the cost of supporting Korean in a multilingual model. |