An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsing (D18-1)
Copied to clipboard
| Challenge: | Existing studies have shown that character models are less important in the presence of word embeddings, but combining them quickly leads to diminishing returns. |
| Approach: | They propose to combine pre-trained word embeddings, character models and POS tags to improve parsing quality by categorising words by frequency, POS tag and language. |
| Outcome: | The proposed system improves on initialised word embeddings but combines them quickly leads to diminishing returns. |
Similar Papers
Entity or Relation Embeddings? An Analysis of Encoding Strategies for Relation Extraction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to relation extraction use concatenating embeddings of head and tail entities . however, such representations capture the types of the entities involved, leading to false positives and confusion between relations involving entities of the same type. |
| Approach: | They propose a model which combines [MASK] embeddings with entity embedds to learn relation embeddations. |
| Outcome: | The proposed model outperforms the state-of-the-art on several benchmarks . it uses a self-supervised pre-training strategy which further improves the results. |
Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing - A Tale of Two Parsers Revisited (D19-1)
Copied to clipboard
| Challenge: | In recent years, dependency parsing has shifted from discrete features to neural networks and continuous representations. |
| Approach: | They propose to use deep contextualized word embeddings to pack information about global sentence structure into local feature representations to make the two approaches virtually equivalent in terms of accuracy and error profile. |
| Outcome: | The proposed model improves the accuracy and error profile of transition-based and graph-based dependency parsers on 13 languages. |
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens (2022.naacl-main)
Copied to clipboard
| Challenge: | Standard pre-trained language models do not see the characters that compose each token's string representation. |
| Approach: | They probe the embedding layer of pretrained language models and show that models learn the internal character composition of whole word and subword tokens without seeing the characters coupled with the tokens. |
| Outcome: | The embedding layers of RoBERTa and GPT2 hold enough information to accurately spell up to a third of the vocabulary and reach high character ngram overlap across all token types. |
Revisiting Tri-training of Dependency Parsers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained word embeddings and self-training have been used in dependency parsing tasks for years. |
| Approach: | They compare tri-training and pretrained word embeddings in dependency parsing . they use language-specific FastText and ELMo embedds and multilingual BERT embedders . |
| Outcome: | The proposed methods are tri-training and pretrained word embeddings. |
From Text to Lexicon: Bridging the Gap between Word Embeddings and Lexical Resources (C18-1)
Copied to clipboard
| Challenge: | Distributional word representations are omnipresent in modern NLP. |
| Approach: | They propose to combine lemmatization and part of speech (POS) typing to improve word embedding performance. |
| Outcome: | The proposed methods improve word embedding performance on verbs and verbs. |
Word Embeddings for Code-Mixed Language Processing (D18-1)
Copied to clipboard
| Challenge: | Existing bilingual word embedding techniques are not ideal for code-mixed text processing and there is a need for learning multilingual word embeds from code-mixed texts. |
| Approach: | They propose to use bilingual word embedding techniques to train skip-grams on synthetic code-mixed text generated through linguistic models of code- mixing to perform two tasks. |
| Outcome: | The proposed embedding technique performs better on semantic and syntactic tasks than the existing embeddable techniques on sentiment analysis and POS tagging tasks. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
Handling Normalization Issues for Part-of-Speech Tagging of Online Conversational Text (L18-1)
Copied to clipboard
Géraldine Damnati, Jeremy Auguste, Alexis Nasr, Delphine Charlet, Johannes Heinecke, Frédéric Béchet
| Challenge: | a new approach to POS tagging noisy user generated text is proposed . word embeddings are trained on a noisy corpus to address both normalization and POS. |
| Approach: | They propose to use word embeddings to normalize text before tagging it, while a gated neural network based tagger handles the remaining errors. |
| Outcome: | The proposed approach normalizes some errors before tagging, while a gated neural network handles the remaining errors. |
How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions (P19-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings (CLEs) are used for downstream NLP tasks . CLEs are based on bilingual lexicon induction (BLI) evaluations vary greatly, hindering ability to interpret performance and properties of different CLE models. |
| Approach: | They evaluate CLE models for a large number of language pairs on bilingual lexicon induction and three downstream tasks. |
| Outcome: | The proposed model performance is based on supervised and unsupervised models on bilingual lexicon induction and three downstream tasks. |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |