Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging (2022.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks, but they fail to achieve state-of-the-art (SOTA) performance. |
| Approach: | They propose a Guassian HMM variant for unsupervised POS tagging that incorporates contexualized word representations into the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art models on Penn Treebank and multilingual Universal Dependencies treebank v2.0. |
Similar Papers
Unsupervised Cross-Lingual Part-of-Speech Tagging for Truly Low-Resource Scenarios (2020.emnlp-main)
Copied to clipboard
| Challenge: | a limited set of translations into one or more high-resource languages are available for POS tagging . a bi-LSTM architecture that uses contextualized word embeddings improves performance . |
| Approach: | They propose an unsupervised cross-lingual transfer approach for part-of-speech tagging . they use the Bible as parallel data to learn POS taggers for target languages . |
| Outcome: | The proposed approach improves accuracy on 12 diverse languages . the Bible is used as a parallel corpus for the study . |
A Grounded Unsupervised Universal Part-of-Speech Tagger for Low-Resource Languages (N19-1)
Copied to clipboard
| Challenge: | Unsupervised part of speech (POS) tagging is often framed as a clustering problem, but taggers need to ground their clusters as well. |
| Approach: | They propose an approach for low-resource unsupervised part of speech (POS) tagging that yields fully grounded output and requires no labeled training data. |
| Outcome: | The proposed method achieves reasonable performance across languages, including Sinhalese and Kinyarwanda, with no labeled training data. |
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent Part-Of-Speech (POS) induction models assume certain independence assumptions that do not hold in real languages. |
| Approach: | They propose a Masked Part-of-Speech Model (MPoSM) that can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
| Outcome: | The proposed model can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
Neural Factor Graph Models for Cross-lingual Morphological Tagging (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to morphological tagging are limited by the assumption that tag sets overlap . a limited amount of data is available for most languages to learn these morphology taggers. |
| Approach: | They propose a method for cross-lingual morphological tagging that relaxes this assumption . they use factorial conditional random fields with neural network potentials to smooth over superficial differences in the surface forms . |
| Outcome: | The proposed model can smooth over superficial differences in the surface forms and generate unseen or rare tag sets. |
Generating Datasets with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles. |
| Approach: | They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models. |
| Outcome: | The proposed approach outperforms baselines on several semantic textual similarity datasets. |
Unsupervised Cross-Lingual Adaptation of Dependency Parsers Using CRF Autoencoders (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on cross-lingual adaptation of dependency parsers without annotated target corpora focuses on discriminative source parser ignoring unannotated corporata . |
| Approach: | They propose to use unsupervised discriminative parsers to adapt dependency parser to unannotated target corpora without a supervised generative parsing method. |
| Outcome: | The proposed method significantly outperforms previous methods. |
XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-training language models have been successful in natural language understanding and autoregressive generation tasks, but non-autoregressive models have not been sufficiently successful. |
| Approach: | They propose a pre-trained masked language model (MLM) and a non-autoregressive generation model with a lightweight decorator. |
| Outcome: | The proposed model outperforms the previous mask-predict model on translation datasets by 19.9x. |
LaoPLM: Pre-trained Language Models for Lao (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) can capture different levels of concepts in context . previous work on Lao has been hampered by the lack of annotated datasets . |
| Approach: | They construct a text classification dataset to alleviate the resource-scarce situation of Lao . they evaluate them on two downstream tasks: part-of-speech tagging and text classification . |
| Outcome: | The proposed model can capture different levels of concepts in context and generate universal language representations. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
A Fine-Grained Domain Adaption Model for Joint Word Segmentation and POS Tagging (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that joint models of word segmentation and POS tagging can lead to better performance because they are closely related. |
| Approach: | They propose a domain adaption method for Chinese word segmentation and POS tagging that uses a simple metric to model the gaps between target and target domains. |
| Outcome: | The proposed method can gain significant performance improvements over baselines on a benchmark dataset. |