Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging (N19-1)
Copied to clipboard
| Challenge: | Character-level models of tokens are effective at dealing with within-token noise and out-of-vocabulary words. |
| Approach: | They propose to eliminate the need for tokenizers by using a character-level semi-Markov conditional random field that uses neural networks for its character and segment representations. |
| Outcome: | The proposed model outperforms state-of-the-art part-of speech taggers on a noisy English dataset. |
Similar Papers
Hybrid semi-Markov CRF for Neural Sequence Labeling (P18-2)
Copied to clipboard
| Challenge: | Existing conditional random fields (CRFs) use hand-crafted features to perform sequence labeling tasks. |
| Approach: | They propose to use semi-Markov conditional random fields for neural sequence labeling in natural language processing to extract features from segments instead of words. |
| Outcome: | The proposed model achieves state-of-the-art when no external knowledge is used. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Stochastic Tokenization with a Language Model for Neural Text Classification (P19-1)
Copied to clipboard
| Challenge: | Sentences segmented with words or subwords can be difficult to perform text classification tasks. |
| Approach: | They propose a method to learn tokenization and text classification simultaneously to address these problems. |
| Outcome: | The proposed method improves on sentiment analysis in Japanese and Chinese using tokenization and text classification models. |
Filtered Semi-Markov CRF (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for sequence labeling tasks such as Named Entity Recognition (NER) suffer from quadratic complexity over sequence length and poor performance compared to CRF. |
| Approach: | They propose a variant of Semi-Markov CRF that incorporates a filtering step to eliminate irrelevant segments, reducing complexity and search space. |
| Outcome: | The proposed method outperforms both CRF and Semi-CRF on several NER benchmarks while being significantly faster. |
Sub-Character Tokenization for Chinese Pretrained Language Models (2023.tacl-1)
Copied to clipboard
Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun
| Challenge: | Existing tokenization methods for Chinese PLMs treat each character as an indivisible token, but ignore the unique feature of the writing system where additional linguistic information exists below the character level. |
| Approach: | They propose to encode Chinese characters into short sequences and construct Chinese vocabulary based on the encoded text. |
| Outcome: | The proposed tokenizers can tokenize inputs into much shorter sequences, improving computational efficiency. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging (2022.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks, but they fail to achieve state-of-the-art (SOTA) performance. |
| Approach: | They propose a Guassian HMM variant for unsupervised POS tagging that incorporates contexualized word representations into the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art models on Penn Treebank and multilingual Universal Dependencies treebank v2.0. |
Token-level and sequence-level loss smoothing for RNN language models (P18-1)
Copied to clipboard
| Challenge: | Maximum likelihood estimation treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space. |
| Approach: | They propose to extend the reward augmented maximum likelihood approach to token-level loss smoothing by using token-based approaches to improve the model's performance. |
| Outcome: | The proposed model improves on image captioning and machine translation tasks and treats all sentences that do not match the ground truth as poor . |
Neural semi-Markov CRF for Monolingual Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our proposed model outperforms all previous approaches for monolingual word alignment. |
| Approach: | They propose a neural semi-Markov CRF alignment model which unifies word and phrase alignments through variable-length spans. |
| Outcome: | The proposed model outperforms existing models on in-domain and out-of-domain evaluations and a QA-based benchmark with human annotations. |