| Challenge: | Maximum likelihood estimation treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space. |
| Approach: | They propose to extend the reward augmented maximum likelihood approach to token-level loss smoothing by using token-based approaches to improve the model's performance. |
| Outcome: | The proposed model improves on image captioning and machine translation tasks and treats all sentences that do not match the ground truth as poor . |
Similar Papers
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models rely on frequency information because they maximize the likelihood of tokens during training. |
| Approach: | They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. |
| Outcome: | The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. |
Token-Level Self-Evolution Training for Sequence-to-Sequence Learning (2023.acl-short)
Copied to clipboard
| Challenge: | Adaptive training approaches do not consider the variation of learning difficulty in different training steps, making the learning deterministic and sub-optimal. |
| Approach: | They propose a dynamic token-level self-evolution training method that reweighs the training losses of different target tokens based on priors. |
| Outcome: | Empirically, the proposed method yields significant improvements on three translation tasks. |
A Natural Bias for Language Generation Models (2023.acl-short)
Copied to clipboard
| Challenge: | a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. |
| Approach: | They propose to initialise bias terms in a model's final linear layer with the log-unigram distribution and use it to output the unigram frequency statistics as prior knowledge. |
| Outcome: | The proposed method improves learning efficiency and improves overall performance. |
Sequence-level Large Language Model Training with Contrastive Preference Optimization (2025.findings-naacl)
Copied to clipboard
| Challenge: | a new method to improve the performance of large language models requires a small computational cost. |
| Approach: | They propose a CPO procedure that can inject sequence-level information into the model at any training stage without expensive human labeled data. |
| Outcome: | The proposed objective surpasses the next token prediction in terms of win rate in instruction-following and text generation tasks. |
Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens (N18-1)
Copied to clipboard
| Challenge: | Recent work has used attention weights to visualize the focus of neural models in input data. |
| Approach: | They propose to use attention-based visualization techniques to infer token-level labels from a network trained only on sentence-level labeling. |
| Outcome: | The proposed approach outperforms gradient-based methods on four datasets and is expected to outperfect supervised methods. |
Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks (2022.acl-short)
Copied to clipboard
| Challenge: | Experimental results show text smoothing outperforms data augmentation methods by a substantial margin. |
| Approach: | They propose to use a masked language model to convert a token to a smoothed representation by converting a sentence from its one-hot representation to 'controllable smoothes' they propose to combine text smoothing with other data augmentation methods to achieve better performance. |
| Outcome: | The proposed method outperforms mainstream data augmentation methods by a substantial margin on different datasets in a low-resource regime. |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
Language Modeling with a General Second-Order RNN (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of RNNs update their state as the input sequence is processed . second-order RNN architectures show promising performance in language modeling . |
| Approach: | They propose a second-order RNN architecture that generalizes existing ones . they use a Penn Treebank dataset to analyze how their different components affect performance . |
| Outcome: | The proposed architecture generalizes existing RNNs on a Penn Treebank dataset . it shows that removing the first-order terms does not hinder performance . |
Trainable, Multiword-aware Linguistic Tokenization Using Modern Neural Networks (2026.eacl-srw)
Copied to clipboard
| Challenge: | Tokenization is a fundamental task in natural language processing that forms the first step of many pipelines. |
| Approach: | They propose to use a standard tokenizer trained without MWE-awareness as a baseline and a character-level SRN+CRF model to train token-level models. |
| Outcome: | The proposed tokenizers are based on a character-level and token-level sequence labeling problem and are consistent with the proposed pipelines. |