On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work uses linear models and neural networks for word ordering, yet pre-trained language models have not been studied in word ordering. |
| Approach: | They propose a constrained language generation task using unordered words as input. |
| Outcome: | The proposed model is able to perform better than existing models and proves to be reliable. |
Similar Papers
Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese (2020.acl-main)
Copied to clipboard
| Challenge: | a method using neural language models (LMs) for analyzing the word order of language is currently lacking. |
| Approach: | They propose a method using neural language models to analyze the word order in Japanese . they test whether there is a parallel between LMs and human word order preference . |
| Outcome: | The proposed method is validated by comparing it with other linguistic studies. |
Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to encode natural languages without orders are lacking. |
| Approach: | They conduct a comprehensive analysis of the ability of neural models to organize sentences from a bag of words under three typical scenarios. |
| Outcome: | The proposed models can reorder or reconstruct sentences from a bag of words under three typical scenarios. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to generative commonsense reasoning hypothesize that pre-trained models lack sufficient parametric knowledge for this task. |
| Approach: | They propose to use order-agnostic input to elaborately manipulate the order of the given concepts before generation to evaluate their commonsense knowledge. |
| Outcome: | The proposed approach outperforms more sophisticated models with a lot of external data and resources in the task of generating a logical sentence from a set of concepts. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)
Copied to clipboard
| Challenge: | masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say . |
| Approach: | They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks. |
| Outcome: | The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models . |
Constrained Language Models Yield Few-Shot Semantic Parsers (2021.emnlp-main)
Copied to clipboard
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme
| Challenge: | Large pretrained language models excel at generating natural language, but they are not efficient for task specific semantic parsing. |
| Approach: | They propose to use large pretrained language models as few-shot semantic parsers . they paraphrase inputs into a controlled sublanguage resembling English . |
| Outcome: | The proposed model can generate surprisingly accurate models on multiple tasks with minimal code and data. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020.emnlp-main)
Copied to clipboard
Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan
| Challenge: | Existing models for temporal ordering of events rely on pretrained representations, transfer and multitask learning, and self-training techniques. |
| Approach: | They propose a neural architecture and a set of training methods for ordering events by predicting temporal relations by pre-training models. |
| Outcome: | The proposed models can predict temporal relations between two pairs of events within a span of text and identify temporal relationships between them. |
Implicit Representations of Meaning in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Neural language models (NLMs) encode lexical relations and syntactic structure, but their effectiveness is still unclear. |
| Approach: | They propose to use text as a model to model entities and situations as they evolve throughout a discourse. |
| Outcome: | The proposed models have functional similarities to linguistic models of dynamic semantics and can be learned with only text as training data. |