Learning Spoken Language Representations with Neural Lattice Language Modeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining language models do not consider spoken language properties. |
| Approach: | They propose a framework that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. |
| Outcome: | The proposed framework outperforms baselines on spoken inputs on intent detection and dialogue act recognition datasets. |
Similar Papers
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words. |
| Approach: | They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner. |
| Outcome: | The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Pretrained Language Models for Sequential Sentence Classification (D19-1)
Copied to clipboard
| Challenge: | Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels. |
| Approach: | They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF. |
| Outcome: | The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Neural Speech Translation using Lattice Transformations and Graph Networks (D19-53)
Copied to clipboard
| Challenge: | Existing work on end-to-end systems bypass the need for intermediate representations, but this approach is limited in practical applications. |
| Approach: | They propose a lattice-tosequence model which uses lattics as encoders and graph networks to address two problems by applying latticae transformations and a neural model. |
| Outcome: | The proposed model beats pipeline approaches while being orders of magnitude faster than previous work. |
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Can LLMs Facilitate Interpretation of Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to uncover knowledge encoded within pre-trained language models are limited in terms of scalability and scope of interpretation. |
| Approach: | They propose to use a large language model, ChatGPT, as an annotation tool . they demonstrate that ChatGPt produces accurate and semantically richer annotations . |
| Outcome: | The proposed method produces accurate and semantically richer annotations compared to human annotations. |