| Challenge: | Pre-trained language models have improved performance for many NLP tasks in finance and healthcare. |
| Approach: | They propose a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages. |
| Outcome: | The proposed model outperforms other models on commercial generation tasks and on other markets, languages, and tasks. |
Similar Papers
MVP: Multi-task Supervised Pre-training for Natural Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved remarkable success in natural language generation tasks. |
| Approach: | They propose to use a large-scale natural language generation corpus to pre-train a text generation model MVP in a supervised manner. |
| Outcome: | The proposed model outperforms BART and Flan-T5 on 13 out of 17 datasets and outperformed BART by 9.3% and FlaN-T5. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
LaoPLM: Pre-trained Language Models for Lao (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) can capture different levels of concepts in context . previous work on Lao has been hampered by the lack of annotated datasets . |
| Approach: | They construct a text classification dataset to alleviate the resource-scarce situation of Lao . they evaluate them on two downstream tasks: part-of-speech tagging and text classification . |
| Outcome: | The proposed model can capture different levels of concepts in context and generate universal language representations. |
Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data (2020.coling-industry)
Copied to clipboard
Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei, Karthik Mohan, Michael White
| Challenge: | Natural language generation (NLG) is a critical component in conversational systems . Traditionally, NLG components have been deployed using template-based solutions . however, deployment of such model-based systems has been challenging due to high latency and data needs. |
| Approach: | They propose a family of techniques to deploy data-efficient neural solutions for NLG in conversational systems to production. |
| Outcome: | The proposed techniques achieve production quality with light-weight neural network models using fraction of the data needed otherwise. |
An Empirical Study of Generating Texts for Search Engine Advertising (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing studies on neural language generation have not evaluated the effect of generated ads with actual serving included because it requires a large amount of training data and a particular environment. |
| Approach: | They propose to integrate a reinforcement learning framework into an end-to-end sequence-tosequence (Seq2S) model and demonstrate how to improve the ads’ impact, deploy models to a product, and evaluate the generated ads. |
| Outcome: | The proposed method improves the ads’ impact, deploys the models to a product, and evaluates the generated ads. |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |
Benchmarking Large Language Model Capabilities for Conditional Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Autoregressive and pre-trained large language models have shifted the field from application-specific to generation-based approaches. |
| Approach: | They propose to adapt existing application-specific generation benchmarks to pre-trained large language models to better suit different tasks. |
| Outcome: | The proposed models differ in their applicability to different data regimes and their generalization to multiple languages. |
ClinicalT5: A Generative Language Model for Clinical Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent generative language models like BART and T5 are gaining popularity with their competitive performance on text generation and tasks cast as generative problems. |
| Approach: | They propose to build domain-specific PLMs through fine-tuning or pre-training from scratch over domain corpora. |
| Outcome: | The proposed model outperforms existing models on domain-specific tasks and compares favorably with its close baselines. |
Large-Scale Transfer Learning for Natural Language Generation (P19-1)
Copied to clipboard
Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, Thomas Wolf
| Challenge: | Large-scale pretrained language models define state of the art in natural language processing, achieving outstanding performance on a variety of tasks. |
| Approach: | They propose to apply large-scale pretrained language models to natural language generation tasks by comparing architectural and training schemes. |
| Outcome: | The proposed architectures perform well on open-domain dialog as a typical high entropy generation task. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |