Papers by Andy Rosenbaum
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained encoder-only and sequence-to-sequence models are computationally expensive. |
| Approach: | They propose a recipe to initialize one model from the other to improve pre-training efficiency. |
| Outcome: | The proposed method matches the performance of a from-scratch model with a multilingual encoder while reducing the total compute cost by 27%. |
PLACES: Prompting Language Models for Social Conversation Synthesis (2023.findings-eacl)
Copied to clipboard
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, Dilek Hakkani-Tur
| Challenge: | Currently, collecting high quality conversational data is expensive and infeasible for many applications . a promising direction is to generate synthetic dialogues by prompting large language models . |
| Approach: | They propose to use expert-written conversations as in-context examples to generate synthetic dialogues by prompting large language models. |
| Outcome: | The proposed approach is generalizable to multi-party conversations, compared to human-collected conversations. |
LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging (2022.coling-1)
Copied to clipboard
| Challenge: | LINGUIST generates annotated data for Intent Classification and Slot Tagging (IC+ST) we demonstrate fine-tuning of a large-scale seq2seq model to control outputs of multilingual data generation. |
| Approach: | They propose a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST) they use a 5-billion-parameter multilingual sequence-to-sequence model to fine-tune it on a flexible instruction prompt. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on a SNIPS intent setting and shows significant improvement on IC+ST in a cross-lingual setting. |
CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing (2022.aacl-short)
Copied to clipboard
| Challenge: | Large Language Models excel at a low-resource level given limited data, but are unsuitable for runtime systems which require low latency. |
| Approach: | They propose a method to augment training data for a model 40x smaller (500M parameters) they use Alexa to generate synthetic data from Alexa 20B to augment the training set . |
| Outcome: | The proposed method improves low-resource SP on two datasets in low-source settings. |