Papers by Colin Raffel
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022.acl-demo)
Copied to clipboard
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, Alexander Rush
| Challenge: | PromptSource is a system for creating, sharing, and using natural language prompts . prompts are used to train and query language models in zero-shot learning settings . |
| Approach: | PromptSource is a system for creating, sharing, and using natural language prompts . et al.: using prompts to train and query language models is emerging area in NLP . they propose a templating language for defining data-linked prompts, a user interface that iterates on prompt development . |
| Outcome: | PromptSource is a system for creating, sharing, and using natural language prompts . it has a templating language for defining data-linked prompts and a community-driven set of guidelines . |
What Language Model to Train if You Have One Million GPU Hours? (2022.findings-emnlp)
Copied to clipboard
Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy
| Challenge: | Recent years have seen the advent of large language models characterized by emergent capabilities arising from sheer scale alone. |
| Approach: | They propose to use a multilingual model to compare performance to the English-only model by ablation at the billion-parameter scale. |
| Outcome: | The proposed model is based on a multilingual model and its performance against the English-only model. |
Knowledge is a Region in Weight Space for Fine-tuned Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuning on different datasets from the same underlying task form s looser clusters. |
| Approach: | They analyze the weight space and the underlying loss landscape of different models to determine how they are interconnected and how they perform. |
| Outcome: | The proposed method improves accuracy on 11 out of 12 datasets by starting from the center of the region, if not more, than using the pretrained model. |
Learning with Limited Text Data (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Natural Language Processing (NLP) relies on labeled data to perform state-of-the-art performance . labeles are often required to label large amounts of textual data . this tutorial will provide an overview of labeleing in NLP . |
| Approach: | This tutorial will provide a systematic overview of methods for learning from limited labeled data. |
| Outcome: | This tutorial will provide a systematic and up-to-date overview of the proposed methods . it will highlight current challenges and future directions . |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning (2023.acl-long)
Copied to clipboard
| Challenge: | Pretraining has been shown to scale well with compute, data size and data diversity. |
| Approach: | They propose a method that provides benefits of multitask learning but leverages distributed computation . they propose 'coldfusion' can create synergistic loop where finetuned models can be "recycled" |
| Outcome: | The proposed method outperforms RoBERTa and previous multitask models on 35 datasets. |
DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have become a dominant tool for NLP researchers in a wide range of tasks. |
| Approach: | They propose an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. |
| Outcome: | The proposed library is open source and can be used to implement powerful LLM workflows. |
How Much Knowledge Can You Pack Into the Parameters of a Language Model? (2020.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we show that large neural language models trained on unstructured text can attain competitive results on open-domain question answering benchmarks without access to external knowledge. |
| Approach: | They propose to fine-tune pre-trained neural language models to answer questions without external knowledge . they show that this approach scales with model size and performs competitively . |
| Outcome: | The proposed approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. |
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models generate text that is problematic or lacks a desired attribute. |
| Approach: | They propose a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text with certain properties. |
| Outcome: | The proposed procedure outperforms prior weighted decoding methods and matches state-of-the-art techniques that require additional training. |
Do Transformer Modifications Transfer Across Implementations and Applications? (2021.emnlp-main)
Copied to clipboard
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
| Challenge: | Currently, the Transformer is the de facto architecture of choice for processing sequential data. |
| Approach: | They evaluate the Transformer architecture and its modifications in a shared experimental setting . they conjecture that performance improvements may strongly depend on implementation details . |
| Outcome: | The proposed improvements do not significantly improve performance, the authors find . the proposed improvements are either developed in the same codebase or are minor changes . |
Crosslingual Generalization through Multitask Finetuning (2023.acl-long)
Copied to clipboard
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel
| Challenge: | Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. |
| Approach: | They apply multitask prompted finetuning to pretrained multilingual models and generate variants called BLOOMZ and mT0. |
| Outcome: | The proposed models can generalize to non-English languages that have never been seen before. |
Evaluating the Factual Consistency of Large Language Models Through News Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs generally assign a higher score to factually consistent summaries than to factualally inconsistent summary. |
| Approach: | They propose a benchmark to measure whether large language models prefer factually consistent continuations of inputs. |
| Outcome: | The proposed benchmark compares the scores an LLM assigns to a factually consistent versus a inconsistent summary for an input news article. |
Improving and Simplifying Pattern Exploiting Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained language models can learn well when primed with only a few labeled examples. |
| Approach: | They propose a method that uses task-specific unlabeled data to provide denser supervision during fine-tuning. |
| Outcome: | The proposed approach outperforms GPT-3 on SuperGLUE without any unlabeled data. |
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for enhancing data efficiency in limited labeled data are limited. |
| Approach: | They propose to use data augmentation methods to increase the efficiency of limited data learning in NLP. |
| Outcome: | The proposed methods perform well on topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. |
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022.tacl-1)
Copied to clipboard
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
| Challenge: | a number of pre-trained language models use sequences of tokens corresponding to word units . token-free models that operate directly on raw text have many advantages . |
| Approach: | They propose a standard Transformer architecture that can be used to process byte sequences . they also characterize trade-offs in terms of parameter count, training FLOPs, and inference speed . |
| Outcome: | The proposed model is more robust to noise and more robust on spelling and pronunciation tasks. |
Petals: Collaborative Inference and Fine-tuning of Large Models (2023.acl-demo)
Copied to clipboard
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel
| Challenge: | Recent studies show that pretrained language models can solve practical tasks using more than 100 billion parameters. |
| Approach: | They propose a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. |
| Outcome: | The proposed system outperforms offloading for very large models running on consumer GPUs with 1 step per second, enough for many interactive LLM applications. |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)
Copied to clipboard
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
| Challenge: | Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English. |
| Approach: | They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. |
| Outcome: | The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting. |
Monotonic Infinite Lookback Attention for Simultaneous Machine Translation (P19-1)
Copied to clipboard
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, Colin Raffel
| Challenge: | Simultaneous machine translation begins to translate each source sentence before the source speaker has finished speaking, with applications to live and streaming scenarios. |
| Approach: | They propose a simultaneous translation system that learns an adaptive schedule with a neural machine translation model that attends over all source tokens read thus far. |
| Outcome: | The proposed system can achieve latency-quality trade-offs favorable to a proposed wait-k strategy for many latency values. |