Papers by Richard Bai
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024.acl-long)
Copied to clipboard
| Challenge: | Large language model pre-training is infeasible due to the large compute costs and duration associated with pre- training and the impending scarcity of high-quality data on the web. |
| Approach: | They propose to use an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as “like Wikipedia” or in “question-answer format” to jointly pre-train LLMs on real and synthetic rephrases. |
| Outcome: | The proposed model speeds up pre-training by 3x on the C4 dataset, and improves perplexity by 50% on average across different subsets of the Pile. |
Divide-or-Conquer? Which Part Should You Distill Your LLM? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first. |
| Approach: | They propose a strategy that breaks down reasoning tasks into a problem decomposition phase and a solution phase and propose 'smaller' models that can achieve good generalization. |
| Outcome: | The proposed approach outperforms a single stage solution in two tasks and their impact on reasoning outcomes and inference cost. |