Papers with toxicity
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models with instruction tuning are resource-intensive . a recent study suggests that the performance of LLMs scales proportionally with the size of the model. |
| Approach: | They propose to distill knowledge from instruction-tuned LLMs into much smaller ones . they develop a large set of 2.58M instructions based on existing and newly-generated instructions . |
| Outcome: | The proposed models are comparable to strong baselines while being much smaller in size. |