Papers by Zhenxuan Yu

Copied to clipboard

Challenge:	Large language models (LLMs) have achieved significant performance improvements in natural language processing domain, but require large computational resources for training and inference.
Approach:	They propose to use a language model architecture based on State-Space Models to quantify embedding and projection layers of a model with 150 B tokens from scratch.
Outcome:	The proposed language model architecture reduces costs by compressing context windows during inference while reducing the cost of training and inference.