Papers by William Brandon
Fast Matrix Multiplications for Lookup Table-Quantized LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | generative (tokenby-token) inference is memory-bound and requires a large amount of memory to perform. |
| Approach: | They propose a lookup table engine for weight-quantized large language models that uses offline restructuring of the quantized weight matrix to minimize bit manipulations associated with unpacking. |
| Outcome: | The proposed kernel can be 2-4x faster than existing GEMM kernels while achieving performance gains of 1.5 to 2 times. |