Papers by David Woodruff
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads. |
| Approach: | They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates. |
| Outcome: | The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements. |