Papers by Brandon Reagen
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |