Papers by Juanxi Tian
Taming LLMs with Gradient Grouping (2025.acl-long)
Copied to clipboard
| Challenge: | a new study presents scaling with gradient grouping (SGG) the adaptive learning rate scaling approach is based on per-parameter statistics, which incurs memory overhead. |
| Approach: | They propose an optimizer wrapper that improves adaptive learning rate estimation by dynamic grouping and group-specific scaling. |
| Outcome: | The proposed algorithm improves learning rate estimation on diverse models with different model sizes and batch sizes. |