Papers by Keigo Shibata
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle layers . |
| Approach: | They propose a jump-suppressing regularizer which penalizes large hidden state displacements near the final layer during pre-training. |
| Outcome: | The proposed method significantly reduces hidden state jumps in the final layer and increases model capacity. |