Papers by Guoxuan Chen
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |