Papers by Xinwen Hou
Highway Transformer: Self-Gating Enhanced Self-Attentive Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, attending to all the global contexts at different locations. |
| Approach: | They propose a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. |
| Outcome: | The proposed system could boost the performance of the Transformer modules by allowing them to skip connections and step towards suboptimal points during the optimization process. |