Papers by Selim Jerad

1 papers

Copied to clipboard

Challenge:	Understanding the expressive power of transformers has attracted attention . many studies analyze unique hard attention transformers, where attention selects a single position that maximizes the attention scores.
Approach:	They propose to use unique hard attention to select a single position that maximizes attention scores . they show that models with leftmost-hard attention are equivalent to soft attention .
Outcome:	The proposed models with leftmost-hard attention are equivalent to soft attention, suggesting they may better approximate real-world transformers than right-attention models.