Papers by Jerome Bolte

Copied to clipboard

Challenge:	Despite the success of in-context learning, recent studies have identified systematic limitations in its generalization behavior.
Approach:	They propose a new attention scoring function that mitigates failures in transformer models . they use Scaled Signed Averaging to train the scoring function instead of Softmax .
Outcome:	The proposed scoring function outperforms transformer models with Softmax on NLP benchmarks and linguistic probing tasks.