Papers by Wang Peixu

Copied to clipboard

Challenge:	Large pre-trained language models have demonstrated impressive capabilities, but there is still much to learn about how they operate.
Approach:	They investigate the ability of the autoregressive transformer to perform basic addition operations by using causal analysis to find that a few different attention heads in the middle layers control the addition carry . they found that due to the lack of global focus on the sequence within these attention heads, the model struggles to handle long-sequence addition tasks.
Outcome:	The model performs basic addition tasks, but it still faces challenges with length generalization.