Papers by Dana Angluin
Simulating Hard Attention Using Soft Attention (2026.tacl-1)
Copied to clipboard
| Challenge: | a central element of hard attention is attention, which computes a weighted average of values from all unmasked positions. |
| Approach: | They propose transformers that can simulate hard attention by using temperature scaling and positional embeddings. |
| Outcome: | The proposed transformers can effectively focus all attention on a subset of positions. |
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity (2022.tacl-1)
Copied to clipboard
| Challenge: | averaging hard attention is used to recognize formal languages that UHAT and GUHAT cannot recognize. |
| Approach: | They analyze three formal Transformer encoders that differ in the form of their self-attention mechanism . they find that UHAT and GUHAT Transformers can only recognize formal languages in AC0 . |
| Outcome: | The proposed models can recognize languages that UHAT and GUHAT cannot . the proposed models are based on the DYCK and PARITY languages . |
Transformers as Transducers (2025.tacl-1)
Copied to clipboard
| Challenge: | Using finite transducers, we find that transformers can express large classes of (total functional) transductions. |
| Approach: | They extend existing RASP programming language to sequence-to-sequence transductions and introduce two new extensions. |
| Outcome: | The proposed model can express surprisingly large classes of (total functional) transductions. |