Papers by Jiaoda Li
Unique Hard Attention: A Tale of Two Sides (2025.acl-short)
Copied to clipboard
| Challenge: | Understanding the expressive power of transformers has attracted attention . many studies analyze unique hard attention transformers, where attention selects a single position that maximizes the attention scores. |
| Approach: | They propose to use unique hard attention to select a single position that maximizes attention scores . they show that models with leftmost-hard attention are equivalent to soft attention . |
| Outcome: | The proposed models with leftmost-hard attention are equivalent to soft attention, suggesting they may better approximate real-world transformers than right-attention models. |
A Transformer with Stack Attention (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent research suggests that transformer-based language models fail to learn basic algorithmic patterns. |
| Approach: | They propose to augment transformer-based language models with a differentiable stack-based attention mechanism that adds a level of interpretability to the model. |
| Outcome: | The proposed model can model some, but not all, deterministic context-freelanguages. |
Characterizing the Expressivity of Local Attention in Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that global and local attention are expressively complementary. |
| Approach: | They propose to restrict global attention to a fixed-size window of preceding tokens . they also propose to add local attention to local-only transformers to increase model quality . |
| Outcome: | The proposed model outperforms the global–local transformers on formal language recognition and natural language modeling. |
Differentiable Subset Pruning of Transformer Heads (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model. |
| Approach: | They propose a method that prunes a Transformer's multi-head attention mechanism away without significantly harming its performance. |
| Outcome: | The proposed method improves on natural language inference and machine translation tasks while offering precise control of sparsity level. |
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
Probing via Prompting (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have increased the performance of data-driven natural language processing (NLP) models on a wide variety of tasks. |
| Approach: | They propose a model-free approach to probing via prompting which formulates probing as a prompting task and combine pruning to analyze where the model stores the linguistic information in its architecture. |
| Outcome: | The proposed approach extracts information from pre-trained models while learning much less on its own. |
Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal machine translation models outperform text-only models when visual context is available, but recent studies have shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise. |
| Approach: | They propose to use visual data to highlight the importance of visual inputs in MMT models to enhance their leverage. |
| Outcome: | The proposed models outperform text-only models when visual context is available, but the results show that the visual context might not be exploited by the models at all. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |