Papers by David Bau
Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current language models process text as sequences of tokens that roughly correspond to words . individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise . |
| Approach: | They propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers. |
| Outcome: | The proposed method "reads out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers. |
Position-aware Automatic Circuit Discovery (2025.acl-long)
Copied to clipboard
| Challenge: | Existing circuit analysis methods treat component positions as equally relevant across input positions, limiting their ability to capture cross-positional interactions or mechanisms that vary across positions. |
| Approach: | They propose to extend edge attribution patching to differentiate between token positions and introduce a dataset schema to enable position-aware circuit discovery in datasets with variable length examples. |
| Outcome: | The proposed approach enables position-aware circuit discovery even on tasks with variable length examples. |
Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has shown that LLMs encode social biases and manifest in clinical tasks. |
| Approach: | They use mechanistic interpretability to uncover biases within LLMs . they find gender information is highly localized in MLP layers . |
| Outcome: | The proposed method can reveal biases and representations within LLMs in healthcare. |