Papers by William Timkey
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021.emnlp-main)
Copied to clipboard
| Challenge: | Similarity measures are a vital tool for understanding how language models represent and process language. |
| Approach: | They propose to use cosine similarity and Euclidean distance to understand how words cluster in semantic space. |
| Outcome: | The proposed measures show that rogue dimensions dominate similarity measures and reveal representational quality. |
To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text (2021.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models have seen great improvements in recent years, but there is limited understanding of the strategies different models employ and how they relate their understanding of language. |
| Approach: | They characterize how one popular abstractive model uses an explicit copy/generation switch to control its level of abstraction vs extraction . they find that abstractive summarization models lack the semantic understanding necessary to generate paraphrases that are both abstractive and faithful to the source document. |
| Outcome: | The proposed model uses syntactic boundaries to truncate sentences that are often copied verbatim. |
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theories of human sentence processing can be divided into two broad categories: expectation-based theories and memory-based ones. |
| Approach: | They propose to integrate expectations and retrieval from working memory into a unified cognitive model that can capture syntactic and semantic interference effects observed in human experiments. |
| Outcome: | The proposed model captures syntactic and semantic interference effects observed in human experiments. |