Papers by Yichu Zhou
DirectProbe: Studying Representations without Classifiers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches for probing opaque representations often use training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. |
| Approach: | They propose a heuristic that directly studies the geometry of a representation by building upon the notion of 'version space' they argue that doing so can be unreliable because different representations may need different classifiers . |
| Outcome: | Experiments with linguistic tasks and contextualized embeddings show that even without training classifiers, DirectProbe can shine lights on how an embeddable space represents labels and anticipate the classifier performance for the representation. |
A Closer Look at How Fine-tuning Changes BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations are used to analyze information in NLP . however, how fine-tuning changes the underlying embedding space is less studied . |
| Approach: | They propose to use probing techniques to analyze how fine-tuning changes the embedding space of pre-trained contextualized representations. |
| Outcome: | The proposed model improves classification performance by increasing the distances between examples associated with different labels. |
Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new technique for exploring contextualized vector space is proposed . masked prediction of a word in a sentence allows controlled exploration of the space . |
| Approach: | They propose a method for exploring regions around individual points in a contextualized vector space . they use a static embedding to induce a "pseudoword" vector and masked prediction of a word . |
| Outcome: | The proposed method investigates the geometry of the contextualized space around individual instances of a word . it uses a static embedding to induce a contextualized "pseudoword" vector . |