Papers by John Hewitt
Learning Translations via Images with a Massively Multilingual Image Dataset (P18-1)
Copied to clipboard
| Challenge: | Existing datasets for learning translations of words are limited to a few high-resource languages and unrealistically easy settings. |
| Approach: | They propose a large-scale multilingual corpus of images labeled with the word they represent to facilitate translation research. |
| Outcome: | The proposed method improves on an unsupervised technique that has been limited to a few languages and unrealistic settings. |
Backpack Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | We train a 170Mparameter Backpack language model on OpenWebText, matching the loss of a 6Bparameter Transformer. |
| Approach: | They propose a neural architecture that learns multiple non-contextual sense vectors for each word in a vocabulary and represents a word as a context-dependent, non-negative linear combination of sense vector. |
| Outcome: | The proposed model outperforms a GPT-2's word embeddings on lexical similarity evaluations and can be used to perform controllable text generation and debiasing. |
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
A Structural Probe for Finding Syntax in Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Approach: | They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Outcome: | The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry. |
Truncation Sampling as Language Model Desmoothing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Long samples of text from neural language models can be of poor quality. |
| Approach: | They propose to think of a neural language model as a mixture of k and a true distribution that avoids infinite perplexity. |
| Outcome: | The proposed methods generate more plausible long documents according to humans and break out of repetition. |
RNNs can generate bounded hierarchical languages with optimal memory (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that RNNs can efficiently generate bounded hierarchical languages with high syntactic fidelity, but their success is not well-understood theoretically. |
| Approach: | They propose a language of well-nested brackets and m-bounded nesting depth . they prove that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction. |
| Outcome: | The proposed language is well-nested brackets and has m-bounded nesting depth . it shows that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction. |
A Distributional and Orthographic Aggregation Model for English Derivational Morphology (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to derived word generation model derivational morphology to generate words with particular semantics are not effective. |
| Approach: | They propose a novel aggregation model that learns derivational transformations as orthographic functions and as functions in distributional word embedding space. |
| Outcome: | The proposed model learns to choose between the hypothesis of each system and the hypothesis from the model. |
Refining Targeted Syntactic Evaluation of Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Targeted syntactic evaluation of subject-verb number agreement in English (TSE) evaluates language models’ syntaktic knowledge using hand-crafted minimal pairs of sentences that differ only in the main verb’s conjugation. |
| Approach: | They propose to use templates to evaluate language models' syntactic knowledge to assess their ability to conjugate arbitrary verbs and their likely behavior to measure their likelihood of conjugating grammatical sentences. |
| Outcome: | The proposed evaluations overestimate systematicity of language models, but score up to 40% better on verbs that they predict are likely in context. |
JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset (2022.findings-emnlp)
Copied to clipboard
| Challenge: | JamPatoisNLI provides the first dataset for natural language inference in a creole language, Jamaican Patois. |
| Approach: | They propose to use JamPatoisNLI to provide a dataset for natural language inference in a creole language, Jamaican Patois. |
| Outcome: | The proposed dataset shows that the language's relatedness to English significantly boosts the effectiveness of cross-lingual transfer even for the higher order task. |
Generating Text from Language Models (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models. |
| Approach: | They will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
| Outcome: | This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
Conditional probing: measuring usable information beyond a baseline (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for probing representations are limited to predicting part-of-speech . current methods cannot detect when a representation is predictive of just aspects of part- of-seech not explainable by the word identity. |
| Approach: | They propose to condition on the information in a baseline representation to test whether it is predictive of part-of-speech. |
| Outcome: | The proposed method is based on a theory of usable information called V-information and conditions on the information in the baseline. |
Designing and Interpreting Probes with Control Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing studies on supervised models to predict properties from representations have shown high accuracy on a range of linguistic tasks. |
| Approach: | They propose control tasks which associate word types with random outputs to complement linguistic tasks by construction . they find that popular probes on ELMo representations are not selective . |
| Outcome: | The proposed tasks associate word types with random outputs to complement linguistic tasks. |