Papers by Dan Garrette
How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual language models can learn generalisations useful for other languages . yet, it remains unclear to what extent and under which conditions these models benefit from multilingual data and cross-lingual sharing. |
| Approach: | They propose a training data attribution method to retrieve training samples from multilingual data that are most influential for test predictions in a given language. |
| Outcome: | The proposed method exploits the ability to learn generalisations useful for other languages on zero-shot cross-lingual transfer for many languages. |
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (2023.tacl-1)
Copied to clipboard
Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant
| Challenge: | a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation is presented . FRMT is a type of style-targeted translation that uses labeled training data to perform tasks. |
| Approach: | They propose a dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. |
| Outcome: | The proposed model is based on two translations from English into Portuguese and Mandarin Chinese. |
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2022.tacl-1)
Copied to clipboard
| Challenge: | End-to-end neural models have replaced the traditional pipeline and require an explicit tokenization step. |
| Approach: | They propose a neural encoder that operates directly on character sequences without explicit tokenization or vocabulary and a pre-training strategy that optionally uses subwords as a soft inductive bias. |
| Outcome: | The proposed model outperforms a comparable mBert model on a multilingual benchmark by 5.7 F1 on the TyDi QA benchmark. |
Frequency Effects on Syntactic Rule Learning in Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. |
| Approach: | They investigate the performance of BERT on English subject–verb agreement by analyzing word frequency and absolute frequency of verb forms. |
| Outcome: | The proposed model generalizes well to subject–verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. |
Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification (D18-1)
Copied to clipboard
| Challenge: | Code-switching is a challenge for NLP due to the lack of representative data for training models. |
| Approach: | They propose a model that is trained exclusively on monolingual resources but can be applied to unseen code-switched text at inference time. |
| Outcome: | The proposed model outperforms standard models on Hindi-English part-of-speech tagging and on unannotated code-switched text with alternate scripts. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)
Copied to clipboard
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki
| Challenge: | Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English. |
| Approach: | They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs. |
| Outcome: | The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs. |
Improving Multilingual Models with Language-Clustered Vocabularies (2020.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art multilingual models depend on vocabularies that cover all languages . but the methods for generating those vocalaries are not ideal for massively multilingual applications. |
| Approach: | They propose a procedure for multilingual vocabulary generation that combines separately trained vocabularies of several automatically derived language clusters. |
| Outcome: | The proposed procedure shows improvements across languages on multilingual benchmark tasks . the proposed procedure reduces out-of-vocabulary rate by a factor of 8 . |
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, Partha Talukdar
| Challenge: | Existing datasets are often informed by established research directions in the NLP community. |
| Approach: | They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
| Outcome: | The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. |
| Approach: | They propose to explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning on per-language subnetworks to better guide cross-lingual sharing. |
| Outcome: | The proposed approach can increase language specialization of subnetworks in favor of more cross-lingual sharing. |
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021.emnlp-main)
Copied to clipboard
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, Melvin Johnson
| Challenge: | Recent advances in multilingual natural language processing have improved performance on benchmarks such as XTREME and XGLUE by 13 points . however, improvements have been easier to achieve in some tasks than others . |
| Approach: | They extend XTREME to XTRAME-R, which includes ten natural language understanding tasks and covers 50 typologically diverse languages. |
| Outcome: | The proposed framework improves the performance on the XTREME multilingual benchmark by 13 points compared to human-level performance on English transfer learning. |
Character-Aware Models Improve Visual Text Rendering (2023.acl-long)
Copied to clipboard
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, Noah Constant
| Challenge: | Current image generation models struggle to produce well-formed visual text due to lack of character-level input features. |
| Approach: | They conduct a series of experiments to compare character-aware vs. character-blind text encoders to determine their spelling ability. |
| Outcome: | The character-aware models outperform character-blind models on a range of novel text rendering tasks. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
Dialect-robust Evaluation of Generated Text (2023.acl-long)
Copied to clipboard
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
| Challenge: | Existing evaluation metrics that are not robust to dialect variation are difficult to measure for many groups of users and can penalize systems for producing text in lower-resource dialects. |
| Approach: | They propose a dialect-robust evaluation metric that produces the same score for system outputs that share the same semantics but are expressed in different dialects. |
| Outcome: | The proposed method significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. |