Papers by Daniel Simig
Understanding In-Context Learning via Supportive Pretraining Data (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a form of learning that provides a handful of examples at inference time, but it is not well understood why it emerges as the model has never been specifically trained on such demonstrations. |
| Approach: | They adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL and compare it with random subsets of pretrain data. |
| Outcome: | The proposed method improves the model's ICL ability by 18% if it is continued on a small subset of pretraining data. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Open Vocabulary Extreme Classification Using Generative Models (2022.findings-acl)
Copied to clipboard
Daniel Simig, Fabio Petroni, Pouya Yanki, Kashyap Popat, Christina Du, Sebastian Riedel, Majid Yazdani
| Challenge: | Extreme multi-label classification (XMC) aims at tagging content with subset of labels from an extremely large label set. |
| Approach: | They propose a model that predicts a set of labels outside of the known vocabulary by using a loss-dependent loss-based loss-free model. |
| Outcome: | The proposed model can predict labels outside the known vocabulary while performing on par with state-of-the-art solutions for known labels. |
Text Characterization Toolkit (TCT) (2022.aacl-demo)
Copied to clipboard
Daniel Simig, Tianlu Wang, Verna Dankers, Peter Henderson, Khuyagbaatar Batsuren, Dieuwke Hupkes, Mona Diab
| Challenge: | Text Characterization Toolkit (TCT) is a tool that researchers can use to study characteristics of large datasets. |
| Approach: | They propose a text characterization toolkit that researchers can use to study characteristics of large datasets. |
| Outcome: | The proposed tool can be used to study characteristics of large datasets and to understand the influence of attributes on models’ behaviour. |