Papers by Todor Mihaylov
Understanding In-Context Learning via Supportive Pretraining Data (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a form of learning that provides a handful of examples at inference time, but it is not well understood why it emerges as the model has never been specifically trained on such demonstrations. |
| Approach: | They adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL and compare it with random subsets of pretrain data. |
| Outcome: | The proposed method improves the model's ICL ability by 18% if it is continued on a small subset of pretraining data. |
Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing work on discourse-aware self-attention models for reading comprehension uses annotations . |
| Approach: | They propose to use linguistic annotations as a basis for a Discourse-Aware Semantic Self-Attention encoder for reading comprehension on narrative texts. |
| Outcome: | The proposed model improves reading comprehension performance on narrative texts up to +3.4 Rouge-L . it also improves inter- and cross-sentential discourse relations, sentence-internal semantic role relations, and long-distance coreference relations. |
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022.naacl-main)
Copied to clipboard
Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva
| Challenge: | Existing approaches to improve in-context few-shot learning are pretraining and downstream fewshot evaluation. |
| Approach: | They propose to use self-supervision as an intermediate training stage between pretraining and downstream fewshot usage to train models to perform in-context few shot learning. |
| Outcome: | The proposed model outperforms baseline models on two benchmarks. |
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)
Copied to clipboard
Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev
| Challenge: | bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian. |
| Approach: | They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian. |
| Outcome: | The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning. |
EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | EXAMS is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations. |
| Approach: | They propose to use EXAMS to evaluate cross-lingual and multilingual question answering for high school examinations. |
| Outcome: | The proposed model can be used to explore multilingual reasoning and knowledge transfer methods and pre-trained models in schools in different languages, which was not possible by now. |
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (P18-1)
Copied to clipboard
| Challenge: | a new model for reading comprehension integrates external commonsense knowledge . cloze-style reading comprehension is a language understanding task similar to question answering . |
| Approach: | They propose a reading comprehension model that integrates external commonsense knowledge in a cloze-style setting. |
| Outcome: | The proposed model improves results over a very strong baseline on a hard Common Nouns dataset, making it a strong competitor of more complex models. |
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |