Papers by Veselin Stoyanov
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021.acl-long)
Copied to clipboard
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, Gargi Ghosh
| Challenge: | Knowledge-intensive tasks require large amounts of knowledge about the world . recent neural retrieval models achieve better results by learning directly from task-specific training data. |
| Approach: | They propose a multi-task trained neural retrieval model that can be universally trained on a wide variety of problems. |
| Outcome: | The proposed model outperforms specialised retrievers on a few-shot setting and matches or improves state-of-the-art on multiple benchmarks. |
Conversational Semantic Parsing (2020.emnlp-main)
Copied to clipboard
Armen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick, Michael Haeger, Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, Sonal Gupta
| Challenge: | Structured representations for task-oriented assistant systems are limited due to the limitations of the representation. |
| Approach: | They propose a semantic representation for task-oriented conversational systems that can represent co-reference and context carryover. |
| Outcome: | The proposed model improves the best results on ATIS, SNIPS, TOP and DSTC2 by up to 5 points for slot-carryover. |
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform few-shot learning, but discriminative models like ELECTRA do not fit into the paradigm. |
| Approach: | They propose to use ELECTRA to train pre-trained models to score originality of target options without introducing new parameters. |
| Outcome: | The proposed model outperforms masked language models in a wide range of tasks without adding new parameters. |
Continual Few-Shot Learning for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large number of end-to-end systems are needed for many tasks in natural language processing. |
| Approach: | They propose a continual few-shot learning task where a system is asked to correct mistakes with a few training examples. |
| Outcome: | The proposed task compares two NLI and one sentiment analysis datasets with baselines from diverse paradigms. |
Complementary Explanations for Effective In-Context Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable capabilities in learning from expla- nations in prompts, but there has been limited understanding of exactly how these explana- tions function or why they are effective. |
| Approach: | They propose a maximal marginal relevance-based exemplar selection approach to construct exemplar sets that are both relevant and comple- mentary. |
| Outcome: | The proposed model improves in- context learning performance across three tasks on multiple LLMs. |
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Self-training Improves Pre-training for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, Alexis Conneau
| Challenge: | Unsupervised pretraining has led to improvements in natural language understanding . a data augmentation method can be used to generate labels for unlabeled examples . |
| Approach: | They propose a semi-supervised method which uses unlabeled data to retrieve sentences from a database of billions of unlabed sentences crawled from the web. |
| Outcome: | The proposed method improves on standard text classification benchmarks by 2.6% and knowledge distillation by few shots. |
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning (2026.acl-long)
Copied to clipboard
Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Nenkov Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Jinyan Su, Aaryamonvikram Singh, Rui Xing, Rania Elbadry, Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty, Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou, Preslav Nakov
| Challenge: | Existing benchmarks emphasize final numerical answers while neglecting intermediate reasoning steps. |
| Approach: | They propose a symbolic benchmark for verifiable Chain-of-Thought evaluation in finance . FINCHAIN spans 58 topics across 12 financial domains and three difficulty levels . |
| Outcome: | The proposed benchmark aims to bridge symbolic reasoning and factual verification. |
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022.naacl-main)
Copied to clipboard
Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva
| Challenge: | Existing approaches to improve in-context few-shot learning are pretraining and downstream fewshot evaluation. |
| Approach: | They propose to use self-supervision as an intermediate training stage between pretraining and downstream fewshot usage to train models to perform in-context few shot learning. |
| Outcome: | The proposed model outperforms baseline models on two benchmarks. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
On the Role of Bidirectionality in Language Model Pre-Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on language model pre-training explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. |
| Approach: | They propose a framework that generalizes prior approaches to pre-training language models by focusing on bidirectionality and controlling each of them separately. |
| Outcome: | The proposed framework generalizes prior approaches including fully unidirectional models like GPT, fully bidirectional models and hybrid models like CM3 and prefix LM. |
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)
Copied to clipboard
Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev
| Challenge: | bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian. |
| Approach: | They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian. |
| Outcome: | The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning. |
Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani
| Challenge: | Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score. |
| Approach: | They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks. |
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models are currently used for many NLP tasks . however, inference for these models requires significant computational resources . |
| Approach: | They propose to use a shared text encoder to amortize the computational cost of inference over multiple tasks. |
| Outcome: | The proposed method reduces the size of the extracted representations by a factor of 16 to store them for later use. |
SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning (2026.acl-long)
Copied to clipboard
Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov, Zhuohan Xie
| Challenge: | English financial NLP has progressed rapidly through benchmarks for sentiment, document understanding, and financial question answering. |
| Approach: | They propose a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari’ah-compliant reasoning. |
| Outcome: | The proposed dataset contains 14,380 expert-verified instances spanning seven tasks . it includes financial sentiment analysis, extractive summarization, and event–cause reasoning . |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models. |
| Approach: | They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states . |
| Outcome: | The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures. |
Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models (2023.eacl-main)
Copied to clipboard
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer
| Challenge: | Pretrained language models store a large amount of factual information that can be elicited by prompting or finetuning. |
| Approach: | They propose methods to measure model factual beliefs and update incorrect beliefs in models . they propose a new visualization tool that shows relationships between stored model beliefs . |
| Outcome: | The proposed methods improve models' consistency and accuracy, the authors show . their methods outperform existing methods in more difficult settings, the paper shows . |
Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition (N19-1)
Copied to clipboard
| Challenge: | Current language models are unable to efficiently model entity names observed in text providing insufficient context. |
| Approach: | They propose to augment a traditional model with an external knowledge base to model entity names observed in text. |
| Outcome: | The proposed model improves on a Named Entity Recognition (NER) task by requiring no additional information such as named entity tags. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)
Copied to clipboard
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov
| Challenge: | State-of-the-art natural language processing systems rely on annotated data to learn competent models. |
| Approach: | They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu. |
| Outcome: | The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |
ToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection (2022.emnlp-main)
Copied to clipboard
Badr AlKhamissi, Faisal Ladhak, Srinivasan Iyer, Veselin Stoyanov, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, Mona Diab
| Challenge: | Hate speech detection is complex and requires commonsense reasoning and social nuance . prior work has shown that even humans cannot achieve a high agreement on whether a post constitutes HS . |
| Approach: | They frame a few-shot learning task to decompose a hate speech detection task into its "constituent" parts. they show that infusing commonsense knowledge from reasoning datasets improves the performance even further. |
| Outcome: | The proposed method outperforms baseline methods in the 16-shot case. |
A Multi-lingual Multi-task Architecture for Low-resource Sequence Labeling (P18-1)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-task learning can boost the performance of related tasks such as MT and abstractive text summarization. |
| Approach: | They propose a multi-lingual multi-task architecture to develop supervised models with a minimal amount of labeled data for sequence labeling. |
| Outcome: | The proposed architecture achieves 4.3%-50.5% absolute gains compared to mono-lingual model . the proposed model is particularly effective in low-resource settings . |