Papers by Olga Kovaleva
Down and Across: Introducing Crossword-Solving as a New NLP Benchmark (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in NLP have led to a growing demand for challenging tasks. |
| Approach: | They propose to solve crossword puzzles as a natural language understanding task . they release a corpus of crossword clues from the daily crossword spanning 25 years . |
| Outcome: | The proposed task is based on a corpus of crossword puzzles from the new york times daily crossword spanning 25 years . the dataset contains over half a million unique clue-answer pairs . |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |
Similarity-Based Reconstruction Loss for Meaning Representation (D18-1)
Copied to clipboard
| Challenge: | a new paper proposes and evaluates a set of loss functions that can be used to train models for representation learning . cross-entropy loss penalizes models when they fail to generate the exact word from ground truth data . |
| Approach: | They propose and evaluate loss functions that can be used to train any neural model for representation learning. |
| Outcome: | The proposed loss functions amplify semantic diversity while preserving original meaning . they show performance improvement on paraphrase detection and language inference tasks . |
Calls to Action on Social Media: Detection, Social Impact, and Censorship Potential (D19-50)
Copied to clipboard
| Challenge: | Calls to action are effective means of mobilization in social networks, but their potential for censorship and predicting offline protest events has not yet been evaluated. |
| Approach: | They examine the possibility of their automatic detection on historical data from the 2011-2013 protests in Bolotnaya, Russia. |
| Outcome: | The political calls to action can be annotated and detected with relatively high accuracy and have a moderate positive correlation with actual rally attendance. |
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained Transformers are remarkably robust to pruning. |
| Approach: | They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance. |
| Outcome: | The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |