Papers by Vinit Ravishankar
Attention Can Reflect Syntactic Structure (If You Let It) (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has attempted to decode linguistic structure from the Transformer . but, much of the work focused on English, a language with rigid word order and a lack of inflectional morphology. |
| Approach: | They propose to fine-tune a feature encoder for BERT to learn linguistic structure from its multi-head attention mechanism. |
| Outcome: | The proposed model can decode full trees above baseline accuracy from single attention heads across languages. |
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
The Impact of Positional Encodings on Multilingual Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Several modifications have been proposed to improve monolingual language models, but none of them result in better multilingual models. |
| Approach: | They propose to add positional encodings to token embeddings to preserve word-order information in a non-autoregressive setting. |
| Outcome: | The proposed modifications tend to improve monolingual models, but none improve multilingual models. |
A Closer Look at Parameter Contributions When Training Neural Language and Translation Models (2022.coling-1)
Copied to clipboard
| Challenge: | Neural models and Transformers have been used for almost every NLP task . however, the intrinsic dynamics of the training procedure have not been studied in depth for highly complex network architectures. |
| Approach: | They analyze the learning dynamics of neural language and translation models using Loss Change Allocation indicator . they use a standard Transformer architecture to train a model with three learning objectives . |
| Outcome: | The proposed model is based on a standard model that is used for training tasks. |
MGAD: Multilingual Generation of Analogy Datasets (L18-1)
Copied to clipboard
| Challenge: | Existing methods for word embedding evaluation are computationally expensive and task-specific. |
| Approach: | They propose a minimally supervised method for generating word embedding evaluation datasets for a large number of languages using existing dependency treebanks and parsers. |
| Outcome: | The proposed method evaluates three popular word embedding algorithms against these datasets and shows that their performance varies between syntactic categories. |
The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on inductive biases of language models towards natural language has focused on quantifying their ability to build multilingual spaces. |
| Approach: | They propose to use linguistically motivated tasks as a proxy to study inductive biases of language models with respect to natural language phenomena to build multilingual embedding spaces. |
| Outcome: | The proposed model performance is compared with other models using a set of linguistically motivated tasks and a training corpus in 15 languages. |
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual transformers are less effective in resource-lean scenarios and for distant languages. |
| Approach: | They propose to use massively multilingual transformers to pretrain languages . they show that MMTs are less effective in resource-lean scenarios and distant languages if they are pre-trained via language modeling . |
| Outcome: | The proposed model is less effective in resource-lean scenarios and for distant languages than cross-lingual word embeddings. |
Word Order Does Matter and Shuffled Language Models Know It (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. |
| Approach: | They propose a transformer-based BERT architecture that uses a fixed, sinusoidal position embedding added to each token embeddable to compensate for this absence of linear order. |
| Outcome: | The proposed model retains word order information because of the dependencies between sentence length and unigram probabilities. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |