Papers by Mostafa Abdou
Attention Can Reflect Syntactic Structure (If You Let It) (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has attempted to decode linguistic structure from the Transformer . but, much of the work focused on English, a language with rigid word order and a lack of inflectional morphology. |
| Approach: | They propose to fine-tune a feature encoder for BERT to learn linguistic structure from its multi-head attention mechanism. |
| Outcome: | The proposed model can decode full trees above baseline accuracy from single attention heads across languages. |
Better, Faster, Stronger Sequence Tagging Constituent Parsers (N19-1)
Copied to clipboard
| Challenge: | Existing efforts to speed up constituent parsing have focused on chart-based or shift-reduce parsers. |
| Approach: | They propose to use auxiliary losses and sentence-level fine-tuning to mitigate greedy decoding issues. |
| Outcome: | The proposed model surpasses the performance of sequence tagging constituent parsers on the English and Chinese Penn Treebank datasets and reduces their parsing time even further. |
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
Mapping Brains with Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | accumulated evidence for brain and language model activations remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism. |
| Approach: | They examine the evidence accumulated by 30 studies spanning 10 datasets and 8 metrics to determine whether there is any overlap between brain and language model activations. |
| Outcome: | The findings suggest that representations extracted from NLP models can (partially) explain the signal found in neural data. |
MGAD: Multilingual Generation of Analogy Datasets (L18-1)
Copied to clipboard
| Challenge: | Existing methods for word embedding evaluation are computationally expensive and task-specific. |
| Approach: | They propose a minimally supervised method for generating word embedding evaluation datasets for a large number of languages using existing dependency treebanks and parsers. |
| Outcome: | The proposed method evaluates three popular word embedding algorithms against these datasets and shows that their performance varies between syntactic categories. |
Higher-order Comparisons of Sentence Encoder Representations (D19-1)
Copied to clipboard
| Challenge: | a technique developed by neuroscientists compares activity patterns of different measurement modalities . a recent study examined the correspondence between popular pretrained language encoders and human processing difficulty . |
| Approach: | They employ a technique to compare activity patterns of different measurement modalities . they establish a correspondence between widely-employed pretrained language encoders and human processing difficulty . |
| Outcome: | The proposed technique can be used to compare representational geometries of neural models . it does not require large training samples and is not prone to overfitting, authors say . |
Word Order Does Matter and Shuffled Language Models Know It (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. |
| Approach: | They propose a transformer-based BERT architecture that uses a fixed, sinusoidal position embedding added to each token embeddable to compensate for this absence of linear order. |
| Outcome: | The proposed model retains word order information because of the dependencies between sentence length and unigram probabilities. |
X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension (D19-61)
Copied to clipboard
| Challenge: | Existing knowledge bases are heavily biased towards English, but Wikipedias cover very different topics in different languages. |
| Approach: | They propose a multilingual dataset that frams relation extraction as a machine reading problem. |
| Outcome: | The proposed model can be used to transfer models cross-lingually and improves knowledge base completion across languages. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Challenges and Strategies in Cross-Cultural NLP (2022.acl-long)
Copied to clipboard
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, Anders Søgaard
| Challenge: | Various efforts have been made to accommodate linguistic diversity and serve speakers of many different languages. |
| Approach: | They propose a framework to examine cultural differences in NLP to better serve users . they argue that cultural knowledge, preferences and values can affect NLP practices . |
| Outcome: | The proposed framework examines how cultural knowledge, preferences and values can affect NLP practices. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |