Papers by Phillip Smith
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
The Engage Corpus: A Social Media Dataset for Text-Based Recommender Systems (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies have examined the impact of recommendation algorithms on how users discover and join online groups, but there are few standardized datasets for generating such models. |
| Approach: | They propose to use Reddit to build a dataset that can be used to build models of user engagement with online groups. |
| Outcome: | The proposed model is based on the behavior of subreddits banned in June 2020 as part of Reddit's efforts to stop the dissemination of hate speech. |
Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability (2021.acl-long)
Copied to clipboard
| Challenge: | In-domain experts are recruited to reannotate augmented samples and determine to what extent each strategy preserves the original rating. |
| Approach: | They implement 7 different data augmentation strategies for the task of automatic scoring of children’s ability to understand others’ thoughts, feelings, and desires. |
| Outcome: | The data augmentation strategies outperform task-agnostic augmentations and automatic augmentation systems perform worst on the MIND-CA corpus. |
The Multilingual Amazon Reviews Corpus (2020.emnlp-main)
Copied to clipboard
| Challenge: | The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019 . |
| Approach: | They propose to use mean absolute error (MAE) instead of classification accuracy for this task since MAE accounts for ordinal nature of the ratings. |
| Outcome: | The proposed model uses mean absolute error (MAE) instead of classification accuracy since MAE accounts for ordinal nature of the ratings. |
“What is on your mind?” Automated Scoring of Mindreading in Childhood and Early Adolescence (2020.coling-main)
Copied to clipboard
Venelin Kovatchev, Phillip Smith, Mark Lee, Imogen Grumley Traynor, Irene Luque Aguilera, Rory Devine
| Challenge: | Existing studies show that children who excel at mindreading are more likely to be identified as popular by classmates and have reciprocated friendships. |
| Approach: | They propose to automate the scoring of mindreading ability in middle childhood and early adolescence using a new corpus of 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . |
| Outcome: | The proposed scoring system is based on 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . the results demonstrate the applicability of state-of-the-art NLP solutions to a new domain and task. |
Domain Mismatch Doesn’t Always Prevent Cross-lingual Transfer Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent studies have reported that domain mismatch prevents cross-lingual transfer . UBLI and UNMT do not work well when underlying monolingual corpora come from different domains . |
| Approach: | They show that a simple initialization regimen can overcome domain mismatch in cross-lingual transfer . they pre-train word embeddings on concatenated domain-mismatched corpora and use them as initializations . |
| Outcome: | The initialization regimen can overcome the domain mismatch effect in cross-lingual transfer learning . the initializations were used for MUSE UBLI, UN Parallel UNMT, and the SemEval 2017 task . |
Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to extract parallel sentences from unaligned text yield surprisingly good results. |
| Approach: | They propose an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text using multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. |
| Outcome: | The proposed method outperforms existing methods and outperformed previous state-of-the-art methods by boosting translation performance by up to 3.5 BLEU on the WMT’14 French-English and WMT'16 German-English tasks. |