Papers by Phillip Keung
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER (D19-1)
Copied to clipboard
| Challenge: | Contextual word embeddings have demonstrated state-of-the-art performance on various NLP tasks. |
| Approach: | They propose to use adversarial learning to improve upon multilingual BERT's zero-resource cross-lingual performance by aligning embeddings of English documents and their translations. |
| Outcome: | The multilingual version of BERT performs surprisingly well in cross-lingual settings, even when only labeled English data is used to finetune the model. |
The Engage Corpus: A Social Media Dataset for Text-Based Recommender Systems (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies have examined the impact of recommendation algorithms on how users discover and join online groups, but there are few standardized datasets for generating such models. |
| Approach: | They propose to use Reddit to build a dataset that can be used to build models of user engagement with online groups. |
| Outcome: | The proposed model is based on the behavior of subreddits banned in June 2020 as part of Reddit's efforts to stop the dissemination of hate speech. |
The Multilingual Amazon Reviews Corpus (2020.emnlp-main)
Copied to clipboard
| Challenge: | The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019 . |
| Approach: | They propose to use mean absolute error (MAE) instead of classification accuracy for this task since MAE accounts for ordinal nature of the ratings. |
| Outcome: | The proposed model uses mean absolute error (MAE) instead of classification accuracy since MAE accounts for ordinal nature of the ratings. |
Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning. |
| Approach: | They show that English dev accuracy makes it difficult to obtain reproducible results . they recommend providing oracle scores alongside zero-shot results if possible . |
| Outcome: | mBERT and XLM have shown strong performance on cross-lingual recognition, text classification, dependency parsing, and other tasks. |
Domain Mismatch Doesn’t Always Prevent Cross-lingual Transfer Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent studies have reported that domain mismatch prevents cross-lingual transfer . UBLI and UNMT do not work well when underlying monolingual corpora come from different domains . |
| Approach: | They show that a simple initialization regimen can overcome domain mismatch in cross-lingual transfer . they pre-train word embeddings on concatenated domain-mismatched corpora and use them as initializations . |
| Outcome: | The initialization regimen can overcome the domain mismatch effect in cross-lingual transfer learning . the initializations were used for MUSE UBLI, UN Parallel UNMT, and the SemEval 2017 task . |
Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to extract parallel sentences from unaligned text yield surprisingly good results. |
| Approach: | They propose an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text using multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. |
| Outcome: | The proposed method outperforms existing methods and outperformed previous state-of-the-art methods by boosting translation performance by up to 3.5 BLEU on the WMT’14 French-English and WMT'16 German-English tasks. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |