Papers by Benjamin Bolte
On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)
Copied to clipboard
Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa
| Challenge: | Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models. |
| Approach: | They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. |
| Outcome: | The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task. |
On Generative Spoken Language Modeling from Raw Audio (2021.tacl-1)
Copied to clipboard
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux
| Challenge: | Using a set of metrics to evaluate the learned representations, we aim to create a system that learns from natural interactions as infants learn their first language. |
| Approach: | They propose a task of learning acoustic and linguistic characteristics from raw audio and a set of metrics to evaluate the learned representations at acustic, linguistic and encoding levels. |
| Outcome: | The proposed models evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. |