Papers by Michael Chung
RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)
Copied to clipboard
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
| Challenge: | recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability. |
| Approach: | They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. |
| Outcome: | The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models. |
The Alexa Meaning Representation Language (N18-3)
Copied to clipboard
Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, Spyros Matsoukas
| Challenge: | a new meaning representation language for spoken language is introduced for Alexa . AMRL provides a common representation for how people communicate in spoken language . there is no mechanism to represent ambiguity, forcing the choice of a fixed interpretation for ambiguous utterances. |
| Approach: | They introduce a meaning representation for spoken language, the Alexa meaning represent language . they use a spoken language dataset to collect a sample of utterances from eight domains . |
| Outcome: | The proposed representation provides a common representation for spoken language understanding . it supports cross-domain queries, fine-grained types, complex utterances and composition . the proposed representation was released to developers at a trade show in 2016 . |
Do Transformer Modifications Transfer Across Implementations and Applications? (2021.emnlp-main)
Copied to clipboard
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
| Challenge: | Currently, the Transformer is the de facto architecture of choice for processing sequential data. |
| Approach: | They evaluate the Transformer architecture and its modifications in a shared experimental setting . they conjecture that performance improvements may strongly depend on implementation details . |
| Outcome: | The proposed improvements do not significantly improve performance, the authors find . the proposed improvements are either developed in the same codebase or are minor changes . |
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Experimental results show that SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%. |
| Approach: | They propose a semi-supervised learning framework to jointly pre-train the speech and language modules using unpaired speech and text. |
| Outcome: | The proposed framework improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%. |