Papers by SiYu Ding
ERNIE-Doc: A Retrospective Long-Document Modeling Transformer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for document-level language pretraining are not suitable for long documents due to their quadratically increasing memory and time consumption. |
| Approach: | They propose a document-level language pretraining model based on Recurrence Transformers. |
| Outcome: | The proposed model outperforms existing models on language understanding tasks. |