Papers by Jiezhong Qiu

GenGO

Omakase Read Later

Search

Papers by Jiezhong Qiu

ACL
Findings

2022.acl-long
2020.findings-emnlp

downstream tasks
BlockBERT
language model pre-training
BERT-based model
RoBERTa
memory consumption
inference time
prediction accuracy
modeling long-distance dependencies
BERT

2022
2020

Language Models

2 papers

GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022.acl-long)

Copied to clipboard

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang

Language Models

Challenge:	Existing pretraining frameworks do not perform well for all tasks of three main categories, such as natural language understanding (NLU), unconditional generation, and conditional generation.
Approach:	They propose a general language model based on autoregressive blank infilling to address this challenge.
Outcome:	The proposed model outperforms BERT, T5, and GPT on a wide range of tasks across NLU, conditional and unconditional generation tasks.

Blockwise Self-Attention for Long Document Understanding (2020.findings-emnlp)

Copied to clipboard

Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, Jie Tang

Language Models

Challenge:	Recent advances in pre-training and fine-tuning methods have drastically reshaped the landscape of natural language processing research.
Approach:	They propose a lightweight BERT model that introduces sparse block structures into the attention matrix to reduce memory consumption and training/inference time.
Outcome:	The proposed model uses 18.7-36.1% less memory and 12.0-25.1% more time to learn compared to an advanced BERT-based model, RoBERTa.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations