Papers by Hangbo Bao
Attention Temperature Matters in Abstractive Summarization Distillation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent progress of abstractive text summarization relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. |
| Approach: | They propose to distill large Transformer summarization models into smaller ones with minimal performance loss by manipulating attention temperatures in Transformers. |
| Outcome: | The proposed method outperforms vanilla pseudo-labeling based methods on three summarization datasets and is shorter and more abstractive. |
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | Experimental results show that unified model outperforms other models that treat encoding and matching separately. |
| Approach: | They evaluate a unified model with Transformer layers for machine reading comprehension . they find that the model learns different modeling strategies compared with previous models . |
| Outcome: | The unified model outperforms models with Transformer layers on the machine reading comprehension task. |
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption (2022.findings-acl)
Copied to clipboard
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei
| Challenge: | enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks. |
| Approach: | They propose an approximation approach for transformers which enables inference on ciphertext data. |
| Outcome: | The proposed approach can infer pre-trained models on encrypted data with negligible performance drop but enjoy theory-guaranteed privacy-preserving advantage. |
Learning to Sample Replacements for ELECTRA Pre-Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that ELECTRA pretrains a discriminator to detect replaced tokens . despite compelling performance, there is no direct feedback loop from discriminator and generator to generator, making replacements biased to correct tokens. |
| Approach: | They propose to augment sampling with a hardness prediction mechanism to encourage the discriminator to learn what it has not acquired. |
| Outcome: | The proposed method improves ELECTRA pre-training on various downstream tasks. |
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on deep self-attention distillation for natural language processing tasks is limited by computational resources and latency. |
| Approach: | They generalize deep self-attention distillation in MINILM by using only self- attention relation distillation for taskagnostic compression of pretrained Transformers. |
| Outcome: | The proposed model outperforms the state-of-the-art in a multilingual and multilingual teacher model. |