Papers by Minchul Lee
LittleBird: Efficient Faster & Longer Transformer for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for BERT have a limitation dealing with long inputs due to its attention mechanism. |
| Approach: | They propose a model based on BigBird with improved speed and memory footprint . they propose 'pack and unpack attention' to replace global attention . |
| Outcome: | The proposed model can work on long inputs even after being pre-trained on short inputs. |