Papers with exBERT
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020.acl-demos)
Copied to clipboard
| Challenge: | Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. |
| Approach: | They propose a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. |
| Outcome: | Using exBERT, we can analyze the representations and attentions of large language models and extend them to previously not analyzed models. |
exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models with limited corpus and computational resources are limited by the complexity of the training resources. |
| Approach: | They propose a method to extend BERT pre-trained models from a general domain to a new pre-train model for a specific domain with a different additive vocabulary. |
| Outcome: | The proposed method outperforms existing methods on biomedical benchmark tasks using the MTL-Bioinformatics-2016 dataset. |