| Challenge: | ProFormer is a projection based transformer architecture that is faster and lighter making it suitable to deploy to memory constraint devices such as mobile phones, watches and IoT. |
| Approach: | They propose a projection based transformer architecture that generates word representations on-the-fly without embedding lookup tables and a local projection attention layer that transforms the input sequence of N LSH word projections into a sequence of K representations. |
| Outcome: | The proposed architecture reduces memory footprint from 92.16 MB to 1.7 KB and requires 16x less computation overhead making it suitable to deploy to memory constraint devices and preserve user privacy. |
Similar Papers
On-Device Text Representations Robust To Misspellings via Projections (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in Locality-Sensitive Hashing (LSH)-based projection networks have demonstrated state-of-the-art performance in various classification tasks without explicit word embedding lookup tables by computing on-the fly text representations. |
| Approach: | They propose to use locality-sensitive hashing to compute on-the-fly text representations without explicit word embedding tables. |
| Outcome: | The proposed classifiers are more robust to common misspellings and perturbations of the input text compared to biLSTMs and fine-tuned BERT based methods. |
PRADO: Projection Attention Networks for Document Classification On-Device (D19-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning have improved the performance of on-device neural networks for long text classification. |
| Approach: | They propose a projection attention neural network PRADO that combines trainable projections with attention and convolutions to train tiny neural networks that achieve high performance on multiple long document classification tasks. |
| Outcome: | The proposed model achieves high performance on multiple long document classification tasks while maintaining compact size. |
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | DeFormer is a transformer-based QA model that uses input-wide self-attention at all layers, causing them to be slow and memory-intensive. |
| Approach: | They propose a transformer which substitutes the full self-attention with question-wide and passage-wide self- attentions in the lower layers. |
| Outcome: | The proposed model can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
HashFormers: Towards Vocabulary-independent Pre-trained Transformers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. |
| Approach: | They propose a family of vocabulary-independent pre-trained transformers that support unlimited vocabulary . they propose to map each token to its corresponding embedding by default . |
| Outcome: | The proposed models are more memory efficient than existing models while achieving comparable performance on multiple text classification tasks. |
Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, Dongsoo Lee
| Challenge: | Quantization is an effective technique to address heavy computation load and memory overhead during inference. |
| Approach: | They propose a low-bit quantization strategy to represent Transformer weights by an extremely low number of bits. |
| Outcome: | The proposed model achieves 11.8 smaller model size than baseline model, with less than -0.5 BLEU. |
Fast Word Predictor for On-Device Application (2020.coling-demos)
Copied to clipboard
| Challenge: | a proposed word prediction model is developed for a chat application serving more than 100 million users. |
| Approach: | They propose a fast word predictor that reduces memory size and inference time on mobile devices. |
| Outcome: | The proposed model reduces memory size and inference time on a mobile device compared with a standard neural network . it achieves robust performance by learning on large text corpora and is available on microsoft's chat app . |
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)
Copied to clipboard
| Challenge: | Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks. |
| Approach: | They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks. |
| Outcome: | The proposed model is based on a self-attention-based model with a high input length threshold. |
Modularized Transfomer-based Ranking Framework (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. |
| Approach: | They propose to modularize a Transformer ranker into separate modules for text representation and interaction. |
| Outcome: | The proposed model is faster than previous models and is easier to interpret and understand. |
A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection tasks (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Virtual assistants often can handle only a limited scope of intents. |
| Approach: | They propose to combine out-of-scope (OOS) rejection with intent classification . they propose to regularize cross-entropy loss with an in-scope embedding reconstruction loss . |
| Outcome: | The proposed method achieves a 1-4% improvement in the area under the precision-recall curve for rejecting out-of-sample instances without compromising intent classification performance. |