Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units (P19-1)
Copied to clipboard
| Challenge: | Graphics Processing Units (GPUs) are used to train and evaluate neural networks efficiently. |
| Approach: | They propose two new GPU algorithms for multiplying a matrix by a few-hot vector and fused softmax and top-N selection. |
| Outcome: | The proposed algorithms achieve speedups over state-of-the-art parallel GPU baselines of up to 7x and 50x, respectively. |
Similar Papers
Effective Batching for Recurrent Neural Network Grammars (2021.findings-acl)
Copied to clipboard
| Challenge: | RNNGs are hard to scale due to the difficulty of batched training. |
| Approach: | They propose to batch RNNGs where every operation is computed in parallel with tensors across multiple sentences. |
| Outcome: | The proposed RNNG scales faster than existing models and achieves x6 speedup compared to existing C++ DyNet implementation . |
Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve accuracy of neural networks are slow due to computational complexity. |
| Approach: | They propose a vector-vector-matrix architecture which greatly reduces latency at inference time for NLP applications by a factor of four. |
| Outcome: | The proposed framework reduces the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. |
SparseFlow: Accelerating Transformers by Sparsifying Information Flows (2024.acl-long)
Copied to clipboard
| Challenge: | SparseFlow is an efficient method to sparsify the dense information flows within transformers. |
| Approach: | They propose a method to sparsify the dense pathways of token representations across all transformer blocks by parameterizing them to be sparse. |
| Outcome: | The proposed method reduces computational costs by half on average without compromising task accuracy. |
Optimized Speculative Sampling for GPU Hardware Accelerators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large foundational speech and language models require more memory and computational resources to generate long sequences. |
| Approach: | They propose to optimize speculative sampling for parallel hardware accelerators by combining multiple GPU threads to reduce profiling time. |
| Outcome: | The proposed approach improves profiling time from 6% to 13% without compromising accuracy. |
Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Ryan Solgi, Kai Zhen, Rupak Vignesh Swaminathan, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
| Challenge: | Low-rank tensor compression techniques are used for over-parameterized neural networks, but their applications to compress pre-trained LLMs for downstream tasks remain challenging due to the high-rank nature of pre-training data. |
| Approach: | They propose sparse augmented tensor networks to enhance low-rank tenorized LLMs . they also propose a framework that enables full model compression . |
| Outcome: | The proposed framework improves accuracy and efficiency in tensorized language models. |
Understanding and Improving Hidden Representations for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Existing studies have explored some methods for understanding hidden representations, but they have not sought to improve the translation quality rationally according to their understanding. |
| Approach: | They propose to construct a sequence of nested relative tasks and measure the feature generalization ability of the learned hidden representation over these tasks. |
| Outcome: | The proposed methods achieve consistent improvements (up to +1.3 BLEU) on two widely-used datasets. |
Sparse Sequence-to-Sequence Models (P19-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence models are dense and assigning nonzero probability to implausible outputs. |
| Approach: | They propose a new family of -entmax transformations that includes softmax and sparsemax as particular cases and is sparser for any > 1 . they provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. |
| Outcome: | The proposed models are able to produce sparse alignments and assign nonzero probability to short list of plausible outputs, sometimes rendering beam search exact. |
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads. |
| Approach: | They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates. |
| Outcome: | The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements. |
Neurons in Large Language Models: Dead, N-gram, Positional (2024.findings-acl)
Copied to clipboard
| Challenge: | We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. |
| Approach: | They analyze a family of large language models that rely on whether an FFN neuron is activated or not . they use a single GPU to analyze the OPT network and use it to analyze data . |
| Outcome: | The proposed model is lightweight and can be run on a single GPU. |
Tricks for Training Sparse Translation Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Multitask learning with an unbalanced data distribution skews model learning towards high resource tasks. |
| Approach: | They propose to use a temperature heating mechanism and dense pre-training to mitigate this by training models with a fixed model capacity. |
| Outcome: | The proposed techniques improve performance on two multilingual translation benchmarks compared to BASELayers and Dense scaling baselines and in combination, more than 2x model convergence speed. |