Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks (D18-1)
Copied to clipboard
| Challenge: | Activation functions are nonlinearities which have been attributed to the success story of deep learning. |
| Approach: | They propose to use a penalized tanh function to replace the sigmoid and tansh gates in LSTM cells and to improve the performance of the activation function. |
| Outcome: | The proposed activation function performs best on all tasks and can replace the sigmoid and tanh gates in LSTM cells. |
Similar Papers
Are Sample-Efficient NLP Models More Robust? (2023.acl-short)
Copied to clipboard
| Challenge: | Recent work in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distortion performance. |
| Approach: | They conduct a large empirical study to investigate the relationship between sample efficiency and robustness. |
| Outcome: | The results show that pre-trained models with lower sample efficiency perform better on some tasks but not others. |
Similarity-Distance-Magnitude Activations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for interpreting neural network-based language models (LMs) are limited to approximately conditional quantities. |
| Approach: | They introduce a similarity-distance-magnitude activation function and an SDM estimator to control class- and prediction-conditional accuracy among selective classifications. |
| Outcome: | The proposed estimator is more robust to covariate shifts and out-of-distribution inputs while remaining informative over in-difference data. |
Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations (C18-1)
Copied to clipboard
| Challenge: | Neural activation models have been proposed to map word semantics to localized neural activations. |
| Approach: | They propose a computational model that estimates semantic similarity in the neural activation space and investigate its performance for various natural language processing tasks. |
| Outcome: | The proposed model performs better than state-of-the-art word embeddings for the task of semantic similarity estimation between very similar or very dissimilar words while performing well on other tasks such as entailment and word categorization. |
Even the Simplest Baseline Needs Careful Re-investigation: A Case Study on XML-CNN (2022.naacl-main)
Copied to clipboard
| Challenge: | XML-CNN has been a popular research topic in NLP due to its superior performance . however, the increasing complexity brings difficulties to ensure the true architectural progress . |
| Approach: | They propose to re-examine an influential multi-label text classification method . they propose suitable baselines for multi-level text classification tasks . |
| Outcome: | The proposed method performs better than the original model, the authors show . they show that the re-implementation reveals contradictory results to the original work . |
Activation Reward Models for Few-Shot Model Alignment (2026.findings-acl)
Copied to clipboard
Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig
| Challenge: | A common approach is to use reward models that enable reinforcement-learning post-training. |
| Approach: | They propose a method that steers LLM activations to align with few-shot preference data without finetuning. |
| Outcome: | The proposed method surpasses zero-shot, few-shot and voting-based benchmarks on reward hacking and noise signals. |
Motivating Next-Gen Accelerators with Flexible N:M Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches (2026.acl-industry)
Copied to clipboard
Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina, Vladimir Smirnov, Redko Dmitry, Alexey Dontsov, Maxim Zhelnin, Evgeny Burnaev, Egor Shvetsov
| Challenge: | Recent studies show that sparsification is not supported in large language models. |
| Approach: | They propose to use activation pruning to accelerate large language models with sparsification . they compare activation pruners with weight pruner and activater pruning with activation . |
| Outcome: | The proposed approach outperforms weight pruning at matched sparsity levels. |
Unraveling Feature Extraction Mechanisms in Neural Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks have become indispensable across a variety of natural language processing tasks. |
| Approach: | They propose a theoretical approach based on Neural Tangent Kernels to investigate neural networks' internal mechanisms. |
| Outcome: | The proposed approach can be applied to analyze language modeling tasks . it shows that the choice of activation function can affect feature extraction . |
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | In neural sequence-to-sequence learning, Reinforcement Learning (RL) has gained popularity due to the suitability of Policy Gradient (PG) methods for the end-to end training paradigm. |
| Approach: | They propose to let the model explore the output space beyond the reference output that is used for standard cross-entropy minimization by reinforcing model outputs according to their quality, effectively increasing the likelihood of higher-quality samples. |
| Outcome: | The proposed model explores the output space beyond the reference output that is used for cross-entropy minimization, increasing the likelihood of higher-quality samples. |
Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019) (D19-61)
Copied to clipboard
| Challenge: | EMNLP-IJCNLP 2019 Workshop on Deep Learning Approaches for Low-Resource Natural Language Processing takes place in Hong Kong, China . |
| Approach: | EMNLP-IJCNLP 2019 Workshop on Deep Learning Approaches for Low-Resource Natural Language Processing takes place in Hong Kong, China . call for papers for this second workshop met with a strong response . |
| Outcome: | the EMNLP-IJCNLP 2019 workshop on deep learning approaches for low-resource natural language processing takes place in Hong Kong, China. |
Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling Baselines (N19-1)
Copied to clipboard
| Challenge: | Experimental results show that standalone n-gram models lend themselves as natural choices for resource-lean or morphologically rich languages. |
| Approach: | They run experiments on 50 languages covering all morphological language families to compare n-gram models with lstm models. |
| Outcome: | The proposed extension outperforms an lstm language model on 42 languages while its extension which explicitly injects linguistic knowledge outperformed the character-aware neural model on 8 languages. |