Papers with convolutions
Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection (2020.acl-main)
Copied to clipboard
| Challenge: | Off-topic spoken response detection is crucial for an automated speaking assessment system. |
| Approach: | They propose a novel approach for off-topic spoken response detection with high off-top recall on both seen and unseen prompts. |
| Outcome: | The proposed model achieves significant improvements in detecting off-topic responses with extremely high on-topic recall on both seen and unseen prompts. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Efficient Convolutional Neural Networks for Diacritic Restoration (D19-1)
Copied to clipboard
| Challenge: | Diacritic restoration is a computational task that requires a computer to understand written texts. |
| Approach: | They propose to use Temporal Convolutional Neural Networks (TCN) to restore missing diacritics for each character in written text. |
| Outcome: | The proposed model improves on TCN in Arabic, Yoruba, and Vietnamese. |
Adaptive Convolution for Text Classification (N19-1)
Copied to clipboard
| Challenge: | Existing convolutional neural networks (CNNs) use sparse representations of text, such as bag-of-words. |
| Approach: | They propose an adaptive convolution for text classification to give flexibility to convolutional neural networks (CNNs) they attach filter-generating networks to convevolution blocks in existing CNNs . |
| Outcome: | The proposed convolution improves performance in seven benchmark datasets by 2.6 percentage points . the proposed conversions can be likened to players of the twenty questions . |
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that convolutions have been successful in natural language learning. |
| Approach: | They propose a convolutional approach to construct relative position embeddings in self-attention layers and propose 'compact attention' they propose multiple ways to integrate convolutions into Transformer self- attention. |
| Outcome: | The proposed composite attention improves performance on multiple downstream tasks, replacing absolute position embeddings, and is more expressive than convolutions in NLP. |
PRADO: Projection Attention Networks for Document Classification On-Device (D19-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning have improved the performance of on-device neural networks for long text classification. |
| Approach: | They propose a projection attention neural network PRADO that combines trainable projections with attention and convolutions to train tiny neural networks that achieve high performance on multiple long document classification tasks. |
| Outcome: | The proposed model achieves high performance on multiple long document classification tasks while maintaining compact size. |