Cascaded Semantic and Positional Self-Attention Network for Document Classification

Challenge:	Existing approaches to document classification combine semantic information with positional information (word orders) . document classification is one of the fundamental problems in natural language processing .
Approach:	They propose a new architecture to combine semantic and positional information using a semantic self-attention layer cascaded with Bi-LSTM.
Outcome:	The proposed model can exploit the interaction between semantics and word positions in a more interpretable and adaptive manner while preserving a compact model size and high convergence rate.

Similar Papers

Overview

Challenge

Approach

Outcome

Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (D18-1)

Copied to clipboard

Dinghan Shen, Xinyuan Zhang, Ricardo Henao, Lawrence Carin

Representation Learning

Challenge:	Existing approaches to network embeddings focus on one-hot representations of vertices, which are not able to capture relationships between verti- ces.
Approach:	They propose to integrate semantic features into network embeddings by matching important words between text sequences for all pairs of vertices.
Outcome:	The proposed framework outperforms state-of-the-art embedding methods on three real-world benchmarks for downstream tasks including link prediction and multi-label vertex classification.

DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures (2026.eacl-long)

Copied to clipboard

Benno Uthayasooriyar, Antoine Ly, Franck Vermet, Caio Corro

Semantic Text Processing

Challenge:	Existing models that take text block positions into account are not efficient for document understanding.
Approach:	They propose a layout-aware BERT model that takes into account text block positions in relative polar coordinate system rather than the Cartesian one.
Outcome:	The proposed model eliminates the need for absolute positional embeddings on a dataset more than six times smaller than the widely used IIT-CDIP corpus.

Assessing the Ability of Self-Attention Networks to Learn Word Order (P19-1)

Copied to clipboard

Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, Zhaopeng Tu

Semantic Text Processing

Challenge:	Existing studies have attributed SAN to being weak at learning positional information for sequence modeling due to lack of recurrence structure.
Approach:	They propose a word reordering detection task to quantify how well word order information is learned by SAN and RNN.
Outcome:	The proposed task quantifies how well word order information learned by SAN and RNN is learned.

Investigating Capsule Network and Semantic Feature on Hyperplanes for Text Classification (D19-1)

Copied to clipboard

Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao, Chun Wang, Bing Ma

Information Retrieval

Challenge:	Various neural networks are designed for text classification on the basis of word embedding, but polysemy is a fundamental feature of the natural language, which brings challenges to text classification.
Approach:	They propose to use capsule networks to construct the vectorized representation of semantics and utilize hyperplanes to decompose each capsule to acquire the specific senses.
Outcome:	The proposed model extracts more discriminative semantic features and yields significant performance gain compared to baseline methods.

Joint Semantic and Distributional Word Representations with Multi-Graph Embeddings (D19-53)

Copied to clipboard

Pierre Daix-Moreux, Matthias Gallé

Semantic Text Processing

Challenge:	Prior work has shown that word embeddings can be improved by using semantic knowledge-bases.
Approach:	They propose a way to combine distributional and semantic information while preserving lexical information of co-occurrences of words.
Outcome:	The proposed method improves word embeddings on a variety of word similarities.

Recurrent Attention Networks for Long-text Modeling (2023.findings-acl)

Copied to clipboard

Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, Qing Li

Information Extraction & Text Mining

Challenge:	Existing approaches to encoding long documents using self-attention have been limited by quadratic computational complexities and limited application in long text processing.
Approach:	They propose a long-document encoding model that allows the recurrent operation of self-attention.
Outcome:	The proposed model extracts global semantics in token-level and document-level representations, making it inherently compatible with both sequential and sequential tasks.

HIT - A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation (2021.findings-acl)

Copied to clipboard

Ayan Sengupta, Sourabh Kumar Bhattacharjee, Tanmoy Chakraborty, Md. Shad Akhtar

Semantic Text Processing

Challenge:	linguistics and morphology of resource-short code-mixed texts remain a key challenge in text processing.
Approach:	They propose a hierarchical transformer-based framework that captures the semantic relationship among words and hierarchically learns sentencelevel semantics using a fused attention mechanism.
Outcome:	The proposed framework improves on one European and five Indic languages on four NLP tasks on eleven datasets.

Self-Attention with Structural Position Representations (D19-1)

Copied to clipboard

Xing Wang, Zhaopeng Tu, Longyue Wang, Shuming Shi

Semantic Text Processing

Challenge:	Experimental results show that SANs can't encode positions of input words . SAN's are currently lacking in encoding positions of words based on position-unaware "bagof-words" theory .
Approach:	They propose to augment SANs with structural position representations to capture latent structure of input sentence.
Outcome:	The proposed approach consistently outperforms the sequential representations on translation tasks.

Enhancing Self-Attention with Knowledge-Assisted Attention Maps (2022.naacl-main)

Copied to clipboard

Jiangang Bai, Yujing Wang, Hong Sun, Ruonan Wu, Tianmeng Yang, Pengfei Tang, Defu Cao, Mingliang Zhang1, Yunhai Tong, Yaming Yang, Jing Bai, Ruofei Zhang, Hao Sun, Wei Shen

Semantic Text Processing

Challenge:	Existing works of knowledge infusion depend on multi-task learning frameworks, which are inefficient and require large-scale retraining when new knowledge is considered.
Approach:	They propose a method which integrates knowledge-generated attention maps into the self-attention mechanism and integrates it into the model.
Outcome:	The proposed model outperforms existing methods on academic datasets and industry-scale ad relevance applications.

Hierarchical Transformers for Multi-Document Summarization (P19-1)

Copied to clipboard

Yang Liu, Mirella Lapata

Language Models

Challenge:	Existing models for multidocument summarization have been developed that can process multiple documents in a hierarchical manner.
Approach:	They propose a neural summarization model which can process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner.
Outcome:	The proposed model improves on the WikiSum dataset and can process multiple documents in a hierarchical manner.