Improving Textual Network Embedding with Global Attention via Optimal Transport (P19-1)
Copied to clipboard
Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, Lawrence Carin
| Challenge: | Existing methods for learning textual network embeddings are noisy and sparse. |
| Approach: | They propose to use text-based attention parsing to learn context-aware network embeddings. |
| Outcome: | The proposed model outperforms state-of-the-art methods in a number of domains. |
Similar Papers
Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to network embeddings focus on one-hot representations of vertices, which are not able to capture relationships between verti- ces. |
| Approach: | They propose to integrate semantic features into network embeddings by matching important words between text sequences for all pairs of vertices. |
| Outcome: | The proposed framework outperforms state-of-the-art embedding methods on three real-world benchmarks for downstream tasks including link prediction and multi-label vertex classification. |
Be More with Less: Hypergraph Attention Networks for Inductive Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text classification is a critical research topic with broad applications in natural language processing. graph neural networks (GNNs) have received increasing attention but their performance is jeopardized in practice. |
| Approach: | They propose a model which captures long-distance interactions between words and a graph-based model which can be used to perform text classification. |
| Outcome: | The proposed model can achieve more expressive power with less computational consumption on the text classification task. |
Contrastive Document Representation Learning with Graph Attention Networks (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document representation learning are significantly affected by the scarcity of document-level data. |
| Approach: | They propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. |
| Outcome: | Empirically, the proposed approach is effective in document classification and document retrieval tasks. |
Global Textual Relation Embedding for Relational Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing methods to learn textual relation embeddings are lacking in large open-domain corpora. |
| Approach: | They propose to learn a general-purpose embedding of textual relations using a large dataset from Freebase. |
| Outcome: | The proposed embedding can facilitate downstream tasks requiring relational understanding of the text. |
Deep Attention Diffusion Graph Neural Networks for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text classification based on graph neural networks (GNNs) consider only one-hop neighborhoods and low-frequency information within texts, which suffer from over-smoothing issues if many graph layers are stacked. |
| Approach: | They propose a deep attention diffusion Graph Neural Network model to learn text representations by bridging the chasm of interaction difficulties between a word and its distant neighbors. |
| Outcome: | The proposed model outperforms existing methods on standard benchmark datasets on a set of textual features. |
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent graph-to-text models generate text from graph data using global or local aggregation . global node encoding allows explicit communication between two distant nodes, but fails to capture long-range relationships. |
| Approach: | They propose to combine global and local aggregation to learn node representations . they propose to use global and locally encoding to learn contextualized node embeddings based on graph data . |
| Outcome: | The proposed models outperform state-of-the-art models on two graph-to-text datasets by 18.01 and 63.69 points. |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
A Deep Neural Information Fusion Architecture for Textual Network Embeddings (D19-1)
Copied to clipboard
| Challenge: | Textual network embeddings aim to learn a low-dimensional representation for every node in the network while seeking to retain the original network information. |
| Approach: | They propose a deep neural architecture to fuse the two kinds of informations into one representation. |
| Outcome: | The proposed model outperforms the comparing methods on all three datasets. |
Recurrent Attention Networks for Long-text Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to encoding long documents using self-attention have been limited by quadratic computational complexities and limited application in long text processing. |
| Approach: | They propose a long-document encoding model that allows the recurrent operation of self-attention. |
| Outcome: | The proposed model extracts global semantics in token-level and document-level representations, making it inherently compatible with both sequential and sequential tasks. |
Leveraging Local and Global Patterns for Self-Attention Networks (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to integrate local and global information into self-attention networks have been criticized for overlooking neighboring information. |
| Approach: | They propose a hybrid attention mechanism to leverage local and global information . they use a gating scalar to integrate both sources of information based on local contexts . |
| Outcome: | The proposed approach improves on translation tasks and shows that the two types of contexts are complementary. |