GAProtoNet: A Multi-head Graph Attention-based Prototypical Network for Interpretable Text Classification

Challenge:	Existing models with black-box nature obscure decision-making process and lack interpretability.
Approach:	They propose a multi-head graph attention-based prototypical network that uses a vector and prototypes to learn an interpretable prototypical representation.
Outcome:	The proposed model achieves superior results without sacrificing the accuracy of the original black-box LMs.

Similar Papers

Overview

Challenge

Approach

Outcome

Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models (2023.findings-emnlp)

Copied to clipboard

Sean Xie, Soroush Vosoughi, Saeed Hassanpour

Explainability & Interpretability in NLP

Challenge:	Existing methods for interpreting LLMs are post hoc and focus on low-level features and lack of explainability at higher-level text units.
Approach:	They propose a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings during the fine-tuning stage while maintaining competitive performance.
Outcome:	The proposed framework can learn interpretable embeddings during the fine-tuning stage while maintaining competitive performance.

GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation (2022.coling-1)

Copied to clipboard

Anthony Colas, Mehrdad Alvandipour, Daisy Zhe Wang

Semantic Text Processing

Challenge:	Recent improvements in KG-to-text generation are due to additional pre-training tasks . these tasks require extensive computational resources while only suggesting marginal improvements.
Approach:	They propose a mask structure to capture neighborhood information and a type encoder that adds a bias to the graph-attention weights depending on the connection type.
Outcome:	The proposed model outperforms state-of-the-art models while requiring no additional pre-training tasks.

Pretraining Language Models with Text-Attributed Heterogeneous Graphs (2023.findings-emnlp)

Copied to clipboard

Tao Zou, Le Yu, Yifei Huang, Leilei Sun, Bowen Du

Language Models

Challenge:	Existing pretraining tasks for Language Models (LMs) focus on learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs.
Approach:	They propose a topology-aware pretraining task that explicitly considers the topological and heterogeneous information in TAHGs by optimizing an LM and an auxiliary heterogenous graph neural network.
Outcome:	The proposed framework explicitly considers the topological and heterogeneous information in TAHGs.

Roles and Utilization of Attention Heads in Transformer-based Neural Language Models (2020.acl-main)

Copied to clipboard

Jae-young Jo, Sung-Hyon Myaeng

Language Models

Challenge:	Sentence encoders based on transformer architectures have shown promising results on various natural language understanding tasks.
Approach:	They propose a sentence representation method that takes advantage of most influential attention heads.
Outcome:	The proposed method improves performance on the downstream tasks.

Syntax-Enhanced Pre-trained Model (2021.acl-long)

Copied to clipboard

Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, Nan Duan

Syntactic Text Processing

Challenge:	Existing methods that use syntax of text in pre-training and fine-tuning suffer from discrepancy between the two stages.
Approach:	They propose a model that utilizes the syntactic structure of text in pre-training and fine-tuning stages.
Outcome:	The proposed model achieves state-of-the-art on six public benchmark datasets.

PROMINET: Prototype-based Multi-View Network for Interpretable Email Response Prediction (2023.emnlp-industry)

Copied to clipboard

Yuqing Wang, Prashanth Vijayaraghavan, Ehsan Degan

Responsible & Trustworthy NLP

Challenge:	a new study examines email marketing performance by considering email content and metadata.
Approach:	They propose a model that incorporates semantic and structural information from email data to generate latent exemplars for email response prediction.
Outcome:	The proposed model outperforms baseline models on two real-world email datasets . it provides interpretability through prototypes at different granularity levels while maintaining comparable performance to non-interpretable models.

Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)

Copied to clipboard

Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi

Language Models

Challenge:	Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples.
Approach:	They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses.
Outcome:	The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance.

Contrastive Document Representation Learning with Graph Attention Networks (2021.findings-emnlp)

Copied to clipboard

Peng Xu, Xinchi Chen, Xiaofei Ma, Zhiheng Huang, Bing Xiang

Semantic Text Processing

Challenge:	Existing methods for document representation learning are significantly affected by the scarcity of document-level data.
Approach:	They propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings.
Outcome:	Empirically, the proposed approach is effective in document classification and document retrieval tasks.

Explanation Graph Generation via Generative Pre-training over Synthetic Graphs (2023.findings-acl)

Copied to clipboard

Han Cui, Shangzhan Li, Yu Zhang, Qi Shi

Multimodality

Challenge:	Existing frameworks for explanation graph generation are limited due to the large number of datasets available.
Approach:	They propose a text-to-graph generative task to pre-train a model to bridge the text-graph gap.
Outcome:	The proposed framework surpasses all baseline systems with remarkable margins on ExplaGraphs and CommonsenseQA.

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models (2021.emnlp-main)

Copied to clipboard

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui

Language Models

Challenge:	Transformer architecture is composed of multi-head attention, which has been extensively analyzed.
Approach:	They extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization.
Outcome:	The proposed method incorporates the whole attention block, i.e., multi-head attention, residual connection, and layer normalization into the analysis.