Enhancing Chinese Pre-trained Language Model via Heterogeneous Linguistics Graph (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained Chinese language models ignore linguistics knowledge to learn representations. |
| Approach: | They propose a task-free enhancement module to integrate linguistics knowledge into Chinese pre-trained language models. |
| Outcome: | The proposed model improves Chinese pre-trained language models on 6 tasks with 10 benchmark datasets. |
Similar Papers
Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training (2022.emnlp-industry)
Copied to clipboard
Taolin Zhang, Junwei Dong, Jianing Wang, Chengyu Wang, Ang Wang, Yinghui Liu, Jun Huang, Yong Li, Xiaofeng He
| Challenge: | Existing knowledge-enhanced pre-trained language models (KEPLMs) can capture internal knowledge, but can't understand external background knowledge. |
| Approach: | They propose to use Chinese knowledge-enhanced pre-trained language models to improve context-aware representations via learning from structured relations in knowledge bases. |
| Outcome: | Experiments show that Chinese knowledge-enhanced pre-trained language models outperform strong baselines over various benchmark NLP tasks and in different model sizes. |
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information (2021.acl-long)
Copied to clipboard
| Challenge: | ChineseBERT model incorporates glyph and pinyin information of Chinese characters into pretraining . proposed model achieves new performance boost over baseline models with fewer training steps . |
| Approach: | They propose a ChineseBERT model that incorporates glyph and pinyin information into pretraining . the glyph embedding is obtained based on different fonts of a character, and the pinyink embeddment characterizes the pronunciation of Chinese characters. |
| Outcome: | The proposed model achieves new performance boosts over baseline models with fewer training steps. |
Pretraining Language Models with Text-Attributed Heterogeneous Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pretraining tasks for Language Models (LMs) focus on learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs. |
| Approach: | They propose a topology-aware pretraining task that explicitly considers the topological and heterogeneous information in TAHGs by optimizing an LM and an auxiliary heterogenous graph neural network. |
| Outcome: | The proposed framework explicitly considers the topological and heterogeneous information in TAHGs. |
GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent improvements in KG-to-text generation are due to additional pre-training tasks . these tasks require extensive computational resources while only suggesting marginal improvements. |
| Approach: | They propose a mask structure to capture neighborhood information and a type encoder that adds a bias to the graph-attention weights depending on the connection type. |
| Outcome: | The proposed model outperforms state-of-the-art models while requiring no additional pre-training tasks. |
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words. |
| Approach: | They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner. |
| Outcome: | The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. |
Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding models suffer from limited knowledge representation due to sparse and noisy dataset annotations. |
| Approach: | They propose to use pretrained language models to enhance knowledge representation by leveraging world knowledge from pretrained models. |
| Outcome: | Extensive experiments show that the proposed framework can improve results over existing models. |
Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing models of language understanding are based on explicit representations of hierarchical structure, but there are good reasons to doubt that they can be said to understand language in any meaningful way. |
| Approach: | They examine whether syntactic and semantic graph representations can complement and improve neural language modeling. |
| Outcome: | The proposed model outperforms pretrained models on English WSJ in perplexity and other metrics. |
An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to explicitly incorporate human-defined linguistic priors into fine-tuning tasks. |
| Approach: | They replace parsed graphs or trees with trivial ones to investigate linguistic priors . they propose to use trivial graphs as baselines to design advanced knowledge fusion methods . |
| Outcome: | The use of trivial graphs improves performance in fully-supervised and few-shot settings. |
Cross-lingual Text Classification with Heterogeneous Graph Neural Network (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for cross-lingual text classification only consider factors beyond semantic similarity, causing performance degradation between some language pairs. |
| Approach: | They propose a method to incorporate heterogeneous information within and across languages for cross-lingual text classification using graph convolutional networks. |
| Outcome: | The proposed method significantly outperforms state-of-the-art models on all tasks and achieves consistent performance gain over baselines in low-resource settings. |
Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification require task-specific training data in high-resource sources . labeling cost, task characteristics, and privacy concerns can hinder the use of cross-linguistic training . |
| Approach: | They propose a dictionary-based heterogeneous graph (DHGNet) that uses bilingual dictionaries for task-independent word embeddings. |
| Outcome: | The proposed method outperforms pretrained models even though it does not access to large corpora. |