Distilling Structured Knowledge for Text-Based Relational Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-based relational reasoning models lack a symbolic representation of text . performance gap between NLP models and structured models remains . |
| Approach: | They first pre-train a GNN on a reasoning task using structured inputs and then incorporate its knowledge into an NLP model. |
| Outcome: | The proposed model improves on two state-of-the-art NLP models on 13 different inductive reasoning datasets from the CLUTRR benchmark. |
Similar Papers
Distilling Knowledge for Search-based Structured Prediction (P18-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy. |
| Approach: | They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space. |
| Outcome: | The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines. |
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown overwhelming performances on many tasks, but their large size and slow inference speed have hindered practical deployments. |
| Approach: | They propose a hierarchical relational knowledge distillation method to capture hierarchic and domain relational information. |
| Outcome: | The proposed method outperforms existing methods on multi-domain datasets and is highly reproducible. |
DISK-CSV: Distilling Interpretable Semantic Knowledge with a Class Semantic Vector (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural networks (NNs) are becoming deeper and more complex, making them difficult to understand and interpret. |
| Approach: | They propose a method to distill knowledge concurrently from any neural network architecture for text classification. |
| Outcome: | The proposed method achieves better performance than the target black-box and provides better explanations than existing techniques. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Distilling Linguistic Context for Language Model Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation is a major technique for deploying vast language models in resource-strapped environments. |
| Approach: | They propose a method that transfers contextual knowledge via Word Relation and Layer Transforming Relation. |
| Outcome: | The proposed method is able to transfer contextual knowledge without restrictions on architectural changes between teacher and student on language understanding tasks. |
Distilling Relation Embeddings from Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture a surprisingly rich amount of lexical knowledge, but it is unclear to what extent relation embeddings can be used to encode relational knowledge. |
| Approach: | They found that word vector differences capture lexical relations . relationship embeddings can be used to encode relational knowledge . |
| Outcome: | The results are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. |
Scalable Syntax-Aware Language Models Using Knowledge Distillation (P19-1)
Copied to clipboard
| Challenge: | Prior work has shown that syntactic neural language models learn from small amounts of training data more effectively than sequential models. |
| Approach: | They propose a knowledge distillation technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model and enables it to develop a more structurally sensitive representation of the larger training data. |
| Outcome: | The proposed method improves on baseline syntactic evaluations on LSTMs with a higher level of accuracy than previous methods. |
DistillCSE: Distilled Contrastive Learning for Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings are based on contrastive learning (CL) . |
| Approach: | They propose a framework which performs contrastive learning under the self-training paradigm with knowledge distillation and propose 'Group-P shuffling strategy' and averaging logits from multiple teacher components. |
| Outcome: | The proposed framework outperforms many strong baseline methods and yields a new state-of-the-art performance. |
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent graph-to-text models generate text from graph data using global or local aggregation . global node encoding allows explicit communication between two distant nodes, but fails to capture long-range relationships. |
| Approach: | They propose to combine global and local aggregation to learn node representations . they propose to use global and locally encoding to learn contextualized node embeddings based on graph data . |
| Outcome: | The proposed models outperform state-of-the-art models on two graph-to-text datasets by 18.01 and 63.69 points. |
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2022.naacl-main)
Copied to clipboard
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi
| Challenge: | Prior studies suggested pre-trained language models possess limited understanding of commonsense knowledge despite otherwise stellar performance on leaderboards. |
| Approach: | They propose a framework that uses larger models to teach smaller models by distilling knowledge symbolically as text in addition to the neural model. |
| Outcome: | The proposed framework is based on a general language model teacher's commonsense knowledge graphs and a neural commonsensing model surpassing the teacher model's in all three criteria. |