| Challenge: | Recent work probes PLMs for the extent of factual knowledge through prompts . however, these methods do not consider symmetry of the task: object and subject prediction. |
| Approach: | They propose a continuous prompt-based method that leverages symmetry of the task by constructing symmetrical prompts for subject and object prediction. |
| Outcome: | The proposed method improves on a popular factual probing dataset on lAMA. |
Similar Papers
Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found prompt-based probing evaluations inaccurate, inconsistent and unreliable. |
| Approach: | They propose to conduct debiasing via causal intervention to uncover biases in probing evaluations . authors argue that prompt-based probing is inaccurate, inconsistent and unreliable . |
| Outcome: | This paper examines the effectiveness of prompt-based probing in pretrained language models . it highlights critical biases which could induce biased results and conclusions . authors suggest rethinking criteria for evaluating better pretrained models based on such evaluations . |
Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research shows that pre-trained language models suffer from “prompt bias” in factual knowledge extraction. |
| Approach: | They propose a representation-based approach to mitigate prompt bias during inference time by querying the model and removing it from its internal representations to generate debiased representations. |
| Outcome: | The proposed approach corrects the overfitted performance caused by prompt bias and significantly improves prompt retrieval capability. |
Towards Alleviating the Object Bias in Prompt Tuning-based Factual Knowledge Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize prompts for factual knowledge extraction are undesirable object bias. |
| Approach: | They propose a prompt tuning method that reduces object bias and improves factual knowledge extraction. |
| Outcome: | The proposed method reduces object bias and improves accuracy of factual knowledge extraction. |
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are trained on vast unlabeled data, rich in world knowledge. |
| Approach: | They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases . |
| Outcome: | The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work. |
Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)
Copied to clipboard
Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang
| Challenge: | Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks. |
| Approach: | They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem. |
| Outcome: | The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings. |
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for factual probing can interpret the model’s prediction accuracy as a lower bound on the amount of factual information it encodes. |
| Approach: | They propose a method which directly optimizes in continuous embedding space and can predict an additional 6.4% of facts in the LAMA benchmark. |
| Outcome: | The proposed method outperforms the best previous prompt method by 6.4% on the LAMA benchmark. |
Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-enhanced pre-trained language models (PLMs) introduce redundant factual knowledge from knowledge bases and require complex modules. |
| Approach: | They propose a knowledge prompting-based PLM framework that incorporates factual knowledge into PLMs. |
| Outcome: | The proposed framework can be flexibly combined with existing mainstream PLMs. |
Self-Supervised Prompt Optimization (2025.findings-emnlp)
Copied to clipboard
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Xinbing Liang, Fengwei Teng, Jinhao Tu, Fashen Ren, Xiangru Tang, Sirui Hong, Chenglin Wu, Yuyu Luo
| Challenge: | Existing prompt optimization methods rely heavily on external references such as ground truth or by humans, limiting their applicability in real-world scenarios where such data is unavailable or costly to obtain. |
| Approach: | They propose a cost-efficient framework that discovers effective prompts for both closed and open-ended tasks without external reference. |
| Outcome: | The proposed framework outperforms state-of-the-art prompt optimization methods with significantly lower costs and fewer samples. |
Test-time Augmentation for Factual Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve factual probing are relation-specific and do not generalize to unseen relation types. |
| Approach: | They propose to use test-time augmentation to augment and ensemble prompts at test time to reduce sensitivity to prompt variations. |
| Outcome: | The proposed method improves model confidence, but for other models, it leads to degradation. |
PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens. |
| Approach: | They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations. |
| Outcome: | Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods. |