Papers by Qiaoyu Tang
Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models (2024.lrec-main)
Copied to clipboard
Boxi Cao, Qiaoyu Tang, Hongyu Lin, Shanshan Jiang, Bin Dong, Xianpei Han, Jiawei Chen, Tianshu Wang, Le Sun
| Challenge: | Pre-trained language models have shown remarkable memory formation, but vanilla networks without pre-training suffer catastrophic forgetting problem. |
| Approach: | They conduct experiments to investigate the retentive-forgetful contradiction between vanilla and pre-trained language models by controlling the target knowledge types, learning strategies and learning schedules. |
| Outcome: | The results show that pre-trained language models are forgetful and pre-training leads to retentive models . |
Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work neither proves that pre-trained models successfully learn the injected factual knowledge nor proves there is a causal relation between injected knowledge and downstream performance improvements. |
| Approach: | They propose a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm. |
| Outcome: | The proposed framework shows that factual knowledge injection is successful but correctness of injected knowledge only has limited effect on the models’ downstream performance. |
On the Editability of Delta Parameters in Post-Trained Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Several studies have explored delta parameter properties via pruning, quantization, low-rank approximation, and extrapolation, but what properties of delta parameters are essential for maintaining performance? |
| Approach: | They propose to examine delta parameter properties along magnitude and sign . they propose to use a loss-based local surrogate analysis to examine editing effects . |
| Outcome: | The proposed analysis shows that delta parameters can be edited while maintaining performance. |