Papers with TPT
Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactic Language Models (SLMs) have difficulty with inference efficiency due to explicit generation of syntactical structures. |
| Approach: | They propose a method to "plant" trees into attention weights of unidirectional Transformer LMs to implicitly reflect syntactic structures of natural language. |
| Outcome: | The proposed method outperforms SLMs on the SyntaxGym benchmark. |
Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)
Copied to clipboard
Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang
| Challenge: | Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models. |
| Approach: | They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt. |
| Outcome: | The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets. |