Papers by Quanshi Zhang
Identifying Semantic Induction Heads to Understand In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance, but lack of transparency in their inference logic raises concerns about their trustworthiness. |
| Approach: | They conduct a detailed analysis of the operations of attention heads to understand their in-context learning of LLMs. |
| Outcome: | The proposed analysis of attention heads reveals that they increase the output logits of object tokens and recall objects . the proposed model is a novel approach to understand the in-context learning of large language models. |
RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL (2022.emnlp-main)
Copied to clipboard
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, Zhouhan Lin
| Challenge: | Experimental results show RASAT can leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model. |
| Approach: | They propose a Transformer seq2seq architecture augmented with relation-aware self-attention that leverages relational structures while inheriting pretrained parameters from the T5 model. |
| Outcome: | The proposed model can leverage relational structures while inheriting pretrained parameters from the T5 model effectively. |
Challenging the Explanation Based on Preceding Tokens: Discovering Transferable Non-Literal Biasing (2026.acl-short)
Copied to clipboard
| Challenge: | et al. (2017) show that the generated preceding tokens may push the large language model towards the target answer. |
| Approach: | They find that generated preceding tokens may push large language models towards the target answer . they suggest that the LLM may intentionally use the semantically unrelated tokens to help generation of the target . |
| Outcome: | The generated preceding tokens may push the large language model towards the target answer . the biased connotations of the target response can also transfer to other prompts . |