Papers by Yinghan Long
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown dominant performance across a range of domains including language and vision, but their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. |
| Approach: | They propose a segmented recurrent transformer that combines segmente recursion with recursive attention to reduce the computational cost. |
| Outcome: | The proposed model achieves higher ROUGE1 scores and lower computational complexity than current approaches. |
Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning is susceptible to intrinsic bias present in pre-trained language models (LMs), leading to sub-optimal performance in prompt-based zero/few-shot settings. |
| Approach: | They propose a null-input prompting method to calibrate intrinsic bias encoded in pre-trained language models (LMs) they leverage a diverse set of auto-selected null meaning inputs generated from GPT-4 to probe intrinsic bias. |
| Outcome: | The proposed method significantly improves zero/few-shot learning performance of LMs for both in-context learning and prompt-based fine-tuning (on average 9% and 2%, respectively). |