Papers by Junming Shao
Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are overly parameterized and have significant redundancy . recent studies show that PLMs are highly over-parameterized and robust to pruning . |
| Approach: | They propose to re-parameter and fine-tune pre-trained language models from a new perspective: Discovery of intrinsic task-specific subspace. |
| Outcome: | The proposed model can be fine-tuned in the subspace with a small number of free parameters. |
Revisiting Representation Degeneration Problem in Language Modeling (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language modeling is a fundamental task in natural language processing, applications include machine translation, image captioning and speech recognition. |
| Approach: | They propose a cosine regularization method to solve the representation degeneration problem by analyzing the limitations of the proposed method and then propose an alternative regularization technique to tackle the problem. |
| Outcome: | The proposed method is effective in language modeling and image captioning. |
An Online Semantic-enhanced Dirichlet Model for Short Text Stream Clustering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to cluster short text streams exploit short text in a batch way, but determine optimal batch size is difficult since we have no priori knowledge when the topics evolve. |
| Approach: | They propose an online Semantic-enhanced Dirichlet Model for short sext stream clustering which integrates the word-occurance semantic information into a new graphical model and clusters each arriving short text automatically in an online way. |
| Outcome: | The proposed model has better performance than state-of-the-art models on synthetic and real-world data sets. |