Papers by Yukun Ma
MulCode: A Multiplicative Multi-way Model for Compressing Neural Language Model (D19-1)
Copied to clipboard
| Challenge: | a large number of parameters dominate the memory usage of deep neural nets . compression of embedding layers is the key to reducing memory usage . |
| Approach: | They propose a multi-way multiplicative neural compressor to compress embedding layers . they use an adaptively created matrix and multiplicativative compositions to learn them . |
| Outcome: | a new multi-way multiplicative neural compressor can achieve 41.38 times compression rate with little loss in performance. |
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings (2023.emnlp-main)
Copied to clipboard
Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang
| Challenge: | Prior studies diagnose the anisotropy problem in sentence embeddings from pre-trained language models without fine-tuning. |
| Approach: | They propose an unsupervised method that weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. |
| Outcome: | Empirical evaluations show that the proposed method can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks. |