Papers by Seongwan Kim
Riemannian Optimization for LoRA on the Stiefel Manifold (2025.findings-emnlp)
Copied to clipboard
| Challenge: | powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. |
| Approach: | They propose to optimize LoRA’s B matrix by imposing explicit orthogonality constraints that achieve near-perfect orthogonal and full effective rank. |
| Outcome: | The proposed method outperforms AdamW and LoRA in terms of parameter efficiency and representational capacity. |
Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods suffer from a lack of interpretability . we propose a new interpretability framework that decomposes dense embeddings into distinct, interpretable latent concepts. |
| Approach: | They propose a new interpretability framework that leveragesSparse Autoencoders to decompose uninterpretable dense embeddings fromDPR models into distinct, interpretable latent concepts. |
| Outcome: | The proposed interpretability framework achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches. |
Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches for fine-tuning large language models require a trade-off between exact gradients with high memory and low memory with noisy estimates (MeZO). |
| Approach: | They propose a method which derivates gradients from LoRA's low-rank structure and manually deriving backward passes to exploit the low-level structure. |
| Outcome: | The proposed method reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices. |