Papers by Haejun Lee
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking (2023.acl-long)
Copied to clipboard
| Challenge: | Existing schema-guided dialogue state tracking models do not account for schema variations and are not generalized to unseen services. |
| Approach: | They propose a new architecture which allows for rich attention among descriptions and history while keeping computation costs constrained. |
| Outcome: | The proposed model outperforms the more than 30x larger D3ST-XXL model on the SGD-X benchmark by 5.0 points. |
You Only Need One Model for Open-domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to Open-domain Question Answering use external knowledge bases, but have separate parameters and are weakly-coupled during training. |
| Approach: | They propose to use a single question answering model trained end-to-end to retrieve external knowledge and rerank passages with a separate reranked model. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores on the Natural Questions and TriviaQA open datasets. |
Learning to Generate Questions by Learning to Recover Answer-containing Sentences (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent research has focused on synthetically generating a question from a given context and an annotated answer by training an additional generative model. |
| Approach: | They propose a method that learns to generate contextually rich questions by recovering answer-containing sentences. |
| Outcome: | The proposed approach improves the quality and accuracy of existing models and achieves comparable results to the state-of-the-art on MS MARCO and NewsQA. |
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs (2025.acl-long)
Copied to clipboard
Anshumann Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee
| Challenge: | Knowledge distillation is a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. |
| Approach: | They propose an importance-sampling-based method which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits. |
| Outcome: | The proposed method enables faster training of student models with marginal overhead (10%) compared to cross-entropy based training, while maintaining competitive performance compared with full distillation. |
Answering Open-Domain Questions of Varying Reasoning Steps from Text (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark is developed to answer open-domain questions from text . the system uses a single multi-task transformer model to perform all the necessary subtasks . |
| Approach: | They develop a unified system to answer directly from open-domain questions . they use a single multi-task transformer model to perform all the necessary subtasks . |
| Outcome: | The proposed system can answer open-domain questions on any text collection without prior knowledge of reasoning complexity. |
FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoded beam search, and their generated output suffers from hallucinations. |
| Approach: | They propose to extend transformer encoders with the ability to fuse information from multiple passages to provide cross-sample attention over all tokens across samples. |
| Outcome: | The proposed method outperforms the current state-of-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference. |
On-Device Neural Language Model Based Word Prediction (C18-2)
Copied to clipboard
| Challenge: | Currently, on-device keyboards have limited memory and response time for word prediction . a proposed on-device neural language model based word prediction method is available for mobile devices . |
| Approach: | They propose an on-device neural language model based word prediction method that optimizes run-time memory and provides a real-time prediction environment. |
| Outcome: | The proposed model outperforms existing methods for word prediction in keystroke savings and word prediction rate and has been commercialized. |