Papers by Joel Jang
Exploring the Practicality of Generative Retrieval on Dynamic Corpora (2024.emnlp-main)
Copied to clipboard
| Challenge: | a lack of comprehensive comparison between GR and Dual Encoders in IR systems is limiting . GR is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6) and storage footprint (x4) |
| Approach: | They propose to use autoregressive language models to perform information retrieval (IR) their results highlight the potential of GR for future use in practical IR systems . |
| Outcome: | The proposed model is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, efficient inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to the most common model, Dual Encoder (DE). |
Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that adding a instruction tuning stage to training large language models can improve zero-shot task generalization. |
| Approach: | They propose a method that retrieves promptspecific source prompt embeddings from training instances . they train soft prompt embeds for each prompt through prompt tuning and store the samples . |
| Outcome: | The proposed method outperforms hard prompts on unseen tasks by 2.39% points and outperformed 10 out of 11 datasets. |
How Well Do Large Language Models Truly Ground? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing research defines “grounding” as having the correct answer, which does not ensure the reliability of the entire response. |
| Approach: | They propose a stricter definition of grounding: fully utilizes the necessary knowledge from the provided context and stays within the limits of that knowledge. |
| Outcome: | The proposed model can be ground on external contexts and maintain its correct answer. |
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models with less than 100B parameters perform poorly on chain-of-thought reasoning . we aim to equip smaller LMs with the step-by-step reasoning capability . |
| Approach: | They propose to equip smaller LMs with the step-by-step reasoning capability by tuning with CoT rationales. |
| Outcome: | The proposed dataset outperforms large LMs on 4 domain-specific tasks even with demonstrations . |
Fixed Input Parameterization for Efficient Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that attaching prompts to the input is effective at conditioning Language Models (LMs) however, prompts are always included in the input text during inference, thus incurring substantial computational and memory overhead. |
| Approach: | They propose to inject a fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. |
| Outcome: | The proposed method can be up to 280 times more efficient in terms of FLOPs than previous approaches. |
Semiparametric Token-Sequence Co-Supervision (2024.acl-long)
Copied to clipboard
| Challenge: | Using semiparametric token-sequence co-supervision, language models are trained using a finite parametric vocabulary space. |
| Approach: | They propose a semiparametric token-sequence co-supervision training method that leverages supervision from two different supervisions. |
| Outcome: | The proposed method outperforms models trained via each supervision independently and shows that it encourages a broader generalization capability across the model. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
LangBridge: Multilingual Reasoning Without Multilingual Supervision (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to adapt language models for multilingual reasoning tasks require multilingual supervision. |
| Approach: | They propose a zero-shot approach to adapt language models for multilingual reasoning tasks without multilingual supervision by bridging two models by introducing minimal trainable parameters between them. |
| Outcome: | The proposed approach significantly improves multilingual reasoning capabilities on low-resource languages. |
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022.emnlp-main)
Copied to clipboard
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo
| Challenge: | Language Models (LMs) become outdated as the world changes, a phenomenon called temporal misalignment. |
| Approach: | They propose a lifelong benchmark that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
| Outcome: | The proposed benchmark can be trained on the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
Gradient Ascent Post-training Enhances Language Model Generalization (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that language models can perform diverse downstream NLP tasks in a zero-shot manner without any gradient updates. |
| Approach: | They show that updating pretrained LMs with Gradient Ascent Post-training enhances zero-shot generalization capabilities without any task-specific fine-tuning. |
| Outcome: | The proposed method can be comparable to 2-3x times larger LMs across 12 different NLP tasks. |
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that an adversary can extract training data from Pretrained Language Models including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses. |
| Approach: | They propose to use knowledge unlearning to reduce privacy risks for LMs by performing gradient ascent on target token sequences instead of trying to unlearn all the data at once. |
| Outcome: | The proposed method can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. |