Papers by Jun-Hyung Park
SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. |
| Approach: | They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt. |
| Outcome: | The proposed method outperforms baseline methods while preserving memory usage. |
SEED: Semantic Knowledge Transfer for Language Model Adaptation to Materials Science (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to adapt pre-trained language models to materials science rely on frequency information from limited downstream datasets. |
| Approach: | They propose a vocabulary expansion method to adapt pre-trained language models to materials science by incorporating latent materials knowledge of lightweight embeddings into PLMs. |
| Outcome: | The proposed method mitigates the limitations of existing adaptation methods and can be used in materials science. |
DIVE: Towards Descriptive and Diverse Visual Commonsense Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Towards human-level visual understanding, visual commonsense generation has been introduced . but current research on visual commonense generation ignores an important human cognitive ability . |
| Approach: | They propose a visual commonsense generation framework to improve inferences by visual common sense generation. |
| Outcome: | The proposed framework outperforms state-of-the-art models in descriptiveness and diversity . human evaluations confirm that the framework aligns closely with human judgments on descriptiveness . |
Adaptive Convolution for Text Classification (N19-1)
Copied to clipboard
| Challenge: | Existing convolutional neural networks (CNNs) use sparse representations of text, such as bag-of-words. |
| Approach: | They propose an adaptive convolution for text classification to give flexibility to convolutional neural networks (CNNs) they attach filter-generating networks to convevolution blocks in existing CNNs . |
| Outcome: | The proposed convolution improves performance in seven benchmark datasets by 2.6 percentage points . the proposed conversions can be likened to players of the twenty questions . |
Break it Down into BTS: Basic, Tiniest Subword Units for Korean (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing word embeddings for Korean use the internal structure of words with subword information to improve the quality of word representations. |
| Approach: | They introduce Basic, Tiniest Subword (BTS) units for Korean language that are inspired by Hangeul, the Korean writing system. |
| Outcome: | The proposed framework outperforms the state-of-the-art Korean word embedding by 11.8% on all intrinsic and extrinsic tasks. |
Coconut: Contextualized Commonsense Unified Transformers for Graph-Based Commonsense Augmentation of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained language models lack commonsense knowledge . |
| Approach: | They propose a contextualized knowledge prompting scheme to guide the contextualization of structured commonsense knowledge based on large language models. |
| Outcome: | The proposed approach outperforms the state-of-the-art technique by an average of 5.8%. |
Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Inefficient transformers have been a challenge for many years, requiring computational costs that scale quadratically with the length of the input sequence. |
| Approach: | They propose a token reduction approach that dynamically routes tokens within layers to ensure that all tokens remain accessible in subsequent layers. |
| Outcome: | The proposed approach achieves up to 25x faster inference time without significant loss in accuracy. |
Incorporating Domain Knowledge into Materials Tokenization (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in language models have expanded their applications in materials science, but they often produce excessive fragmentation and semantic loss. |
| Approach: | They propose a frequency-centric tokenization approach that integrates material knowledge into tokenization. |
| Outcome: | The proposed tokenization approach outperforms existing tokenization methods and achieves an average performance gain of 4% and 2% in the generation and classification tasks. |
Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches for language models have overlooked the difficulty of training examples. |
| Approach: | They propose a framework that controls difficulty of training examples during pre-training by a tutor network. |
| Outcome: | The proposed framework outperforms state-of-the-art KD methods with student models on the GLUE benchmark. |
MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chemical representation learning often lead to overfitting and limited scalability due to early convergence. |
| Approach: | They propose a framework to train Transformers on SMILES sequences to learn from structural examples and integrate external materials embedding to enrich molecular representations. |
| Outcome: | The proposed model outperforms state-of-the-art models on molecular property prediction tasks. |
Client-Customized Adaptation for Parameter-Efficient Federated Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have a large memory footprint and are difficult to use in federated learning (FL) |
| Approach: | They propose a hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. |
| Outcome: | The proposed framework maximizes the utility of shared model parameters while minimizing divergence caused by client heterogeneity. |
Learning from Missing Relations: Contrastive Learning with Commonsense Knowledge Graphs for Commonsense Inference (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to commonsense inference lack coverage and expressive diversity of commonsensense knowledge graphs. |
| Approach: | They propose a framework that contrasts sets of semantically similar and dissimilar events . they propose 'solar' framework that can be used to learn commonsense inference . |
| Outcome: | The proposed framework outperforms the state-of-the-art commonsense transformer on commonsensense inference by 1.84% on average among 8 metrics. |
MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focused on constructing domain-specific corpus focus on a limited and scarce nature of datasets in materials science poses significant challenges for developing models that generalize well across a broad range of materials entities. |
| Approach: | They propose a method to adapt pre-trained language models for materials science by continuously pre-training them on a materials science corpus. |
| Outcome: | The proposed method is able to adapt pre-trained language models for materials science tasks. |
Moleco: Molecular Contrastive Learning with Chemical Language Models for Molecular Property Prediction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Pre-trained chemical language models (CLMs) are highly effective in predicting molecular properties, but implicitly contain limited structural information. |
| Approach: | They propose a contrastive learning framework to enhance the understanding of molecular structures within chemical language models by measuring the similarity of fingerprint vectors among different molecules. |
| Outcome: | The proposed framework outperforms state-of-the-art models in molecular property prediction. |
KOAS: Korean Text Offensiveness Analysis System (2021.emnlp-demo)
Copied to clipboard
San-Hee Park, Kang-Min Kim, Seonhee Cho, Jun-Hyung Park, Hyuntae Park, Hyuna Kim, Seongwon Chung, SangKeun Lee
| Challenge: | morphological richness and complex syntax of Korean cause difficulties in neural model training. |
| Approach: | They propose a system that exploits contextual and linguistic features and estimates an offensiveness score for a Korean text. |
| Outcome: | The proposed system exploits both contextual and linguistic features and estimates an offensiveness score for a Korean text. |
Zero-shot Commonsense Reasoning over Machine Imagination (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to zero-shot commonsense reasoning have suffered from human reporting bias inherent in textual commonsence knowledge, leading to discrepancies in understanding between PLMs and humans. |
| Approach: | They propose a zero-shot commonsense reasoning framework that integrates machine-generated images into the reasoning process. |
| Outcome: | The proposed framework outperforms existing methods on diverse reasoning benchmarks and analysis. |
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations but comes at a substantial training cost. |
| Approach: | They propose a concept-based curriculum masking method that evaluates the MLM difficulty of each token based on a carefully-designed linguistic difficulty criterion. |
| Outcome: | The proposed method significantly improves pre-training efficiency with the original BERT model at half the training cost. |
Multi-pretraining for Large-scale Text Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for large-scale text classification involve excessive computation and memory overheads. |
| Approach: | They propose a self-supervised and weakly supervised pretraining frameworks for large-scale text classification with multiple categories. |
| Outcome: | The proposed framework improves on the self-supervised and weakly supervised methods while being computationally efficient. |