Papers with CLMs
Cross-lingual Similarity of Multilingual Representations Revisited (2022.aacl-main)
Copied to clipboard
| Challenge: | Similarity indexes like CKA and CCA are not suitable for cross-lingual learning analysis. |
| Approach: | They propose an alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context. |
| Outcome: | The proposed method is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context. |
Moleco: Molecular Contrastive Learning with Chemical Language Models for Molecular Property Prediction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Pre-trained chemical language models (CLMs) are highly effective in predicting molecular properties, but implicitly contain limited structural information. |
| Approach: | They propose a contrastive learning framework to enhance the understanding of molecular structures within chemical language models by measuring the similarity of fingerprint vectors among different molecules. |
| Outcome: | The proposed framework outperforms state-of-the-art models in molecular property prediction. |
SURE: Mutually Visible Objects and Self-generated Candidate Labels For Relation Extraction (2025.coling-main)
Copied to clipboard
| Challenge: | Joint relation extraction models face high computational complexity, complex network architectures, difficult parameter tuning and limited interpretability. |
| Approach: | They develop a candidate label marker mechanism that prioritizes strategic label selection over simple label generation. |
| Outcome: | The proposed candidate label marks improve the SOTA methods by 2.5%, 1.9%, 1.2% . the proposed candidate labels improve the performance of the proposed methods . |
Controllable Text Generation with Residual Memory Transformer (2024.findings-acl)
Copied to clipboard
| Challenge: | Large-scale Causal Language Models (CLMs) have been successful in text generation, but there is still a challenge to control the generation process. |
| Approach: | They propose a non-intrusive, lightweight control plugin to control the generation process of a CLM at arbitrary time steps. |
| Outcome: | The proposed plugin can handle any type of control conditions and cooperate with the base CLM through a residual learning paradigm. |
Form and Meaning in Intrinsic Multilingual Evaluations (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing metrics for conditional language models are not universally comparable . a number of assumptions are made about their quality in multilingual settings . |
| Approach: | They make assumptions about the quality of conditional language models and their semantic meanings that are not consistent with current metrics. |
| Outcome: | The results show that current evaluation metrics are not universally comparable . they use six metrics on two multi-parallel corpora with mono- and multilingual models . |
Better Character Language Modeling through Morphology (P19-1)
Copied to clipboard
| Challenge: | Inflected words benefit more from explicitly modeling morphology than uninflectes . morphological supervision is also used to augment character language models in low-resource languages . |
| Approach: | They add morphological supervision to character language models via multitasking to improve BPC performance across 24 languages even when morphology data and language modeling data are disjointed. |
| Outcome: | The addition improves performance even when morphology data and language modeling data are disjointed. |
Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work on word embeddings reports low correlations with human ratings . contextual language models (CLMs) have been successful in acquiring semantic and world knowledge. |
| Approach: | They propose to use BERT to probe contextual language models for predicting typicality scores. |
| Outcome: | The proposed methods improve on previous studies on word embeddings and their ability to predict typicality scores. |
DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning approaches for attribute-controllable text generation are difficult to implement due to the lack of interpretability of deep neural networks. |
| Approach: | They propose a new approach that incorporates attribute knowledge of discriminator to optimize prompt tuning by steering a frozen CLM to produce attribute-specific texts. |
| Outcome: | The proposed approach can achieve state-of-the-art control performance while maintaining high-quality text generation. |
On the Strength of Character Language Models for Multilingual Named Entity Recognition (D18-1)
Copied to clipboard
| Challenge: | Character-level patterns have been widely used in English Named Entity Recognition systems. |
| Approach: | They propose to use corpus-agnostic character-level language models to capture name tokens . they demonstrate they can capture name and non-name tokens in a diverse set of languages . |
| Outcome: | The proposed model improves the performance of an off-the-shelf NER system for multiple languages. |
Fine-Grained Controllable Text Generation Using Non-Residual Prompting (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to control the text generation process are not expressive enough. |
| Approach: | They propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. |
| Outcome: | The proposed architecture is expressive and versatile on multiple experimental settings. |
FrameEOL: Semantic Frame Induction using Causal Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. |
| Approach: | They propose a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label . |
| Outcome: | The proposed method outperforms existing methods on English and Japanese datasets. |
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection (2026.acl-long)
Copied to clipboard
Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang
| Challenge: | Existing knowledge injection benchmarks for large language models lack standardized testing grounds. |
| Approach: | They propose a knowledge injection benchmark that leverages recently-added and expert-curated facts from Wikipedia’s “Did You Know...” entries. |
| Outcome: | The proposed framework improves reliability accuracy by 29.1%. |
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for token-level KV optimization and grouping of tokens are inefficient and strain compute and storage resources. |
| Approach: | They propose a mixture-of-expert approach that dynamically optimizes token-wise computation and memory allocation by a token-based expert-choice routing mechanism guided by learned importance scores. |
| Outcome: | The proposed approach retains all tokens while adaptively routing them to specialized experts with varying KV group sizes, balancing granularity and efficiency. |