Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions (2026.findings-acl)
Copied to clipboard
| Challenge: | Concept Tokens is a lightweight method that adds a special token to a pretrained LLM . we find that negating the hallucination token reduces hallucines and lowers precision . |
| Approach: | They propose a lightweight method that adds a new special token to a pretrained LLM and learns only its embedding from multiple natural language definitions of a target concept. |
| Outcome: | The proposed method can learn only its embedding from multiple definitions of a target concept . the study shows that it can improve hallucinations and recasting in closed-book questions . |
Similar Papers
Beyond Tokens: Concept-Level Training Objectives for LLMs (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained with a surprisingly narrow objective: predicting the next token in a sequence. |
| Approach: | They propose a shift from token-level to concept-level prediction where concepts group multiple surface forms of the same idea. |
| Outcome: | The proposed model improves on human-level models on diverse NLP benchmarks. |
Token Knowledge: A New Perspective For Knowledge in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Predicting the presence and absence of certain knowledge in large language models could aid hallucination avoidance. |
| Approach: | They propose a token knowledge dataset construction method and use the intermediate states during inference to train probes. |
| Outcome: | The proposed method increases the model's latent potential by 60% to 90% with strong out-of-distribution generalization by training on just a few dozen prompts. |
Extracting Conceptual Spaces from LLMs Using Prototype Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Conceptual spaces represent entities and concepts using cognitively meaningful dimensions . practical methods for extracting conceptual spaces are currently lacking . |
| Approach: | They propose a strategy in which features are encoded by embedding a description of a corresponding prototype. |
| Outcome: | The proposed approach is highly effective. |
AMenDeD: Modelling Concepts by Aligning Mentions, Definitions and Decontextualised Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Contextualised Language Models (LMs) improve on word embeddings by encoding meaning of words in context. |
| Approach: | They propose to learn a unified embedding space in which all three types of representations can be integrated. |
| Outcome: | The proposed model outperforms existing approaches in ontology completion tasks. |
To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)
Copied to clipboard
| Challenge: | State-of-the-art language models perform well on a variety of language tasks, but struggle with understanding negation cues in tasks like natural language inference (NLI). |
| Approach: | They propose a new learning strategy for negation building on ELECTRA’s replaced token detection objective. |
| Outcome: | The proposed approach leads to substantial gains on a variant of RTE with additional negation. |
Tokenizer-Aware Cross-Lingual Adaptation of Decoder-Only LLMs through Embedding Relearning and Swapping (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been primarily focused on English, leaving the multilingual ability unexplored. |
| Approach: | They propose a technique that creates new tokenizers and tunes embeddings on fixed model weights for target language adaptation. |
| Outcome: | The proposed method is light-weight and performant but has limitations for older models and high resource languages. |
CUTE: Measuring LLMs’ Understanding of Their Tokens (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) perform well on a wide variety of tasks, authors say . they lack direct access to characters, which can be difficult to generalize to new languages . |
| Approach: | They propose a benchmark to test the orthographic knowledge of Large Language Models . they find that most LLMs seem to know the spelling of their tokens - yet fail to manipulate text . |
| Outcome: | The proposed benchmark tests the orthographic knowledge of large language models . it finds that most LLMs seem to know the spelling of their tokens, but fail to manipulate text . |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
Learning to Insert [PAUSE] Tokens for Better Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored incorporating special-purpose tokens into the training process to enhance reasoning capabilities. |
| Approach: | They propose a method for inserting dummy tokens consecutively just before reasoning steps to increase model effectiveness. |
| Outcome: | The proposed method outperforms fine-tuning and previous token insertion methods on multiple datasets and models. |
Towards Concept-Aware Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Concepts play a pivotal role in various human cognitive functions, including reasoning and communication. |
| Approach: | They analyze how well contemporary large language models capture human concepts and their structure . they propose a method for pretraining LLMs using concepts and a simpler approach . |
| Outcome: | The proposed method matches human intuition and improves robustness of predictions. |