Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on monosemanticity on its basic units. |
| Approach: | They propose to revisit monosemanticity from the feature decorrelation perspective and advocate for its encouragement. |
| Outcome: | The proposed method improves representation diversity and activation sparsity and improves preference alignment performance. |
Similar Papers
DPN-LE: Dual Personality Neuron Localization and Editing for Large Language Models (2026.findings-acl)
Copied to clipboard
Lifan Zheng, Xue Yang, Jiawei Chen, Chenyan WU, Jingyuan Zhang, Fanheng Kong, Xinyi Zeng, Xiang Chen, Yu Tian
| Challenge: | Current methods for editing personality traits in large language models can change personalities but reduce performance. |
| Approach: | They propose a novel paradigm for personality editing that locates and edits LLM neurons and enables competitive personality control at inference time. |
| Outcome: | Experiments on LLaMA-3-8B-Instruct and Qwen2.5-7B-instruct show that the proposed approach can improve performance and improve performance. |
Endowing Neural Language Learners with Human-like Biases: A Case Study on Dependency Length Minimization (2024.lrec-main)
Copied to clipboard
| Challenge: | Comparing the behavior of models with that of human learners can reveal which aspects affect the emergence of this preference. |
| Approach: | They propose to add three factors to the standard neural-agent language learning and communication framework to make the simulation more realistic. |
| Outcome: | The proposed conditions can contribute to a small but significant learning advantage for listeners of verb-initial languages. |
Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that large language models are struggling to align with human preference in complex tasks and scenarios. |
| Approach: | They propose a low-redundant alignment method that selects the top-10% most updated parameters in LLMs for alignment training. |
| Outcome: | The proposed method improves on 10 datasets and shows that it is redundant . it can be used to train LLMs on QA and ECQA datasets, but it is not feasible to test it on a large dataset. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Constructing Your Model’s Value Distinction: Towards LLM Alignment with Anchor Words Tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models (LLMs) shows that they can generate outputs that are honest, positive, harmless, etc. |
| Approach: | They propose a method that amplifies logits difference between positive and negative tokens . they propose to use the logits gap to generate positive and positive tokens after alignment . |
| Outcome: | The proposed method achieves effective alignment, but requires fewer computational resources compared to training-time alignment methods. |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models. |
| Approach: | They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs. |
| Outcome: | The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets. |
Context Limitations Make Neural Language Models More Human-Like (2022.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) have been used in cognitive modeling and engineering studies to simulate human cognitive load during reading. |
| Approach: | They propose to constrain LMs' context access to improve their simulation of human reading behavior by incorporating syntactic biases into their context access. |
| Outcome: | The proposed model improves the simulation of human reading behavior by incorporating syntactic biases into their context access. |
Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Training (2025.findings-acl)
Copied to clipboard
Yihang Yao, Zhepeng Cen, Miao Li, William Han, Yuyou Zhang, Emerson Liu, Zuxin Liu, Chuang Gan, Ding Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated strong reasoning capabilities across various tasks. |
| Approach: | They propose a data-centric approach that enhances LLMs’ awareness of symmetry in query variations and propose syMmetry-ENhanceD (MEND) data augmentation. |
| Outcome: | Extensive experiments on logical and arithmetic reasoning tasks show that the proposed approach improves model robustness at the knowledge extraction stage through query augmentation. |