Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models are highly effective tools for solving different kinds of problems in natural language processing. |
| Approach: | They propose to use large language models to solve a myriad of problems. |
| Outcome: | The proposed model performs worse on word meaning comprehension than an encoder-only model with vastly fewer parameters. |
Similar Papers
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation (2025.findings-acl)
Copied to clipboard
Yingfeng Luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, JingBo Zhu
| Challenge: | Recent advances in machine translation have focused on a single pre-trained decoder . encoder-decoder architectures have received relatively little attention in NMT . |
| Approach: | They propose a method that leverages LLMs as MT encoders and pairs them with lightweight decoders to develop universal translation models. |
| Outcome: | The proposed method matches or surpasses baselines in terms of translation quality but achieves 75% reduction in memory footprint of the KV cache. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |
How Important is a Language Model for Low-resource ASR? (2024.findings-acl)
Copied to clipboard
| Challenge: | Using an n-gram language model in ASR may seem obvious, but its absence in most implementations suggests otherwise. |
| Approach: | They examine whether using an n-gram language model in ASR can improve accuracy in low-resource languages. |
| Outcome: | The proposed model is absent in most implementations, but it does improve accuracy in English and Mandarin. |
Grammar-Constrained Decoding Makes Large Language Models Better Logical Parsers (2025.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown capabilities in various natural language processing tasks, yet struggle with logical reasoning. |
| Approach: | They propose to combine Large Language Models with symbolic reasoners to improve syntactic correctness and semantic accuracy in logical parsing tasks. |
| Outcome: | The proposed approach improves syntactic correctness and semantic accuracy in logical parsing tasks. |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations of large language models (LLMs) focus on a single output per example, which limits our understanding of LLM performance variability in real-world applications. |
| Approach: | They explore the performance differences between greedy decoding and sampling and identify benchmarks’ consistency regarding non-determinism and examine unique model behaviors. |
| Outcome: | The proposed model outperforms sampling methods and greedy decoding outperformed other models. |
Do Language Models Have Semantics? On the Five Standard Positions (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained to solve the so-called cloze task . solving clozing tasks is essentially a memorization task, says a recent study . |
| Approach: | They propose to use five positions to determine whether large language models exhibit semantic understanding . large language model is trained to solve the so-called cloze task . |
| Outcome: | The proposed theory is based on a pairwise comparison of five positions on semantic understanding in large language models and chatbots. |
LLMs Can Also Do Well! Breaking Barriers in Semantic Role Labeling via Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Semantic role labeling (SRL) is a crucial task of natural language processing (NLP). |
| Approach: | They propose to equip LLMs with retrieval-augmented generation and self-correction mechanisms to enable SRL to perform better in Chinese and English. |
| Outcome: | The proposed method achieves state-of-the-art in Chinese and English on three widely-used benchmarks. |
A Tour of Explicit Multilingual Semantics: Word Sense Disambiguation, Semantic Role Labeling and Semantic Parsing (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | a recent advent of pretrained language models has sparked a revolution in NLP . but, there are still questions about whether current approaches capture explicit, symbolic meaning . this tutorial will review efforts to tackle three key open problems in lexical and sentence-level semantics . |
| Approach: | This tutorial reviews recent efforts to shed light on meaning in NLP . it will focus on three key open problems in lexical and sentence-level semantics . |
| Outcome: | This tutorial reviews recent efforts to shed light on meaning in NLP . it focuses on three key open problems in lexical and sentence-level semantics . |
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism. |
| Approach: | They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models. |
| Outcome: | The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages. |