Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing work proposes dialect adaptation for encoder models or encoder-decoder models. |
| Approach: | They propose to use MD-3 to combine task adapters and dialect adapters to decoder models using a masked word game-playing conversation. |
| Outcome: | The proposed architecture outperforms baselines on Indian English and Nigerian English on a masked conversation with two models. |
Similar Papers
Task-Agnostic Low-Rank Adapters for Unseen English Dialects (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study found that LLMs are trained on corpora disproportionally weighted in favor of Standard American English . prior work on dialect struggle with generalizing to evolving and emerging dialects in a scalable manner. |
| Approach: | They propose a method that leverages linguistic knowledge to enable resource-efficient adaptation . their method disentangles dialect-specific and cross-dialectal information . |
| Outcome: | a new method improves generalization to unseen dialects in a task-agnostic fashion . it achieves the best or most competitive performance across 5 dialects . |
Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) pre-trained on massive text data in many languages are preferred solution for various Natural Language processing tasks. |
| Approach: | They compare tokenization parity and information parity as representational biases in pre-trained models . they find TP is better predictor of performance on tasks reliant on syntactic and morphological cues . |
| Outcome: | The proposed model improves on dialect classification, topic classification, and extractive question answering tasks. |
Toward Inclusive Language Models: Sparsity-Driven Calibration for Systematic and Interpretable Mitigation of Social Biases in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new method to mitigate stereotypical bias in large language models is needed . inherent biases from training on vast Internet datasets can amplify harmful stereotypes . |
| Approach: | They propose a method to identify stereotypical bias in decoder-only transformer models . they apply a localization mechanism that correlates internal activations with a new Context Influence score . |
| Outcome: | The proposed method reduces stereotypical biases on BBQ, StereoSet, and CrowS-Pairs while improving reasoning performance on MMLU by 10%. |
Speculative Contrastive Decoding (2024.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. |
| Approach: | They propose a decoding approach that leverages predictions from smaller language models to achieve both decoding acceleration and quality improvement. |
| Outcome: | The proposed method achieves both decoding acceleration and quality improvement on four diverse language tasks. |
Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It) (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study has shown that pre-trained NLMs can capture syntax- and semantic-sensitive phenomena. |
| Approach: | They investigate whether fine-tuning pre-trained models with linguistic knowledge improves their performance in a target task. |
| Outcome: | The proposed enhancements improve models' performance in a target task, the authors show . the study includes models in Italian and English, and multilingual models in English and Italian . |
Aligners: Decoupling LLMs and Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. |
| Approach: | They propose to decouple LLMs and alignment by training *aligner* models that can be used to align any LLM on an as-needed basis. |
| Outcome: | The proposed model can be used to align any LLM for a given criteria on an as-needed basis. |
Aligning Large Language Models for Controllable Recommendations (2024.acl-long)
Copied to clipboard
| Challenge: | Existing literature focuses on integrating domain-specific knowledge into LLMs to enhance accuracy using a fixed task template. |
| Approach: | They propose a collection of supervised learning tasks augmented with labels derived from a conventional recommender model to improve LLMs’ proficiency in adhering to recommendation-specific instructions. |
| Outcome: | The proposed approach significantly improves the capability of LLMs to respond to instructions within recommender systems, reducing formatting errors while maintaining a high level of accuracy. |
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that denoising word alignment improves cross-lingual transferability . most applications and resources are still English-centric, making non-English users hard to access. |
| Approach: | They propose to denoise word alignment as a cross-lingual pre-training task . they first self-label word alignments for parallel sentences and then mask tokens . |
| Outcome: | The proposed model improves cross-lingual transferability on token-level tasks, especially on question answering, and structured prediction. |
KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation. |
| Approach: | They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance. |
| Outcome: | The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin. |
Structural Contrastive Pretraining for Cross-Lingual Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train multilingual language models using pretraining tasks like mask language modeling have yielded promising results on a wide range of downstream tasks. |
| Approach: | They propose a new task to align the structural words in a parallel sentence, enhancing models’ ability to comprehend cross-lingual representations. |
| Outcome: | The proposed task improves model's ability to comprehend cross-lingual representations by increasing the frequency of negative pairings. |