Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)

Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)

Challenge:	Position bias is a tendency of a model unfairly prioritizing information from certain parts of the input text over others, leading to undesirable behavior.
Approach:	They propose to measure position bias in large language models for zero-shot summarization tasks by measuring position bias.
Outcome:	The proposed model performance and position biases lead to new insights and discussion on zero-shot summarization tasks.

Copied to clipboard

Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, Mark Gerstein

Improving Toponym Resolution by Predicting Attributes to Constrain Geographical Ontology Entries (2024.naacl-short)

Challenge:	Large Language Models (LLMs) have advanced capabilities but produce complex structured data.
Approach:	They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs.
Outcome:	The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats.

Copied to clipboard

Zeyu Zhang, Egoitz Laparra, Steven Bethard

Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)

Challenge:	Existing approaches to geocoding only encode location mentions and their context .
Approach:	They propose a prompt-based approach to geocoding where the machine learning algorithm encodes only the location mention and its context.
Outcome:	The proposed model achieves state-of-the-art performance on multiple datasets.

Copied to clipboard

Ting-Han Fan, Ta-Chung Chi, Alexander Rudnicky

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers (2024.naacl-short)

Challenge:	Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades.
Approach:	They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs.
Outcome:	The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic.

Copied to clipboard

Roy Xie, Orevaoghene Ahia, Yulia Tsvetkov, Antonios Anastasopoulos

Information Extraction & Text Mining

Challenge:	Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis.
Approach:	They propose a method to extract distinguishing lexical features of dialects by utilizing interpretable dialect classifiers in the absence of human experts.
Outcome:	The proposed method extracts key language-specific lexical features that contribute to dialectal variations.

Clear Up Confusion: Advancing Cross-Domain Few-Shot Relation Extraction through Relation-Aware Prompt Learning (2024.naacl-short)

Copied to clipboard

Ge Bai, Chenji Lu, Daichi Guo, Shilong Li, Ying Liu, Zhang Zhang, Guanting Dong, Ruifang Liu, Sun Yong

Low-Resource NLP

Challenge:	Existing approaches to few-shot Relation Extraction (RE) are prone to confusion when applying knowledge to a target domain with entirely new types of relations.
Approach:	They propose a relation-aware prompt learning method with pre-training to clear confusion by decomposing relation types through an innovative label prompt.
Outcome:	The proposed method outperforms previous sota methods and yields better results on cross-domain few-shot RE tasks.

Fusion Makes Perfection: An Efficient Multi-Grained Matching Approach for Zero-Shot Relation Extraction (2024.naacl-short)

Copied to clipboard

Shilong Li, Ge Bai, Zhang Zhang, Ying Liu, Chenji Lu, Daichi Guo, Ruifang Liu, Sun Yong

Personalized Review Recommendation based on Implicit dimension mining (2024.naacl-short)

Challenge:	Existing methods to extract unseen relations require laborious manual annotation . a new approach uses fine-grained matching to reduce manual annotation cost .
Approach:	They propose an efficient multi-grained matching approach that uses virtual entity matching to reduce manual annotation cost.
Outcome:	The proposed approach outperforms the state-of-the-art methods and achieves inference efficiency and accuracy in zero-shot relation extraction tasks.

Copied to clipboard

Bei Xu, Yifan Xu

Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence (2024.naacl-short)

Challenge:	Existing research on review recommendation ignores personalized and implicit requirements.
Approach:	They propose a Large language model driven Personalized Review Recommendation model based on Implicit dimension mining.
Outcome:	The proposed model outperforms state-of-the-art textual models on the Amazon-MRHP dataset.

Copied to clipboard

Yinhong Liu, Yixuan Su, Ehsan Shareghi, Nigel Collier

Challenge:	Existing lexical or semantic metrics cannot accurately capture the discourse coherence of long-form text generation.
Approach:	They propose to use automatic metric to quantify the discourse divergence between two long-form articles . they use a theory of functional discourse structure to model the coherence of long-formed texts .
Outcome:	The proposed metric outperforms existing evaluation methods on three datasets from representative domains.

Returning to the Start: Generating Narratives with Related Endpoints (2024.naacl-short)

Copied to clipboard

Anneliese Brei, Chao Zhao, Snigdha Chaturvedi

Challenge:	RENarGen generates closed narratives by ensuring the first and last sentences are related and then infilling the middle sentences.
Approach:	They propose a novel novel novel that generates closed narratives by ensuring the first and last sentences are related and then infilling the middle sentences.
Outcome:	The proposed paradigm generates closed narratives by ensuring the first and last sentences are related and then infilling the middle sentences.

Unified Examination of Entity Linking in Absence of Candidate Sets (2024.naacl-short)

Copied to clipboard

Nicolas Ong, Hassan Shavarani, Anoop Sarkar

Knowledge Representation

Challenge:	Entity linking systems depend on candidate sets for their performance, but a comprehensive comparative analysis of these systems is lacking.
Approach:	They propose a black-box benchmark and a method to evaluate all state-of-the-art entity linking methods.
Outcome:	The proposed approach reduces the inference time and memory footprint of some models.

MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages (2024.naacl-short)

Copied to clipboard

Daryna Dementieva, Nikolay Babakov, Alexander Panchenko

Challenge:	Text detoxification is a textual style transfer task where a toxic text is paraphrased to the neutral register.
Approach:	They propose to extend ParaDetox pipeline to multiple languages to automate parallel detoxification corpus collection.
Outcome:	The proposed methods have been used in toxic speech combating and toxic speech fighting tasks.

SKICSE: Sentence Knowable Information Prompted by LLMs Improves Contrastive Sentence Embeddings (2024.naacl-short)

Copied to clipboard

Fangwei Ou, Jinan Xu

A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models (2024.naacl-short)

Challenge:	Experimental results show that our model outperforms the previous state-of-the-art model GloVe on STS tasks.
Approach:	They propose a simple and effective prompt template that is able to obtain the knowable information of input sentences from LLMs.
Outcome:	The proposed model outperforms the previous state-of-the-art model PromptBERT on STS tasks.

Copied to clipboard

Jaylen Jones, Lingbo Mo, Eric Fosler-Lussier, Huan Sun

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes (2024.naacl-short)

Challenge:	Existing methods for counter narrative evaluation lack alignment with human judgment as they rely on superficial reference comparisons instead of incorporating key aspects of counter narrative quality as evaluation criteria.
Approach:	They propose to use 5 defined aspects to generate counter narrative candidates using human-annotated scores and feedback from counter narrative specialized NGOs to assess their effectiveness.
Outcome:	The proposed evaluation framework outperforms existing metrics and achieves strong alignment to human-annotated scores and feedback.

Copied to clipboard

Harmon Bhasin, Timothy Ossowski, Yiqiao Zhong, Junjie Hu

CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders. (2024.naacl-short)

Challenge:	Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential.
Approach:	They propose to combine multi-task learning (MTL) with in-context learning (ICL) to build models that can generalize to multiple tasks while being robust to out-of-distribution examples.
Outcome:	The proposed training strategies enable models to learn difficult tasks while mixing in prior tasks, denoted as mixed curriculum.

Copied to clipboard

Crystina Zhang, Minghan Li, Jimmy Lin

ContrastiveMix: Overcoming Code-Mixing Dilemma in Cross-Lingual Transfer for Information Retrieval (2024.naacl-short)

Challenge:	Existing cross-encoders do not capture all information into the [CLS] token . Xiong et al., 2021) find that the out-of-domain approach is less robust.
Approach:	They introduce a cross-encoder with late interaction that incorporates a late interaction layer into existing models.
Outcome:	The proposed method improves BEIR by 5% without compromising in-domain effectiveness or search latency.

Copied to clipboard

Junggeun Do, Jaeseong Lee, Seung-won Hwang

SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window (2024.naacl-short)

Challenge:	Multilingual pretrained language models have been widely adopted in cross-lingual transfer . however, training mPLMs on code-mixed data is counterproductive .
Approach:	They propose a method that disentangles contrastive loss between conflicting objectives . they propose mPLMs that train on code-mixed data for information retrieval .
Outcome:	The proposed model outperforms mDPR in cross-lingual transfer tasks . it is effective in sentence-level and token-level zero-shot tasks despite lack of target language data .

Copied to clipboard

Vikas Raunak, Tom Kocmi, Matt Post

Machine Translation

Challenge:	prevailing evaluation methods for machine translation metrics are at the sentence-level . but there are many linguistic phenomena that cannot be translated without context .
Approach:	They propose a metric that leverages a moving window that slides over a document to feed it into a quality estimation model.
Outcome:	The proposed metric obtains significantly higher pairwise system accuracy than its sentence-level baseline.

Separately Parameterizing Singleton Detection Improves End-to-end Neural Coreference Resolution (2024.naacl-short)

Copied to clipboard

Xiyuan Zou, Yiran Li, Ian Porada, Jackie Cheung

Information Extraction & Text Mining

Challenge:	Current end-to-end coreference resolution models combine detection of singleton mentions and antecedent linking into a single step.
Approach:	They add a singleton detector to a coarse-to-fine coreference model and design an anaphoricity-aware span embedding and singletont detection loss.
Outcome:	The proposed method significantly improves model performance on OntoNotes and four additional datasets.

Unveiling Divergent Inductive Biases of LLMs on Temporal Data (2024.naacl-short)

Copied to clipboard

Sindhu Kishore, Hangfeng He

On Retrieval Augmentation and the Limitations of Language Model Training (2024.naacl-short)

Challenge:	Temporal relations play a crucial role across diverse applications, including event summarization, predicting future events and medical information processing.
Approach:	They evaluate the performance of large language models in the analysis of temporal data using two prompt types, Question Answering and Textual Entailment.
Outcome:	The proposed models show that they are biased towards specific temporal relationships, while GPT-3.5 prefers “AFTER” for implicit and explicit events, while TE models lean towards “BEFORE”.

Copied to clipboard

Ting-Rui Chiang, Xinyan Yu, Joshua Robinson, Ollie Liu, Isabelle Lee, Dani Yogatama

GenDecider: Integrating “None of the Candidates” Judgments in Zero-Shot Entity Linking Re-ranking (2024.naacl-short)

Challenge:	Recent efforts to improve the performance of language models (LMs) have focused on scaling up model and training data size, though with steep accompanying energy and compute resource costs.
Approach:	They propose to augment a language model with k-nearest neighbors retrieval on its training data to reduce its perplexity.
Outcome:	The proposed model reduces storage costs by over 25x compared to traditional retrieval methods for GPT-2 and Mistral 7B .

Copied to clipboard

Kang Zhou, Yuepei Li, Qing Wang, Qiao Qiao, Qi Li

Advancing the Robustness of Large Language Models through Self-Denoised Smoothing (2024.naacl-short)

Challenge:	Existing methods for zero-shot reranking assume the correct entity is always among the retrieved candidates.
Approach:	They propose a novel re-ranking approach for Zero-Shot Entity Linking . they use the Llama model to detect scenarios where the correct entity is not retrieved .
Outcome:	The proposed approach significantly improves disambiguation and accuracy on the ZESHEL dataset.

Copied to clipboard

Jiabao Ji, Bairu Hou, Zhen Zhang, Guanhua Zhang, Wenqi Fan, Qing Li, Yang Zhang, Gaowen Liu, Sijia Liu, Shiyu Chang

Robustness in NLP

Challenge:	Existing adversarial attacks can cause LLMs to make wrong predictions on downstream tasks or generate harmful content misaligned with human values.
Approach:	They propose to use randomized smoothing to add noise to the input and then make predictions based on these denoised versions.
Outcome:	The proposed method surpasses existing methods in both empirical and certified robustness in defending against adversarial perturbations for both downstream tasks and human alignments (i.e., jailbreak attacks).

Can LLM’s Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis (2024.naacl-short)

Copied to clipboard

Vishnu Sashank Dorbala, Sanjoy Chowdhury, Dinesh Manocha

Challenge:	83.3% of users find the synthesized instructions accurately capture the details of the environment and show characteristics similar to those of human-generated instructions.
Approach:	They propose an algorithm that uses in-context learning to condition an LLM to generate instructions using just a few references.
Outcome:	The proposed algorithm is platform-agnostic and 83.3% of users find it to be accurate and similar to human-generated instructions.

On the Role of Summary Content Units in Text Summarization Evaluation (2024.naacl-short)

Copied to clipboard

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo Walenta, Juri Opitz, Leonardo Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou

Summarization

Challenge:	a human written summary content unit (SCU) is used to judge the quality of a summary . a pyramid evaluation method is based on SCUs that decompose a reference summary into concise sentences .
Approach:	They propose to use automated SCUs to evaluate the quality of a candidate summary . they propose to generate SCU approximations from AMR meaning representations and large language models .
Outcome:	The proposed method can be fully automated, but lacks the human effort to validate it.

More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)

Copied to clipboard

David Samuel, Lucas Charpentier, Sondre Wold

Discourse-Aware In-Context Learning for Temporal Expression Normalization (2024.naacl-short)

Challenge:	Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline.
Approach:	They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects.
Outcome:	The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context.

Copied to clipboard

Akash Gautam, Lukas Lange, Jannik Strötgen

Low-Resource NLP

Challenge:	Temporal expression (TE) normalization is a well-studied problem, but upcoming machine learning approaches suffer from a lack of labeled data.
Approach:	They propose to use in-context learning to inject task, document, and example information into a large language model for temporal expression normalization.
Outcome:	The proposed model performs better in non-standard settings by dynamically including relevant examples during inference.

Contextualizing Argument Quality Assessment with Relevant Knowledge (2024.naacl-short)

Copied to clipboard

Darshan Deshpande, Zhivar Sourati, Filip Ilievski, Fred Morstatter

Challenge:	Existing methods for assessing argument quality in isolation analyze their quality in the absence of context, which affects their accuracy and generalizability.
Approach:	They propose a method for scoring argument quality based on contextualization via relevant knowledge that leverages large language models to provide feedback, infer hidden assumptions, supply a similar-quality argument, or give a counter-argument.
Outcome:	The proposed method outperforms existing methods across multiple metrics in both in-domain and zero-shot setups.

Selective Perception: Learning Concise State Descriptions for Language Model Actors (2024.naacl-short)

Copied to clipboard

Kolby Nottingham, Yasaman Razeghi, Kyungmin Kim, Jb Lanier, Pierre Baldi, Roy Fox, Sameer Singh

ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)

Challenge:	Recent large language models support longer contexts, but requiring them to process redundant or irrelevant data increases inference time and cost.
Approach:	They propose a method that leverages a small finetuned LM to sample the minimal set of input features that maximizes the performance of a downstream LM.
Outcome:	The proposed method reduces the length of LM actor input by 87% and 99% while improving task success rates by 158% and 54% on NetHack and robot planning.

Copied to clipboard

Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell

Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels (2024.naacl-short)

Challenge:	Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms.
Approach:	They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations.
Outcome:	The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps.

Copied to clipboard

Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, Michael Bendersky

LLM-Driven Knowledge Injection Advances Zero-Shot and Cross-Target Stance Detection (2024.naacl-short)

Challenge:	Existing pointwise LLMs provide noisy or biased answers for documents that are partially relevant to the query.
Approach:	They propose to incorporate fine-grained relevance labels into the LLM prompt . they propose to better differentiate between documents with different levels of relevance .
Outcome:	The proposed model can differentiate between documents with different levels of relevance to the query and derive a more accurate ranking.

Copied to clipboard

Zhao Zhang, Yiming Li, Jin Zhang, Hui Xu

Sentiment Analysis

Challenge:	Existing methods for stance detection focus on background information and not on the accompanying input texts.
Approach:	They propose to prompt Large Language Models to explicitly extract the relationship between paired text and unseen target as contextual knowledge and inject it into a generation model BART to exploit the rich contexts and semantics.
Outcome:	The proposed model is able to detect stance labels in zero-shot and cross-target scenarios.

Leveraging Prototypical Representations for Mitigating Social Bias without Demographic Information (2024.naacl-short)

Copied to clipboard

Shadi Iskander, Kira Radinsky, Yonatan Belinkov

Representation Learning

Challenge:	Existing approaches to mitigate social biases require explicit annotation of demographic information for each sample.
Approach:	They propose a method that leverages predefined demographic texts and incorporates a regularization term during the fine-tuning process to mitigate bias in language models.
Outcome:	The proposed method outperforms debiasing methods with limited demographic-annotated data.

Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding (2024.naacl-short)

Copied to clipboard

Guangyu Yang, Jinghong Chen, Weizhe Lin, Bill Byrne

EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning (2024.naacl-short)

Challenge:	Recent work shows that MBR decoding can significantly improve translation performance of Multilingual Large Language Models.
Approach:	They propose a method that uses a monolingual fine-tuning set to fine- tune MLLMs to get the gains of MBR without additional computation in inference.
Outcome:	The proposed method outperforms greedy decoding and beam search on multiple NMT tests.

Copied to clipboard

Raja Sekhar Reddy Mekala, Yasaman Razeghi, Sameer Singh

LEAF: Language Learners’ English Essays and Feedback Corpus (2024.naacl-short)

Challenge:	Language models are adopting inference-time prompting techniques such as zero-shot and few-shot prompting.
Approach:	They propose a prompting technique that prompts the model to rephrase its queries before answering them.
Outcome:	The proposed prompt improves zero-shot-CoT performance of code-davinci-002 by 5% . the proposed prompt is tailored for four scenarios in both zero- and few-shot settings .

Copied to clipboard

Shabnam Behzad, Omid Kashefi, Swapna Somasundaran

Challenge:	Current automated essay scoring models lack the granularity desired by learners and instructors seeking more detailed insights.
Approach:	They present a corpus of English essays and their corresponding feedback from the “essayforum” website.
Outcome:	The LEAF corpus provides valuable feedback for students and teachers . it provides insights on argumentative aspects and organizational coherence .

Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps (2024.naacl-short)

Copied to clipboard

Abteen Ebrahimi, Katharina Wense

On the True Distribution Approximation of Minimum Bayes-Risk Decoding (2024.naacl-short)

Challenge:	lexical gaps exist in a variety of domains, such as QA, but they can only be expressed as a combination of words in another language.
Approach:	They compare the current performance and long-term viability of two approaches to cross-lingual transfer . they leverage lexical gaps to create a multilingual question answering dataset .
Outcome:	The proposed model outperforms zero-shot transfer and machine translation (MT) lexical gaps exist in a variety of domains, including linguistics, linguistic coding, and linguistic analysis.

Copied to clipboard

Atsumoto Ohashi, Ukyo Honda, Tetsuro Morimura, Yuu Jinnai

Text Generation

Challenge:	Minimum Bayes-risk (MBR) decoding has recently gained renewed attention in text generation.
Approach:	They propose to use anomaly detection to measure the degree of approximation by sampling texts from a model and selecting the text with the highest similarity to the others.
Outcome:	The proposed method shows that previous hypotheses about samples do not correlate well with the variation, but the results support the core assumption of MBR decoding.

Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024.naacl-short)

Copied to clipboard

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schuetze

Llama meets EU: Investigating the European political spectrum through the lens of LLMs (2024.naacl-short)

Challenge:	Existing methods to overcome catastrophic forgetting are rehearsal-based and parameter isolation-based.
Approach:	They propose a rehearsal-free framework which continuously adds new modules to language models and composes them with existing modules.
Outcome:	Experiments on benchmarks show that MoCL outperforms state-of-the-art and effectively facilitates knowledge transfer.

Copied to clipboard

Ilias Chalkidis, Stephanie Brandl

M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation (2024.naacl-short)

Challenge:	Large Language Models inherit clear political leanings that have been shown to influence downstream task performance.
Approach:	They adapt Llama Chat to a European political context and audit its political leanings based on the EUandI questionnaire to analyze its political knowledge and ability to reason in context.
Outcome:	The proposed model is adapted from speeches of individual euro-parties from debates in the European Parliament to analyze its political leanings.

Copied to clipboard

Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fujinuma, Maria Nadejde, Xing Niu, Ron Litman, Yair Kittenplon, Raghavendra Pappagari

Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata (2024.naacl-short)

Challenge:	Document translation is a challenge for machine translation systems that focus on textual content at the sentence level, ignoring global context and visual layout structure.
Approach:	They propose a benchmark dataset to evaluate document-level NMT systems . they use visual cues to preserve reading order and contiguous blocks of text .
Outcome:	The proposed benchmarks assess document-level NMT systems on the comprehensive task of translating semi-structured documents.

Copied to clipboard

Jinghong Chen, Weizhe Lin, Jingbiao Mei, Bill Byrne

Dialogue Systems & Conversational Agents

Challenge:	Existing non-autoregressive (NAR) models fail to generate specified entity names in up to 40% of responses and produce OOV errors.
Approach:	They propose a constrained decoding algorithm for Directed Acyclic T5 model which offers lexical, vocabulary and length control.
Outcome:	The proposed model significantly improves on Schema Guided Dialogue and DART datasets, establishing strong results for Task-Oriented Dialog and Data-to-Text NLG.

Do Vision-Language Models Understand Compound Nouns? (2024.naacl-short)

Copied to clipboard

Sonal Kumar, Sreyan Ghosh, S Sakshi, Utkarsh Tyagi, Dinesh Manocha

Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (2024.naacl-short)

Challenge:	Open-vocabulary vision-language models (CLIP) are emerging as a promising new paradigm for text-to-image retrieval.
Approach:	They propose a benchmark to evaluate the effectiveness of open-vocabulary vision-language models (CLIP) for text-to-image retrieval using contrastive loss.
Outcome:	The proposed framework improves CN understanding of CLIP by 8.25% on Compun.

Copied to clipboard

Minji Jung, Soyeon Park, Jeewoo Sul, Yong Suk Choi

Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers (2024.naacl-short)

Challenge:	Prompt tuning is an efficient method for initializing pre-trained models . but initialization of prompts is sensitive when the model size is small .
Approach:	They propose a method to measure catastrophic forgetting by analyzing prompts for the first time . they characterize a question answering task based on answer format and prompt initialization .
Outcome:	The proposed approach can help deepen understanding of prompt tuning.

Copied to clipboard

Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, Arash Eshghi

Do Multilingual Language Models Think Better in English? (2024.naacl-short)

Challenge:	Recent approaches for developing vision and language models leverage existing vision and a language expert and try to learn a mapping between them.
Approach:	They propose to use a resampler module to create a ‘visual prompt’ which is then fed to the large language models (LLM) using a textual prompt.
Outcome:	The proposed method has been shown to be effective across coarse-grained tasks like image captioning and visual question answering, but more fine-grounded tasks that require spatial understanding have not been thoroughly examined.

Copied to clipboard

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, Mikel Artetxe

A Continued Pretrained LLM Approach for Automatic Medical Note Generation (2024.naacl-short)

Challenge:	Existing studies show that translation-test improves performance of multilingual models by translating the input into English using an external machine translation system.
Approach:	They propose a new approach that leverages the few-shot translation capabilities of multilingual language models.
Outcome:	The proposed approach outperforms direct inference on 5 tasks.

Copied to clipboard

Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, Jeffrey Ward

Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts (2024.naacl-short)

Challenge:	HEAL is the first continuously trained LLaMA2-based LLM for medical conversations . despite the success of LLMs in general capabilities, they often fall short in niche domains like healthcare .
Approach:	They propose a 13B LLaMA2-based LLM that is purpose-built for medical conversations and measured on automated scribing.
Outcome:	The HEAL LLM outperforms GPT-4 and PMC-LLaMA in PubMedQA with 78.4% accuracy and parity with GPT-LLAMA in generating medical notes.

Copied to clipboard

Michael Saxon, Yiran Luo, Sharon Levy, Chitta Baral, Yezhou Yang, William Yang Wang

Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models (2024.naacl-short)

Challenge:	With growth in the popularity of text-to-image models has come interest in assessing their multilingual capabilities, including multilingual accessibility.
Approach:	They propose to correct translation errors in a concept list translated to seven languages and compare the outputs of the benchmark to those conditioned on the old.
Outcome:	The proposed benchmark contains translation errors in Spanish, Japanese, and Chinese.

Copied to clipboard

Tingyu Xie, Qi Li, Yan Zhang, Zuozhu Liu, Hongwei Wang

Lifelong Event Detection with Embedding Space Separation and Compaction (2024.naacl-short)

Challenge:	Existing studies exploring the performance of large language models on named entity recognition tasks have focused on training task-specific LLMs for NER.
Approach:	They propose a training-free self-improving framework that utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs.
Outcome:	The proposed framework improves performance on the named entity recognition task by using an unlabeled corpus.

Copied to clipboard

Chengwei Qin, Ruirui Chen, Ruochen Zhao, Wenhan Xia, Shafiq Joty

Representation Learning

Challenge:	Existing methods for event detection are prone to forgetting due to overlap between memory data and the previously learned embedding space.
Approach:	They propose a method that embeds feature distributions away from the previous embedding space and mitigates overfitting by a memory calibration mechanism.
Outcome:	The proposed method outperforms existing state-of-the-art methods with extensive experiments.

Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion (2024.naacl-short)

Copied to clipboard

Smriti Singh, Cornelia Caragea, Junyi Jessy Li

CPopQA: Ranking Cultural Concept Popularity by LLMs (2024.naacl-short)

Challenge:	Existing work has sought to identify what triggers or causes a particular emotion, but the relationship between those triggers and the prediction of emotion detection models is little understood.
Approach:	They propose a dataset to evaluate the ability of large language models to identify emotion triggers . they compare features considered important for emotion prediction models to those considered less salient .
Outcome:	The proposed dataset compares large language models and fine-tuned models on social media posts . it shows that emotion triggers are not considered salient features for emotion prediction models .

Copied to clipboard

Ming Jiang, Mansi Joshi

The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation (2024.naacl-short)

Challenge:	Existing studies on the knowledge capacity of large language models (LLMs) focus on knowledge explicitly learned from pretraining data or implicitly inferable from similar contexts.
Approach:	They propose a few-shot question-answering task that examines LLMs’ statistical ranking abilities for long-tail cultural concepts.
Outcome:	The proposed task examines LLMs’ statistical ranking abilities for long-tail cultural concepts, particularly in the United States and the United Kingdom.

Copied to clipboard

Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao

Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learning (2024.naacl-short)

Challenge:	Large language models (LLMs) have garnered significant attention over the past year . previous studies have evaluated LLMs' performance in solving math word problems, but there is little discussion on whether they comprehend the operations they generate.
Approach:	They challenge the notion that arithmetic is language-independent and compare models with cross-agent collaborations to find significant limitations in their performance.
Outcome:	The proposed model outperforms collaborative approaches in basic arithmetic tasks.

Copied to clipboard

Philipp Borchert, Jochen De Weerdt, Marie-Francine Moens

A diverse Multilingual News Headlines Dataset from around the World (2024.naacl-short)

Challenge:	Relation classification (RC) models extract rich information from sentences with limited labeled instances.
Approach:	They propose to combine multiple sentence representations with contrastive learning to enhance information extraction by combining multiple sentence and entity tokens.
Outcome:	The proposed approach is able to extract discriminative information from multiple representations and contrastive learning.

Copied to clipboard

Felix Leeb, Bernhard Schölkopf

The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation (2024.naacl-short)

Challenge:	Babel Briefings features 4.7 million news headlines from August 2020 to November 2021, across 30 languages and 54 locations worldwide with English translations of all articles included.
Approach:	They propose to use a weighted similarity metric to group articles into clusters about the same event and visualize the event signatures of the event.
Outcome:	The proposed dataset features 4.7 million headlines from August 2020 to November 2021 across 30 languages and 54 locations worldwide with English translations of all articles included.

Copied to clipboard

Evgeniia Tokarchuk, Vlad Niculae

Efficient Sample-Specific Encoder Perturbations (2024.naacl-short)

Challenge:	Continuous-output neural machine translation models are trained to predict the continuous representation based on distances between vectors.
Approach:	They propose a continuous-output neural machine translation (CoNMT) approach that uses random output embeddings to outperform laboriously pre-trained models.
Outcome:	The proposed strategy outperforms pre-trained embeddings on large datasets and is strongest for rare words due to the geometry of their embedders.

Copied to clipboard

Yassir Fathullah, Mark Gales

Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twitter (2024.naacl-short)

Challenge:	Existing approaches to training encoder-decoder systems often depend on teacher-forcing with the likelihood criteria, e.g. next token prediction of the reference sequence.
Approach:	They propose an inference-efficient way to modify the behaviour of an encoder-decoder system according to a specific attribute of interest by using a small proxy network.
Outcome:	The proposed framework improves the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition.

Copied to clipboard

Nuredin Ali Abdelkadir, Charles Zhang, Ned Mayo, Stevie Chancellor

Challenge:	Social media data is used for detecting users with mental disorders, but public datasets lack crucial metadata related to this aspect.
Approach:	They use a custom geo-located Twitter dataset to evaluate the generalization of depressiondetection models on cross-cultural Twitter data.
Outcome:	The proposed models perform worse on Global South users compared to Global North.

Removing RLHF Protections in GPT-4 via Fine-Tuning (2024.naacl-short)

Copied to clipboard

Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Daniel Kang

LifeTox: Unveiling Implicit Toxicity in Life Advice (2024.naacl-short)

Challenge:	Large language models (LLMs) have increased in their capabilities, which increases their potential for dual use.
Approach:	They show that fine-tuning can remove RLHFprotections with as few as 340 examples and a 95% success rate.
Outcome:	The proposed method removes RLHFprotections with as few as 340 examples and a 95% success rate on non-censored outputs.

Copied to clipboard

Minbeom Kim, Jahyun Koo, Hwanhee Lee, Joonsuk Park, Hwaran Lee, Kyomin Jung

Arithmetic Reasoning with LLM: Prolog Generation & Permutation (2024.naacl-short)

Challenge:	Existing safety benchmarks and red teaming prompts fail to capture implicit toxicity in complex real-life advice-seeking scenarios.
Approach:	They propose a dataset designed for identifying implicit toxicity within advice-seeking scenarios.
Outcome:	The proposed dataset matches or surpasses the zero-shot performance of large language models in toxicity classification tasks.

Copied to clipboard

Xiaocheng Yang, Bingsen Chen, Yik-Cheung Tam

Verifying Claims About Metaphors with Large-Scale Automatic Metaphor Identification (2024.naacl-short)

Challenge:	Existing work has shown that large language models can generate arithmetic and commonsense reasoning, but they are not native to mathematical operations and symbolic manipulations.
Approach:	They propose to use large language models to generate Prolog programs to solve math problems using a code interpreter to generate arithmetic and symbolic formulas.
Outcome:	The proposed model outperforms CoT generation in the GSM8K benchmark across three LLMs.

Copied to clipboard

Kotaro Aono, Ryohei Sasano, Koichi Takeda

Sentiment Analysis

Challenge:	Existing studies on metaphors have focused on a small number of examples, whereas few studies verify claims with large corpus.
Approach:	They propose to use a large corpus to verify existing claims about verb metaphors . they apply metaphor detection to sentences extracted from Common Crawl .
Outcome:	The proposed method identifies verb metaphors with lower concreteness, imageability, familiarity and more emotional and subjective sentences.

InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis (2024.naacl-short)

Copied to clipboard

Kevin Scaria, Himanshu Gupta, Siddharth Goyal, Saurabh Sawant, Swaroop Mishra, Chitta Baral

Sentiment Analysis

Challenge:	Experimental results on the Sem Eval 2014, 15, and 16 datasets demonstrate that InstructABSA outperforms the previous state-of-the-art (SOTA) approaches on Term Extraction (ATE), Sentiment Classification(ATSC) and Sentimence Pair Extraction(ASPE) subtasks.
Approach:	They introduce positive, negative, and neutral examples to each training sample, and instruction tune the model (Tk-Instruct) for ABSA subtasks.
Outcome:	The proposed model outperforms the state-of-the-art (SOTA) on Term Extraction (ATE), Sentiment Classification (ATSC) and Sentimence Pair Extractions (ASPE) subtasks.

MEMORY-VQ: Compression for Tractable Internet-Scale Memory (2024.naacl-short)

Copied to clipboard

Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontanon, William Cohen, Sumit Sanghai, Joshua Ainslie

Information Retrieval

Challenge:	Memory-based methods like LUMEN pre-compute token representations for retrieved passages to speed up inference.
Approach:	They propose a method to reduce storage requirements of memory-augmented models . they use a vector quantization variational autoencoder to compress token representations .
Outcome:	The proposed method achieves 16x compression rate with comparable performance on KILT benchmark.

Unveiling the Magic: Investigating Attention Distillation in Retrieval-Augmented Generation (2024.naacl-short)

Copied to clipboard

Zizhong Li, Haopeng Zhang, Jiawei Zhang

Improving Factuality in Clinical Abstractive Multi-Document Summarization by Guided Continued Pre-training (2024.naacl-short)

Challenge:	Retrieval-augmented generation framework addresses the limitations of large language models by enabling real-time knowledge updates for more accurate answers.
Approach:	They propose to use attention distillation to improve retrieval-augmented language models' learning performance by identifying key factors influencing their workflow and proposing indicators for optimizing models’ training methods and avoiding ineffective training.
Outcome:	The proposed framework improves the learning performance of large language models in the training phase but also reduces the impact of ineffective training.

Copied to clipboard

Ahmed Elhady, Khaled Elsayed, Eneko Agirre, Mikel Artetxe

Information Extraction & Text Mining

Challenge:	Existing methods for enhancing the factual accuracy of abstractive summarization models are not effective in fact-critical domains like clinical document summarizing.
Approach:	They propose a guided continued pre-training stage for encoder-decoder models followed by supervised fine-tuning on summarization.
Outcome:	The proposed approach improves the quality and factuality of the summaries and achieves the best-published results on the clinical document summarization task.

MuLan: A Study of Fact Mutability in Language Models (2024.naacl-short)

Copied to clipboard

Constanza Fierro, Nicolas Garneau, Emanuele Bugliarello, Yova Kementchedjhieva, Anders Søgaard

Language-Independent Representations Improve Zero-Shot Summarization (2024.naacl-short)

Challenge:	Pretrained and large language models encode factual knowledge, but factual information changes over time and mutates with the passage of time.
Approach:	They propose to use a model to evaluate the ability of English language models to anticipate time-contingency by comparing their models to a benchmark model.
Outcome:	The proposed model can predict the president of a country or the winner of sa championship in time, but it is difficult to update them due to their mutability.

Copied to clipboard

Vladimir Solovyev, Danni Liu, Jan Niehues

Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)

Challenge:	Pretrained models can be fine tuned on downstream generation tasks, but they can fail in zero-shot conditions.
Approach:	They propose query-key finetuning to decouple task-specific knowledge from pretrained models . they propose a variant that more directly enforces language-agnostic representations .
Outcome:	The proposed model decouples task-specific knowledge from pretrained language generation abilities.

Copied to clipboard

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, Wen-tau Yih

GuyLingo: The Republic of Guyana Creole Corpora (2024.naacl-short)

Challenge:	Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations.
Approach:	They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context.
Outcome:	The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks.

Copied to clipboard

Christopher Clarke, Roland Daynauth, Jason Mars, Charlene Wilkinson, Hubert Devonish

Machine Translation

Challenge:	linguistic diversity across the globe encompasses a multitude of smaller, indigenous, and regional languages that lack the same level of computational support.
Approach:	They propose a corpus for advancing NLP research in the domain of Creolese in Guyana . they outline a framework for gathering and digitizing this corpus, including colloquial expressions, idioms, and regional variations in a low-resource language .
Outcome:	The proposed corpus includes colloquial expressions, idioms, and regional variations in a low-resource language.

DoubleLingo: Causal Estimation with Large Language Models (2024.naacl-short)

Copied to clipboard

Marko Veljanovski, Zach Wood-Doughty

Improved Text Emotion Prediction Using Combined Valence and Arousal Ordinal Classification (2024.naacl-short)

Challenge:	Existing methods for causal estimation are inadequate for noisy text data.
Approach:	They propose to use LLM-based nuisance models to estimate causal effects from non-randomized data using assumptions about the underlying data distribution.
Outcome:	The proposed method reduces the relative absolute error by 10.4% over existing methods on the best available dataset.

Copied to clipboard

Michail Mitsios, Georgios Vamvoukakis, Georgia Maniati, Nikolaos Ellinas, Georgios Dimitriou, Konstantinos Markopoulos, Panos Kakoulidis, Alexandra Vioni, Myrsini Christidou, Junkwang Oh, Gunu Jho, Inchul Hwang, Georgios Vardaxoglou, Aimilios Chalamandaris, Pirros Tsiakoulis, Spyros Raptis

Text Classification

Challenge:	Emotion detection in textual data is pivotal for developing empathetic human-computer interaction systems.
Approach:	They propose a method for categorizing emotions from textual data that acknowledges similarities and distinctions of various emotions.
Outcome:	The proposed method preserves high accuracy in emotion prediction and significantly reduces errors in misclassification cases.

On Narrative Question Answering Skills (2024.naacl-short)

Copied to clipboard

Emil Kalbaliyev, Kairit Sirts

Natural Language Interfaces

Challenge:	Existing task-level skill views oversimplify multidimensional nature of tasks, while question-level taxonomies face issues in evaluation and methodology.
Approach:	a new skill taxonomy synthesizes and redefines narrative understanding skills from previous taxonomies . the new taxomy includes a generation skill dimension from the answering perspective .
Outcome:	a new skill taxonomy synthesizes and redefines narrative understanding skills from previous taxonomies and includes a generation skill dimension from the answering perspective.

Order-Based Pre-training Strategies for Procedural Text Understanding (2024.naacl-short)

Copied to clipboard

Abhilash Nandy, Yash Kulkarni, Pawan Goyal, Niloy Ganguly

Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications? (2024.naacl-short)

Challenge:	Procedural text is difficult to understand due to the changing attributes of entities in the context.
Approach:	They propose sequence-based pre-training methods to enhance procedural understanding in natural language processing by using ordered instructions to guide individuals through a task.
Outcome:	The proposed methods improve on two datasets in the datasets NPN-Cooking and ProPara domains respectively.

Copied to clipboard

Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg