Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciences (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for named entity recognition fail in scientific domains such as biomedicine and chemistry. |
| Approach: | They propose a model to transfer knowledge from the biomedical domain to the target domain . they use pseudo labeling and contrastive learning to enhance discrimination . |
| Outcome: | The proposed model outperforms baseline models by up to 5% . the proposed model is based on a biomedical domain model and a chemical domain model . |
Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing diffusion models are applied to continuous feature space while texts are sequences of discrete categorical tokens. |
| Approach: | They propose to use an encoder-decoder Transformer architecture to approach sequence-to-sequence text generation. |
| Outcome: | The proposed model improves on five sequence-to-sequence generation tasks compared to other diffusion-based models regarding text quality and inference time. |
An Interactive Framework for Profiling News Media Sources (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing tools for detecting fake news are difficult for automated systems . e.g., we focus on the source level, and ask: Is this source factual or politically biased? |
| Approach: | They propose an interactive framework for news media profiling that uses graphs and pre-trained large language models to characterize social context on social media. |
| Outcome: | The proposed framework can detect fake and biased news media with as little as 5 human interactions . it can scale better, as often sources publish have same factuality/political bias as source . |
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable proficiency in language understanding and have been successfully applied to a variety of real-world tasks through task-specific fine-tuning or prompt engineering. |
| Approach: | They propose a task that challenges LLMs to identify the locations of mines based on numerical clues provided by adjacent cells. |
| Outcome: | The proposed task requires an understanding of each cell’s state, discerning spatial relationships between clues and mines, and strategizing actions based on logical deductions drawn from the arrangement of the cells. |
TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation (2024.naacl-long)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation (ERC) aims to identify emotions expressed by participants at each turn within a conversation. |
| Approach: | They propose a Teacher-leading Multimodal fusion network for ERC that integrates cross-modal knowledge distillation to transfer information from a lan- guage model acting as the teacher to non- verbal students. |
| Outcome: | The proposed model achieves state-of-the-art in a multi-speaker conversation dataset for ERC. |
Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies use in-context learning with large language models (LLM) to find similar dialogue exemplars for prompt learning. |
| Approach: | They propose to use a conversation retriever to find similar in-context examples for prompt learning. |
| Outcome: | The proposed approach improves on multiWOZ datasets with GPT-Neo-2.7B and LLaMA-7B/30B . |
Promptly Predicting Structures: The Return of Inference (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for structured prediction rely on large labeled datasets. Existing approaches for structured predictions require detailed annotation guidelines about the task, the label set, and the interactions between labels. |
| Approach: | They propose a framework for constructing zero- and few-shot linguistic structure predictors using structural constraints and combinatorial inferences. |
| Outcome: | The proposed framework can be extended to build zero- and few-shot label predictors on two structured prediction tasks and five datasets. |
On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL (2024.naacl-long)
Copied to clipboard
| Challenge: | Structured data is prevalent in tables, databases, and knowledge graphs, but there is a gap in our understanding of how these linearization-based methods handle structured data, which is inherently non-linear. |
| Approach: | They investigate the linear handling of structured data in encoder-decoder language models, specifically T5. |
| Outcome: | The proposed model can mimic human-designed processes such as schema linking and syntax prediction, and it can be compressed due to modality fusion redundancy. |
Extractive Summarization with Text Generator (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing extractive systems lack gold training signals, thereby hindering learning of extractive models. |
| Approach: | They propose to use text generators to train extractive summarizers by approximating outputs of abstractive summaries. |
| Outcome: | The proposed method can be used to train extractive summarizers without training . it is shown that the approximated summaries correlate positively with the auxiliary summary outputs. |
Self-generated Replay Memories for Continual Neural Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation systems exhibit strong performance in several different languages, but their ability to learn continuously is limited by catastrophic forgetting. |
| Approach: | They propose a method that leverages a key property of encoder-decoder Transformers, i.e. their generative ability, to continuously learn Neural Machine Translation systems. |
| Outcome: | The proposed approach can counteract catastrophic forgetting without explicit memorization of training data. |
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated strong efficacy as visual assistants . however, evaluation of their reasoning capabilities requires a costly benchmark . |
| Approach: | They propose a pipeline to measure the reasoning consistency of vision-language models . they propose supervised fine-tuning of VLMs and feedback from LLMs . |
| Outcome: | The proposed framework reduces cost while ensuring the generation of a high-quality dataset. |
Building Knowledge-Guided Lexica to Model Cultural Variation (2024.naacl-long)
Copied to clipboard
| Challenge: | Cultural variation exists between nations, but also within regions . Historically, it has been difficult to computationally model cultural variation due to a lack of training data and scalability constraints. |
| Approach: | They propose a method to measure cultural variation using a knowledge-guided lexical model using geolocated tweets. |
| Outcome: | The proposed method could help us better understand the way people communicate and build more culturally-aware NLP systems. |
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for speech-to-text translation (ST) have achieved impressive supervised and zero-shot performance. |
| Approach: | They propose to use consistency regularization methods to boost end-to-end (E2E) speech-totext translation (ST) by regularizing the intra-modal consistency instead of the modality gap. |
| Outcome: | The proposed training strategies achieve state-of-the-art (SOTA) performance in most translation directions. |
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on LLMs that only enhance reasoning abilities, but which lack factual hallucination and slow-thinking capabilities, argues that SPP is a cognitive synergist. |
| Approach: | They propose a Solo Performance Prompting (SPP) that transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. |
| Outcome: | The proposed model reduces factual hallucination and maintains strong reasoning abilities on three challenging tasks . |
FPT: Feature Prompt Tuning for Few-shot Readability Assessment (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt-based methods lack crucial linguistic knowledge for readability assessment tasks such as word length, sentence length, and usage of different difficulty-level words. |
| Approach: | They propose a new prompt-based tuning framework that incorporates linguistic knowledge and a loss function to calibrate the similarity ranking order between categories. |
| Outcome: | The proposed framework outperforms the large language model gpt-3.5-turbo-16k in most cases. |
Self-Prompting Large Language Models for Zero-Shot Open-Domain QA (2024.naacl-long)
Copied to clipboard
| Challenge: | Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing specific background documents. |
| Approach: | They propose a framework to explicitly utilize the massive knowledge encoded in LLM parameters and their strong instruction understanding abilities. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on three widely-used ODQA datasets and achieves comparable performance with customized fine-tuned models on full training data. |
Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models lack knowledge of nuanced, domain-specific details and are susceptible to hallucinations. |
| Approach: | They construct a benchmark that measures head, torso, and tail facts in terms of popularity. |
| Outcome: | The proposed model is based on 18K question-answer pairs regarding head, torso, and tail facts in terms of popularity. |
kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning (2024.naacl-long)
Copied to clipboard
Wenting Zhao, Ye Liu, Yao Wan, Yibo Wang, Qingyang Wu, Zhongfen Deng, Jiangshu Du, Shuaiqi Liu, Yunlong Xu, Philip Yu
| Challenge: | Recent advances in task-oriented parsing involve formulating the task as a sequence-to-sequence problem, relying on a wealth of labeled data. |
| Approach: | They propose a task-oriented parsing framework that integrates nearest-neighbor learning with a nearest-nearest approach. |
| Outcome: | The proposed model can be used to synthesize computer programs based on a natural-language prompt without additional data or specialized prompts. |
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Evaluating retrieval-augmented generation systems relies on hand annotations for input queries, passages to retrieve, and responses to generate. |
| Approach: | They propose an automated evaluation framework for retrieval-augmented generation (RAG) ARES fine tunes lightweight LLM judges on synthetically generated queries and answers . |
| Outcome: | The proposed framework evaluates RAG systems using only human annotations . it can be used to improve system understanding and create targeted solutions . |
DEMO: A Statistical Perspective for Efficient Image-Text Matching (2024.naacl-long)
Copied to clipboard
| Challenge: | Image-text matching is a problem that seeks to connect vision and language through semantic understanding. |
| Approach: | They propose a deep unsupervised hashing-based approach for image-text matching . they characterize each image using multiple augmented views, which are considered as samples . |
| Outcome: | The proposed approach achieves superior performance on image-text matching datasets compared with state-of-the-art methods. |
SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | a new benchmark for multilingual foundation models is being developed . brittleness of foundation models in the dimensions of semantics and multilinguality is a key limitation . |
| Approach: | They propose a benchmark for multilingual foundation models, SeaEval . they examine how well these models comprehend cultural practices, nuances, and values . |
| Outcome: | The proposed model can be used to evaluate multilingual and multicultural scenarios. |
Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have conjectured that multimodal hallucination is due to the vision encoder failing to ground on the image properly. |
| Approach: | They propose a multimodal self-feedback guided revision model that leverages visual cues to generate feedback to its initial response based on the visual information provided by the vision encoder. |
| Outcome: | The proposed model reduces multimodal hallucination and outperforms previous models on MMHal-Bench, POPE, and GAVIE. |
LLMs Are Few-Shot In-Context Low-Resource Language Learners (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) empowers large language models to perform diverse tasks in underrepresented languages using only short in-contrast information. |
| Approach: | They extensively assess the effectiveness of in-context learning with LLMs in low-resource languages . they also identify the shortcomings of in context label alignment . |
| Outcome: | The proposed approach improves understanding quality of low-resource languages by closing the language gap in the target language. |
Simple and effective data augmentation for compositional generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | Compositional generalization is the ability of a system to correctly predict the meaning of complex sentences when trained on simpler sentences. |
| Approach: | They propose to use data augmentation methods to generate additional training data by sampling from an augmentation distribution to generalize to the out-of-distribution test data. |
| Outcome: | The proposed method outperforms existing methods that sampled from the training distribution and outperformed existing methods. |
Rethinking Tabular Data Understanding with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. |
| Approach: | They propose a method for table structure normalization to improve model performance . they propose aggregation of multiple reasoning pathways to improve performance based on textual and symbolic reasoning. |
| Outcome: | The proposed method improves performance on symbolic reasoning tasks with textual reasoning slightly outperforming symbolic reasoning on tables. |
From Shortcuts to Triggers: Backdoor Defense with Denoised PoE (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing backdoor defense methods focus on specific triggers, leaving a universal defense unexplored. |
| Approach: | They propose an ensemble-based backdoor defense framework that denies backdoor attacks by capturing backdoor shortcuts and preventing learning them. |
| Outcome: | The proposed framework significantly improves defense performance against backdoor attacks . it is also effective under a more challenging but practical setting . |
BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for accounting databases that can be queried using natural language are lacking in some domains. |
| Approach: | They propose a large-scale text-to-SQL dataset for accounting and financial domains . they propose 'bookSQl' to be used to query accounting databases using natural language . |
| Outcome: | The proposed model performs poorly on the existing model, pointing towards a more focused model for this domain. |
FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Flow-adhering planning algorithm for task oriented dialogs (TODs) is a task-oriented dialog (TO) that can be used for task planning and API usage. |
| Approach: | They propose a Flow-Adhering Planning algorithm that follows predefined flows and preserves API dependencies in task oriented dialogs. |
| Outcome: | The proposed algorithm outperforms other decoding and prompting-based baselines in task oriented dialogs. |
DuRE: Dual Contrastive Self Training for Semi-Supervised Relation Extraction (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing document-level relation extraction methods require manual training and labeled data to obtain supervised learning. |
| Approach: | They propose a document-level relation extraction framework that integrates RE and text generation as a dual process. |
| Outcome: | The proposed framework significantly boosts recall and F1 score with comparable precision on two document-level RE tasks against several strong baselines. |
Query-Efficient Textual Adversarial Example Generation for Black-Box Attacks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing black-box attacks require thousands of queries on the target model, making them expensive in real-world applications. |
| Approach: | They propose a new approach that guides word substitutions using prior knowledge from the training set to improve the attack efficiency. |
| Outcome: | The proposed approach reduces query-free attack and guided search attacks by a factor of 10 500 . it improves transferability and generalization by the ensemble of the ABPens in NLP . |
Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles (2024.naacl-long)
Copied to clipboard
Kung-Hsiang Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu
| Challenge: | Existing studies on multi-document summarization focus on collating information that all sources agree upon, but the task of summarizing diverse information remains underexplored. |
| Approach: | They propose a task of summarizing diverse information encountered in multiple news articles encompassing the same event using a dataset curated by a large language model. |
| Outcome: | The proposed task aims to summarize diverse information in multiple news articles encompassing the same event . the proposed task is difficult due to its limited coverage and verbosity biases . |
AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency of abstractive summarization lack coherence or error-type coverage. |
| Approach: | They propose a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) they use a selection module NegFilter to ensure the quality of the generated negative examples . |
| Outcome: | The proposed framework outperforms existing systems on the AggreFact-SOTA benchmark and provides high error-type coverage. |
PILOT: Legal Case Outcome Prediction with Case Law (2024.naacl-long)
Copied to clipboard
| Challenge: | predicting legal case outcomes requires identifying relevant precedent cases . predicting case outcomes in case law systems presents unique challenges . |
| Approach: | They propose a framework for making legal case outcome predictions with case law . they propose to use two modules for relevant case retrieval and temporal pattern handling . |
| Outcome: | The proposed framework shows significant improvement over previous models based on civil law cases . it is crucial to identify relevant precedent cases that serve as evidence for judges . |
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks. |
| Approach: | They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks with comparable parameters. |
R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for speaker and noise-invariant speech representations use unlabeled audio data to pretrain encoders, generating good representations for downstream tasks like automatic speech recognition (ASR) and speaker identification. |
| Approach: | They propose a domain-specific self-supervision method for speaker and noise-invariant speech representations by learning discrete acoustic units with speaker-in-variant clustering. |
| Outcome: | The proposed method reduces computational resources by 12X compared to state-of-the-art methods while outperforming them in severely distorted speech scenarios. |
InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions (2024.naacl-long)
Copied to clipboard
| Challenge: | In order to perform downstream tasks, Large Language Models (LLMs) need continual adaptation without catastrophic forgetting. |
| Approach: | They propose a new paradigm that allows for continual adaptation without catastrophic forgetting . they propose to replay previous data based on task similarity with instructions . |
| Outcome: | The proposed method improves performance over 16 tasks with different training orders. |
Language Agnostic Code Embeddings (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that code language models have strong cross-lingual traits, but their multilingual representations can be dissected into a language-specific syntax component and a semantic component. |
| Approach: | They propose to isolate and eliminate language-specific components from multilingual code embeddings to improve downstream code retrieval tasks. |
| Outcome: | The proposed model improves retrieval tasks by removing language-specific components . the proposed model can be used to perform a variety of code generation tasks . |
An Examination of the Compositionality of Large Generative Vision-Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the compositionality of vision-language models (VLMs) however, the performance of GVLMs in multimodal compositional reasoning remains under-explored. |
| Approach: | They propose a syntactical bias score to quantify GVLMs' syntaktical bias . they propose 'SADE' task to assess GVLs's robustness against inclination toward syntical correctness. |
| Outcome: | The proposed benchmarks are based on evaluation metrics and current benchmarks. |
Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoors (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing defense mechanisms assume that only one type of trigger is adopted by the attacker, while defending against multiple simultaneous and independent trigger types necessitates general defense frameworks. |
| Approach: | They propose a framework that uses a mixture of experts as a trigger-only ensemble to defend against multiple trigger types. |
| Outcome: | The proposed framework defends against multiple trigger types in a single ensemble and in combination of models. |
VertAttack: Taking Advantage of Text Classifiers’ Horizontal Vision (2024.naacl-long)
Copied to clipboard
| Challenge: | Current SOTA classifiers are subject to problems like bias and are vulnerable to adversarial attacks. |
| Approach: | They propose an attack to mimic a classifier's character based attack and thenrewrite those words vertically. |
| Outcome: | The proposed attack can drop the accuracy of 4 different transformer models on 5 datasets and preserve meaning. |
KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on multimodal sentence embeddings took negative samples without reviewing, resulting in noisy and noisy negative samples. |
| Approach: | They propose a multimodal contrastive learning approach that inherits the knowledge from the teacher model to learn the difference between positive and negative instances. |
| Outcome: | The proposed approach can detect noisy and wrong negative samples before they are calculated in the contrastive objective. |
The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that multilingual speech processing systems can generalize to unseen languages without adaptation. |
| Approach: | They propose a phoneme-based phoneme embedding model that can be generalized to unseen languages by using a neural forced aligner. |
| Outcome: | The proposed model can generalize to unseen languages without adaptation. |
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing vision-language models focus on salient attributes but ignore contextualized nuances, resulting in gender bias. |
| Approach: | They propose a task-agnostic generation framework to mitigate gender bias in vision-language models. |
| Outcome: | The proposed framework can mitigate gender bias in vision-language models . it yields all-sided but gender-obfuscated narratives, which prevents concentration on localized image features, especially gender attributes. |
BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing LLMs adopt autoregressive architectures without explicit backward dependency modeling. |
| Approach: | They propose a backward dependency enhanced large language model that transforms attention layers from uni-to-bi-directional to learn sentence embeddings. |
| Outcome: | The proposed model achieves state-of-the-art performance in varying scenarios. |
Assessing Factual Reliability of Large Language Model Knowledge (2024.naacl-long)
Copied to clipboard
| Challenge: | Factual knowledge of LLMs is typically evaluated using accuracy, yet this metric does not capture the vulnerability of LRMs to hallucination-inducing factors like prompt and context variability. |
| Approach: | They propose a metric designed to measure LLMs’ factual reliability by comparing the distance between the probability distributions of a valid output and its counterparts produced by the same LLM probing the same fact using different styles of prompts and contexts. |
| Outcome: | The proposed metric measures the distance between the probability distributions of a valid output and its counterparts produced by the same LLM probing the same fact using different styles of prompts and contexts. |
Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on dialogue response selection focus on post-training and fine-tuning for cross-encoders. |
| Approach: | They propose a post-training technique tailored for dense encoders in dialogue response selection . they propose 'Dialogue Contextual Masking Auto-Encoder' which compresses dialogue semantics into dense vectors . |
| Outcome: | The proposed technique achieves state-of-the-art on two commonly evaluated benchmarks. |
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptability. |
| Approach: | They propose a framework that performs task-solving by first creating a toolkit and then integrating the planning and calling of tools through a chain-of-solve approach. |
| Outcome: | The proposed model performs task-solving by harnessing Toolink's creativity and CoS ability on ChatGPT and finetunes the LLaMA-7B model. |
Create! Don’t Repeat: A Paradigm Shift in Multi-Label Augmentation through Label Creative Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for multi-label data augmentation have been ineffective, authors say . a mere 1.5% of labels have more than 100 training instances, a problem that persists for years . |
| Approach: | They propose a new paradigm for multi-label data augmentation called Label Creative Generation . they propose tail-driven conditional augmentation with tail-based sampling and label-conditioned generation . |
| Outcome: | The proposed approach has shown a 10% increase in PSP@1 across three datasets . it effectively mitigates the long-tail effect and enhances model performance . |
Neurocache: Efficient Vector Retrieval for Long-range Language Modeling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent research shows that retrieval-augmented models with shorter contexts (4K tokens) can match the performance of models with longer contexts (16K/32K token) |
| Approach: | They introduce an approach to extend the effective context size of large language models by using an external vector cache to store past states. |
| Outcome: | The proposed method improves on models trained from scratch and pre-trained models. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models of large language models struggle with complex logical reasoning problems. |
| Approach: | They propose to use large language models to identify their own errors to improve their models' performance. |
| Outcome: | The proposed models can identify logical fallacies accurately and improve by themselves. |
Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. |
| Approach: | They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning. |
| Outcome: | The proposed framework achieves comparable results to existing models on three language understanding benchmarks. |
MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | TALMs have been successfully employed in question-answering benchmarks, but their efficacy on complex mathematical reasoning benchmarks are open research questions. |
| Approach: | They propose a tool-augmented large language model for mathematical reasoning that enhances the skillset of large language models (LLMs) by 13.5%. |
| Outcome: | The proposed model achieves better accuracy and better knowledge retrieval performance than existing tools. |
CoUDA: Coherence Evaluation via Unified Data Augmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing data augmentations for coherence evaluation rely on heuristic rules and lack designing criteria. |
| Approach: | They propose a data augmentation framework that breaks down coherence into global and local aspects and designs augmentation strategies for both aspects. |
| Outcome: | The proposed framework surpasses recent models in scoring and ranking tasks with 233M parameters. |
mEdIT: Multilingual Text Editing via Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | mEdIT is a multi-lingual extension to CoEdit for writing assistance. |
| Approach: | They propose to train multi-lingual large language models (LLMs) by fine-tuning them via instruction tuning. |
| Outcome: | The proposed model performs well on multilingual text editing benchmarks and generalizes well to new languages. |
Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Towards Byzantine-robust federated embodied agent learning, we study the attack and defense for the task of vision-and-language navigation (VLN) |
| Approach: | They propose a new method to defend against a navigation-and-language navigation attack using navigation as wish (NAW) the method provides the server with a 'prompt' of the vision-and language alignment variance between benign and malicious clients so they can be distinguished during training. |
| Outcome: | The proposed method outperforms other state-of-the-art defense methods on two VLN datasets. |
In-context Learning and Gradient Descent Revisited (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism remains elusive. |
| Approach: | They propose a simple gradient descent-based optimization procedure that respects layer causality and improves similarity scores significantly. |
| Outcome: | The proposed procedure improves similarity scores on untrained models despite not showing ICL. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
On Large Language Models’ Hallucination with Regard to Known Facts (2024.naacl-long)
Copied to clipboard
Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, Jie Zhou
| Challenge: | Large language models are successful in answering factoid questions but are also prone to hallucination. |
| Approach: | They propose self-reporting to the model when faced with such limitations. |
| Outcome: | The proposed classifier can detect hallucinations with an 88% success rate and can be used to answer factoid questions with correct answer knowledge. |
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors (2024.naacl-long)
Copied to clipboard
| Challenge: | Natural language generation models are used for many downstream applications involving interpersonal communication, such as text completion, "smart" reply suggestions, and chatbot assistants. |
| Approach: | They conduct five case studies that perturb identity-related language features in NLG inputs to examine their assumptions about fairness. |
| Outcome: | The findings highlight open challenges around what constitutes “fair” or “good” NLG system behaviors. |
Language Models Hallucinate, but May Excel at Fact Verification (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have produced non-factual outputs . however, current LLMs suffer from the hallucination issue . |
| Approach: | They propose to use instruction-tuned LLMs to generate factual outputs . they find that FLAN-T5-11B performs best as a fact verifier . |
| Outcome: | The proposed method outperforms more capable LLMs like GPT3.5 and ChatGPT in the human evaluation. |
A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing state-of-the-art event coreference resolution systems rely on spurious and spurious associations in the input mention pair text. |
| Approach: | They propose a rationale-centric counterfactual data augmentation method that leverages the debiasing capability of counterfact data haussed by LLM-in-the-loop to mitigate spurious association while emphasizing causation. |
| Outcome: | The proposed method achieves state-of-the-art on three popular cross-document benchmarks and demonstrates robustness in out-of domain scenarios. |
TrojFSP: Trojan Insertion in Few-shot Prompt Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt tuning on a few data samples presents security issues, e.g., Trojan attacks. |
| Approach: | They propose a method to transfer established data poisoning attacks directly to few-shot prompt tuning, a technique to address the poisoned imbalance issue. |
| Outcome: | The proposed method achieves an ASR of over 99% while maintaining negligible decreases in CDA. |
Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models (2024.naacl-long)
Copied to clipboard
Yi Luo, Zhenghao Lin, YuHao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong
| Challenge: | Guide-Align is a guideline-oriented approach to augment the safety and quality of Large Language Models. |
| Approach: | They propose a guideline-oriented method to augment the safety and quality of large language models. |
| Outcome: | The proposed method outperforms existing methods on three benchmarks and shows significant improvements in security and quality. |
X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs (2024.naacl-long)
Copied to clipboard
| Challenge: | X-parade is the first cross-lingual dataset of paragraph-level information divergences and entailments . ability to recognize differences in meaning underlies many NLP tasks . |
| Approach: | They propose a cross-lingual dataset of paragraph-level information divergences . they use a dictionary to identify new or inferred information in Wikipedia pages . |
| Outcome: | The proposed dataset shows that the proposed methods fail to handle inferable information . the dataset contains fine-grained span-level annotations for content in different languages . |
Topics, Authors, and Institutions in Large Language Model Research: Trends from 17K arXiv Papers (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in language modeling have caused disruptive shifts throughout AI research, spurring discussion about how the field is changing and how it should change. |
| Approach: | They analyze a dataset of 16,979 LLM-related arXiv papers and examine industry and academic publishing trends. |
| Outcome: | The authors examine the impact of large language models on AI research in 2023 and 2022. |
E5: Zero-shot Hierarchical Table Analysis using Augmented LLMs via Explain, Extract, Execute, Exhibit and Extrapolate (2024.naacl-long)
Copied to clipboard
| Challenge: | Hierarchical tables with multi-level headers are confusing for models due to their complex structure, implicit semantics, and calculation relationships. |
| Approach: | They propose a code-augmented framework for zero-shot hierarchical table question answering that combines self-explaining the table's hierarchic structures with external code execution to prevent hallucinations. |
| Outcome: | The proposed framework outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement. |
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate extremely long-context LLMs or analyze their limitations. |
| Approach: | They propose a Synthetic, Scalable, Systematic evaluation suite for LLMs using SQL execution. |
| Outcome: | The proposed evaluation suite is able to scale text length and difficulty across scenarios and provides strong correlations with real-world benchmarks. |
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning (2024.naacl-long)
Copied to clipboard
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu
| Challenge: | Existing large language models have limited ability to perform tasks effectively. |
| Approach: | They propose a large-scale multimodal chart instruction dataset with 600k instances supporting diverse tasks and chart types. |
| Outcome: | The proposed LMM achieves state-of-the-art performance on existing chart QA benchmarks. |
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Accurate Knowledge Distillation via n-best Reranking (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies using sequencelevel knowledge distillation (KD) have adopted this approach. |
| Approach: | They propose to utilize n-best reranking to enhance Sequence-Level Knowledge Distillation by utilizing a diverse set of models with different inductive biases, objective functions or architectures to pick the highest-quality hypotheses as labels. |
| Outcome: | The proposed approach is validated on the WMT’21 German English and Chinese english translation tasks. |
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet relying on extensive manual labeling to provide procedural feedback remains a significant impediment. |
| Approach: | They propose a self-supervised framework that decomposes complex problems into manageable subquestions with a controllable granularity switch and sequentially applies reinforcement learning to iteratively improve the subquest solver. |
| Outcome: | The proposed framework improves performance on mathematical and commonsense reasoning tasks over SOTA. |
SEMQA: Semi-Extractive Multi-Source Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent proposed long-form question answering systems have shown promising capabilities, but attributing and verifying their generated abstractive answers can be difficult. |
| Approach: | They propose a task that summarises multiple sources in a semi-extractive fashion . they create a dataset with human-written semi-extractive answers to natural and generated questions . |
| Outcome: | The proposed task summarizes multiple sources in a semi-extractive fashion and produces fine in-line attributions by-design that are easy to verify, interpret, and evaluate. |
Fine-Tuning Language Models with Reward Learning on Policy (2024.naacl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is an effective approach to align large language models (LLMs) to human preferences. |
| Approach: | They propose a framework that refines a reward model using policy samples to keep it on-distribution. |
| Outcome: | The proposed framework outperforms the state-of-the-art on three benchmark datasets showing that it can learn robust representations of policy samples. |
A Universal Dependencies Treebank for Highland Puebla Nahuatl (2024.naacl-long)
Copied to clipboard
| Challenge: | Annotated linguistic corpora are essential component of natural language processing (NLP) Annotation frameworks are used for morphological and dependency-based syntactic phenomena in endangered, indigenous, and/or marginalized languages. |
| Approach: | They propose a Universal Dependencies (UD) treebank for Highland Puebla Nahuatl . they describe the process of data collection, annotation decisions and interesting syntactic constructions . |
| Outcome: | The proposed treebank is the second such UD treebank for a Mexican language . it is a significant addition to an existing treebank of another Nahuatl language based on the framework . |
COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing multilingual language models struggle to capture local nuances and contexts that vary from culture to culture. |
| Approach: | They propose a public Indonesian language common sense reasoning dataset COPAL-ID . it incorporates Indonesian local and cultural nuances and provides a more natural portrayal of causal reasoning . |
| Outcome: | The proposed dataset is fluent and free from awkward phrases, unlike the previous dataset. |
IterAlign: Iterative Constitutional Alignment of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to 13.5% in harmlessness. |
| Approach: | They propose a data-driven constitution discovery and self-alignment framework called IterAlign to overcome these drawbacks by leveraging red teaming to uncover weaknesses of an LLM. |
| Outcome: | Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty by up to 13.5%. |
OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are computationally expensive and often require computational resources. |
| Approach: | They propose a routing framework that seamlessly integrates a SLM and an LLM, or-lm, or a LLM into a single framework. |
| Outcome: | The proposed routing framework reduces the computational costs by over 50% in dialogue state tracking tasks. |
Multi-Operational Mathematical Derivations in Latent Space (2024.naacl-long)
Copied to clipboard
| Challenge: | Using a symbolic engine, we investigate the possibility of approximating multiple mathematical operations in latent space for expression derivation. |
| Approach: | They propose to model mathematical operations as explicit geometric transformations by leveraging a symbolic engine and a large-scale dataset. |
| Outcome: | The proposed paradigms can be used to approximate multiple mathematical operations in latent space, while discriminating the conclusions for a single operation is achievable in the original expression encoder. |
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |
XferBench: a Data-Driven Benchmark for Emergent Language (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to teach models to "language" are full of bias, toxicity, and potential intellectual property violations. |
| Approach: | They propose a benchmark for evaluating the overall quality of emergent languages using data-driven methods. |
| Outcome: | The proposed benchmark is based on utterances from the emergent language and is validated using human, synthetic, and emergentic language baselines. |
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. |
| Approach: | They propose a protocol to evaluate the degree to which language models can accurately emulate human behavior in conversational recommendation systems. |
| Outcome: | The proposed protocol evaluates five tasks to reveal deviations of language models from human behavior and offers insights on how to reduce deviations with model selection and prompting strategies. |
A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | evaluating the generalisability of Transformers to out-of-distribution mathematical reasoning problems is a challenge for many open-source models. |
| Approach: | They propose a method for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, and compare their results to sequence classification tasks. |
| Outcome: | The proposed framework outperforms GPT-4, GPT-3.5 and a canon of fine-tuned BERT models in classification tasks . perturbations to input reasoning can reduce their performance by up to 80 F1 points . |
Identifying Linear Relational Concepts in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a technique for finding concept directions for human-interpretable concepts is needed to find their direction in the latent space . a linear relational concept (LRC) can be used to locate concepts in hidden activations . |
| Approach: | They propose a method for finding human-interpretable concepts by inverting a linear relational embedding and using earlier object layers. |
| Outcome: | The proposed method outperforms standard probing classifiers on performance as concept classifier and ability to causally change model output. |
Benchmark Transparency: Measuring the Impact of Data on Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we quantify the impact that data distribution has on the performance and evaluation of NLP models. |
| Approach: | They propose to use disproportional stratified sampling to measure the data distribution across 6 different dimensions to quantify model performance. |
| Outcome: | The proposed framework measures the data distribution across 6 different dimensions and shows that it is statistically significant and predicts model performance. |
JAMDEC: Unsupervised Authorship Obfuscation using Constrained Decoding over Small Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to protect the identity and privacy of online authorship are lacking supervision data for diverse authorship and domains. |
| Approach: | They propose an unsupervised inference-time approach to authorship obfuscation that uses a user-controlled, inference time algorithm to oblige the authorship. |
| Outcome: | The proposed method outperforms state-of-the-art methods while performing competitively against a propriety model two orders of magnitudes larger. |
REST: Retrieval-Based Speculative Decoding (2024.naacl-long)
Copied to clipboard
| Challenge: | Retrieval-based speculative decoding (REST) is a new language model generation algorithm . it uses existing knowledge to generate draft tokens, allowing for seamless integration and acceleration of any language model. |
| Approach: | They propose a new algorithm that uses a draft language model to generate tokens from existing knowledge. |
| Outcome: | The proposed method achieves a speedup of 1.62 to 2.36 on code or text generation. |
Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations (2024.naacl-long)
Copied to clipboard
Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, Dong Yu
| Challenge: | Sentence embeddings are typically learned to recognize the semantic relation between two text inputs. |
| Approach: | They introduce a contrastively-learned contextual embedding model for fine-grained semantic representation of text. |
| Outcome: | The proposed model is able to produce contextual embeddings corresponding to different atomic propositions, i.e. semantic equivalence between propositions across different text sequences. |
MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference (2024.naacl-long)
Copied to clipboard
| Challenge: | a dataset containing 132,320 sentence pairs from five new scientific domains is used for scientific Natural Language Inference (NLI) the availability of multiple domains makes it possible to study domain shift for scientific NLI. |
| Approach: | They propose a dataset with 132,320 sentence pairs from five new scientific domains to introduce diversity in scientific NLI. |
| Outcome: | The proposed dataset contains 132,320 sentence pairs extracted from five new scientific domains. |
Causal Inference for Human-Language Model Collaboration (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we examine the collaborative dynamics between humans and language models where the interaction involves LMs proposing text segments and humans editing or responding to these segments. |
| Approach: | They propose a causal estimand to estimate the incremental stylistic effect (ISE) of various interaction strategies in dynamic human-LM collaborations. |
| Outcome: | The proposed estimand reduces confounding and significantly improves counterfactual estimation over a set of competitive baselines. |
SELF-GUARD: Empower the LLM to Safeguard Itself (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have investigated methods to improve the safety of large language models (LLMs) safety training involves fine-tuning the LLM with adversarial samples, which activate the LRM’s capabilities against jailbreak. |
| Approach: | They propose a safety training approach that integrates safety training and safeguards to train the LLM to perform harmfulness detection on its own outputs. |
| Outcome: | The proposed method reduces harmful output and adds a [harmful] or [harmless] tag to the end of the LLM's response. |
COSIGN: Contextual Facts Guided Generation for Knowledge Graph Completion (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for knowledge graph completion (KGC) are limited in generality and scalability due to poor contextual facts. |
| Approach: | They propose a contextual facts collector and contextual facts organizer to enhance the inference ability of GM-based methods for various KGC tasks. |
| Outcome: | The proposed model outperforms state-of-the-art methods in terms of performance. |
Toward Informal Language Processing: Knowledge of Slang in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have offered a strong potential for natural language systems to process informal language. |
| Approach: | They propose to use movie subtitles to evaluate slang in large language models . they find that smaller LLMs finetuned on the dataset achieve comparable performance . |
| Outcome: | The proposed dataset can be used to evaluate LLMs on slang detection and identification of regional and historical sources for interpretive insights. |
Ghostbuster: Detecting Text Ghostwritten by Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Ghostbuster is a system that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features. |
| Approach: | They propose a method that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features. |
| Outcome: | The proposed method outperforms existing detectors and a new baseline on student essays, creative writing, and news articles. |
End-to-End Beam Retrieval for Multi-Hop Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing beam retrieval frameworks for multi-hop question answering were customized for two-hop questions and were poorly supervised. |
| Approach: | They propose an end-to-end beam retrieval framework for multi-hop question answering . they combine an encoder and two classification heads to optimize the retrieval process . |
| Outcome: | The proposed framework improves on MuSiQue-Ans and surpasses all previous retrievers on HotpotQA and achieves 99.9% precision on 2WikiMultiHopQA. |
Leveraging Generative Large Language Models with Visual Instruction and Demonstration Retrieval for Multimodal Sarcasm Detection (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sarcasm detection do not fully utilize cross-modal features, limiting their performance on in-domain datasets. |
| Approach: | They propose a multimodal sarcasm detection model with a designed instruction template and a demonstration retrieval module. |
| Outcome: | The proposed model outperforms existing methods on in-domain datasets and achieves state-of-the-art performance. |
Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification require numerous LMs’ calls to search optimal prompts, thus resulting in overfitting performance and increasing computational cost. |
| Approach: | They propose a multi-scale knowledge prompt-based memory model that extracts instance-level and class-level knowledge and stores them in memory banks during training. |
| Outcome: | Experiments on different benchmarks and parameter analysis demonstrate the effectiveness and efficiency of MuSKPrompt in black-box few-shot text classification tasks. |
Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) has shown impressive results on many tasks, but applying LLMs to grammatical error correction (GEC) is still a challenging task. |
| Approach: | They propose an ungrammatical-syntax-based in-context example selection strategy that measures similarity of sentences based on their syntactic structures and identify optimal ICL examples sharing the most similar ill-formed syntax to the test input. |
| Outcome: | The proposed strategy outperforms word-matching and semantics-based methods on a syntax-oriented task like GEC on benchmark English datasets. |
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024.naacl-long)
Copied to clipboard
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi
| Challenge: | Recent advances in few-shot generalization in natural language processing focus on English. |
| Approach: | They propose a benchmark that unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
| Outcome: | The proposed framework unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
TISE: A Tripartite In-context Selection Method for Event Argument Extraction (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that LLMs can finish inference by providing several examples. |
| Approach: | They propose a method which integrates three requirements when selecting an in-context example and integrates them into a set of determinantal point processes to enhance the reasoning capabilities of LLMs. |
| Outcome: | The proposed method can achieve superior performance with fewer examples and outperform some supervised methods. |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
TRUE-UIE: Two Universal Relations Unify Information Extraction Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Information extraction (IE) tasks have a variety of schemas and objectives that differ across tasks. |
| Approach: | They propose a paradigm where all IE tasks are aligned to learn the same goals . they use two universal relations to extract mention spans and type recognition . |
| Outcome: | The proposed model achieves state-of-the-art on established benchmarks spanning 16 datasets, spanning 7 diverse IE tasks. |
zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to forecast links on temporal knowledge graphs are embedding-based . but they face a strong challenge in modeling the unseen zero-shot relations . |
| Approach: | They propose to embed knowledge graphs (TKGF) entities and relations based on observed contexts into embedding-based methods to model unseen zero-shot relations. |
| Outcome: | The proposed methods show strong performance on traditional TKG forecasting benchmarks, but they face a strong challenge in modeling unseen zero-shot relations that have no prior graph context. |
Embodied Executable Policy Learning with Language-based Scene Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing Large Language models with text inputs lack the capability to evolve with non-expert interactions with environments. |
| Approach: | They propose a novel learning paradigm that generates robots’ executable actions in the form of text, derived solely from visual observations. |
| Outcome: | The proposed learning paradigm surpasses baselines and can adapt to the target tasks effectively. |
Metacognitive Prompting Improves Understanding in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in prompting have enhanced reasoning in logic-intensive tasks for LLMs, yet the nuanced understanding abilities of these models remain underexplored. |
| Approach: | They propose a strategy inspired by human introspective reasoning processes to enhance LLMs' understanding abilities. |
| Outcome: | The proposed method outperforms chain-of-thought prompting and its advanced versions on ten natural language understanding (NLU) datasets. |
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing red-teaming methods for large language models often discover safety risks without addressing them. |
| Approach: | They propose a multi-round automatic red-teaming method that incorporates both adversarial prompt writing and safe response generation. |
| Outcome: | The proposed method significantly increases red-teaming scalability and the safety of the target LLM. |
DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing multi-modal dialogue datasets that focus on image-based dialogues have low quality and limited diversity of images per dialogue. |
| Approach: | They propose to construct a multi-modal dialogue dataset that guarantees both dialogue quality and image diversity without requiring minimum human effort. |
| Outcome: | The proposed dataset outperforms existing datasets in terms of quality and diversity in human evaluation. |
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing ensemble methods for Large Language Models focus on reward model ranking of outputs, leading to significant computation overhead. |
| Approach: | They propose a reward-guided routing method distilling rewards on training queries to train a routing function. |
| Outcome: | The proposed method outperforms the best single model and ranks first on 44% of tasks. |
Automatic Generation of Model and Data Cards: A Step Towards Responsible AI (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models and datasets are incomplete and lack consistent documentation. |
| Approach: | They propose an automated generation approach using Large Language Models (LLMs) their paper establishes a comprehensive dataset and develops 'CardGen' pipeline . |
| Outcome: | The proposed approach exhibits enhanced completeness, objectivity, and faithfulness in generated model and data cards, a significant step in responsible AI documentation practices ensuring better accountability and traceability. |
FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing (2024.naacl-long)
Copied to clipboard
| Challenge: | Standard fine-tuning of language models suffers with generalization to distribution shifts. |
| Approach: | They propose to use Fisher Information to investigate scheduled unfreezing algorithms for adapter-based cross-lingual task transfer to improve generalization to distribution shifts. |
| Outcome: | The proposed method achieves an average of 2 points improvement over four datasets compared to standard fine-tuning and provides empirical evidence for a theory-based justification of the proposed method. |
Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are adept at question answering and reasoning tasks, but when reasoning in situational context, human expectations vary depending on the relevant cultural common ground. |
| Approach: | They construct and evaluate a dataset for proverb understanding with conversational context for six different languages and their usage within the context. |
| Outcome: | The proposed model is able to reason with proverbs and sayings in conversational contexts. |
The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youth (2024.naacl-long)
Copied to clipboard
Shir Lissak, Nitay Calderon, Geva Shenkman, Yaakov Ophir, Eyal Fruchter, Anat Brunstein Klomek, Roi Reichart
| Challenge: | Queer youth face increased mental health risks, such as depression, anxiety, and suicidal ideation. |
| Approach: | They propose a scale that is inspired by psychological standards and expert input to evaluate LLM's interactions with queer-related content. |
| Outcome: | The proposed scale outperforms human responses to queer-related content and outperformed LLMs in the qualitative and quantitative analysis. |
IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to extract relational triples have inherent shortcomings such as redundant information and incomplete triple recognition. |
| Approach: | They propose an Implicit Perspective for relational triple Extraction based on Diffusion model that uses block coverage to complete tables. |
| Outcome: | The proposed method achieves state-of-the-art performance while gaining low computational complexity. |
QualEval: Qualitative Evaluation for Model Improvement (2024.naacl-long)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, Ashwin Kalyan
| Challenge: | Quantitative evaluation metrics are inadequate for large language models due to complexity of tasks and cannot provide actionable diagnostics. |
| Approach: | They propose a quantitative evaluation tool called QualEval that uses automated qualitative evaluation as a vehicle for model improvement. |
| Outcome: | The proposed method improves the performance of the Llama 2 model by 15% compared to baselines. |
Quantum-inspired Language Model with Lindblad Master Equation and Interference Measurement for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | Quantum-inspired models have demonstrated superior performance in many downstream language tasks, such as question answering and sentiment analysis. |
| Approach: | They propose a quantum-inspired neural network that integrates the Lindblad Master Equation to model the evolution process and the interferometry to the measurement process, providing more physical meaning to strengthen the interpretability. |
| Outcome: | The proposed model outperforms existing models on sentiment analysis datasets and shows that it is more accurate and performs better than existing models. |
VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization (2024.naacl-long)
Copied to clipboard
Dongsheng Zhu, Daniel Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, Dawei Yin
| Challenge: | Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. |
| Approach: | They propose a novel approach to advancing multi-modal language models in zero-shot learning by evaluating and optimizing instructional texts through In-Context Learning. |
| Outcome: | The proposed approach improves zero-shot performance in multi-modal tasks by evaluating and optimizing instructional texts. |
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for generating 'jailbreaks' suffer from manual design or require optimization on other white-box models, which compromises either generalization or efficiency. |
| Approach: | They propose a framework that leverages LLMs to generate effective jailbreak prompts and a generalized framework that can be used to generate prompts. |
| Outcome: | The proposed framework improves the attack success rate while reducing the time cost compared to baselines. |
P3Sum: Preserving Author’s Perspective in News Summarization with Diffusion Language Models (2024.naacl-long)
Copied to clipboard
Yuhan Liu, Shangbin Feng, Xiaochuang Han, Vidhisha Balachandran, Chan Young Park, Sachin Kumar, Yulia Tsvetkov
| Challenge: | Existing summarization systems alter the political opinions and stances of news articles in more than 50% of summaries, misrepresenting the intent and perspectives of the authors. |
| Approach: | They propose a model-based summarization approach controlled by political perspective classifiers that preserves the political stance of a generated summary. |
| Outcome: | The proposed model outperforms state-of-the-art summarization systems and large language models by up to 13.7% in terms of success rate of stance preservation, with competitive performance on standard metrics of summarizing quality. |
Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes (2024.naacl-long)
Copied to clipboard
| Challenge: | Our work explores the potential of large language models (LLMs) to close the novice-expert knowledge gap in remediating math mistakes. |
| Approach: | They propose a method that uses cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation. |
| Outcome: | The proposed model can bridge the novice-expert knowledge gap by using cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation. |
RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to integrate rhetorical structure theory into long document summarization models are unexplored. |
| Approach: | They propose to integrate rhetorical structure theory into a long document summarization model by explicitly incorporating rhetorical uncertainty into the model. |
| Outcome: | The proposed models outperform the vanilla LoRA and full-parameter fine-tuning models and outperformed previous state-of-the-art methods. |
Strings from the Library of Babel: Random Sampling as a Strong Baseline for Prompt Optimisation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent prompt optimisation approaches use the generative nature of language models to produce prompts, even rivaling the performance of human-curated prompts. |
| Approach: | They propose to randomly sample tokens from the model vocabulary as "separators" they show that random separators are competitive baselines, having less than a 1% difference compared to previous self-optimisation methods. |
| Outcome: | The proposed method outperforms human-curated prompts in nine text classification tasks and eight language models and has a 40% chance of performing better than human-generated separators. |
ReTA: Recursively Thinking Ahead to Improve the Strategic Reasoning of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing logical reasoning evaluations of Large Language Models (LLMs) focus on single-turn and static environments, such as arithmetic problems. |
| Approach: | They propose a Recursively Thinking-Ahead agent that analyzes the opponents’ future moves/actions and assigns reward signals for these situations. |
| Outcome: | The proposed agent is based on two scenarios: Online Racing and Offline Probing. |
Fact Checking Beyond Training Set (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing fact checking systems are unsuitable for evaluating the veracity of everyday claims due to the availability of evidence resources. |
| Approach: | They propose an adversarial algorithm to make the retriever component robust against distribution shift. |
| Outcome: | The proposed method is insensitive to the order of claims and evidence documents. |
Program-Aided Reasoners (Better) Know What They Know (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work shows that program-aided reasoning improves accuracy but also requires reasoners to "know what they know". |
| Approach: | They compare the calibration of program-aided language models (PAL) and text-based Chain-of-thought (COT) prompting techniques over 5 datasets and 2 model types . |
| Outcome: | The proposed methods improve accuracy and calibrate the models over 5 datasets and 2 model types. |
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue . |
| Approach: | They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information. |
| Outcome: | The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research. |
Principles from Clinical Research for NLP Model Generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | In clinical research, generalizability depends on (a) internal validity of experiments and (b) external validity or transportability of the results to the wider population. |
| Approach: | They propose to ensure internal validity when building machine learning models in NLP by incorporating learning spurious correlations into their models. |
| Outcome: | The proposed model can perform well on data unseen during training, but drawn from the same distribution or population. |
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
| Approach: | They argue that disparities in scale are transient and researchers can work to reduce them . they argue that data, rather than hardware, is still a bottleneck for many applications . |
| Outcome: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit positional bias in how they use context, which affects listwise ranking. |
| Approach: | They propose a method to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias. |
| Outcome: | The proposed method improves on five datasets in sorting and passage reranking by 34-52% . it marginalizes out different list orders in the prompt to produce an order-independent ranking . |
From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved remarkable success in aligning with user intentions. |
| Approach: | They develop local and global explanation methods and a feed-forward-based method for input-output attribution to investigate the impact of instruction tuning on user intentions. |
| Outcome: | The proposed method compares explanations from pre-trained and instruction-tuned models . it empowers LLMs to recognize the instruction parts of user prompts, it encourages response generation . |
POLYIE: A Dataset of Information Extraction from Polymer Material Scientific Literature (2024.naacl-long)
Copied to clipboard
Jerry Cheung, Yuchen Zhuang, Yinghao Li, Pranav Shetty, Wantian Zhao, Sanjeev Grampurohit, Rampi Ramprasad, Chao Zhang
| Challenge: | SciIE datasets for polymer materials are lacking for this class of materials . POLYIE is curated from 146 full-length polymer scholarly articles . |
| Approach: | They propose a SciIE dataset for polymer materials that uses entity annotations from 146 full-length articles. |
| Outcome: | The proposed dataset is curated from 146 full-length polymer scholarly articles . it presents challenges due to diverse lexical formats of entities and ambiguity between entities . |
LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordination (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on learning and enhancing large language models to understand and generate natural language. |
| Approach: | They propose a computational bionic memory mechanism equipped with a parameter-efficient fine-tuning schema to personalize medical assistants. |
| Outcome: | The proposed method can enhance the response with aware of previous mistakes for new queries during a dialogue session, but the training costs are prohibitive. |
SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization are limited due to limited training data. |
| Approach: | They propose to re-use existing multilingual summarization and translation pipelines to perform cross-lingual summaries in a sequence. |
| Outcome: | The proposed approach outperforms existing methods in many languages with only 10% of the fine-tuning samples. |
KTRL+F: Knowledge-Augmented In-Document Search (2024.naacl-long)
Copied to clipboard
| Challenge: | KTRL+F is a knowledge-augmented in-document search that requires real-time identification of all semantic targets within a document with the awareness of external sources through a single natural query. |
| Approach: | They propose a knowledge-augmented in-document search that requires real-time identification of all semantic targets within a document with the awareness of external sources through a single natural query. |
| Outcome: | The proposed model reduces time for searching with less queries and reduced extra visits to other sources for collecting evidence. |
How Well Do Large Language Models Truly Ground? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing research defines “grounding” as having the correct answer, which does not ensure the reliability of the entire response. |
| Approach: | They propose a stricter definition of grounding: fully utilizes the necessary knowledge from the provided context and stays within the limits of that knowledge. |
| Outcome: | The proposed model can be ground on external contexts and maintain its correct answer. |
ALBA: Adaptive Language-Based Assessments for Mental Health (2024.naacl-long)
Copied to clipboard
| Challenge: | Adaptive language-based assessments require a substantial sample of words per person for accuracy. |
| Approach: | They propose an adaptive language-based assessment task that involves ordering questions and scoring latent psychological trait using limited language responses to previous questions. |
| Outcome: | The proposed methods improve over non-adaptive baselines, but are more accurate and scalable with fewer questions. |
FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Question Answering lack robustness . current systems generate inconsistent responses by performing coarse-grained changes in tables and questions. |
| Approach: | They formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems . they aim to evaluate TQA responses on the content of relevant cells rather than on biases . |
| Outcome: | The proposed benchmarks show that state-of-the-art TQA systems fail in three aspects . the benchmarks are released publicly and are expected to improve over time . |
MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for query expansion lack corpus-specific knowledge and cost. |
| Approach: | They propose a query-query-document generation method that leverages large language models for mutual verification to produce diverse sub-queries and corresponding documents. |
| Outcome: | The proposed method is fully zero-shot and extensive experiments on three public benchmark datasets demonstrate its effectiveness over existing methods. |
Efficient Benchmarking (of Language Models) (2024.naacl-long)
Copied to clipboard
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen
| Challenge: | Efficient Benchmarking is a method for reducing computation costs of LM evaluation without compromising reliability. |
| Approach: | They propose to reduce the computation costs of LM evaluation without compromising reliability by using a new measure - Decision Impact on Reliability. |
| Outcome: | The proposed benchmarks reduce computation costs by x100 or more, while maintaining reliability. |
ReFACT: Updating Text-to-Image Models by Editing the Text Encoder (2024.naacl-long)
Copied to clipboard
| Challenge: | Text-to-image generative models encode factual associations that can quickly become outdated, diminishing their utility for end-users. |
| Approach: | They propose a method for editing factual associations in text-to-image models without retraining or explicit input from end-users. |
| Outcome: | The proposed method improves generalization and preservation of unrelated concepts on an existing dataset and compares with other methods. |
A Likelihood Ratio Test of Genetic Relationship among Languages (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing tests of significance for bilateral comparisons are infeasible by design or yield false positives when applied to groups of languages or language families. |
| Approach: | They propose a likelihood ratio test to determine if given languages are related based on the proportion of invariant character sites in aligned wordlists. |
| Outcome: | The proposed test solves the problem of false positives on some language families. |
PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models excel in various tasks, but their huge size and inaccessibility of parameters present challenges for practical deployment. |
| Approach: | They propose to use CoT data to distill task-specific ability from large language models to smaller models . they use reasoning programs to suppress errors in distilled data and improve distillation quality . |
| Outcome: | The proposed model outperforms LLMs on arithmetic reasoning, symbolic reasoning, and general ability. |
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)
Copied to clipboard
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Several new LLMs have been introduced necessitating their evaluation on non-English languages. |
| Approach: | They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets. |
| Outcome: | The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages. |
Unlocking Emergent Modularity in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing MNNs are explicit, with predefined modular architectures and individual modules expected to implement distinct functions. |
| Approach: | They propose to unlock emergent modularity in language models by fine-tuning them as Mixture-of-Experts (MoEs) EMoE is robust to various configurations and can scale up to Large Language Models . |
| Outcome: | The proposed models can be fine-tuned as Mixture-of-Expert (MoE) counterparts without introducing any extra parameters. |
A School Student Essay Corpus for Analyzing Interactions of Argumentative Structure and Quality (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing arguments mining corpus with ground-truth quality annotations is lacking . authors propose baseline approaches to argument mining and essay scoring . |
| Approach: | They propose to use argumentative structure to support argumentative writing . they use an annotated german corpus to analyze interactions between the two tasks . |
| Outcome: | The proposed methods can be used to support argumentative writing . they analyze interactions between argumentative structure and quality annotations . |
Adjusting Interpretable Dimensions in Embedding Space with Human Judgments (2024.naacl-long)
Copied to clipboard
| Challenge: | Embedding spaces contain interpretable dimensions indicating gender, formality in style, or even object properties. |
| Approach: | They combine seed-based vectors with human ratings of where words fall along a specific dimension to evaluate on predicting object properties and stylistic properties. |
| Outcome: | The proposed model improves on seed-based vectors and human ratings on object properties and stylistic properties. |
PatentEval: Understanding Errors in Patent Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | a patent is a legal instrument that grants inventors or entities exclusive rights over their invention for a designated period. |
| Approach: | They propose a typology specifically designed for evaluating two distinct tasks in machine-generated patent texts. |
| Outcome: | The proposed approach provides valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation. |
Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated considerable success in various natural language processing tasks, but their performance in NMT tasks is still underexplored. |
| Approach: | They propose to use LLMs as automatic post-editors rather than direct translators to improve BLEU and COMET performance. |
| Outcome: | The proposed approach improves BLEU but COMET performance compared to in-context learning. |
Metaphor Detection with Context Enhancement and Curriculum Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Metaphor detection is a challenging task for natural language processing systems . previous work failed to adequately utilize internal and external semantic relationships . |
| Approach: | They propose a model that leverages the difference between literal and external meanings of words and sentences as the sentence external difference. |
| Outcome: | The proposed model achieves competitive performance across multiple datasets with improved convergence speed compared to other models. |
What Causes the Failure of Explicit to Implicit Discourse Relation Recognition? (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work claimed that explicit classifiers perform poorly in implicit scenarios . a label shift occurs after connectives are removed, but no empirical evidence supports this claim . |
| Approach: | They propose to prove that the discourse relations expressed by some explicit instances will change when connectives disappear. |
| Outcome: | The proposed methods outperform strong baselines on PDTB 2.0, PDTT 3.0, and the GUM dataset. |
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions (2024.naacl-long)
Copied to clipboard
Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Livescu, Shinji Watanabe
| Challenge: | Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models. |
| Approach: | They adapt a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers. |
| Outcome: | The proposed model can generalize to new datasets and languages for seen task types. |
How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid progress in open-source Large Language Models (LLMs) is driving AI development, but lacks sufficient trustworthiness to detect and mitigate adversarial demonstrations. |
| Approach: | They propose an extended Chain of Utterances-based (CoU) prompting strategy to attack open-source LLMs. |
| Outcome: | The proposed attack strategy is based on malicious demonstrations and toxicity tests on open-source models. |
Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Despite the impressive performance of large-scale language models, their ability to reason through complex problems remains a bottleneck. |
| Approach: | They propose a method which diversifies reasoning paths from specific surface forms of the problem to improve mathematical reasoning performance. |
| Outcome: | The proposed approach improves mathematical reasoning performance over vanilla self-consistency, especially for problems initially deemed unsolvable. |
TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced tasks like text summarization, but their size and computational demands limit their use in resource-constrained and privacy-centric settings. |
| Approach: | They propose a framework for distilling LLMs’ text summarization abilities into a compact, local model using a curriculum learning strategy that evolves from simple to complex tasks. |
| Outcome: | The proposed framework outperforms baseline models on CNN/DailyMail, XSum, and ClinicalTrial, and improves interpretability by providing insights into the summarization rationale. |
GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing relation extraction methods rely on exact matching with human-annotated reference relations, while GRE methods produce diverse and semantically accurate relations. |
| Approach: | They propose a multi-dimensional assessment of relation extraction methods using human-annotated reference relations. |
| Outcome: | The proposed method is consistent with human preferences for RE quality. |
Curated Datasets and Neural Models for Machine Translation of Informal Registers between Mayan and Spanish Vernaculars (2024.naacl-long)
Copied to clipboard
| Challenge: | a set of corpora in several Mayan languages spoken in Guatemala and Mexico is published . the languages are considered to be somewhat in decline in terms of resources and global exposure . |
| Approach: | They develop, curate, and publicly release a set of corpora in several Mayan languages spoken in Guatemala and southern Mexico, which they call MayanV. |
| Outcome: | The proposed datasets are parallel with Spanish, the dominant language of the region, and differ in register from most other available resources. |
The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work to enhance data partitioning strategies for more realistic model evaluations faces challenges in providing a clear optimal choice. |
| Approach: | They analyze morphological segmentation and morphology of ten languages from 19 languages . they use multiple datasets and splits to evaluate models . |
| Outcome: | The proposed model training and evaluation sets and new test data show that models trained from random splits can achieve higher numerical scores and model rankings tend to generalize more consistently. |
Measuring Entrainment in Spontaneous Code-switched Speech (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies of entrainment in code-switched domains have been limited to human-machine textual interactions. |
| Approach: | They propose to use acoustic-prosodic features to identify multiple dimensions and feature sets of entrainment in code-switched speech. |
| Outcome: | The findings give rise to important implications for the potentially “universal” nature of entrainment as a communication phenomenon and potential applications in inclusive and interactive speech technology. |
A Survey of Meaning Representations – From Theory to Practical Utility (2024.naacl-long)
Copied to clipboard
| Challenge: | Symbolic meaning representations of natural language text have been studied since at least the 1960s . with the availability of large annotated corpora, the field has recently seen several new developments . |
| Approach: | They propose a framework for expressing meaning in natural language text using annotated corpora and a set of tools for machine learning. |
| Outcome: | The frameworks are based on a set of theoretical and practical problems and their applications. |
Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale multilingual pretrained language models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLm. |
| Approach: | They propose to leverage the learned knowledge from well-performing languages to guide under-performing ones within the same mPLM. |
| Outcome: | The proposed model shows that it can guide under-performing languages while minimizing language-level performance disparities across different mPLMs. |
Evaluating In-Context Learning of Libraries for Code Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work shows that large proprietary LLMs can learn novel library usage in-context from demonstrations. |
| Approach: | They evaluate large proprietary LLMs to understand library usage in-context . they find they are able to generate code based on library specification presented in-constext - a promising area . |
| Outcome: | The proposed models can learn library usage in-context from demonstrations . the results pave the way for more adaptable and dynamic coding environments. |
Visually-Aware Context Modeling for News Image Captioning (2024.naacl-long)
Copied to clipboard
| Challenge: | a new framework for News Image Captioning emphasizes the connection between textual context and visual elements. |
| Approach: | They propose a face-naming module for learning better name embeddings from news images . they use CLIP to retrieve sentences that are semantically close to the image . |
| Outcome: | The proposed framework outperforms the current state-of-the-art by 7.97/5.80 CIDEr scores on GoodNews/NYTimes800k. |
Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing computational models of pragmatics are implemented as recursive reasoning procedures, in which listeners interpret utterances by reasoning about the intentions of less-sophisticated speakers. |
| Approach: | They propose a game-theoretic model of pragmatics that combines a recursive reasoning model with a regularized conventions model to obtain principled pragmatic language generation procedures. |
| Outcome: | The proposed model matches or improves predictions made by iterated best response and Rational Speech Acts models of language understanding. |
TopicGPT: A Prompt-based Topic Modeling Framework (2024.naacl-long)
Copied to clipboard
| Challenge: | TopicGPT uses large language models to uncover latent topics in text . topic models represent topics as bags of words that require "reading the tea leaves" topic models also offer limited control over formatting and specificity of topics . |
| Approach: | TopicGPT uses large language models to uncover latent topics in text . authors propose a prompt-based framework that produces topics that align better with human categorizations . |
| Outcome: | TopicGPT produces topics that align better with human categorizations compared to competing methods. |
ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger (2024.naacl-long)
Copied to clipboard
| Challenge: | Textual backdoor attacks are increasingly challenging to detect due to the use of advanced generative models such as GPT-4. |
| Approach: | They propose a framework that harnesses advanced generative models to execute stealthier backdoor attacks on text classifiers. |
| Outcome: | The proposed framework achieves state-of-the-art attack success rate of 97.35% over four sentiment classification tasks and four human cognition stealthiness tests. |
Social Meme-ing: Measuring Linguistic Variation in Memes (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we analyze memes as a form of language subject to the same kinds of sociolinguistic variation as other modalities, such as written language and speech. |
| Approach: | They propose a computational pipeline to cluster memes into templates and semantic variables, taking advantage of their multimodal structure to learn meme semantics from an unstructured dataset. |
| Outcome: | The proposed method uses 3.8M images from a reddit meme database to analyze linguistic variation in memes. |
ExpertQA: Expert-Curated Questions and Attributed Answers (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study examined the attribution and factuality of language models in domains . experts from various fields are using large language models for information-seeking scenarios . |
| Approach: | They evaluate language models' attribution and factuality by bringing domain experts in the loop . they collect expert-curated questions from 484 participants across 32 fields of study . |
| Outcome: | The results show that language models can provide factually correct answers in high-stakes fields, but they can also be harmful to experts. |
What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception (2024.naacl-long)
Copied to clipboard
| Challenge: | Question answering models can often be black boxes, as their reasoning process is mostly opaque. |
| Approach: | They analyze the effect of rationales generated by QA models on user feedback and how well they enable users to understand and trust model answers. |
| Outcome: | The proposed model can be used to improve model responses by removing feedback from end users and enhancing model outputs by using natural language feedback. |
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue models are primarily trained on human-human conversations . thumb ups/downs and gold corrections are often sparse in real-life deployment settings . |
| Approach: | They propose a framework to make use of binary and free-form textual human feedback. |
| Outcome: | The proposed framework improves the final dialogue model by using model-corrected replies. |
Kreyòl-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages (2024.naacl-long)
Copied to clipboard
Nathaniel Robinson, Raj Dabre, Ammon Shurtz, Rasul Dent, Onenamiyi Onesi, Claire Monroc, Loïc Grobol, Hasan Muhammad, Ashi Garg, Naome Etori, Vijay Murari Tiyyala, Olanrewaju Samuel, Matthew Stutzman, Bismarck Odoom, Sanjeev Khudanpur, Stephen Richardson, Kenton Murray
| Challenge: | Creole languages are used in much of Latin America, Africa and the Caribbean . a large multilingual bitext like ours has potential to build the best yet or first ever MT models for many languages . |
| Approach: | They present the largest cumulative dataset to date for Creole language MT . they provide MT models supporting all 41 Creoles in 172 translation directions . |
| Outcome: | The proposed model outperforms a genre-specific Creole MT model on its own benchmark for 23 of 34 translation directions. |
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | et al., 2021) show that instruction models can be trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Approach: | They examine security concerns of emergent instruction tuning paradigm that models are trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Outcome: | The proposed model can achieve 90% success rate across four commonly used datasets. |
Modeling Empathetic Alignment in Conversation (2024.naacl-long)
Copied to clipboard
| Challenge: | Empathy requires perspective-taking and is not explicitly modelled in NLP . |
| Approach: | They propose a new approach to recognizing alignment in empathetic speech, grounded in Appraisal Theory, and use reddit to study emotional conversations to examine alignment. |
| Outcome: | The proposed approach can recognize appraisals and alignments in empathetic speech, and mental health professionals engage with substantially more emotional alignment. |
Native Language Identification in Texts: A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Native language identification is the task of automatically identifying an author’s native language (L1) based on their second language production. |
| Approach: | They present a survey of native language identification applied to texts . authors describe several text representations and computational techniques used in the task . |
| Outcome: | The proposed task has been widely studied for both text and speech, particularly for L2 English due to the availability of suitable corpora. |
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). |
| Approach: | They propose a parameter-efficient framework that reduces trainable parameters through tensor-train decomposition. |
| Outcome: | The proposed methods achieve comparable or better performance than most widely used methods with up to 100 fewer parameters on the LLaMA-2-7B models. |
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for identifying object referents of language expressions consider target and distractor objects independently and pool multiple views before grounding. |
| Approach: | They propose a model that selects an object referent based on language that distinguishes between two similar objects and a multi-view approach to grounding in context model which reduces the relative error by 12.9% . |
| Outcome: | The proposed model improves on the SNARE object reference task with a relative error reduction of 12.9% and an absolute improvement of 2.7%. |
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of localization methods to pinpoint LLM components for memorized data are lacking. |
| Approach: | They propose to use a subset of LLM weights to evaluate localization methods . they propose to measure how much dropping out identified neurons deletes a memorized sequence. |
| Outcome: | The proposed methods show promising localization ability, despite differences in their evaluations. |
PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models can be backdoored such that model behavior is manipulated when trigger tokens are presented. |
| Approach: | They propose a backdoor mitigation strategy for NLP models via adversarial prompt-tuning in few-shot settings that uses two extra sets of soft tokens which approximate the trigger and counteract it respectively. |
| Outcome: | The proposed method keeps model parameters intact and only utilizes two extra sets of soft tokens which approximate the trigger and counteract it respectively. |
Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Previous studies have investigated how different factors affect faithfulness of model explanations . |
| Approach: | They find that the larger the multilingual model, the less faithful FAs are compared to its counterpart monolingual models. |
| Outcome: | The results show that the larger the multilingual model, the less faithful the FAs are compared to its counterpart monolingual models. |
A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (2024.naacl-long)
Copied to clipboard
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito
| Challenge: | a large number of pretraining data design practices are under-documented, authors say . authors: strong performance of modern language models depends on selfsupervised pretraining . |
| Approach: | They propose to pretrain models on data curated at different collection times . they find temporal shift between evaluation data and pretraining data leads to performance degradation . |
| Outcome: | The results validate, quantify, and expose many undocumented intuitions about text pretraining . authors say this practice has outperformed other models in the field . |
Instructional Fingerprinting of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) require considerable cost to train from scratch . fingerprinting is essential to protect intellectual property and to ensure downstream users and developers adhere to their license terms. |
| Approach: | They propose a method for fingerprinting Large language models that implants a private key into the model to generate specific text when the key is present. |
| Outcome: | The proposed method prevents publisher overclaim and maintains robustness against fingerprint guessing and parameter-efficient training. |
Reinforced Multiple Instance Selection for Speaker Attribute Prediction (2024.naacl-long)
Copied to clipboard
Alireza Salkhordeh Ziabari, Ali Omrani, Parsa Hejabi, Preni Golazizian, Brendan Kennedy, Payam Piray, Morteza Dehghani
| Challenge: | Current methods for predicting speaker attributes take a speaker’s utterances as input and provide a prediction per speaker attribute. |
| Approach: | They propose a Multiple Instance Learning approach that uses Reinforcement Learning to predict speaker attributes using a set of utterances from social media posts and political ideologies from transcribed speeches. |
| Outcome: | The proposed approach outperforms existing methods on a range of related tasks including predicting speakers’ psychographics and demographics from social media posts and political ideologies from transcribed speeches. |
DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid explosion in model sizes has resulted in high inference times . open-source LLMs are democratizing research in natural language processing . |
| Approach: | They propose a suite of multi-token prediction language models that reduce net inference times by leveraging traditional autoregressive weights. |
| Outcome: | The proposed model achieves same-quality generated text as baseline (Pythia-6.9B) with only 5.87% and 2.67% parameter and training time overheads. |
Few-shot Knowledge Graph Relational Reasoning via Subgraph Adaptation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to predict unseen triplets from knowledge graphs are limited by spurious information in KGs. |
| Approach: | They propose a framework that adapts contextualized graphs to subgraphs generated from support and query triplets to perform the prediction. |
| Outcome: | The proposed framework extracts more comprehensive information from support triplets while minimizing spurious information when predicting query triplet. |
Uncertainty Quantification for In-Context Learning of Large Language Models (2024.naacl-long)
Copied to clipboard
Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen
| Challenge: | Existing studies on in-context learning have focused on quantifying the uncertainty associated with the model's response, but they neglect the complexity of the LLM and the uniqueness of in-constitut learning. |
| Approach: | They propose a method to quantify the uncertainty associated with in-context learning and propose corresponding estimation method to quantify both types of uncertainties. |
| Outcome: | The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. |
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM (2024.naacl-long)
Copied to clipboard
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, Oleksii Kuchaiev
| Challenge: | Existing helpfulness preference datasets do not specify what makes some responses more helpful and others less helpful. |
| Approach: | They use a dataset that has annotated for correctness, coherence, complexity, and verbosity. |
| Outcome: | The dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. |
A Preference-driven Paradigm for Enhanced Translation with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent research shows that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) however, SFT simply instructs the model to imitate reference translations token by token, making it vulnerable to the noise present in the data. |
| Approach: | They propose a preference-based approach to supervised fine-tuning that trains the model to imitate reference translations token by token, making it vulnerable to noise. |
| Outcome: | The proposed approach overcomes the plateau associated with imitation-based SFT and is more resilient in the absence of gold translations. |
Fair Abstractive Summarization of Diverse Perspectives (2024.naacl-long)
Copied to clipboard
Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang
| Challenge: | Existing work on summarization metrics and large language models has not explored fair abstractive summarizing. |
| Approach: | They propose four reference-free automatic metrics to measure the differences between target and source perspectives. |
| Outcome: | The proposed methods alleviate fair abstractive summarization on user-generated data. |
What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases (2024.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models have broad competence that is difficult to evaluate . current evaluation benchmarks focus on only assessing one or a few capabilities . |
| Approach: | They perform a large-scale transfer learning experiment to discover latent VL skills from data. |
| Outcome: | The results suggest that factor analysis can identify reasonable yet surprising VL skill factors . the results contribute to the design of balanced and broad-coverage vision-language evaluation methods. |
Show Your Work with Confidence: Confidence Bands for Tuning Curves (2024.naacl-long)
Copied to clipboard
| Challenge: | a rush to scale up has left us with large, costly language models and little understanding of how different designs compare. |
| Approach: | They propose a method to construct valid confidence bands for tuning curves . they validated their method with ablations and analyze the effect of sample size . |
| Outcome: | The proposed method shows that bootstrap confidence bands do not approximate their target confidence. |
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives (2024.naacl-long)
Copied to clipboard
Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, Gregory Serapio-García
| Challenge: | Recent work shows that ignoring rater subjectivity is problematic within specific tasks and for specific subgroups. |
| Approach: | They propose a disagreement analysis framework to measure group association in perspectives among different rater subgroups. |
| Outcome: | The proposed framework reveals specific rater groups that have significantly different perspectives than others on certain tasks and helps identify demographic axes that are crucial to consider in specific task contexts. |
Event Causality Is Key to Computational Story Understanding (2024.naacl-long)
Copied to clipboard
| Challenge: | Cognitive science and symbolic AI research suggest that event causality provides vital information for story understanding. |
| Approach: | They propose a method for event causality identification that leads to material improvements in story understanding. |
| Outcome: | The proposed method improves story understanding on the COPES dataset . it achieves 4.1-10.9% increase on Clip Accuracy and 4.2-13.5% increase on Sentence IoU . |
Subspace Representations for Soft Set Operations and Sentence Similarities (2024.naacl-long)
Copied to clipboard
| Challenge: | Embedding-based word representations are crucial for capturing the semantic meanings of individual words. |
| Approach: | They propose to embed word sets and corresponding set operations within pre-trained word embedding spaces. |
| Outcome: | The proposed representations outperform vector-based representations in sentence similarity and set retrieval tasks. |
My Heart Skipped a Beat! Recognizing Expressions of Embodied Emotion in Natural Language (2024.naacl-long)
Copied to clipboard
| Challenge: | a new task is needed to recognize physical manifestations of emotions in natural language . physical manifestation of emotions affects not only our mental state but also our physical state . |
| Approach: | They propose a task to recognize expressions of embodied emotion in natural language . they use body part mentions with human annotations to extract emotional manner expressions . |
| Outcome: | The proposed model can train without gold data and improve performance with gold data. |
Low-Cost Generation and Evaluation of Dictionary Example Sentences (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior studies have shown that language models can be trained to generate example sentences, but they relied on costly customized models and word sense datasets for generation and evaluation. |
| Approach: | They propose a new automatic evaluation metric called OxfordEval that measures the win-rate of generated sentences against existing Oxford Dictionary sentences. |
| Outcome: | The proposed model achieves over 85.1% win rate against baseline sentences compared to 39.8% win rate for prior model-generated sentences. |
Making Language Models Better Tool Learners with Execution Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing tool learning methodologies induce large language models to utilize tools indiscriminately . Existing frameworks that teach language models when and how to use tools propagate errors rather than enhance performance. |
| Approach: | They propose a framework that enables large language models to continually learn through feedback derived from tool execution. |
| Outcome: | The proposed framework can make large language models selectively use tools . it improves accuracy while enhancing insufficient tool learning, it shows . |
Complex Claim Verification with Evidence Retrieved in the Wild (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work makes simplifying assumptions in retrieval that depart from real-world use cases: no access to evidence, access to curated evidence, or access to published evidence after a claim was made. |
| Approach: | They propose a pipeline to check claims using raw evidence from the web . they restrict their retriever to only search documents available prior to the claim's making . |
| Outcome: | The proposed method is based on a political claim dataset and shows that the evidence summary produced by the system is reliable and relevant to answering key questions. |
Multimodal Multi-loss Fusion Network for Sentiment Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | This paper examines the optimal selection and fusion of feature encoders across multiple modalities and combines them in one neural network to improve sentiment detection. |
| Approach: | They propose to combine feature encoders across multiple modalities into one neural network to improve sentiment detection. |
| Outcome: | The proposed model achieves state-of-the-art performance for three datasets . it also shows that integrating context significantly improves model performance. |
Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest using large language models to make tabular classifications . however, LLMs have been shown to exhibit harmful social biases based on stereotypes and inequalities present in society. |
| Approach: | They propose to use large language models to make tabular classifications . they show that LLMs inherit biases from their training data . |
| Outcome: | The proposed models exhibit harmful biases that reflect stereotypes and inequalities in society. |
Analyzing the Use of Metaphors in News Editorials for Political Framing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the use of metaphors in political discourses are largely unexplored. |
| Approach: | They propose to use a dataset to study the use of metaphors in political discourses . they identify single and composite metaphors and provide annotations of their source and target domains based on the corpus . |
| Outcome: | The proposed dataset consists of 300 news editorials with single and composite metaphors and annotations of the source and target domains for each metaphor. |
SharpSeq: Empowering Continual Event Detection through Sharpness-Aware Sequential-task Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for continual event detection suffer from catastrophic forgetting . a novel continual learning paradigm leveraging sharpness-aware minimization is needed . |
| Approach: | They propose a continual learning paradigm that leverages sharpness-aware minimization and a generative model to balance training data distribution. |
| Outcome: | The proposed approach outperforms existing methods on real-world datasets. |
Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained language models contain various kinds of knowledge. |
| Approach: | They designed a probe that allows comparison of 34 million distinct paraphrases that follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. |
| Outcome: | Extensive knowledge retrieval experiments show that prompts following clausal syntax have several desirable properties in comparison to appositive syntax. |
Know When To Stop: A Study of Semantic Drift in Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for text generation make predictions token-by-token, without pre-established text structure. |
| Approach: | They develop a semantic drift score that measures the degree of separation between correct and incorrect facts in generated texts. |
| Outcome: | The proposed methods improve factuality by a large margin compared to baseline and early stopping methods. |
Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024.naacl-long)
Copied to clipboard
| Challenge: | masked language modeling is widely used as a pretraining component in Vision and language (V+L) but performance on benchmarks has not received the attention it deserves. |
| Approach: | They propose a curriculum masking scheme that uses a parallel mask selection agent to mask tokens at a frequency proportional to the level of cross modal interaction necessary to reconstruct them. |
| Outcome: | The proposed method improves relational understanding on a wide range of V+L tasks. |
Elote, Choclo and Mazorca: on the Varieties of Spanish (2024.naacl-long)
Copied to clipboard
| Challenge: | Spanish is the official language in 20 countries and the second most-spoken native language . available corpora treat it as one monolithic language, damping prediction power . |
| Approach: | They compile and curate datasets in different varieties of Spanish around the world at an unprecedented scale and create the CEREAL corpus. |
| Outcome: | The results show that Spanish is a multilingual language with a wide range of cultural and cultural influences. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
A Zero-Shot Monolingual Dual Stage Information Retrieval System for Spanish Biomedical Systematic Literature Reviews (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that most SRs are skewed towards English databases, excluding databases in Languages other than English (LoE). |
| Approach: | They propose a zero-shot dual information retrieval baseline system that integrates traditional retrieval methods with pre-trained language models and cross-attention re-rankers for enhanced accuracy in Spanish biomedical literature retrieval. |
| Outcome: | The proposed system improves on three real-life case studies in Spanish biomedical literature retrieval using the LILACS database, which is known for its coverage of Latin American and Caribbean biomedically literature. |
LayoutPointer: A Spatial-Context Adaptive Pointer Network for Visual Information Extraction (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models inadequately utilize spatial information of entities, causing incorrectly linking spatially distant entities. |
| Approach: | They propose a Spatial-Context Adaptive Pointer Network to restore semantic order among entities . they propose XFUND-based tail-to-head pointer to restore the semantic order . |
| Outcome: | The proposed method outperforms existing state-of-the-art methods in F1 scores for RE tasks. |
Long-form evaluation of model editing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of model editing only use the ‘next few tokens’ completions after a prompt. |
| Approach: | They propose a new evaluation protocol that measures the efficacy and impact of model editing in long-form generative settings by using a machine-rated survey and a classifier which correlates well with human ratings. |
| Outcome: | The proposed evaluation protocol has little relationship with short-form metrics despite being designed to extend efficacy, generalization, locality, and portability into a long-form setting. |
Analyzing the Role of Semantic Representations in the Era of Large Language Models (2024.naacl-long)
Copied to clipboard
Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, Mona Diab
| Challenge: | Existing studies show the benefits of semantic representations in NLP tasks . Existing work using AMR is concerned with trainable models . |
| Approach: | They propose an AMR-driven chain-of-thought prompting method that uses AMR . they propose to use it to predict which input examples AMR may help or hurt on . |
| Outcome: | The proposed method hurts performance more than it helps on five different tasks. |
TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate incorrect responses based on made-up facts, which are called hallucinations. |
| Approach: | They propose a framework that combines Retrieval Augmented Generation with conformal prediction to provide the first end-to-end statistical correctness guarantee for RAG. |
| Outcome: | The proposed framework reduces prediction set size by 16.2% on average compared to an ablation. |
MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities (2024.naacl-long)
Copied to clipboard
| Challenge: | Decoding continuous language from brain activity is a formidable but promising field of research . previous attempts to map brain activity to text relied on learning to encode brain activity . |
| Approach: | They propose a method that maps brain activity to text embeddings by directly comparing them with predicted brain responses. |
| Outcome: | The proposed method outperforms the current state-of-the-art model showing improvements on BLEU and METEOR scores. |
On-the-fly Definition Augmentation of LLMs for Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Despite their general capabilities, LLMs struggle on biomedicalNER tasks due to specialized terminology and lack of training data. |
| Approach: | They propose a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. |
| Outcome: | The proposed approach improves performance on biomedicalNER tasks by 15% (on average) The proposed method outperforms fine-tuned language models in few-shot settings. |
This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes (2024.naacl-long)
Copied to clipboard
| Challenge: | Pretrained large language models may answer differently in different languages . this contrasts with a multilingual human, who would likely answer consistently . |
| Approach: | They propose a dataset of territorial disputes which includes multiple-choice questions in 49 languages . they propose metrics to quantify bias and consistency in responses across different languages based on their data . |
| Outcome: | The proposed model recalls certain knowledge inconsistently when asked in different languages. |
Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for constructing event temporal graphs have been suboptimal . authors propose a set-aligning framework for the effective utilisation of Large Language Models . |
| Approach: | They propose a set-aligning framework for the effective utilisation of Large Language Models to alleviate text generation loss penalties. |
| Outcome: | The proposed framework surpasses existing baselines for event temporal graph generation. |
LanguageFlow: Advancing Diffusion Language Generation with Probabilistic Flows (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work has demonstrated success in controlling sentence attributes and structure based on diffusion language models. |
| Approach: | They propose a language-rectified flow method that reformulates standard probabilistic flow models to learn ordinary differential equations to transport between the source and target distributions. |
| Outcome: | The proposed method outperforms baselines on three fine-grained control tasks and multiple high-quality text editing tasks. |
Towards Improved Multi-Source Attribution for Long-Form Answer Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Current LLMs struggle with attribution for long-form answers which require reasoning over multiple evidence sources. |
| Approach: | They propose to improve attribution capability of large language models for long-form answer generation to multiple sources with multiple citations per sentence. |
| Outcome: | The proposed model improves on a wide range of attribution benchmark datasets on PolitiICite, a multi-source attribution dataset based on PolitIcite articles . |
Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | DP-training methods that require per-example gradients are difficult to implement . however, a method that prioritizes query privacy is not feasible. |
| Approach: | They propose a method that prioritizes ensuring query privacy prior to training a deep retrieval system. |
| Outcome: | The proposed method shows that it improves retrieval quality compared to direct DP-training while maintaining query-level privacy guarantees. |
Okay, Let’s Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work shows that generative large language models (LLMs) can be used to solve cross-document coreference problems. |
| Approach: | They propose rationale-oriented event clustering and knowledge distillation methods for event coreference scoring that leverage enriched information from the FTRs for improved CDCR. |
| Outcome: | The proposed model achieves SOTA B3 F1 on the ECB+ and GVC corpora without additional annotation or expensive document clustering. |
Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Increasing the use of knowledge graphs to augment LLMs has led to hallucinations . large language models (LLMs) are prone to producing hallucinosis due to knowledge gaps . |
| Approach: | They review knowledge graph-based augmentation techniques in large language models to assess their effectiveness and examine their performance. |
| Outcome: | The proposed methods have been evaluated against three groups of LLMs and offer methodological comparisons and performance evaluations. |
Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study has focused on the cloze training objective of Masked Language Models . distractors must be distinct and incorrect, and can be biased if the test creator is testing two versions of a text . |
| Approach: | They propose a method that jointly optimizes sets of distractors from Masked Language Models. |
| Outcome: | The proposed method has stronger correlation with teacher-created comprehension tests than state-of-the-art neural method and is more internally consistent. |
Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances of Large Language Models (LLMs) have been pushing the field of Natural Language Processing (NLP) to the next level in many different aspects. |
| Approach: | They propose a novel labeling method which estimates how much incremental knowledge is brought into LLMs by a demonstration. |
| Outcome: | The proposed method estimates how much incremental knowledge is brought into the LLMs by a demonstration. |
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures. |
| Approach: | They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. |
| Outcome: | The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model. |
CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistants (2024.naacl-long)
Copied to clipboard
| Challenge: | A major challenge in deploying LLM-based virtual conversational assistants in real world settings is ensuring they operate within what is admissible for the task. |
| Approach: | They propose to use large language models (LLMs) to generate training data with two key LLM components: scenario-augmented generation and contrastive training examples. |
| Outcome: | The proposed model improves over baselines in multiple dialogue domains. |
Advancing Beyond Identification: Multi-bit Watermark for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to detect machine-generated text focus on detection, but some misuses require tracing the adversary user for counteracting them. |
| Approach: | They propose a method for embedding traceable multi-bit information during language model generation. |
| Outcome: | The proposed method outperforms existing methods in terms of robustness and latency while maintaining text quality. |
HTCCN: Temporal Causal Convolutional Networks with Hawkes Process for Extrapolation Reasoning in Temporal Knowledge Graphs (2024.naacl-long)
Copied to clipboard
| Challenge: | Temporal knowledge graphs (TKGs) are powerful tools for storing and modeling dynamic facts. |
| Approach: | They propose a Hawkes process-based temporal causal convolutional network for temporal reasoning under extrapolation settings. |
| Outcome: | The proposed network is based on Hawkes process-based temporal causal convolutional network and captures the temporal evolution of facts. |
SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation (2024.naacl-long)
Copied to clipboard
Abe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, Yulia Tsvetkov
| Challenge: | Existing watermarked generation algorithms employ token-level designs and are vulnerable to paraphrase attacks. |
| Approach: | They propose a sentence-level watermarking algorithm that uses locality-sensitive hashing to partition the semantic space of sentences. |
| Outcome: | The proposed algorithm is more robust than the existing state-of-the-art method on paraphrasers and domains, while posing only minor degradations to SemStamp. |
Media Bias Detection Across Families of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional NLP models have shown good performance in classifying media bias, but require careful model design and extensive tuning. |
| Approach: | They ask how well prompting of large language models can recognize media bias. |
| Outcome: | The prompt-based models deliver comparable performance to traditional models with greatly reduced effort and the availability of context substantially improves results. |
Better Zero-Shot Reasoning with Role-Play Prompting (2024.naacl-long)
Copied to clipboard
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, Xiaohang Dong
| Challenge: | Recent years have witnessed a paradigm shift in natural language processing, driven by large language models such as GPT-3, PaLM, and Llama. |
| Approach: | They propose a strategy for role-play prompting and assess its performance under the zero-shot setting. |
| Outcome: | The proposed method outperforms the standard zero-shot prompting approach across 12 reasoning benchmarks. |
Event-Content-Oriented Dialogue Generation in Short Video (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing multi-modal dialogue models are limited to incapacity of reading visual information and multi-dimensional interactions. |
| Approach: | They propose a novel event-oriented video-dialogue dataset called SportsVD to overcome these challenges by generating human-like response according to event contents in the video and related external knowledge. |
| Outcome: | The proposed method outperforms existing methods on SportsVD and other baselines under several automatic metrics. |
DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to collect high-quality instruction-response pairs suffer from unaffordable labor costs or severe hallucinations in the self-generation of LLMs. |
| Approach: | They propose a method that trains LLMs to generate instruction-response pairs based on human-written documents rather than relying solely on self-generation without context. |
| Outcome: | The proposed method outperforms existing typical methods on multiple benchmarks and shows that it is 100% scalable. |
Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | a study explores the cross-jurisdictional generalizability of legal case summarization models . fine-tuning on non-target datasets outperforms unsupervised methods, but success depends on similarity between source and target jurisdictions. |
| Approach: | They explore how to effectively summarize legal cases of a target jurisdiction where reference summaries are not available. |
| Outcome: | The proposed model can be generalized across jurisdictions and improve transfer performance. |
EDC: Effective and Efficient Dialog Comprehension For Dialog State Tracking (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for dialog state tracking face trade-offs between accuracy and efficiency . effective and efficient dialog comprehension (EDC) predicts domains, slot names and slot values of dialog state step-by-step for better accuracy . |
| Approach: | They propose an alternative method that leverages the tree structure of the dialog state. |
| Outcome: | The proposed approach achieves state-of-the-art JGA accuracy and is more efficient than previous models. |
Automatic Restoration of Diacritics for Speech Data Sets (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing text-based diacritic restoration models have high diacritical error rates when applied to speech data . a recent study shows that the lack of diacritized text can cause poor performance for text restoration models. |
| Approach: | They propose to use Arabic scripts as input for automatic diacritic restoration models . they use a pre-trained model to produce rough diacritized Arabic transcripts for the model . |
| Outcome: | The proposed framework consistently improves diacritic restoration performance compared to baseline models. |
XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)
Copied to clipboard
| Challenge: | XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages. |
| Approach: | They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches. |
| Outcome: | The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches. |
MDR: Model-Specific Demonstration Retrieval at Inference Time for In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-based in-context learning ignore model biases and fail to retrieve the most appropriate demonstrations for different LLMs. |
| Approach: | They propose a model-specific demonstration retrieval method that considers the biases of different LLMs at inference time. |
| Outcome: | The proposed method improves performance on seen and unseen tasks with multi-scale inference LLMs by up to 41.2%. |
Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets for hate speech detection neglect the cultural diversity within a single language. |
| Approach: | They propose a CR**oss-cultural **E**nglish **Hate* speech dataset that uses culturally hateful keywords to identify posts from four countries plus the United States. |
| Outcome: | The proposed dataset shows that only 56.2% of the posts in CREHate achieve consensus among all countries, with the highest pairwise label difference rate of 26%. |
Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models lack contextual knowledge, resulting in text with factual inconsistencies or contextually unfaithful content. |
| Approach: | They propose a method that integrates contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. |
| Outcome: | The proposed method improves context grounding during generation without training. |
Generalizable Sarcasm Detection is Just Around the Corner, of Course! (2024.naacl-long)
Copied to clipboard
| Challenge: | sarcasm can be used to hurt, criticize, or deride but also to be mocking, humorous, or to bond. |
| Approach: | They tested the robustness of sarcasm detection models by fine-tuning their behavior on four sarkasmatic datasets . they found that models performed better when fine- tuned with third-party labels than with author labels. |
| Outcome: | The proposed models performed better when fine-tuned with third-party labels than with author labels on the same dataset and across different datasets. |
Encoding of lexical tone in self-supervised models of spoken language (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing research on representations of phonetic and phonological information has focused on segmental features such as phonemes. |
| Approach: | They propose to analyze the tone encoding capabilities of self-supervised Spoken Language Models, using Mandarin and Vietnamese as case studies. |
| Outcome: | The proposed models encode lexical tone even when trained on non-tonal languages. |
A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change (2024.naacl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are the preferred tool for modeling Lexical Semantic Change (LSC) current evaluations focus on a specific task known as Graded Change Detection (GCD) however, performance comparisons between different approaches are often misleading due to diverse settings. |
| Approach: | They evaluate the performance of contextualized embeddings for Lexical Semantic Change (LSC) they break the problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks . |
| Outcome: | The proposed model outperforms other models on eight available benchmarks for Lexical Semantic Change (LSC) while comparable to GPT-4. |
iACOS: Advancing Implicit Sentiment Extraction with Informative and Adaptive Negative Examples (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for extracting aspects and opinions from text are incomplete. |
| Approach: | They propose a method for extracting Implicit Aspects with Categories and Opinions with Sentiments using implicit tokens. |
| Outcome: | The proposed method outperforms baseline methods on two public benchmark datasets. |
Rectifying Demonstration Shortcut in In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve tasks with a few demonstrations, but often rely on their pre-trained semantic priors rather than the input-label relationships to proceed with ICL prediction. |
| Approach: | They propose a demonstration-aware calibration method to improve LLMs' ability to learn new input-label relationships from demonstrations. |
| Outcome: | The proposed method improves the original ICL task and the task learning setting, and the results are generalized across three LLM families. |
Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark (2024.naacl-long)
Copied to clipboard
Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, Börje Karlsson, Peiqin Lin, Nikola Ljubešić, Lester James Miranda, Barbara Plank, Arij Riabi, Yuval Pinter
| Challenge: | In named entity recognition, the majority of annotation efforts are centered on English, and cross-lingual transfer performance remains brittle. |
| Approach: | They propose to develop gold-standard named entity recognition benchmarks in many languages using a cross-lingual consistent schema. |
| Outcome: | The proposed benchmarks will be released to the public in 2022 . they will provide baselines on in-language and cross-lingual learning settings. |
ODD: A Benchmark Dataset for the Natural Language Processing Based Opioid Related Aberrant Behavior Detection (2024.naacl-long)
Copied to clipboard
Sunjae Kwon, Xun Wang, Weisong Liu, Emily Druhl, Minhee Sung, Joel Reisman, Wenjun Li, Robert Kerns, William Becker, Hong Yu
| Challenge: | Opioid related aberrant behaviors (ORABs) present novel risk factors for opioid overdose. |
| Approach: | They propose to use a biomedical natural language processing benchmark dataset to classify ORABs from patients’ EHR notes into nine categories: confirmed aberrant behavior, suggested aberrant behaviors, Opioids, indication, diagnosed opioid dependency, Benzodiazepines, medication changes, and Central Nervous System-related. |
| Outcome: | The proposed dataset outperforms two state-of-the-art models in most categories and the gains are especially higher among uncommon classes. |
A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Chemical named entity recognition (NER) models are used in many downstream tasks, but it is unknown whether they work the same for everyone. |
| Approach: | They develop a framework for measuring gender bias in chemical NER models . they analyze a corpus of 92,405 words with self-identified gender information from reddit . |
| Outcome: | The proposed framework measures gender bias in chemical NER models using synthetic data and a newly annotated corpus of over 92,405 words with self-identified gender information from Reddit. |
The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to assess instruction quality require trained raters to observe classrooms based on established criteria. |
| Approach: | They propose to use Natural Language Processing techniques to assess multiple high-inference instructional practices in in-person K-12 classrooms and simulated performance tasks for pre-service teachers. |
| Outcome: | The proposed method is able to assess multiple high-inference instructional practices in two educational settings: in-person K-12 classrooms and simulated performance tasks for pre-service teachers. |
Differentially Private Next-Token Prediction of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming increasingly important for ensuring privacy, but DP-SGD overestimates an adversary’s capabilities in having white box access. |
| Approach: | They propose a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-toning and a public model to achieve Differential Privacy. |
| Outcome: | The proposed protocol outperforms DP-SGD and DP training methods for privacy on large datasets. |
Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset (2024.naacl-long)
Copied to clipboard
| Challenge: | Hate speech detection models are only as good as the data they are trained on, but adversarial datasets are slow and costly . data sourced from social media suffer from systematic gaps and biases, leading to unreliable models with simplistic decision boundaries. |
| Approach: | They propose a German Adversarial Hate speech Dataset comprising 11k examples . they explore new strategies for supporting annotators and provide manual analysis of disagreements for each strategy . |
| Outcome: | The proposed dataset is challenging even for state-of-the-art hate speech detection models and it significantly improves model robustness. |
Memory Augmented Language Models through Mixture of Word Experts (2024.naacl-long)
Copied to clipboard
| Challenge: | Increasing the parameter count of language models has been a primary driver of improved model quality, but increasing the model size also increases the cost of training and serving the model. |
| Approach: | They propose to decouple learning capacity and FLOPs by using a mixture-of-experts approach with large knowledge-rich vocabulary based routing functions. |
| Outcome: | The proposed model outperforms the T5 family of models with similar number of FLOPs on knowledge intensive tasks and similar performance to memory augmented approaches. |
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2024.naacl-long)
Copied to clipboard
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi
| Challenge: | Impossible Distillation is a framework for paraphrasing and sentence summarization that can be trained from a low-quality teacher model. |
| Approach: | They propose a framework that distills a high-quality dataset from a low-quality teacher . they hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs . |
| Outcome: | The proposed framework outperforms baseline models on unconstrained paraphrase generation and sentence summarization benchmarks. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
MOKA: Moral Knowledge Augmentation for Moral Event Extraction (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for discerning moral values are limited due to lack of context, lack of moral reasoning capabilities and complexity of moral stances. |
| Approach: | They propose a framework for moral event extraction using moral words and moral scenarios. |
| Outcome: | The proposed framework outperforms baselines across three moral event understanding tasks. |
Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing fine-tuning of large high-resource language models into multilingual machine translators is difficult for extremely lowresource languages. |
| Approach: | They propose to fine-tune large high-resource language models into multilingual machine translators for extremely-lowresource languages such as endangered Indigenous languages. |
| Outcome: | The proposed model halls are reformulated to improve translation accuracy and improve translation quality. |
Backdoor Attacks on Multilingual Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that multilingual machine translation systems are vulnerable to backdoor attacks through data poisoning. |
| Approach: | They propose to investigate the security of multilingual machine translation systems by exposing poisoned data into low-resource languages to cause malicious translations. |
| Outcome: | The proposed method achieves an average of 20% success rate in attacking high-resource languages. |
Personalized Jargon Identification for Enhanced Interdisciplinary Communication (2024.naacl-long)
Copied to clipboard
| Challenge: | Identifying and translating scientific jargon for individual researchers could speed up research, but current methods of jaron identification rely on corpus-level familiarity indicators rather than modeling researcher-specific needs. |
| Approach: | They collect over 10K term familiarity annotations from 11 computer science researchers and investigate supervised and prompt-based methods to predict individual jargon familiarity. |
| Outcome: | The proposed method improves jargon familiarity prediction by using domain, subdomain, and individual knowledge. |
Flames: Benchmarking Value Alignment of LLMs in Chinese (2024.naacl-long)
Copied to clipboard
Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, Dahua Lin
| Challenge: | Existing benchmarks for large language models (LLMs) do not accurately uncover safety vulnerabilities in LLMs. |
| Approach: | They propose a value alignment benchmark called Flames that encompasses both harmlessness principles and a unique morality dimension that integrates specific Chinese values such as harmony. |
| Outcome: | The proposed model performs poorly on Flames, particularly in safety and fairness dimensions. |
Mitigating Bias for Question Answering Models by Tracking Bias Influence (2024.naacl-long)
Copied to clipboard
Mingyu Ma, Jiun-Yu Kao, Arpit Gupta, Yu-Hsiang Lin, Wenbo Zhao, Tagyoung Chung, Wei Wang, Kai-Wei Chang, Nanyun Peng
| Challenge: | Existing literature observes bias in question answering (QA) models, but there is no method to mitigate it. |
| Approach: | They propose an approach to mitigate the bias of question answering models by observing the influence of a query instance on another instance. |
| Outcome: | The proposed method reduces bias level in all 9 bias categories while maintaining comparable QA accuracy. |
Extending CLIP’s Image-Text Alignment to Referring Image Segmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. |
| Approach: | They propose a framework that leverages the cross-modal nature of CLIP for RIS by leveraging image-text alignment knowledge in CLIP's image-embedding space. |
| Outcome: | The proposed framework outperforms CLIP-based methods on all three major RIS benchmarks and outperformed previous CLIP methods. |
Generating Attractive and Authentic Copywriting from Customer Reviews (2024.naacl-long)
Copied to clipboard
| Challenge: | Typical approaches to copywriting focus on product attributes, leading to dull and repetitive content. |
| Approach: | They propose to generate copywriting based on customer reviews as they provide firsthand practical experiences with products, offering a richer source of information than just product attributes. |
| Outcome: | The proposed framework outperforms baseline and zero-shot large language models in terms of both attractiveness and faithfulness. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
Empowering Diffusion Models on the Embedding Space for Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work adapts diffusion models to textual data by diffusing on the embedding space. |
| Approach: | They propose an embedding diffusion model based on Transformer to solve the problem of embeddable space and denoising model. |
| Outcome: | The proposed model is more efficient than previous methods on seminal text generation tasks and is superior to existing models. |
Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce LLMs' biased outputs rely on reward signals from current model outputs without considering the source of biases. |
| Approach: | They propose to leverage the reward model in RL alignment as an instrumental variable to perform causal intervention on LLMs. |
| Outcome: | The proposed method reduces biases by using human feedback to fine tune LLMs to human values. |
Fake Alignment: Are LLMs Really Aligned Well? (2024.naacl-long)
Copied to clipboard
Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, Yingchun Wang
| Challenge: | Existing studies on large language models have shown that they are poorly aligned in practice. |
| Approach: | They propose a framework to evaluate safety in large language models . they propose two new metrics to quantify fake alignment and obtain corrected performance estimation. |
| Outcome: | The proposed framework and two metrics show that some models with purported safety are poorly aligned in practice. |
Visually Guided Generative Text-Layout Pre-training for Document Intelligence (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work shows that pre-training techniques can boost the performance of visual document understanding (VDU) . Xu et al., 2020;; Gu e t al, 2021;; Appalaraju e al. 2022) |
| Approach: | They propose a visually guided generative text-layout pre-training method that optimizes hierarchical language and layout modeling objectives to generate interleaved text and layout sequences. |
| Outcome: | The proposed model can process word-intensive documents of any length and achieves competitive performance over baselines on VDU tasks. |
HILL: Hierarchy-aware Information Lossless Contrastive Learning for Hierarchical Text Classification (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing self-supervised methods in natural language processing rely on augmentation rules to generate contrastive samples. |
| Approach: | They propose a hierarchy-aware information lossless contrastive learning scheme that uses syntactic information reserved in the input sample and fused during the learning process. |
| Outcome: | The proposed learning scheme is superior to existing methods in hierarchical text classification . the proposed learning system is based on a structure encoder and a text encoder . |
Investigating the Emergent Audio Classification Ability of ASR Foundation Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Text and vision foundation models can perform many tasks in a zero-shot setting . however, there has been little work on the zero-shoot abilities of ASR foundation models . |
| Approach: | They investigate the ability of ASR foundation models to perform zero-shot audio classification using text prompts and a decoding probability generator. |
| Outcome: | The proposed model outperforms state-of-the-art models on audio classification datasets without training them on extra data or adding any parameters. |
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is a common method for teaching large language models new tasks . given labeled examples in the input context, the model learns to perform the task without weight updates. |
| Approach: | They examine whether models guided via ICL infer the underlying structure of the task defined by the context or rely on superficial heuristics that only generalize to identically distributed examples. |
| Outcome: | The proposed model generalizes syntactically or linearly on out-of-distribution examples . the proposed model is able to generalize better on pre-trained models . |
Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt (2024.naacl-long)
Copied to clipboard
Yongqi Wang, Ruofan Hu, Rongjie Huang, Zhiqing Hong, Ruiqi Li, Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao
| Challenge: | Recent singing-voice-synthesis methods lack ability to control style attributes of synthesized singing. |
| Approach: | They propose a singing-voice-synthesis method that enables attribute controlling on singer gender, vocal range and volume with natural language. |
| Outcome: | The proposed method achieves favorable control ability and audio quality. |
Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024.naacl-long)
Copied to clipboard
| Challenge: | Automatic speech recognition systems fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. |
| Approach: | They evaluate six leading automatic speech recognition systems based on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
| Outcome: | The six leading speech recognition systems were evaluated on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification (2024.naacl-long)
Copied to clipboard
| Challenge: | Fallacy classification is a task of broad importance due to advances in deep learning and availability of more data. |
| Approach: | They propose a new annotation scheme tailored for subjective NLP tasks and a method designed to handle subjectivity. |
| Outcome: | The proposed approach integrates existing fallacy classification datasets with new ones. |
Diffusion Glancing Transformer for Parallel Sequence-to-Sequence Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that non-autoregressive generation models are superior in generation efficiency but inferior in generation quality. |
| Approach: | They propose a diffusion glancing transformer which employs a modality diffusion process and residual glancy sampling to improve multi-modality modeling. |
| Outcome: | The proposed model outperforms autoregressive and non-autoregressive models on machine translation and text generation benchmarks. |
No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | idiomatic expressions (IEs) are a non-compositional aspect of a text that makes it difficult for a model to comprehend . general purpose PTLMs are negatively affected by the context, as performance increases with its removal. |
| Approach: | They propose to use idiomatic expressions to infer additional meaning from IEs . they argue that only IE-aware models are suitable for idiom- matic reasoning tasks . |
| Outcome: | The proposed models can reason in the presence of idiomatic expressions, the authors show . they show that general purpose PTLMs are negatively affected by the context . |
Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for ICD indexing have a heavy label distribution and a manual process . Xie and Xing (2017) propose a new approach to ICD re-ranking . |
| Approach: | They propose a "retrieve and re-rank" framework to allocate subsets of ICD codes to medical records . they leverage auxiliary knowledge of the electronic health records (EHR) and a discrete retrieval method . |
| Outcome: | The proposed method achieves state-of-the-art performance on the MIMIC-III benchmark. |
Anisotropy is Not Inherent to Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has shown that Transformer embeddings are anisotropic, which is called the representation degradation problem. |
| Approach: | They identify a set of Transformer models with isotropic embedding spaces, the large Pythia models. |
| Outcome: | The proposed model sets show that isotropic models do not develop as previously theorized. |
Finding Replicable Human Evaluations via Stable Ranking Probability (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that human evaluation is the best way to rank natural language generation systems . human raters can exhibit different behaviors when rating outputs, causing ranking to be unstable . stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Approach: | They propose to evaluate results through the lens of stability: stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Outcome: | The proposed model is based on a dataset of multi-segment translations rated by multiple professionals . human raters can exhibit different behaviors when rating NLG outputs, the study shows . |
Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have demonstrated significant improvements in their capacity for generalization. |
| Approach: | They propose to conduct stealthy and persistent unalignment on large language models via backdoor injections. |
| Outcome: | The proposed approach passes the safety evaluation while maintaining strong persistence against re-alignment defense. |
Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts (2024.naacl-long)
Copied to clipboard
| Challenge: | Pretrained language models have advanced natural language processing tasks significantly, but finetuning them on low-resource datasets presents significant challenges such as instability and overfitting. |
| Approach: | They propose a regularization method based on attention-guided weight mixup for finetuning PLMs on low-resource datasets. |
| Outcome: | The proposed method improves generalization and combats overfitting on two splits of the training dataset. |
Detecting Bipolar Disorder from Misdiagnosed Major Depressive Disorder with Mood-Aware Multi-Task Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Bipolar Disorder (BD) is a mental disorder characterized by intense mood swings, ranging from depression to manic states. |
| Approach: | They propose to use social media data to identify BD risk in individuals misdiagnosed as MDD by multi-task learning. |
| Outcome: | The proposed approach outperforms state-of-the-art baselines and can provide insights into the impact of BD mood on future risk. |
Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is an attractive approach for semantic parsing, but learning to parse to rare domain-specific languages from a few demonstrations is challenging. |
| Approach: | They propose to use Python instead of DSLs to augment prompts with a structured domain description. |
| Outcome: | The proposed approach improves accuracy and generalization across three datasets. |
Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevailing practice for adapting language models (LMs) to new domains. |
| Approach: | They propose a mask specific language model that weights the importance of domain-specific terms during fine-tuning to avoid insensitivity. |
| Outcome: | The proposed approach outperforms advanced masking strategies such as span- and PMI-based masking. |
Language Models Implement Simple Word2Vec-style Vector Arithmetic (2024.naacl-long)
Copied to clipboard
| Challenge: | a primary criticism of language models is their inscrutability. |
| Approach: | They propose to use a vector arithmetic style mechanism to solve relational tasks . they find that this mechanism is specific to tasks that require retrieval from pretraining memory . |
| Outcome: | The proposed model reduces to a simple additive update for a variety of tasks . the findings contribute to proving that the models are interpretable and reliable . |
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. |
| Approach: | They propose a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. |
| Outcome: | The proposed framework is based on a meta learning based framework that can identify the optimal rank of each LoRA layer. |
SportQA: A Benchmark for Sports Understanding in Large Language Models (2024.naacl-long)
Copied to clipboard
Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, Weining Shen
| Challenge: | SportQA is a benchmark specifically designed for evaluating Large Language Models (LLMs) sports knowledge is characterized by its fast pace, variety of types, abundance of strategies, and rich player narratives . |
| Approach: | They propose a benchmark specifically designed for evaluating Large Language Models in the context of sports understanding. |
| Outcome: | The proposed benchmark aims to bridge the gap between existing and specialized benchmarks in sports understanding. |
Revisiting subword tokenization: A case study on affixal negation in large language models (2024.naacl-long)
Copied to clipboard
| Challenge: | Negation is central to language understanding but is not properly captured by modern NLP methods. |
| Approach: | They propose to use subword tokenization methods to detect negation in large language models . they find that models can reliably recognize negation, despite mismatches in tokenization accuracy . |
| Outcome: | The proposed models can detect negation in English using subword tokenization methods despite some mismatches in tokenization accuracy and negation detection performance. |
Generating Mental Health Transcripts with SAPE (Spanish Adaptive Prompt Engineering) (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models can generate synthetic data resembling real-world data, but their generative performance depends on the quality of the prompt used to instruct the model. |
| Approach: | They propose a Spanish Adaptive Prompt Engineering method that uses genetic algorithms to generate and select prompts that resemble real-world data. |
| Outcome: | The proposed method produces Spanish therapy transcripts that more closely resemble authentic therapy transcript compared to other prompt engineering techniques that are based on Reflexion and Chain-of-Thought. |
Where are you from? Geolocating Speech and Applications to Language Identification (2024.naacl-long)
Copied to clipboard
Patrick Foley, Matthew Wiesner, Bismarck Odoom, Leibny Paola Garcia Perera, Kenton Murray, Philipp Koehn
| Challenge: | Language identification (LID) is a critical component in many modern multilingual speech technologies. |
| Approach: | They propose to use radio broadcasts with known origin to train regression models . they also propose to explore using geolocation as a proxy task for LID . |
| Outcome: | The proposed model outperforms pretrained models on the FLEURS benchmark and on the VoxLingua benchmark. |
Teaching Language Models to Self-Improve through Interactive Demonstrations (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been shown to improve performance on downstream tasks by prompting them to analyze and revise their outputs. |
| Approach: | They propose a training algorithm that prompts large language models to analyze and revise their own outputs and uses this feedback to train the small model. |
| Outcome: | The proposed approach improves LLaMA-7B's performance on math and reasoning tasks by up to 7.13%. |
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets (2024.naacl-long)
Copied to clipboard
Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Lijia Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
| Challenge: | Existing approaches to augment textual dialogues with retrieved images pose privacy, diversity, and quality constraints. |
| Approach: | They propose a framework to augment text-only dialogues with diverse and high-quality images by using a diffusion model and a feedback loop. |
| Outcome: | The proposed framework is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small. |
Zero-shot Generative Linguistic Steganography (2024.naacl-long)
Copied to clipboard
| Challenge: | Generative linguistic steganography attempts to hide secret messages into covertext . previous studies focused on the statistical differences between the covertext and stegotext - however, ill-formed stegotas can readily be identified by humans . |
| Approach: | They propose a zero-shot approach based on in-context learning for linguistic steganography to achieve better perceptual and statistical imperceptibility. |
| Outcome: | The proposed method produces 1.926 more innocent and intelligible stegotext than any other method. |
Does GPT-4 pass the Turing test? (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) such as GPT-4 seem well designed for Turing's game. |
| Approach: | They evaluated GPT-4 in a public online Turing test and found it passed in 49.7% of games. |
| Outcome: | The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%) participants’ decisions were based mainly on linguistic style (35%) and socioemotional traits (27%). |
Polarity Calibration for Opinion Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing opinions summarization models emphasize the majority opinions while ignoring the minority opinions. |
| Approach: | They propose a method to align output summary and input text to achieve polarity calibration. |
| Outcome: | The proposed model can mitigate the polarity mismatch between output summary and input text, and maintain the content semantic and language quality. |
Sentence-level Media Bias Analysis with Event Relation Graph (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on media bias at the article level have identified media biases but only a few have been done on article level. |
| Approach: | They propose to construct an event relation graph to explicitly reason about event-event relations for sentence-level bias identification. |
| Outcome: | The proposed model improves both precision and recall of bias sentence identification. |
EMONA: Event-level Moral Opinions in News Articles (2024.naacl-long)
Copied to clipboard
Yuanyuan Lei, Md Messal Monem Miah, Ayesha Qamar, Sai Ramana Reddy, Jonathan Tong, Haotian Xu, Ruihong Huang
| Challenge: | Recent work on news articles has focused on social media short texts, but little has explored moral sentiment within news articles. |
| Approach: | They propose to extract event-level moral opinions from news articles using a new dataset . they use annotated event-based moral opinions to analyze news articles . |
| Outcome: | The proposed dataset consists of 400 news articles containing over 10k sentences and 45k events, among which 9,613 events received moral foundation labels. |
DLM: A Decoupled Learning Model for Long-tailed Polyphone Disambiguation in Mandarin (2024.naacl-long)
Copied to clipboard
| Challenge: | Grapheme-to-phoneme conversion datasets suffer from the long-tail problem . context learning for polyphonic characters often stems from a single dimension . |
| Approach: | They propose a model for long-tailed polyphone disambiguation in Mandarin that decouples representation and classification learnings. |
| Outcome: | The proposed model can decouple representation and classification learnings . it achieves transition learning of context from local to global . |
You don’t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments (2024.naacl-long)
Copied to clipboard
Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, David Jurgens
| Challenge: | Large Language Models (LLMs) are popular for research in social sciences . currently, prompting LLMs is insufficient to accurately and reliably capture model perceptions, and we discuss potential alternatives to improve this. |
| Approach: | They construct a dataset that contains 693 questions encompassing 39 different instruments of persona measurement on 115 persona axes and a set of questions containing minor variations. |
| Outcome: | The proposed model can generate answers and negate statements in a consistent and robust manner. |
CASA: Causality-driven Argument Sufficiency Assessment (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to assess the sufficiency of arguments are laborious and inconsistent due to subjective criteria. |
| Approach: | They propose a causality-driven argument sufficiency assessment framework that uses the probability of sufficience to estimate the probability that a premise event would lead to a conclusion when both premise and conclusion events are absent. |
| Outcome: | The proposed framework identifies insufficient arguments and improves them in a writing aid application. |
MacGyver: Are Large Language Models Creative Problem Solvers? (2024.naacl-long)
Copied to clipboard
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, Faeze Brahman
| Challenge: | a new study examines the creative problem-solving capabilities of modern LLMs . it provides insight into the constrained problem- solving capabilities of both humans and AI . |
| Approach: | They use an automatically generated dataset to compare and contrast LLMs and humans to find out their creative problem-solving abilities. |
| Outcome: | The proposed dataset compares LLMs and humans in a constrained setting . it shows that humans excel in tasks they are familiar with but struggle with domain-specific knowledge . |
To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages (2024.naacl-long)
Copied to clipboard
| Challenge: | XLT with multilingual language models is superfluous, says a new study . mBERT, XLM-R and mT5 are effective for cross-lingual transfer, authors say . |
| Approach: | They propose to use multilingual language models to improve cross-lingual transfer (XLT) they propose to add reliable translations to training data for XLT even for non-MT languages . |
| Outcome: | The proposed approaches outperform zero-shot XLT with mLMs, the authors show . the authors believe their findings warrant a broader inclusion of more robust translation-based baselines in XL research. |
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on how LLMs handle inductive instructions, which may stem from users’ false beliefs or malicious intents. |
| Approach: | They propose a benchmark of Inductive Instructions where false knowledge is incorporated into instructions in multiple different styles. |
| Outcome: | The proposed model improves robustness against inductive instructions, despite different inductive styles and complexity. |
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer (2024.naacl-long)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models are limited to a set of predefined entity types. Large language models (LLMs) can extract arbitrary entities through natural language instructions. |
| Approach: | They propose a model that can identify any type of entity using a transformer encoder. |
| Outcome: | The proposed model outperforms existing models on NER benchmarks on a set of predefined entities. |
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are now being used by millions of people across the world. |
| Approach: | They propose a test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way. |
| Outcome: | The proposed test suite identifies eXaggerated Safety behaviours in a systematic way. |
Carpe diem: On the Evaluation of World Knowledge in Lifelong Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Current language models are trained on static data, implying that the encoded knowledge could go wrong as time passes. |
| Approach: | They propose a temporally evolving question-answering benchmark for language models . they use Wikipedia databases to test language models for dynamic knowledge in ever-changing world . |
| Outcome: | The proposed task aims to model the evolution-adaptability of language models in the real world. |
Fine-grained Gender Control in Machine Translation with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on controlled translation has only considered a simplified setup of one target gender for input. |
| Approach: | They propose a Gender-of-Entity prompting method for machine translation that takes the gender of the ambiguous entity as additional input and propose to use it to translate with correct gender inflections. |
| Outcome: | The proposed method instructs the model with fine-grained entity-level gender information to translate with correct gender inflections. |
DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade (2024.naacl-long)
Copied to clipboard
Zefan Cai, Xin Zheng, Tianyu Liu, Haoran Meng, Jiaqi Han, Gang Yuan, Binghuai Lin, Baobao Chang, Yunbo Cao
| Challenge: | Existing models for natural language understanding are based on a well-defined intent 1 ontology. |
| Approach: | They propose to retrain the natural language understanding model as new data from real users are merged into existing data. |
| Outcome: | The proposed model shows that the semantically entangled intents can be recognized with an automatic workflow. |
LLatrieval: LLM-Verified Retrieval for Verifiable Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models struggle with factual errors and often produce non-factual and fabricated content. |
| Approach: | They propose to use large language models to generate text with supporting documents to enable the user to flexibly verify the answer. |
| Outcome: | Experiments on ALCE show that LLatrieval significantly outperforms extensive baselines and achieves state-of-the-art results. |
Mapping Long-term Causalities in Psychiatric Symptomatology and Life Events from Social Media (2024.naacl-long)
Copied to clipboard
Siyuan Chen, Meilin Wang, Minghao Lv, Zhiling Zhang, Juqianqian Juqianqian, Dejiyangla Dejiyangla, Yujia Peng, Kenny Zhu, Mengyue Wu
| Challenge: | Existing studies focus on the semantic content of social media posts, overlooking the evolving nature of mental disorders and symptoms. |
| Approach: | They extract causality between psychiatric symptoms and life events from social media posts and extract temporal attributes to improve diagnosis and treatment planning. |
| Outcome: | The extracted causality features improve diagnostic and treatment planning and improve performance in tasks such as depression and diagnosis point detection. |
Multimodal Chart Retrieval: A Comparison of Text, Table and Image Based Approaches (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal chart retrieval is a task of retrieving information from a multimodal dataset, given a query. |
| Approach: | They compare four different approaches to multimodal chart retrieval . they use chart derendering, chart derending, table retrieval and Tab-GTR . |
| Outcome: | The proposed model outperforms the previous methods on the NQ-Tables benchmark and is significantly more efficient on in-distribution data. |
Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LMs) excel in retrieving popular facts, but encounter difficulty with infrequent entity-relation pairs compared to retrievers. |
| Approach: | They propose to use a WiTQA dataset to explore the effects of combinations of entities and relations on LMs. |
| Outcome: | The proposed model can retain popular relations of less common entities while retaining the same popular relations. |
AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs (2024.naacl-long)
Copied to clipboard
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer
| Challenge: | a new model for speech processing and reasoning uses curated data instead of text. |
| Approach: | They extend the instruction-tuned Llama-2 model with end-to-end speech processing and reasoning abilities without using any carefully curated paired data. |
| Outcome: | The proposed model outperforms or outperfects existing models on synthesized and recorded speech QA tests. |
Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to measure robustness are problematic, and out-of-domain evaluations are no longer relevant. |
| Approach: | They examine models of different sizes spanning different architectural choices and pretraining objectives. |
| Outcome: | The results show that not all out-of-domain tests provide insight into robustness . merely scaling models does not make them adequately robust . |
Sequential Compositional Generalization in Multimodal Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a growing number of multimodal models have a limited capacity for generalization . however, prior studies into compositionality have focused on visual grounding and downstream tasks like image captioning. |
| Approach: | They examine compositional generalization using egocentric kitchen activity videos . they find bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts . |
| Outcome: | The proposed model outperforms text-only models in a multimodal setting. |
Generating Uncontextualized and Contextualized Questions for Document-Level Event Argument Extraction (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for document-level argument extraction do not require human involvement and combine uncontextualized and contextualized questions. |
| Approach: | They propose multiple question generation strategies for document-level event argument extraction that do not require human involvement and combine uncontextualized and contextualized questions. |
| Outcome: | The proposed questions do not require human involvement and are suitable for document-level argument extraction. |
Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to generate counter-misinformation responses are often trained end-to-end without external knowledge, resulting in subpar text quality and excessively repetitive responses. |
| Approach: | They propose retrieval augmented response generation for online misinformation (RARG) that collects supporting evidence and generates counter-misinformation responses via reinforcement learning from human feedback. |
| Outcome: | The proposed method outperforms baselines with extensive experiments with in- and cross-domain datasets and consistently generates high-quality counter-misinformation responses. |
Open-Vocabulary Federated Learning with Multimodal Prototyping (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies assume the label space of training data and test data is identical. |
| Approach: | They propose a framework for adaptation to a federated learning (FL) query that uses arbitrary unknown classes. |
| Outcome: | The proposed framework exploits the knowledge learned from seen classes and robustifies the adapted framework to unseen categories. |
Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for key point analysis rely on semantic similarity instead of measuring the existence of shared key points . |
| Approach: | They propose a key point analysis approach with pairwise generation and graph partitioning to summarize arguments into a concise set of key points. |
| Outcome: | The proposed model surpasses existing models on ArgKP and QAM datasets. |
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. |
| Approach: | They conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. |
| Outcome: | The language used to query the LLMs can impact their performance on cultural-related tasks. |
Code Models are Zero-shot Precondition Reasoners (2024.naacl-long)
Copied to clipboard
Lajanugen Logeswaran, Sungryull Sohn, Yiwei Lyu, Anthony Liu, Dong-Ki Kim, Dongsub Shim, Moontae Lee, Honglak Lee
| Challenge: | Existing methods to reason about action preconditions are lacking for agent to complete tasks. |
| Approach: | They propose a method to reason about action preconditions using pre-trained code models. |
| Outcome: | The proposed approach improves few-shot policy learning approaches across task-oriented dialog and embodied textworld benchmarks. |
Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding (2024.naacl-long)
Copied to clipboard
| Challenge: | End-to-end learning models require large volume of speech data with intent labels . however, models are sensitive to inconsistencies between training and evaluation conditions . |
| Approach: | They propose a module-based approach to learn intent in a noisy-channel model . they correlate error patterns between clean and noisy ASR transcripts . |
| Outcome: | The proposed method outperforms existing methods and improves in noisy environments. |
Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) are more efficient in natural language understanding tasks. |
| Approach: | They evaluate large language models (LLMs) using a TREC Fair Ranking dataset . they assess fairness from both user and content perspectives . |
| Outcome: | The proposed model outperforms the existing models in the fair ranking task. |
TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models struggle with large tables due to their limited input length . a novel method that decomposes tables into smaller and relevant sub-tables reduces the computational load on LLMs . |
| Approach: | They propose a method that leverages text-to-SQL generation to decompose tables into smaller and relevant sub-tables . the method can reduce the input context length significantly, making it more scalable and efficient . |
| Outcome: | The proposed method performs remarkably well on the WikiTQ benchmark and on the TabFact benchmark. |
Contextual Label Projection for Cross-Lingual Structured Prediction (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work favors simplified label translation or relying on word-level alignments for label projection. |
| Approach: | They propose a novel approach CLaP which translates text to target language and performs *contextual translation* on the labels using the translated text as the context. |
| Outcome: | The proposed approach improves translation accuracy on two prediction tasks and shows 2.4 F1 improvement for EAE and 1.4 F1 for named entity recognition. |
Event Detection from Social Media for Epidemic Prediction (2024.naacl-long)
Copied to clipboard
Tanmay Parekh, Anh Mac, Jiarui Yu, Yuxuan Dong, Syed Shahriar, Bonnie Liu, Eric Yang, Kuan-Hao Huang, Wei Wang, Nanyun Peng, Kai-Wei Chang
| Challenge: | Social media is an easy-to-access platform providing timely updates about societal trends and events. |
| Approach: | They propose a framework to extract epidemic-related events from social media posts to provide early warnings. |
| Outcome: | The proposed framework can detect epidemic events for three unseen epidemics of Monkeypox, Zika, and Dengue while existing models fail miserably. |
RESPROMPT: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models (2024.naacl-long)
Copied to clipboard
Song Jiang, Zahra Shakeri, Aaron Chan, Maziar Sanjabi, Hamed Firooz, Yinglong Xia, Bugra Akyildiz, Yizhou Sun, Jinchao Li, Qifan Wang, Asli Celikyilmaz
| Challenge: | Chain-of-thought (CoT) has impressively unlocked the reasoning potential of large language models (LLMs), but it falls short when tackling problems that require multiple reasoning steps. |
| Approach: | They propose a new prompting strategy that advances multi-step reasoning in LLMs by integrating necessary connections into prompts. |
| Outcome: | The proposed strategy improves multi-step reasoning accuracy and improves reasoning accuracy across math, sequential, and commonsense domains. |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
How are Prompts Different in Terms of Sensitivity? (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) has become one of the most popular learning paradigms due to the rapid development of large language models (LLMs). |
| Approach: | They propose a prompt analysis based on sensitivity and introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. |
| Outcome: | The proposed approach is particularly useful when information in the input is scarce. |
LSTDial: Enhancing Dialogue Generation via Long- and Short-Term Measurement Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue systems do not utilize quality dimensions specifically designed for dialogue evaluation to guide the response generation during training. |
| Approach: | They propose a two-stage framework which generates and utilizes conversation evaluation as explicit feedback during training. |
| Outcome: | The proposed framework generates and utilizes conversation evaluation as explicit feedback during training. |
The ART of LLM Refinement: Ask, Refine, and Trust (2024.naacl-long)
Copied to clipboard
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, Asli Celikyilmaz
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve? |
| Approach: | They propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust* that asks necessary questions to decide when an LLM should refine its output and uses it to affirm or deny trust. |
| Outcome: | The proposed reasoning with a refinement strategy achieves a performance gain of +5 points over baselines on two multistep reasoning tasks. |
Modularized Multilingual NMT with Fine-grained Interlingua (2024.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation (MNMT) systems lack layer-sharing to generate interlingua features . however, layer-share structure does not guarantee explicit propagation of language-specific features to respective decoders. |
| Approach: | They propose to share top of language-specific encoder layers to enable interlingua features . their method demonstrates an improved average BLEU score by "+2.90" in En-to-Any directions . |
| Outcome: | The proposed approach improves the BLEU score by "+2.90" in En-to-Any directions and by "+1.06" in zero-shot translation. |
ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies (2024.naacl-long)
Copied to clipboard
| Challenge: | Analogy-making is a central to human cognition, allowing us to abstract information and understand novel situations in terms of familiar ones. |
| Approach: | They propose a pipeline to generate paragraph-based analogies using large language models and large language distractors. |
| Outcome: | The proposed pipeline outperforms existing models in binary and multiple-choice settings and shows that humans outperformed the best models after a light supervision. |
AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing solutions focus on efficient attentions or divide-and-conquer strategies, but these methods sacrifice global context, leading to incoherent and uninformative summaries. |
| Approach: | They propose to leverage the memory-efficient nature of divide-and-conquer methods while preserving global context. |
| Outcome: | The proposed framework improves informativeness, faithfulness, and coherence over baselines on government reports, meeting transcripts, screenplays, scientific papers, and novels. |
NLP Systems That Can’t Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing language models fail to distinguish use from mention, leading to misinformation and hate speech detection, resulting in censorship of counterspeech. |
| Approach: | They propose prompting mitigations that teach the use-mention distinction and show they reduce these errors. |
| Outcome: | The proposed model reduces misinformation and hate speech detection errors by reducing misinformation, and reducing hate speech. |
Debiasing with Sufficient Projection: A General Theoretical Framework for Vector Representations (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained vector representations can inadvertently encode undesirable social biases. |
| Approach: | They propose a framework for reducing bias by transforming vector representations to an unbiased subspace using sufficient projection. |
| Outcome: | The proposed framework mitigates bias across debiasing and fairness tasks and across various vector representation types, including word embeddings and output representations of transformer models. |
Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection (2024.naacl-long)
Copied to clipboard
| Challenge: | Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data. |
| Approach: | They propose a scoring approach that encapsulates three primary dimensions of summarization model quality. |
| Outcome: | The proposed method reduces reliance on human-labeled data and improves the performance of summarization models. |
AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages (2024.naacl-long)
Copied to clipboard
Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Mohamed, Hassan Ayinde, Oluwabusayo Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Toadoum Sari Sakayo, Lyse Naomi Wamba, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Iro, Saheed Abdullahi, Stephen Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Ogbu, Sam Ochieng’, Verrah Otiende, Chinedu Mbonu, Yao Lu, Pontus Stenetorp
| Challenge: | Recent advances in machine translation (MT) have focused on scaling multilingual machine translation models and evaluation data to hundreds of languages, including multiple under-resourced languages. |
| Approach: | They propose to use n-gram matching metrics to measure progress in multilingual machine translation to 13 typologically diverse African languages to create high-quality human evaluation data with simplified MQM guidelines. |
| Outcome: | The proposed metrics have a higher correlation with human judgments than n-gram matching metrics such as BLEU and METEOR. |
TableLlama: Towards Open Large Generalist Models for Tables (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for interpreting, augmenting, and querying semi-structured tables require pretraining on tables or special model architecture design. |
| Approach: | They construct a dataset with a variety of tables and tasks for instruction tuning and evaluating LLMs. |
| Outcome: | The proposed model achieves comparable or better performance on 7 out of 8 in-domain tasks compared with the base model on 6 out-of-domain datasets. |
PEMA: An Offsite-Tunable Plug-in External Memory Adaptation for Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) show impressive performance in various downstream NLP tasks. |
| Approach: | They propose a Parameter-Efficient Fine-Tuning method that integrates with context representations from test data to perform downstream tasks. |
| Outcome: | The proposed method outperforms other methods in memory and latency efficiency and maintains sentence meaning and generating appropriate language and styles. |
Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection (2024.naacl-long)
Copied to clipboard
Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, Hongxia Jin
| Challenge: | Instruction-tuned Large Language Models (LLMs) can modulate responses based on human instructions, but they can be maliciously steered to impact society in subtle but persistent ways. |
| Approach: | They propose a backdoor attack setting that allows an attacker to inject a virtual prompt into an LLM to steer it without any explicit injection at its input. |
| Outcome: | The proposed method is able to poison the model's instruction tuning data and show that it is highly effective in steering the model. |
Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | LLMs generate responses following user's instructions, which requires high dialogue comprehension ability. |
| Approach: | They propose to evaluate LLMs' dialogue comprehension ability using a dialogue summarization task to derive factual questions from the generated summaries and use them as a more flexible measurement of dialogue comprehension. |
| Outcome: | The proposed model reduces the error rate by 11% on the dialogue summarization task. |
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly (2024.naacl-long)
Copied to clipboard
| Challenge: | Current large language models show imbalance abilities in different languages . authors propose two approaches to improve cross-lingual knowledge alignment . |
| Approach: | They propose a framework to assess cross-lingual knowledge alignment of large language models . they propose multilingual pretraining and multilingual instruction tuning to address this problem . |
| Outcome: | The proposed framework assesses the cross-lingual knowledge alignment of LLMs in performance, consistency and conductivity levels. |
A Study on the Calibration of In-context Learning (2024.naacl-long)
Copied to clipboard
Hanlin Zhang, YiFan Zhang, Yaodong Yu, Dhruv Madeka, Dean Foster, Eric Xing, Himabindu Lakkaraju, Sham Kakade
| Challenge: | Prior research has demonstrated improvements in the calibration of language models (LMs) in-context learning is a popular method for adapting static LMs to safety-critical domains. |
| Approach: | They use in-context learning to adapt static language models through tailored prompts to a wide range of tasks and find that miscalibration occurs in low-shot settings. |
| Outcome: | The proposed calibrations show that models exhibit increased miscalibration before achieving better calibration in low-shot settings. |
DialogBench: Evaluating LLMs as Human-like Dialogue Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate LLMs' abilities for task completion as assistant AI. |
| Approach: | They propose a dialogue evaluation benchmark that contains 12 dialogue tasks to evaluate LLMs' capabilities as human-like dialogue systems. |
| Outcome: | The proposed benchmark contains 12 tasks to evaluate LLMs' capabilities . it shows that instruction tuning improves human likeness, but not as human-like systems . |
GINopic: Topic Modeling with Graph Isomorphism Network (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies focus on the representation of documents as a sequence of words, but word dependency patterns are not captured in topic modeling. |
| Approach: | They propose a topic modeling framework based on graph isomorphism networks to capture word dependencies between words. |
| Outcome: | The proposed framework is compared with existing topic models on a dataset of a large text collection and shows that it can uncover the underlying topics in an unsupervised manner. |
CMB: A Comprehensive Medical Benchmark in Chinese (2024.naacl-long)
Copied to clipboard
Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, Haizhou Li
| Challenge: | Large Language Models (LLMs) provide a great breakthrough in medicine, says a new study . existing studies on LLMs leverage subjective evaluation, but evaluation in medicine is professional . |
| Approach: | They propose a localized medical benchmark in Chinese rooted in native Chinese . they propose to use traditional Chinese medicine to evaluate large-scale LLMs . |
| Outcome: | a new benchmark is developed to evaluate large-scale LLMs in china . the proposed model is rooted in the native Chinese linguistic and cultural framework . |
Massive End-to-end Speech Recognition Models with Time Reduction (2024.naacl-long)
Copied to clipboard
Weiran Wang, Rohit Prabhavalkar, Haozhe Shan, Zhong Meng, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Chengjian Zheng, Yanzhang He, Tara Sainath, Pedro Moreno Mengibar
| Challenge: | Using the neural architecture of Google’s universal speech model, we reduce the frame rate and speed up training and inference. |
| Approach: | They propose to use the neural architecture of Google’s universal speech model with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. |
| Outcome: | The proposed methods work with both connectionist temporal classification (CTC) and RNN-Transducer (RNN-T) and over two domains. |
SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics (2024.naacl-long)
Copied to clipboard
Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, Koushik Sen
| Challenge: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
| Approach: | They propose a tool that analyzes transformer-based models and freezes less-contributory layers during fine-tuning to reduce the overall on-device memory usage. |
| Outcome: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
Effective Large Language Model Adaptation for Improved Grounding and Citation Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models generate "hallucinated" answers that are not factual . despite their widespread adoption, they can generate plausiblesounding but nonfactual information. |
| Approach: | They propose a framework that tunes large language models to self-ground claims and provide citations to retrieved documents. |
| Outcome: | The proposed framework generates superior grounded responses with more accurate citations compared to prompting-based approaches and post-hoc citing-based methods. |
Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to write grounded, long-form articles have limited planning capacity and require extensive research and planning in the pre-writing stage. |
| Approach: | They propose a system for the Synthesis of Topic Outlines throughRetrieval and Multi-perspective Question Asking that models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. |
| Outcome: | The proposed system is based on a dataset of high-quality Wikipedia articles and evaluates the pre-writing stage. |
Grounding Gaps in Language Model Generations (2024.naacl-long)
Copied to clipboard
| Challenge: | Effective conversation requires common ground, but it does not emerge spontaneously. |
| Approach: | They propose a set of grounding acts and metrics that quantify attempted grounding . they find that large language models generate language with less conversational grounding than humans . |
| Outcome: | The proposed models generate language with less conversational grounding than humans . compared to humans, they generate language that appears to presume common ground . |
When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale (2024.naacl-long)
Copied to clipboard
| Challenge: | Multilingual machine translation (MMT) is a key tool for improving translation in low-resource languages. |
| Approach: | They examine how denoising autoencoding and backtranslation impact multilingual machine translation under different data conditions and model scales. |
| Outcome: | The proposed method improves translation efficiency in low-resource languages by using denoising autoencoding (DAE) and backtranslation (BT) . |
ContraSim – Analyzing Neural Representations Based on Contrastive Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing similarity measures perform mediocrely on standard benchmarks . |
| Approach: | They develop a similarity measure based on contrastive learning that learns a parameterized measure by using both similar and dissimilar examples. |
| Outcome: | The proposed measure achieves much higher accuracy than previous similarity measures . it is more suitable for the analysis of neural networks, revealing new insights . |
Universal Prompt Optimizer for Safe Text-to-Image Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies based on image checker, model fine-tuning and embedding blocking are impractical in real-world applications. |
| Approach: | They propose a novel reward function measuring toxicity and text alignment of generated images and train the optimizer through Proximal Policy Optimization. |
| Outcome: | The proposed model reduces the likelihood of various models in generating inappropriate images, with no significant impact on text alignment. |
Language Model Based Unsupervised Dependency Parsing with Conditional Mutual Information and Grammatical Constraints (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for unsupervised dependency parsing use difficult to interpret dependence scores. |
| Approach: | They propose to use Conditional Mutual Information (CMI) to measure bi-lexical dependence and incorporate grammatical constraints into unsupervised parsing. |
| Outcome: | The proposed model outperforms state-of-the-art models and grammar-based models in five languages and eight datasets. |
The Bias Amplification Paradox in Text-to-Image Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | amplification is a phenomenon in which models exacerbate biases or stereotypes in training data. |
| Approach: | They compare gender ratios in training vs. generated images to investigate bias amplification . they find that a model amplifys gender-occupation biases considerably . |
| Outcome: | The proposed model amplifys gender-occupation biases in training data, but it can be attributed to discrepancies between training captions and model prompts. |
Grammar-based Data Augmentation for Low-Resource Languages: The Case of Guarani-Spanish Neural Machine Translation (2024.naacl-long)
Copied to clipboard
Agustín Lucas, Alexis Baladón, Victoria Pardiñas, Marvin Agüero-Torales, Santiago Góngora, Luis Chiruzzo
| Challenge: | Low-resource languages suffer from a vicious circle: data is needed to build tools, but available text is scarce. |
| Approach: | They propose to use a grammar-based system to generate Spanish text and syntactically transfer it to Guarani to boost its performance. |
| Outcome: | The proposed system outperforms existing models by pretraining models with synthetic text. |
Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | This study examines the ability of Large Language Models to encapsulate cultural nuances across diverse linguistic landscapes. |
| Approach: | They examine the efficacy of language-specific instruction tuning and the impact of pretraining on dominant language data in Large Language Models. |
| Outcome: | The findings highlight a nuanced landscape, with inconsistencies and biases, particularly in non-Western cultures. |
Toward Interactive Regional Understanding in Vision-Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing image-text pairs capture only coarse and global information of an image, leading to a limitation in their regional understanding ability. |
| Approach: | They propose a model with explicit regional modeling capabilities that allows VLP models to understand user-indicated image regions. |
| Outcome: | The proposed model performs better on zero-shot region understanding tasks without compromising its ability for global image understanding. |
ScriptMix: Mixing Scripts for Low-resource Language Parsing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has considered transliteration and vocabulary augmentation, but the consideration of combining the two has been lacking. |
| Approach: | They propose a multilingual pretrained language model that combines two strengths and overcomes the hurdle of combining them. |
| Outcome: | The proposed model improves POS accuracy by 14% and improves DEP LAS score by 5.6%. |
MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown their strong ability in the field of machine translation, yet they suffer from high computational cost and latency. |
| Approach: | They propose a framework which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
| Outcome: | The proposed framework transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
ToXCL: A Unified Framework for Toxic Speech Detection and Explanation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models that focus on explicit toxic speech detection and explanation are prone to error propagation problems . et al., 2018) show that toxic speech models can be prone for generating errors . |
| Approach: | They propose a framework that can detect and explain toxic speech using a target group generator and an encoder-decoder model. |
| Outcome: | The proposed model outperforms baseline models and achieves state-of-the-art effectiveness . the proposed model generates a toxic explanation that matches the ground truth explanation . |
LinkPrompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt-based learning is a new language model training paradigm that adapts Pre-trained Language Models (PLMs) to downstream tasks. |
| Approach: | They propose a prompt-based learning paradigm that adapts Pre-trained Language Models to downstream tasks . they use a gradient-based beam search algorithm to generate adversarial triggers . |
| Outcome: | The proposed model improves performance on various natural language processing tasks by optimizing the prompt template. |
CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that Large Language Models (LLMs) have impressive capabilities in a variety of domains and tasks. |
| Approach: | They propose a method which prompts LLMs to generate SQL queries based on the previously generated SQL query with an edition chain. |
| Outcome: | The proposed method outperforms different in-context learning baselines and achieves state-of-the-art performance on two benchmarks SParC and CoSQL using LLMs. |
ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Detecting contradictions in texts is often regarded as determining relation between hypothesis and piece of premise. |
| Approach: | They propose a human-annotated dataset to study self-contradictions in long documents . they analyze the capabilities of four open-source and commercially available LLMs . |
| Outcome: | The proposed dataset outperforms open-source LLMs on document-level tasks but struggles with self-contradictions that require more nuance and context. |
Entity Disambiguation via Fusion Entity Decoding (2024.naacl-long)
Copied to clipboard
Junxiong Wang, Ali Mousavi, Omar Attia, Ronak Pradeep, Saloni Potdar, Alexander Rush, Umar Farooq Minhas, Yunyao Li
| Challenge: | Existing generative approaches demonstrate improved accuracy compared to classification approaches under the standardized ZELDA benchmark. |
| Approach: | They propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. |
| Outcome: | The proposed model outperforms existing classification models on the ZELDA benchmark and on retrieval/reader frameworks. |
PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for decision making require complex data analysis. |
| Approach: | They propose a method that generates the plan for decision making as the first step and retrieves the queries for data analysis as the second step. |
| Outcome: | The proposed method outperforms the state-of-the-art iterative plan-then-retrieval augmented generation method by 15.8% and 7.4% respectively. |
GPTScore: Evaluate as You Desire (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for text generation are not adequate to assess the quality of the generated outputs. |
| Approach: | They propose a framework that utilizes emergent abilities of generative pre-trained models to evaluate generated texts. |
| Outcome: | The proposed evaluation framework can achieve what one desires to evaluate for texts simply by natural language instructions. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References (2024.naacl-long)
Copied to clipboard
Tianyi Tang, Hongyuan Lu, Yuchen Jiang, Haoyang Huang, Dongdong Zhang, Xin Zhao, Tom Kocmi, Furu Wei
| Challenge: | Existing evaluation benchmarks with limited references may not accurately reflect the quality of the model’s hypotheses. |
| Approach: | They propose a method to enrich evaluation benchmarks by diversifying the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
| Outcome: | The proposed method can enhance evaluation benchmarks by diversifying the expression of reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
Separation and Fusion: A Novel Multiple Token Linking Model for Event Argument Extraction (2024.naacl-long)
Copied to clipboard
Jing Xu, Dandan Song, Siu Hui, Zhijing Wu, Meihuizi Jia, Hao Wang, Yanru Zhou, Changzhi Zhou, Ziyi Yang
| Challenge: | Existing methods for event argument extraction (EAE) lack cross-event information and require longer role sequences . et al. (2017): outperforms state-of-the-art methods for EE. |
| Approach: | They propose a separation-and-fusion paradigm to separate the acquisition of cross-event information and fuse it into the argument extraction of a target event. |
| Outcome: | The proposed model outperforms the state-of-the-art models on four widely used datasets. |
The Integration of Semantic and Structural Knowledge in Knowledge Graph Entity Typing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to predict missing type annotations for knowledge graphs use only structural knowledge in the local neighborhood of entities. |
| Approach: | They propose a model for KG Entity Typing that integrates semantic and structural knowledge to infer missing types. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods in the Knowledge Graph Entity Typing task. |
ComCLIP: Training-Free Compositional Image and Text Matching (2024.naacl-long)
Copied to clipboard
| Challenge: | erroneous semantics of individual entities are essentially confounders that cause the matching failure. |
| Approach: | They propose a training-free compositional CLIP model which disentangles input images into subjects, objects, and action subimages and composes CLIP’s vision encoder and text encoder to perform evolving matching over compositional text embedding and subimage embeddments. |
| Outcome: | The proposed model mitigates spurious correlations introduced by the pretrained CLIP models and dynamically evaluates the importance of each component. |
ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing statistical phrasal or hierarchical machine translation systems relies on a large set of translation rules which results in engineering challenges. |
| Approach: | They propose to use factorized grammar from the field of linguistics as more general translation rules from XTAG English Grammar to generate a manually crafted summarization dataset. |
| Outcome: | The proposed method outperforms existing methods on low-resource language translation tasks with less training data. |
XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for active learning rely on model uncertainty or disagreement to pick unlabeled data, leading to over-confidence in superficial patterns and lack of exploration. |
| Approach: | They propose to use a bi-directional encoder and a uni-directional decoder to generate and score an explanation for low-resource text classification. |
| Outcome: | The proposed model improves on 9 strong baselines on six datasets and can generate explanations for its predictions. |
LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text-to-image generation have been underperforming in image-totext generation tasks. |
| Approach: | They propose a framework that uses a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr . |
Intent-conditioned and Non-toxic Counterspeech Generation using Multi-Task Instruction Tuning with RLAIF (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing systems that target hate speech with intent-conditioned counterspeech generate better results with longer contexts. |
| Approach: | They propose a framework that enables counterspeech generation by modeling the pragmatic implications underlying social biases in hateful statements. |
| Outcome: | The proposed framework outperforms existing benchmarks in intent-conditioned counterspeech generation. |
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are now commonplace in conversation applications, but their misuse for generating harmful responses has raised serious societal concerns. |
| Approach: | They provide a comprehensive overview of recent studies covering attacks, defenses, and evaluations of Large Language Models (LLMs) . |
| Outcome: | The proposed review summarizes three aspects of LLM conversation safety: attacks, defenses, and evaluations. |
Mind’s Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved significant advances in natural language processing, but their scale and computational demands pose challenges to their practical application. |
| Approach: | They propose a method for distilling the self-evaluation capability from LLMs into SLMs and advocate for more comprehensive thinking by incorporating multiple distinct CoTs and self-estimation outputs. |
| Outcome: | The proposed method significantly improves the performance of distilled SLMs on three NLP benchmarks. |
Divergent Token Metrics: Measuring degradation to prune away LLM components – and optimize quantization (2024.naacl-long)
Copied to clipboard
Björn Deiseroth, Max Meuer, Nikolas Gritsch, Constantin Eichenberg, Patrick Schramowski, Matthias Aßenmacher, Kristian Kersting
| Challenge: | Large Language Models (LLMs) have reshaped natural language processing with impressive capabilities, but their ever-increasing size has raised concerns about their effective deployment and the need for LLM compression. |
| Approach: | This study introduces the Divergent Token Metrics (DTMs) that measure token divergences that allow deeper insights into the subtleties of model compression. |
| Outcome: | The proposed measures can identify outliers and improve performance in the sparseness of the LLMs. |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Instructing Large Language Models to Identify and Ignore Irrelevant Conditions (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing CoT prompting methods elicited multi-step reasoning abilities of large language models (LLMs) but they were seriously confused by the irrelevant conditions, resulting in low accuracy. |
| Approach: | They propose a method that instructs large language models to identify and ignore irrelevant conditions and prompts them to verify the irrelevant conditions. |
| Outcome: | The proposed approach outperforms existing methods on MWPs with GPT-3.5-Turbo and I3C-Select. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
Transformers Can Represent n-gram Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work studies the transformer architecture in terms of language acceptance . n-gram LMs can represent any ngram lm, giving a concrete lower bound on their probabilistic representational capacity. |
| Approach: | They propose to use transformer LMs to represent n-gram LM, a class of language models . they show transformer lms can exactly represent any ngram lmm, giving them a lower bound . |
| Outcome: | The proposed model can represent any n-gram LM, giving a lower bound on their probabilistic representational capacity. |
The Role of n-gram Smoothing in the Age of Neural Networks (2024.naacl-long)
Copied to clipboard
| Challenge: | n-gram smoothing techniques were used to overcome overfitting problems in neural language models for decades. |
| Approach: | They propose to convert any n-gram smoothing technique into a regularizer compatible with neural language models. |
| Outcome: | The proposed regularizers outperform label smoothing on language modeling and machine translation. |
Reliability Estimation of News Media Sources: Birds of a Feather Flock Together (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent research has shown that predicting sources’ reliability is an important first-prior step in addressing additional challenges such as fake news detection and fact-checking. |
| Approach: | They propose a method that leverages reinforcement learning strategies to estimate the reliability degree of news sources based on how all the news media sources interact with each other on the Web. |
| Outcome: | The proposed method can predict reliability labels on a large news media reliability dataset. |
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism. |
| Approach: | They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models. |
| Outcome: | The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages. |
NLP Progress in Indigenous Latin American Languages (2024.naacl-long)
Copied to clipboard
Atnafu Tonja, Fazlourrahman Balouchzahi, Sabur Butt, Olga Kolesnikova, Hector Ceballos, Alexander Gelbukh, Thamar Solorio
| Challenge: | a new study examines the marginalization of indigenous languages in the face of rapid technological advancements. |
| Approach: | They highlight the cultural richness of indigenous languages and the risk they face of being overlooked in the realm of natural language processing. |
| Outcome: | The authors highlight the cultural richness of indigenous languages and their risk of being overlooked in the realm of natural language processing. |
On the Effectiveness of Adversarial Robustness for Abuse Mitigation with Counterspeech (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work on automated counterspeech systems focused on synthetic data but rarely looked into how the public deals with abuse. |
| Approach: | They propose to curate a new dataset of abuse and replies from footballers for study of public figure abuse and use it to examine how models can handle adversarial attacks. |
| Outcome: | The proposed model is robust against adversarial attacks across domains and can handle abuse in the real world. |
Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for text classification are based on fine-tuning the representations computed by large language models. |
| Approach: | They propose to exploit structural properties of pre-trained embeddings to spread information . they use a semisupervised strategy to train models with minimal annotation effort . |
| Outcome: | The proposed method outperforms self-training and random walk labels on different datasets. |
UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have investigated the potential of language models as knowledge bases and the existence of severe biases when extracting factual knowledge. |
| Approach: | They propose an adapter-based framework for generalised factual knowledge extraction using simple methods without introducing extra parameters. |
| Outcome: | The proposed framework improves the model’s out-of-domain generalisation and consistency under various prompts. |
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) generate factually incorrect answers based on their parametric memory. |
| Approach: | They propose a retrieval-augmented large language model that can dynamically select the most suitable strategy based on query complexity. |
| Outcome: | The proposed approach improves the performance of QA systems on open-domain QA datasets. |
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method (2024.naacl-long)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Recent literature reveals that Large Language Models (LLMs) hallucinate intermittently, which impedes their reliability for further utilization. |
| Approach: | They propose a self-detection method to detect which questions an LLM does not know by combining the two components to identify whether the model generates a non-factual response to the question. |
| Outcome: | The proposed method can detect which questions an LLM does not know across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks. |
Are Large Language Model Temporally Grounded? (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent large language models lack a consistent temporal model of textual narratives . sentence ordering in unlabelled texts is only weakly correlated with event ordering . |
| Approach: | They evaluate LLMs with textual narratives and evaluate their common-sense knowledge . they find that LLM models struggle the most with self-consistency . |
| Outcome: | The proposed models lack a consistent temporal model of textual narratives. |
Document Image Machine Translation with Dynamic Multi-pre-trained Models Assembling (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing TIMT tasks focus on text-line-level images. |
| Approach: | They propose to extend the existing TIMT task and introduce a new framework to translate a source document image to markdown-formatted target translation. |
| Outcome: | The proposed task aims to translate a source document image with long context and complex layout structure to markdown-formatted target translation. |
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Current-day large language models generate coherent, grammatical, and seemingly meaningful text, but are prone to hallucinating incorrect information. |
| Approach: | They propose to ‘subtract’ parameters of a model trained to hallucinate from a dialogue response generation model to ‘negate’ the contribution of such hallucinatedexamples from it. |
| Outcome: | The proposed method reduces hallucinations and discourages extractive responses, which are often a consequence of reducing hallucines by encouraging copy-pasting of document spans. |
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’ (2024.naacl-long)
Copied to clipboard
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, Tong Zhang
| Challenge: | Existing methods for instruction tuning force the model to complete a sentence no matter whether it knows the knowledge or not. |
| Approach: | They propose a new approach to tuning large language models to refrain from answering questions beyond its parametric knowledge by identifying the disparity in parametric and parametric information. |
| Outcome: | The proposed approach improves a model’s ability to answer known questions and refrain from answering unknown questions. |
Bridging the Gap between Different Vocabularies for LLM Ensemble (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to combine large language models with complete outputs have limited effectiveness . lexical gaps between different LLMs hinder dynamic correction and enhancement . |
| Approach: | They propose a method to ensemble large language models via Vocabulary Alignment (EVA) they learn mappings between vocabularies of different LLMs with overlapping tokens . |
| Outcome: | The proposed method bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. |
KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient finetuning (PEFT) are limited and only finetune a small number of parameters using limited instruction data. |
| Approach: | They propose a method that inserts an adaptation layer into an LLM to integrate embeddings of entities appearing in the input text. |
| Outcome: | The proposed method can activate parameterized knowledge in an LLM without changing its parameters or input prompts. |
Extremely Weakly-supervised Text Classification with Wordsets Mining and Sync-Denoising (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for weakly-supervised text classification use only class names as supervision . Existing approaches to classify texts without labeled data have significant flaws, including zero-shot instability and context-dependent ambiguities. |
| Approach: | They propose to use wordsets to generate pseudo-labels for unlabeled texts . they propose to train the classifier using a hybrid learning strategy called sync-denoising . |
| Outcome: | The proposed method outperforms all existing prompt and seed methods on 11 datasets by an impressive average of 8 points. |
F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to address Catastrophic Forgetting (CF) have been developed to avoid forgetting and maintain system extensibility. |
| Approach: | They propose a method to reduce Catastrophic Forgetting (CF) by decomposing feed-forward layers into discrete memory cells and ensuring robust extendability. |
| Outcome: | The proposed method achieves higher BLEU scores and almost zero forgetting while maintaining robust extendability. |
Towards Reducing Diagnostic Errors with Interpretable Risk Prediction (2024.naacl-long)
Copied to clipboard
Denis McInerney, William Dickinson, Lucy Flynn, Andrea Young, Geoffrey Young, Jan-Willem van de Meent, Byron Wallace
| Challenge: | Diagnostic errors occur because clinicians cannot easily access relevant information in EHRs. |
| Approach: | They propose a method to use LLMs to identify pieces of evidence that indicate increased or decreased risk of specific diagnoses in patient EHRs. |
| Outcome: | The proposed method reduces diagnostic errors by identifying evidence in patient EHRs . it uses a Neural Additive Model to make predictions backed by evidence at time-points where clinicians are uncertain . |
Generalizable Multilingual Hate Speech Detection on Low Resource Indian Languages using Fair Selection in Federated Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting hate speech in Indian languages with linguistic diversity and cultural nuances are undesirable and involving potential risk to their privacy. |
| Approach: | They propose a federated approach that utilizes continuous adaptation and fine-tuning to aid generalization using subsets of multilingual data. |
| Outcome: | The proposed approach outperforms the state-of-the-art models on 13 Indic datasets across five different pre-trained models. |
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering (2024.naacl-long)
Copied to clipboard
Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, Jordan Boyd-Graber
| Challenge: | a question-answering system must address pragmatic inferences to answer usefully, says a new study . human information needs are often inferred from the surface form, but answers must address the pragmatic needs of the question. |
| Approach: | They examine assumptions and implications made when mothers ask questions . they find that incorporating these inferences into QA pipelines produces more complete answers . |
| Outcome: | a study shows that incorporating inferences from questions helps to address harmful beliefs . human needs vary when asking questions, but a complete answer can address them . a QA pipeline can be more effective in addressing these needs, the study finds . |
Towards Explainability in Legal Outcome Prediction Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Current legal outcome prediction models do not explain their reasoning in the real world, but human legal actors need to understand the model’s decisions. |
| Approach: | They propose a method for identifying the precedent employed by legal outcome prediction models and a taxonomy of legal precedent to compare human judges and neural models. |
| Outcome: | The proposed model learns to predict outcomes reasonably well, but its use of precedent is unlike that of human judges. |
The steerability of large language models toward data-driven personas (2024.naacl-long)
Copied to clipboard
Junyi Li, Charith Peris, Ninareh Mehrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, Rahul Gupta
| Challenge: | Large language models generate biased responses where opinions of certain groups and populations are underrepresented. |
| Approach: | They propose a data-driven notion of persona that allows for a more nuanced understanding of different (latent) social groups present in the population. |
| Outcome: | The proposed method improves model steerability by 57% over baselines. |
CCSum: A Large-Scale and High-Quality Dataset for Abstractive News Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets for supervised news summarization contain considerable amount of noise and expensive training data. |
| Approach: | They propose a large-scale and high-quality dataset for supervised abstractive news summarization containing 1.3 million training samples. |
| Outcome: | The proposed dataset is more factual and informative than established summarization datasets. |
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to label aggregation fail to capture subjective annotations and can lead to biases. |
| Approach: | They propose annotator-aware representations for text for subjective classification tasks that involve learning representations of annotators. |
| Outcome: | The proposed model improves on metrics that assess the performance on capturing individual annotators’ perspectives. |
Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo (2024.naacl-long)
Copied to clipboard
| Challenge: | Neural Table-to-Text models produce hallucinated outputs that are factually incorrect or unrelated to the input data. |
| Approach: | They manually annotated 1,837 texts generated by multiple Neural Table-to-Text models in the politics domain of the ToTTo dataset. |
| Outcome: | The proposed model reduces factual errors by 52% to 76% . the proposed model also struggles with tabular inputs that are structured in a non-standard way, especially when the input lacks distinct row and column values or the column headers are not correctly mapped to corresponding values. |
CERET: Cost-Effective Extrinsic Refinement for Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate incomplete, biased or misleading outputs in their initial attempts. |
| Approach: | They propose a method for refining text generation that takes into account semantic stability, entailment and inter-sample uncertainty measures. |
| Outcome: | The proposed method outperforms self-consistency and self-rerank baselines under various task setups by 1.6% and 3.5% respectively. |
Parameter-Efficient Instruction Tuning of Large Language Models For Extreme Financial Numeral Labelling (2024.naacl-long)
Copied to clipboard
Subhendu Khatuya, Rajdeep Mukherjee, Akash Ghosh, Manjunath Hegde, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, Pawan Goyal
| Challenge: | Existing methods to automatically annotate relevant numerals (GAAP metrics) occurring in financial documents are not cost-effective nor scalable. |
| Approach: | They propose a generative paradigm for annotating GAAP metrics with XBRL tags using metric metadata and a parameter efficient model using LoRA. |
| Outcome: | The proposed model outperforms baseline models on two financial numeric labeling datasets and outperformed several strong baseline models. |
Analysis of State-Level Legislative Process in Enhanced Linguistic and Nationwide Network Contexts (2024.naacl-long)
Copied to clipboard
| Challenge: | a new framework for understanding state-level legislative process improves understanding of state legislation and its implications. |
| Approach: | They propose to use generative large language models to decode legislators' behavior and implications of state policies by establishing a shared nationwide network. |
| Outcome: | The framework decodes legislators’ behavior and implications of state policies by establishing a shared nationwide network enriched with diverse contexts, such as information on interest groups influencing public policy and legislators' courage test results, which reflect their political positions. |
DeMuX: Data-efficient Multilingual Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained multilingual models have enabled deployment of NLP technologies for multiple languages, but their performance under an annotation budget remains an open question. |
| Approach: | They propose a framework that prescribes the exact data-points to label from vast amounts of unlabelled multilingual data, having unknown degrees of overlap with the target set. |
| Outcome: | The proposed framework outperforms strong baselines in 84% of the test cases in the zero-shot setting of disjoint source and target language sets. |
DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | State-of-the-art rankers pre-trained on large task-specific training data such as MS-MARCO exhibit strong performance on various ranking tasks without domain adaptation, also called zero-shot. |
| Approach: | They propose a method to generate unsupervised domain adaptation for ranking using large-scale task-specific training data such as MS-MARCO and Wikipedia retrieval. |
| Outcome: | The proposed method outperforms all zero-shot baselines and significantly outperfies the SOTA baselines on 16 out of 18 datasets, for an average of 4% relative improvement across all datasets. |
How did we get here? Summarizing conversation dynamics (2024.naacl-long)
Copied to clipboard
Yilun Hua, Nicholas Chernogor, Yuzhe Gu, Seoyeon Jeong, Miranda Luo, Cristian Danescu-Niculescu-Mizil
| Challenge: | Throughout a conversation, the way participants interact with each other is in constant flux. |
| Approach: | They propose to summarize conversations by constructing human-written summaries and exploring automated baselines. |
| Outcome: | The summarizing tools help both humans and automated systems forecast toxic behavior in conversations. |
Can Language Model Moderators Improve the Health of Online Discourse? (2024.naacl-long)
Copied to clipboard
Hyundong Cho, Shuai Liu, Taiwei Shi, Darpan Jain, Basem Rizk, Yuyang Huang, Zixun Lu, Nuan Wen, Jonathan Gratch, Emilio Ferrara, Jonathan May
| Challenge: | Existing efforts to automate conversational moderation have focused on banning harmful comments or deleting them, but such efforts can inadvertently push users towards echo chambers that exacerbate polarization. |
| Approach: | They propose a framework to assess models’ moderation capabilities independently of human intervention and propose 'conversational moderation' they propose to use language models as conversational moderators to provide specific feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation. |
| Outcome: | The proposed framework assesses models’ moderation capabilities independently of human intervention and shows that appropriately prompted models provide specific and fair feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation. |
LeanReasoner: Boosting Complex Logical Reasoning with Lean (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often struggle with complex logical reasoning due to logical inconsistencies and the inherent difficulty of such reasoning. |
| Approach: | They propose a method that formalizes logical reasoning problems into theorems within Lean and then proves or disproving the corresponding theorels. |
| Outcome: | The proposed method achieves state-of-the-art performance on the FOLIO dataset and near this level on ProofWriter. |
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve UI code generation rely on expensive human feedback or distilling a proprietary model. |
| Approach: | They propose to use automated feedback to guide large language models to generate UI code . they use a large synthetic dataset to generate improved models and refine them . |
| Outcome: | The proposed model outperforms baseline models and larger proprietary models . the model outpersforms models with automated metrics and human preferences . |
Measuring Cross-lingual Transfer in Bytes (2024.naacl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models can transfer knowledge to target languages with minimal or no examples . underlying mechanisms for this transfer remain unclear, with hypotheses ranging from language contamination to syntactic similarity. |
| Approach: | They conducted an experiment to investigate whether multilingual models transfer knowledge to target languages . they found that models initialized from diverse languages perform similarly to a target language . |
| Outcome: | a new study shows that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. |
MisgenderMender: A Community-Informed Approach to Interventions for Misgendering (2024.naacl-long)
Copied to clipboard
| Challenge: | Misgendering is the act of incorrectly addressing someone’s gender and is pervasive in everyday use platforms and technologies. |
| Approach: | They propose a task and evaluation dataset to assess the effectiveness of automated misgendering interventions for text-based misgending in the US. |
| Outcome: | The proposed dataset includes 3790 instances of social media content and LLM-generations about non-cisgender public figures, annotated for the presence of misgendering, with additional annotations for correcting misgending in LLM generated text. |
Interplay of Machine Translation, Diacritics, and Diacritization (2024.naacl-long)
Copied to clipboard
| Challenge: | MT and diacritization influence performance in a multi-task learning setting, but keeping diacritics is harmful for some languages. |
| Approach: | They propose two classes of metrics to measure the complexity of a diacritical system and propose to use them to compare performance. |
| Outcome: | The proposed metrics correlate positively with the performance of the models. |
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2024.naacl-long)
Copied to clipboard
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao
| Challenge: | Large Language Models (LLMs) have revolutionized the landscape of artificial intelligence. |
| Approach: | They propose a self-guided method to identify and select cherry samples from open-source datasets, minimizing manual curation and potential cost for instruction tuning an LLM. |
| Outcome: | The proposed method enables LLMs to identify discrepancies between expected responses and intrinsic generation capability, and a marked uptick in model training efficiency. |
Safer-Instruct: Aligning Language Models with Automated Preference Data (2024.naacl-long)
Copied to clipboard
| Challenge: | annotating preference data by humans is resource-intensive and creativity-demanding . existing methods face limitations in data diversity and quality . |
| Approach: | They propose a pipeline for annotating large-scale preference data without human annotators. |
| Outcome: | The proposed pipeline outperforms models fine-tuned on human-annotated safety preference data while maintaining a competitive edge in downstream tasks. |
PELMS: Pre-training for Effective Low-Shot Multi-Document Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for abstractive multi-document summarization fail to generate concise, reflective summaries. |
| Approach: | They propose a pre-trained abstractive multi-document summarization model that uses unlabeled multi-doctoral inputs to generate concise, reflective summaries. |
| Outcome: | The proposed model outperforms competing models on a wide range of MDS datasets. |
Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) suffer from hallucinations and unfaithful reasoning due to keyword/entity biases. |
| Approach: | They propose a new probing method and benchmark to quantify this phenomenon by using a keyword/entity biases-based probing technique called EUREQA. |
| Outcome: | The proposed method achieves 62% accuracy on multi-hop and complex QA benchmarks. |
IndiSentiment140: Sentiment Analysis Dataset for Indian Languages with Emphasis on Low-Resource Languages using Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing solutions to bridge the gap between resource-rich and resource-poor languages are being explored. |
| Approach: | They examine the feasibility of machine translation for creating sentiment analysis datasets in 22 Indian languages. |
| Outcome: | The proposed dataset can be used to tackle low-resource challenges in sentiment analysis for Indian languages. |
Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval (2024.naacl-long)
Copied to clipboard
| Challenge: | et al., 2020: performance of dense retrieval models in multilingual retrieval is limited due to uneven and scarce training data available across multiple languages. |
| Approach: | They propose a synthetic retrieval training dataset containing 33 languages for fine-tuning multilingual retrievers without human supervision. |
| Outcome: | The proposed model outperforms human-supervised retrieval models on three retrieval benchmarks. |
SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities (2024.naacl-long)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task to identify textual spans that correspond to named entities in the given text. |
| Approach: | They propose a model that can generalize to entities unseen during training and handle noisy annotations. |
| Outcome: | The proposed model outperforms existing methods on both MNER and GMNER benchmarks and is robust and accurate. |
A Theory Guided Scaffolding Instruction Framework for LLM-Enabled Metaphor Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for metaphor detection and reasoning struggle to explain the underlying reasoning process behind the metaphorical/literal judgment. |
| Approach: | They propose a Theory guided Scaffolding Instruction framework that instructs an LLM to infer the underlying reasoning process of metaphor detection guided by metaphor theories for the first time. |
| Outcome: | The proposed method significantly outperforms both the LLM-based reasoning methods and the SOTA methods in metaphor detection. |
Learning to Compress Prompt in Natural Language Formats (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work rely on compressing long contexts into soft prompts, but soft prompt compression encounters limitations in transferability . natural language (NL) prompts are incompatible with back-propagation, and NL prompts lack flexibility in imposing length constraints. |
| Approach: | They propose a framework that compresses long prompts into NL formatted Capsule Prompts. |
| Outcome: | The proposed framework reduces 81.4% of the original length, decreases inference latency up to 4.5x, and saves 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets. |
Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal summarization with multimodal output (MSMO) has attracted increasing research interest . evaluation is an emerging yet underexplored research topic . |
| Approach: | They propose a framework that studies three research questions of MSMO evaluation . they propose an automatic evaluation metric and a meta-evaluation benchmark dataset . |
| Outcome: | The proposed evaluation metric and human-annotated meta-evaluation benchmark are used to assess the quality of evaluation metrics and show the framework is effective. |
Naive Bayes-based Context Extension for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising in-context learning abilities, but are often impeded by length limitations of transformer architecture. |
| Approach: | They propose a framework to enable existing Large Language Models to perform ICL with an increased number of demonstration examples by significantly expanding their context size. |
| Outcome: | The proposed framework outperforms existing methods in in-context learning with an increased number of demonstration examples while preserving linear efficiency. |
Leitner-Guided Memory Replay for Cross-lingual Continual Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Various continual learning approaches have proposed to mitigate catastrophic forgetting by restricting the data buffer or limiting the data size of a model. |
| Approach: | They propose to use a human-inspired spaced-repetition technique to prioritize examples for cross-lingual continual learning. |
| Outcome: | The proposed approach significantly and consistently decreases forgetting while maintaining accuracy across natural language understanding tasks, language orders, and languages. |
Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure (2024.naacl-long)
Copied to clipboard
| Challenge: | a number of studies have focused on making explicit the linguistic information encoded in language models (LMs) however, this method has been criticized for various reasons. |
| Approach: | They introduce a framework for creating nonce treebanks for multilingual UD corpora . they investigate word co-occurrence statistics and show how nonce data affects the performance of syntactic dependency probes. |
| Outcome: | The proposed framework satisfies syntactic argument structure and ensures grammaticality via language-specific rules. |
Actively Learn from LLMs with Uncertainty Propagation for Generalized Category Discovery (2024.naacl-long)
Copied to clipboard
| Challenge: | Generalized category discovery (GCD) is a crucial task in open-world computing, where new categories frequently emerge, necessitating models that can adapt and learn continually. |
| Approach: | They propose to integrate the feedback from LLMs into an active learning paradigm to simplify the labeling task and minimize the spread of inaccurate feedback. |
| Outcome: | The proposed approach significantly improves baseline models at a nominal average cost. |
Explaining Text Similarity in Transformer Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern foundation models provide flexible text representations that enable the detection of semantic structure in vast amounts of unlabeled data. |
| Approach: | They propose to leverage layer-wise relevance propagation to understand the inner prediction mechanisms of NLP models by analyzing grammatical interactions, multilingual semantics, and biomedical text retrieval. |
| Outcome: | The proposed methods demonstrate their utility in three corpus-level use cases, analyzing grammatical interactions, multilingual semantics, and biomedical text retrieval. |
Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for training contrastive learning based sentence embedding models are largely influenced by the quality of sentence pairs. |
| Approach: | They propose a framework that decomposes LLMs into three stages for training . they propose to refine the generated content at these stages to ensure only high-quality sentence pairs are utilized to train a base contrastive learning model. |
| Outcome: | The proposed framework surpasses ChatGPT and ChatGPP in terms of performance. |
HIL: Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in dense retrieval models have brought ColBERT to prominence in information retrieval, but it is underperforming in zero-shot tasks. |
| Approach: | They propose a Hybrid Isotropy Learning architecture that integrates isotropic and anisotropic representations to improve zero-shot retrieval performance. |
| Outcome: | The proposed model outperforms the baseline ColBERT model in BEIR benchmarks. |
SuperGLEBer: German Language Understanding Evaluation Benchmark (2024.naacl-long)
Copied to clipboard
| Challenge: | a new set of German-pretrained models are being released, but no established, diverse and systematic evaluation suite is available for them. |
| Approach: | They assemble a Natural Language Understanding benchmark suite for the German language and evaluate 10 existing German-pretrained models. |
| Outcome: | The proposed benchmark suite evaluates 10 German-pretrained models on 29 tasks . the results show that encoder models are good choices for most tasks, but not all . |
“You are an expert annotator”: Automatic Best–Worst-Scaling Annotations for Emotion Intensity Modeling (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models mitigate the issue with automatic corpus labeling methods, but there is no work on automating annotations for continuous labels. |
| Approach: | They propose to use a transformer regressor to automate emotion intensity predictions and compare rating scale predictions with best–worst scaling. |
| Outcome: | The proposed method performs better on rating scale annotation tasks than on comparative annotation tasks. |
What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? (2024.naacl-long)
Copied to clipboard
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, Ruihua Song
| Challenge: | Recent advances in GPT-4V have demonstrated remarkable multi-modal capabilities in processing image inputs and following open-ended instructions. |
| Approach: | They propose a plug-and-play technique to enhance multi-modal LLMs . they propose 'lynx' to train multi-modal LLM models . |
| Outcome: | The proposed training strategy improves understanding accuracy and instruction-following proficiency of multi-modal models. |
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. |
| Approach: | They propose to collect annotated human evaluation guidelines and a method for detecting guideline vulnerabilities using Large Language Models. |
| Outcome: | The proposed dataset includes eight vulnerabilities and a method for detecting guideline vulnerabilities. |
MOSAICo: a Multilingual Open-text Semantically Annotated Interlinked Corpus (2024.naacl-long)
Copied to clipboard
Simone Conia, Edoardo Barba, Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Riccardo Orlando, Luigi Procopio, Roberto Navigli
| Challenge: | Existing approaches to integrate semantics into Natural Language Understanding (NLP) systems are cost-effective and environmental impact-related. |
| Approach: | They propose to provide semantically-annotated corpora for four NLU tasks across five languages and to drop the requirement of closed datasets. |
| Outcome: | The proposed model provides hundreds of millions of silver yet high-quality annotations for four NLU tasks across five languages. |
SemRoDe: Macro Adversarial Training to Learn Representations that are Robust to Word-Level Attacks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to defend against word-level attacks have been limited. |
| Approach: | They propose a new approach called Semantic Robust Defence to enhance the robustness of language models by aligning the domains with a distance-based objective. |
| Outcome: | The proposed approach can be generalized across word embeddings, even when they share minimal overlap at both vocabulary and word-substitution levels. |
BUST: Benchmark for the evaluation of detectors of LLM-Generated Text (2024.naacl-long)
Copied to clipboard
| Challenge: | Using the benchmark, we evaluated 5 detectors and found substantial performance variance across tasks. |
| Approach: | They propose to evaluate detectors of texts generated by instruction-tuned large language models (LLMs) using a benchmark dataset, they evaluated 5 detectors and found substantial performance variance across tasks. |
| Outcome: | The proposed benchmarks evaluated 5 detectors and found substantial performance variance across tasks. |
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that multilingual generative models exhibit a strong language bias toward high-resource languages. |
| Approach: | They propose a cross-lingual alignment framework exploiting pairs of translation sentences to improve cross-linguistic abilities. |
| Outcome: | The proposed framework improves cross-lingual abilities and mitigates performance gap. |
MaCSC: Towards Multimodal-augmented Pre-trained Language Models via Conceptual Prototypes and Self-balancing Calibration (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to training pre-trained language models (PLMs) focus on static image modality; inevitably encounter modality gaps and noise; and treat all modalities. |
| Approach: | They propose a multimodal-augmented framework that can infuse multimodal semantics into PLMs and facilitate a self-balancing calibration of information allocation. |
| Outcome: | The proposed framework outperforms baselines on multiple NLP tasks and outperformed existing frameworks. |
Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? (2024.naacl-long)
Copied to clipboard
| Challenge: | Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities . traditional embedding-based methods infer missing links using only training data . a pre-trained language model (PLM)-based KGC may be ineffective in practical applications . |
| Approach: | They propose to use knowledge Graph Completion (KGC) to infer unseen relationships . traditional embedding-based KGC methods infer missing links only from training data . they argue that pre-trained language models acquire inference abilities through pre-training . |
| Outcome: | The proposed method improves performance even though it does not use memorized knowledge. |
Discovering Lobby-Parliamentarian Alignments through NLP (2024.naacl-long)
Copied to clipboard
Aswin Suresh, Lazar Radojević, Francesco Salvi, Antoine Magron, Victor Kristof, Matthias Grossglauser
| Challenge: | Influence of interest groups on parliamentarians and subversion of electorate to determine policy has led to demands from groups such as Transparency International . |
| Approach: | They collect datasets of lobbies’ position papers and MEPs’ speeches and compare them on the basis of semantic similarity and entailment. |
| Outcome: | The proposed method performs significantly better than baselines and matches the public meetings of MEPs with retweet links. |
IterCQR: Iterative Conversational Query Reformulation with Retrieval Guidance (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for conversational query reformulation depend on human annotations. |
| Approach: | They propose a method that reformulates context-dependent conversational queries without relying on human rewrites. |
| Outcome: | The proposed method shows state-of-the-art performance on two widely-used datasets. |
AceGPT, Localizing Large Language Models in Arabic (2024.naacl-long)
Copied to clipboard
Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu
| Challenge: | Significant concerns emerge when addressing cultural sensitivity and local values. |
| Approach: | They propose a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. |
| Outcome: | The proposed model sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. |
Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve translation quality using human feedback have not been validated. |
| Approach: | They propose to use quality estimation to predict human preferences for feedback training . they propose to detect incorrect translations and assign a penalty term to the reward scores . |
| Outcome: | The proposed method outperforms systems using larger parallel corpora by a small amount of monolingual data. |
Depression Detection in Clinical Interviews with LLM-Empowered Structural Element Graph (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for assessing depression only capture part of relevant elements . scarcity of participant data constrains interview modeling due to privacy concerns . |
| Approach: | They propose a structural element graph (SEGA) that transforms clinical interviews into an expertise-inspired directed acyclic graph for comprehensive modeling. |
| Outcome: | The proposed model outperforms baseline methods and powerful LLMs on two real-world clinical datasets. |
SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU (2024.naacl-long)
Copied to clipboard
| Challenge: | Task-oriented dialogue (TOD) systems support users in execution of specific, well-defined tasks through natural language interaction. |
| Approach: | They propose a framework for dialog NLU based on instruction tuning and question-answering-based formulation of ID and VE tasks. |
| Outcome: | The proposed framework surpasses existing models in training and cross-domain transfer and significantly outperforms existing large language models in performance and inference efficiency. |
Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing arguments are incapable of distinguishing between generated key points of different qualities. |
| Approach: | They propose an extractive approach that generates concise, high quality key points . they propose to use a clustering approach to generate key points from raw arguments . |
| Outcome: | The proposed method outperforms state-of-the-art methods for key point generation . it offers concise, high quality generated key points with higher coverage of reference summaries . |
ARM: Alignment with Residual Energy-Based Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) acquire a wide range of abilities and abilities, but their behavior does not align with human preferences. |
| Approach: | They propose to minimize a forward Kullback–Leibler divergence from a target policy to a parameteric policy instead of a reverse KL as in RLHF methods. |
| Outcome: | The proposed method can learn an aligned policy by minimizing a forward Kullback–Leibler divergence from a target policy to a parameteric policy instead of a reverse KL as in RLHF methods. |
HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants (2024.naacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) are popular conversational assistants, but evaluation of such models is not scalable. |
| Approach: | They propose a task that performs automatic evaluation using human judgement and a large-scale set of questions with multiple answers authored and scored by humans. |
| Outcome: | The proposed task performs well with human judgements and is particularly responsive to model changes following instruction-tuning. |
FAMuS: Frames Across Multiple Sources (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work in document-level event and argument extraction tasks suffer from two key shortcomings. |
| Approach: | They propose to combine Wikipedia passages with underlying, genre-diverse source articles for an event . they propose two key task enabled by FAMuS: source validation and cross-document argument extraction . |
| Outcome: | The proposed system can extract event arguments from document and report documents. |
Rationale-based Opinion Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to generate concise summaries of reviews are generic and lack supporting details. |
| Approach: | They propose a rationale-based opinion summarization paradigm that outputs representative opinions and corresponding rationales. |
| Outcome: | The proposed method is more useful than conventional summarizations. |
Mustango: Toward Controllable Text-to-Music Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Mustango is a text-to-music system that allows music-domain-knowledge-informed text-based music generation. |
| Approach: | They propose a music-domain-knowledge-inspired text-to-music system based on diffusion that generates music with captions that include specific instructions related to chords, beats, key and tempo. |
| Outcome: | The proposed system outperforms existing models in music generation tasks. |
Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations (2024.naacl-long)
Copied to clipboard
| Challenge: | Zero-Shot Cross-lingual transfer (ZS-XLT) uses a model trained in a source language to make predictions in another language, often with a performance loss. |
| Approach: | They propose a new approach that uses In-Context Tuning to train a model to learn from context examples and adapt it to a target language by prepending a One-Shot context demonstration. |
| Outcome: | The proposed approach outperforms prompt-based models in Zero-Shot and Few-shot scenarios with target-language examples. |
CNER: Concept and Named Entity Recognition (2024.naacl-long)
Copied to clipboard
| Challenge: | Concept and Named Entity Recognition (CNER) is a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Approach: | They propose a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Outcome: | The proposed task gains +5.4 and +8 macro F1 points when performed as a unified task compared to specialized named entity and concept recognition systems. |
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often used for multi-faceted language generation and evaluation tasks that require complex user constraints or taking into account multiple aspects and criteria. |
| Approach: | They propose a Large Language Model program that consists of branch, solve, and merge modules that are parameterized with specific prompts to the base LLM. |
| Outcome: | The proposed program improves the evaluation correctness and consistency for each LLM by up to 26%, reducing length and pairwise position biases by up 50%, and allowing LLaMA-2-chat to match or outperform GPT-4 on most domains. |
REPLUG: Retrieval-Augmented Black-Box Language Models (2024.naacl-long)
Copied to clipboard
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
| Challenge: | Existing retrieval-augmented language models require access to internal representations to enhance performance. |
| Approach: | They introduce a retrieval-augmented language modeling framework that treats the language model as a black box and augments it with a tuneable retrieval model. |
| Outcome: | The proposed framework improves performance on language modeling tasks by 6.3% and 5.1%. |
David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies of diffusion-based language models have been conducted on a smaller scale. |
| Approach: | They propose to scale an autoregressive diffusion model from 0.4B to 13B parameters and propose techniques to improve its training and inference efficiency. |
| Outcome: | The proposed model is able to combine a large general-purpose diffusion model with smaller, but specialized and contextualized diffusion models at inference time. |
Efficient End-to-End Visual Document Understanding with Rationale Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-processing tools such as optical character recognition (OCR) can map document image inputs to textual tokens, then large language models (LLMs) can reason over text. |
| Approach: | They propose a method that integrates outputs of OCR tools and larger multimodal models as intermediate "rationales" a student model is trained to predict rationales and answers based on visual documents . |
| Outcome: | The proposed model outperforms the base model on three visual document understanding benchmarks with only 1% higher computational cost. |
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Psychologists have documented several ways in which humans’ inferences deviate from the rules of logic. |
| Approach: | They focus on syllogisms, which are inferences from two simple premises, and show that larger models are more logical than smaller ones. |
| Outcome: | The results show that language models often mimic human biases, but overcome them in some cases. |
AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets (2024.naacl-long)
Copied to clipboard
| Challenge: | Standard pool-based active learning is computationally expensive on large pools and often reaches low accuracy by overfitting the initial decision boundary. |
| Approach: | They propose a pool-based active learning method that selects class-specific instances from a labelled set and retrieves the most similar unlabelled instances from the pool. |
| Outcome: | Experiments with AnchorAL show that it is faster, often reducing runtime from hours to minutes, and trains more performant models. |
ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in automated essay scoring have limited the generalizability of models trained on ASAP. |
| Approach: | They propose to annotate persuasive student essays with holistic and trait-specific scores in a corpus of persuasive student essay annotated with ICLE++. |
| Outcome: | The proposed model can be used to evaluate models for newer AES problems such as multi-trait scoring and cross-prompt scoring. |
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations (2024.naacl-long)
Copied to clipboard
Wenting Zhao, Justin Chiu, Jena Hwang, Faeze Brahman, Jack Hessel, Sanjiban Choudhury, Yejin Choi, Xiang Li, Alane Suhr
| Challenge: | Existing work evaluating commonsense reasoning focuses on making inferences about common, everyday situations. |
| Approach: | They propose to use an English language corpus to investigate commonsense reasoning . they characterize performance differences between human explainers and best-performing large language models . |
| Outcome: | The proposed method reduces the loss rate of human-written explanations on commonsense reasoning compared with the vanilla supervised fine-tuning approach . |
To Tell The Truth: Language of Deception and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evidence of people’s ability to discern truth from text-based false information is scarce. |
| Approach: | They propose to use a large language model to learn discernible cues from TV game show data to investigate whether textual cue is more likely to detect fraud . |
| Outcome: | The proposed model detects novel but accurate language cues in many cases where humans failed to detect deception. |
Multilingual Models for ASR in Chibchan Languages (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing algorithms for low resource-intensive languages are not available for these languages . a paper comparing the performance of different models and algorithms for these extremely low resource languages is presented. |
| Approach: | They propose to fine-tune four ASR algorithms to create monolingual models for Bribri and Cabécar . they then use the best performing algorithm to train joint and transfer learning models for both languages . |
| Outcome: | The proposed algorithms are effective in both Bribri and Cabécar, but especially in Bribri. |
LegalDiscourse: Interpreting When Laws Apply and To Whom (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in NLP and information retrieval have already enabled practical applications. |
| Approach: | They propose a 'discourse' taxonomy for span-and-relation parsing of legal texts . they use a dataset of 602 state-level law paragraphs with 3,715 discourse spans and 1,671 relations to investigate the increase in liquor licenses and decrease in applicable laws. |
| Outcome: | The proposed model performs poorly at span identification and relation classification, but lags far below human level. |
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024.naacl-long)
Copied to clipboard
| Challenge: | X-Eval is a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Approach: | They introduce a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Outcome: | The proposed framework improves the model’s ability to follow evaluation instructions and enhances the learning stage to better assess text quality. |
Is Reference Necessary in the Evaluation of NLG Systems? When and Where? (2024.naacl-long)
Copied to clipboard
| Challenge: | Despite recent advances in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. |
| Approach: | They propose to use reference-free metrics to evaluate NLG systems . they find they have a higher correlation with human judgment and greater sensitivity to deficiencies in language quality . |
| Outcome: | The proposed metrics exhibit higher correlation with human judgment and greater sensitivity to deficiencies in language quality. |
Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing prompting methods rely on only one or two of these sources, or require repeatedly invoking large language models to generate similar or identical content. |
| Approach: | They propose a semi-structured prompting approach that integrates parametric memory with unstructured knowledge from text documents and structured knowledge from knowledge graphs. |
| Outcome: | The proposed prompting method surpasses existing prompting methods even exceeding those that require fine-tuning on open-domain multi-hop question answering datasets. |
Evaluating the Deductive Competence of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models have limited abilities to solve deductive reasoning problems . performance differences between conditions do not improve overall performance . |
| Approach: | They investigate whether several large language models can solve a deductive reasoning problem in their conventional form. |
| Outcome: | The proposed models can solve a classic type of deductive reasoning problem in their conventional form. |
Large Human Language Models: A Need and the Challenges (2024.naacl-long)
Copied to clipboard
| Challenge: | a growing recognition of the importance of modeling human and social factors into human-centered NLP models . authors advocate for three positions toward creating large human language models based on psychological and behavioral sciences . |
| Approach: | et al. advocate for three positions toward creating large human language models . they argue that LM training should include the human context and recognize that people are more than their group . |
| Outcome: | a new study shows that learning language from linguistic signals alone is not adequate, according to a recent paper . authors advocate for three positions toward creating large human language models . a human-centered model should include the human context, and account for the dynamic nature of the human environment, they say . |
On Learning to Summarize with Large Language Models as References (2024.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan
| Challenge: | Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets. |
| Approach: | They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved. |
| Outcome: | The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance. |
Hallucination Diversity-Aware Active Learning for Text Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for alleviating hallucinations require costly human annotations . Existing approaches focus on a specific type of hallucinism, which limits their effectiveness . |
| Approach: | They propose a method to detect hallucinations from errors in semantic frame, discourse and content verifiability in LLM summarization using HAllucination Diversity-Aware Sampling. |
| Outcome: | The proposed framework reduces the need for costly human annotations to correct hallucinations in LLM outputs. |
Keep it Private: Unsupervised Privatization of Online Text (2024.naacl-long)
Copied to clipboard
| Challenge: | Authorship obfuscation has been evaluated in narrow settings in the NLP literature . superficial edit operations can lead to unnatural outputs, authors say . |
| Approach: | They propose an automatic text privatization framework that fine-tunes a large language model via reinforcement learning to produce rewrites that balance soundness, sense, and privacy. |
| Outcome: | The proposed method maintains high text quality according to automated metrics and human evaluation, and successfully evades several automated authorship attacks. |
Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (2024.naacl-long)
Copied to clipboard
| Challenge: | a new paradigm for low-rank Adaptation (LoRA) uses weight tying and selective training to improve parameter efficiency. |
| Approach: | They propose a paradigm that uses weight tying and selective training to enhance parameter efficiency of Low-rank Adaptation. |
| Outcome: | The proposed paradigm achieves comparable performance to LoRA with reduced model complexity . the proposed paradigm can be used for a variety of tasks and languages . |
Investigating Data Contamination in Modern Benchmarks for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for large language models are inflated and inconsistent with actual performance. |
| Approach: | They propose a retrieval-based system to explore potential overlaps between benchmarks and pretraining corpora and a protocol to investigate testset slot guessing. |
| Outcome: | The proposed method exploits overlaps between evaluation benchmarks and pretraining corpora and masks a wrong answer in a multiple choice question and prompts the model to fill in the gap. |
Pre-trained Language Models for Entity Blocking: A Reproducibility Study (2024.naacl-long)
Copied to clipboard
| Challenge: | Entity Resolution (ER) is an essential task in data integration . state-of-the-art neural IR models that are based on large language models have not been evaluated on the ER task. |
| Approach: | They evaluate state-of-the-art neural IR models that are based on large language models on a wide range of real-world datasets and evaluate their generalization abilities. |
| Outcome: | The proposed methods have been evaluated on a wide range of datasets and their generalization abilities. |
RE2: Region-Aware Relation Extraction from Visually Rich Documents (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on relation extraction from visually rich documents focus on layout structure and Optical Character Recognition (OCR) results. |
| Approach: | They propose a relation extraction tool that leverages layout structure among entity blocks to improve relation prediction. |
| Outcome: | The proposed model outperforms existing models on a wide range of domains and languages. |
Mix-Initiative Response Generation with Dynamic Prefix Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue systems focus on training a holistic response generation model without any distinction between different initiatives. |
| Approach: | They propose a general mix-Initiative Dynamic Prefix Tuning framework to decouple different initiatives from the generation model. |
| Outcome: | The proposed framework outperforms baselines on two public dialogue datasets on human evaluations and automatic metrics. |
Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Value (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work specifies values as risk criteria formulated in the AI community, e.g., fairness and privacy protection, suffering from poor clarity, adaptability and transparency. |
| Approach: | They propose a value alignment paradigm based on Schwartz's Theory of Basic Values as an instantiation and propose 'BaseAlign' to support this paradigm. |
| Outcome: | The proposed model covers existing risks and anticipates unidentified ones with a low-data set. |
IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context (2024.naacl-long)
Copied to clipboard
Nihar Sahoo, Pranamya Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, Pushpak Bhattacharyya
| Challenge: | Existing benchmark datasets focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India’s unique socio-cultural nuances. |
| Approach: | They propose to use CrowS-Pairs to create a benchmark dataset that captures and evaluates social biases in Large Language Models (LLMs). |
| Outcome: | The proposed dataset is available in English and Hindi and leverages LLMs ChatGPT and InstructGPT to augment the existing dataset with diverse societal biases and stereotypes prevalent in India. |