Findings of the Association for Computational Linguistics: NAACL 2024
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
Weight-Inherited Distillation for Task-Agnostic BERT Compression (2024.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a predominant approach for BERT compression. |
| Approach: | They propose a weight-inherited distillation method which directly transfers knowledge from the teacher to a compact student model by inheriting the weights. |
| Outcome: | The proposed method outperforms state-of-the-art KD-based methods on GLUE and SQUAD benchmarks. |
Ignore Me But Don’t Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train domain expertise for cybersecurity text domains are expensive to train and run. |
| Approach: | They propose to use pretraining methods to account for non-linguistic elements in cybersecurity texts and evaluate their effectiveness through downstream tasks and probing tasks. |
| Outcome: | The proposed strategy outperforms the commonly taken approach of replacing NLEs and outperformed other cybersecurity PLMs on most tasks. |
Extremely efficient online query encoding for dense retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing dense retrieval systems use the same model architecture for encoding both passages and queries, even though queries are much shorter and simpler than passages. |
| Approach: | They propose a small efficient RNN query encoder that can reduce latency by 12 with only a minor decrease in quality. |
| Outcome: | The proposed solution reduces latency by up to 12 while achieving 35.5 MRR@10 score. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
SpeedE: Euclidean Geometric Knowledge Graph Embedding Strikes Back (2024.findings-naacl)
Copied to clipboard
| Challenge: | Geometric knowledge graph embedding models (gKGEs) have shown great potential for knowledge graph completion (KGC) however, contemporary gKges require high embeddable dimensionalities or complex embeddances for good KGC performance, drastically limiting their time and space efficiency. |
| Approach: | They propose a lightweight Euclidean gKGE that provides strong inference capabilities and significantly outperforms state-of-the-art gGKGEs. |
| Outcome: | The proposed model outperforms state-of-the-art gKGEs on YAGO3-10 and WN18RR while significantly increasing their efficiency. |
Language Guided Exploration for RL Agents in Text Environments (2024.findings-naacl)
Copied to clipboard
| Challenge: | Real-world sequential decision making is characterized by sparse rewards and large decision spaces. |
| Approach: | They introduce a language-based framework that provides decision-level guidance to an RL agent. |
| Outcome: | The proposed framework outperforms vanilla RL agents on ScienceWorld in 2022. |
GPT-who: An Information Density-based Machine-Generated Text Detector (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate misinformation, memorized content, plagiarized content, toxic speech, and hallucinated content. |
| Approach: | They propose a statistical detector that uses UID to model the unique statistical signature of each LLM and human author for accurate detection. |
| Outcome: | The proposed method outperforms state-of-the-art detectors by over 20% across domains. |
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Encoder-decoder transformer models suffer from high inference latency due to auto-regressive decoding . Typically, the decoder takes up most of the latency because of the auto-decoding - a problem that is not solved by the current model. |
| Approach: | They propose an approach to perform Dynamic Early Exit on Decoder to reduce inference latency by 20%-74% by using a multi-exit encoder-decoder transformer model trained with deep supervision. |
| Outcome: | The proposed model reduces inference latency by 20%-74% with comparable or even higher accuracy compared to baseline models. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Automatic Pair Construction for Contrastive Post-training (2024.findings-naacl)
Copied to clipboard
Canwen Xu, Corby Rosset, Ethan Chau, Luciano Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Awadallah, Nikhil Rao
| Challenge: | Large language models (LLMs) have unprecedented proficiency in a wide array of tasks. |
| Approach: | They propose a way to construct contrastive data using preference pairs from multiple models of varying strengths using SLiC and DPO. |
| Outcome: | The proposed method outperforms existing models like Orca in the comparison of SLiC and DPO with SFT baselines. |
Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for fact-checking text generated by large language models are expensive and time-consuming. |
| Approach: | They propose a plug-and-play framework that harnesses large language models for efficient fact-checking in a few-shot manner. |
| Outcome: | The proposed framework is compared with state-of-the-art models and shows that it can be used to speed up fact-checking in a few-shot manner. |
Low-resource neural machine translation with morphological modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for character-based and sub-word tokenization are limited to the surface forms of the words. |
| Approach: | They propose a framework-solution for modeling complex morphology in low-resource settings using a transformer architecture and beam search-based decoder. |
| Outcome: | The proposed model improves translation performance on Kinyarwanda English translation using public-domain parallel text. |
Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train named entity recognition models on noisy data are expensive and time-intensive to accumulate. |
| Approach: | They propose to denoise noisy NER data with guidance from a small set of clean instances. |
| Outcome: | The proposed method can improve on large-scale datasets with a small guidance set. |
VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | a lack of standard evaluation metrics and benchmarks makes it difficult to identify strengths of Vietnamese NLP models. |
| Approach: | They propose to establish a standardized set of benchmarks for Vietnamese NLU . they propose to evaluate Vietnamese language understanding models using a pre-trained model . |
| Outcome: | The proposed model combines proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. |
LETI: Learning to Generate from Textual Interactions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing techniques fine-tune on input-output pairs or with numerical rewards that gauge the output quality are not effective. |
| Approach: | They propose to fine-tune pre-trained language models with binary labels and a Python interpreter to get textual feedback from the inputs. |
| Outcome: | The proposed model outperforms the base model on unseen problems and achieves comparable or better performance on humanEval. |
Bilateral Masking with prompt for Knowledge Graph Completion (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing word matching methods fail to obtain satisfactory single embedding representations for entities. |
| Approach: | They propose a bi-encoder-based approach to enhance entity representations by using prompts to narrow the distance between the predicted entity and the known entity. |
| Outcome: | The proposed model achieves state-of-the-art performance on the WN18RR dataset. |
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing generative language models neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. |
| Approach: | They propose a function to mitigate the imbalance between frequent and infrequent tokens . authors propose 'MiLe Loss' function to assess learning difficulty of tokens during training . |
| Outcome: | Experiments show that models with proposed model can improve on downstream benchmarks. |
GOLD: Geometry Problem Solver with Natural Language Description (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for solving geometry math problems struggle with accurately interpreting geometry diagrams, posing a challenge for problem-solving. |
| Approach: | They propose a model that extracts geometric relations from diagrams and converts them into natural language descriptions. |
| Outcome: | The proposed model outperforms the previous best method on the UniGeo dataset by 12.7% and 42.1% in calculation and proving subsets. |
RoDia: A New Dataset for Romanian Dialect Identification from Speech (2024.findings-naacl)
Copied to clipboard
| Challenge: | a dataset for Romanian dialect identification from speech is released . the dataset includes speech samples from five distinct regions of Romania . |
| Approach: | They propose a dataset for Romanian dialect identification from speech . they propose competitive models to be used as baselines for future research . |
| Outcome: | The first dataset for Romanian dialect identification from speech is released . the top scoring model achieves 59.83% and 62.08%, respectively . |
Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. |
| Approach: | They propose to explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning on per-language subnetworks to better guide cross-lingual sharing. |
| Outcome: | The proposed approach can increase language specialization of subnetworks in favor of more cross-lingual sharing. |
Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to enable large language models to implement function calling are limited in their tool-use capabilities. |
| Approach: | They propose a controllable, target-driven approach to empower LLMs to operate external APIs only via prompts. |
| Outcome: | The proposed approach limits LLMs to executing simple tasks, e.g., API Selection and Argument Completion. |
Incorporating Exponential Smoothing into MLP: a Simple but Effective Sequence Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Structured State Space models (SSMs) have been used for long-range sequence learning but are limited in their complexity and computational and memory requirements. |
| Approach: | They propose to incorporate a simple SSM into an element-wise MLP to reduce inductive bias. |
| Outcome: | The proposed model achieves comparable results to existing models on the LRA benchmark. |
OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for object navigation are limited to household datasets with close-set objects, and they lack the ability to generalize to new environments in a zero-shot manner. |
| Approach: | They propose a framework that leverages reasoning abilities of large vision language models to extract proposed objects from natural language instructions that meet the user’s demand. |
| Outcome: | The proposed framework surpasses baselines on all metrics and can be used in a HM3D ObjectNav benchmark. |
Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency? (2024.findings-naacl)
Copied to clipboard
| Challenge: | a clinical note is a document that documents a doctor's interaction with a patient . authors show that LLMs can be used to measure quality indicators . |
| Approach: | They analyze two different approaches to generate different sections of a SOAP note . they use PEGASUS-X Transformer models to examine note consistency . |
| Outcome: | The proposed approach leads to similar ROUGE values and no difference in Factuality metric . human reviewers perform the same tasks with roughly the same agreement as the LLMs . |
VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder (2024.findings-naacl)
Copied to clipboard
| Challenge: | generative diversity is a critical yet underexplored issue in natural language generation . previous approaches to enhance diversity of Transformer models have been limited by their latent variables . |
| Approach: | They propose a framework that bridges Transformer with VAE to enhance generative diversity. |
| Outcome: | The proposed framework improves generative diversity while maintaining generative quality. |
EcoSpeak: Cost-Efficient Bias Mitigation for Partially Cross-Lingual Speaker Verification (2024.findings-naacl)
Copied to clipboard
| Challenge: | Linguistic bias is a critical problem that harms the diversity, equity, and inclusiveness of Natural Language Processing tools. |
| Approach: | They propose a low-cost solution that incorporates contrastive linguistic attention to emphasize relevant speaker verification embedding parts. |
| Outcome: | The proposed model is able to mitigate linguistic bias in five partially cross-lingual scenarios. |
Leveraging Contextual Information for Effective Entity Salience Detection (2024.findings-naacl)
Copied to clipboard
Rajarshi Bhowmik, Marco Ponza, Atharva Tendle, Anant Gupta, Rebecca Jiang, Xingyu Lu, Qian Zhao, Daniel Preotiuc-Pietro
| Challenge: | Prior work on salient entity detection focused on machine learning models that require heavy feature engineering. |
| Approach: | They propose to fine-tune medium-sized language models with a cross-encoder style architecture to achieve significant performance gains over feature engineering approaches. |
| Outcome: | The proposed model fine-tunes medium-sized pre-trained language models with a cross-encoder style architecture yields substantial performance gains over feature engineering approaches. |
LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected? (2024.findings-naacl)
Copied to clipboard
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, Lichao Sun
| Challenge: | Current research focuses on purely MGT detection without adequately addressing mixed scenarios including AI-revised Human-Written Text (HWT) and human-revealed MGT. |
| Approach: | They define mixtext, a form of mixed text involving both AI and human-generated content, and then use a MixSet dataset to assess their effectiveness. |
| Outcome: | The proposed detectors struggle to identify mixtext, particularly in dealing with subtle modifications and style adaptability. |
A (More) Realistic Evaluation Setup for Generalisation of Community Models on Malicious Content Detection (2024.findings-naacl)
Copied to clipboard
| Challenge: | despite the performance of community models for malicious content detection, misinformation and hate speech continue to propagate on social media networks. |
| Approach: | They propose a new evaluation setup for community models for malicious content detection based on a few-shot subgraph sampling approach to test generalisation of models using local explorations of a larger graph. |
| Outcome: | The proposed evaluation setup outperforms existing models on real-world graphs on a training graph. |
Citation: A Key to Building Responsible and Accountable Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) bring transformative benefits alongside unique challenges, including intellectual property (IP) and ethical concerns. |
| Approach: | They propose a new approach to mitigate intellectual property and ethical risks associated with large language models. |
| Outcome: | The proposed approach could enhance content transparency and verifiability . it should account for both non-parametric and parametric content . |
Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on syntactic injection in Variational AutoEncoders (VAEs) are limited to LSTM-based VAEs. |
| Approach: | They propose to use latent space separation techniques to inject syntactic information into Variational AutoEncoders (VAEs) using graph-based models. |
| Outcome: | The proposed end-to-end VAE architecture can improve the organisation of the latent space, alleviating the information loss occurring in standard VAE setups, and resulting in enhanced performances on language modelling and downstream generation tasks. |
Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles (2024.findings-naacl)
Copied to clipboard
Weiting Tan, Haoran Xu, Lingfeng Shen, Shuyue Stella Li, Kenton Murray, Philipp Koehn, Benjamin Van Durme, Yunmo Chen
| Challenge: | Recent work shows that large language models can generalize to machine translation using zero-shot examples with in-context learning. |
| Approach: | They investigate the factors contributing to this gap by matching the writing styles of the target corpus. |
| Outcome: | The proposed methods can be enhanced without the need for parallel demonstration examples. |
Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current research typically employs limited setups with small real-world graphs. |
| Approach: | They propose a new approach to encoding a graph with diverse modalities, such as text, image, and motif, coupled with prompts to approximate a diagram’s global connectivity. |
| Outcome: | The proposed approach improves performance of LLMs in graph structure analysis by focusing on homophily, motif presence, and graph difficulty. |
On-the-Fly Fusion of Large Language Models and Machine Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | a weaker-at-translation LLM can improve translations of a NMT model, compared to a strong dedicated model. |
| Approach: | They propose to ensemble a neural machine translation model with a large language model, prompted on the same task and input. |
| Outcome: | The proposed method can be combined with various techniques from LLM prompting, such as in context learning and translation context. |
READ: Improving Relation Extraction from an ADversarial Perspective (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work in relation extraction (RE) has high generalization capability, but adversarial training methods rely on entities. |
| Approach: | They propose an adversarial training method specifically designed for relation extraction that introduces sequence- and token-level perturbations to the sample and uses a separate perturbation vocabulary to improve the search for entity and context perturbations. |
| Outcome: | The proposed method significantly improves accuracy and robustness in low-resource scenarios. |
REQUAL-LM: Reliability and Equity through Aggregation in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new method for finding reliable and equitable LLM outputs is developed . REQUAL-LM does not require specialized hardware and does not impose a significant computing load . |
| Approach: | They propose a method for finding reliable and equitable LLM outputs through aggregation. |
| Outcome: | The proposed method minimizes harmful bias while finding reliable outputs . it does not require specialized hardware and does not impose a significant computing load . |
Addressing Both Statistical and Causal Gender Fairness in NLP Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Statistical fairness stipulates equivalent outcomes for all protected groups, whereas causal fairness prescribes that a model makes the same prediction for an individual regardless of their protected characteristics. |
| Approach: | They propose to use statistical and causal debiasing methods to reduce gender bias in NLP models. |
| Outcome: | The proposed methods reduce gender bias measured by the targeted metric, but not on other bias metrics. |
LLM-Rec: Personalized Recommendation via Prompting Large Language Models (2024.findings-naacl)
Copied to clipboard
Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Chris Leung, Jiajie Tang, Jiebo Luo
| Challenge: | Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. |
| Approach: | They propose a novel approach which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations. |
| Outcome: | The proposed approach improves recommendation quality and even basic MLP models achieve comparable or even better results than complex content-based methods. |
A Robust Semantics-based Watermark for Large Language Model against Paraphrasing (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to detect LLM-generated content use simple hashes of precedent tokens to partition vocabulary. |
| Approach: | They propose a semantics-based watermark framework to enhance the robustness against paraphrase. |
| Outcome: | The proposed framework is robust under different paraphrases and the semantic meaning of the sentences will be likely preserved under paraphrase. |
Solving Data-centric Tasks using Large Language Models (2024.findings-naacl)
Copied to clipboard
Shraddha Barke, Christian Poelitz, Carina Negreanu, Benjamin Zorn, José Cambronero, Andrew Gordon, Vu Le, Elnaz Nouri, Nadia Polikarpova, Advait Sarkar, Brian Slininger, Neil Toronto, Jack Williams
| Challenge: | Large language models are increasingly useful for data-centric tasks, but how do we decide how much data to include in the prompt? |
| Approach: | They propose a cluster-then-select prompting technique that adds the most representative rows from the input data to the LLM prompt. |
| Outcome: | The proposed technique outperforms a baseline for tasks with syntactic variation in the input table. |
A Novel Paradigm Boosting Translation Capabilities of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on LLMs focused on supervised fine-tuning but their effectiveness has been limited. |
| Approach: | They propose a paradigm consisting of three stages: Secondary Pre-training using extensive monolingual data, Continual Pre- training with interlinear text format documents, and Leveraging source-language consistent instruction for supervised fine-tuning. |
| Outcome: | The proposed approach surpasses previous work and achieves superior performance compared to models such as NLLB-54B(CITATION) and GPT3.5-text-davinci-003. |
Measuring Social Norms of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing datasets that evaluate a general understanding of social science are inadequate to understand social norms. |
| Approach: | They propose a multi-agent framework to improve large language models’ ability to understand social norms by comparing them to elementary students. |
| Outcome: | The proposed framework improves large language models to be on par with humans. |
Source-Free Unsupervised Domain Adaptation for Question Answering via Prompt-Assisted Self-learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing SFDA methods focus on the adaptation phase, overlooking the impact of source domain training on model generalizability. |
| Approach: | They propose a source-free domain adaptation approach for Question Answering where a model trained on a domain is adapted to unlabeled target domains without additional source data. |
| Outcome: | The proposed model outperforms existing methods in managing domain gaps and demonstrating greater stability across target domains. |
Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Level (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for document summarization focus on one type of relation, neglecting the simultaneous effective modeling of both relations. |
| Approach: | They propose a graph neural network-based approach to local and global document summarization using hierarchical discourses. |
| Outcome: | The proposed approach improves on two benchmark datasets and shows that hierarchical structures are important for document summarization. |
LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems (2024.findings-naacl)
Copied to clipboard
| Challenge: | Linguistic entrainment is a phenomenon where linguistic patterns employed by conversational participants converge to one another. |
| Approach: | They propose methods for achieving dialogue entrainment in a task-oriented dialogue system using shared vocabulary. |
| Outcome: | The proposed model produces significantly better entrainment than the base model. |
Efficient Dependency Tree Sampling Without Replacement (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing algorithms for dependency tree sampling have been proposed for sampling without replacement. |
| Approach: | They propose an algorithm that adapts the Wilson Reject algorithm for sampling without replacement and combines it with a Trie data structure. |
| Outcome: | The proposed method is efficient in the case of sampling without replacement from dependency graphs with random weights. |
Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-domain Question Answering (OpenQA) aims at answering factual questions using an external large-scale knowledge corpus. |
| Approach: | They propose a retrieval-augmented approach to QA that focuses on retrieving relevant knowledge from an external corpus. |
| Outcome: | The proposed model can generalize to completely different knowledge domains while adapting to updated versions of the same knowledge corpus and switching to completely new knowledge domain. |
GEE! Grammar Error Explanation with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing grammatical error correction tools do not provide natural language explanations of errors . a system needs to provide one-sentence explanations for each grammamatical errors in a pair of erroneous and corrected sentences. |
| Approach: | They propose a grammar error explanation task that uses one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences. |
| Outcome: | The proposed pipeline identifies grammar errors in German, Chinese, and English . human evaluation reveals that 93.9% of German errors, 96.4% of Chinese errors, and 92.20% of English errors are correctly detected and explained. |
AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant success across various domains, but their application in complex decision-making tasks often necessitates intricate prompt engineering or fine-tuning. |
| Approach: | They propose a lightweight Adapter Language Model (LM) which automatically refines task comprehension based on feedback from RL agents. |
| Outcome: | The proposed framework enhances synergy between LLMs and RL feedback while maintaining generalization abilities and enhancing decision-making capabilities in downstream tasks. |
DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing language models pre-trained on general text overlook the one-to-many property of task-oriented dialogues, where multiple responses can be appropriate given the same context. |
| Approach: | They propose a model that pre-trains LLMs to learn diverse task-oriented dialogue representations by removing domain knowledge that contradicts TODs. |
| Outcome: | The proposed model outperforms strong TOD baselines on various downstream dialogue tasks and learns the intrinsic diversity of task-oriented dialogues. |
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to the emergence of large language models capable ofvarious natural language processing tasks. |
| Approach: | They propose a multi-instructional training approach that integrates a large language model with a speech encoder to harness the capabilities of LLMs for speech recognition and beyond. |
| Outcome: | The proposed model can be trained and aligned with a multilingual LLM on 1900 hours of transcribed data from 139 languages. |
CLEAN–EVAL: Clean Evaluation on Contaminated Large Language Models (2024.findings-naacl)
Copied to clipboard
Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, Hongyuan Lu
| Challenge: | Existing methods to evaluate large language models are prone to data contamination. |
| Approach: | They propose a method which parses contaminated data and back-translates it into a candidate set. |
| Outcome: | The proposed method reduces data contamination and evaluates the LLMs more cleanly. |
R-BASS : Relevance-aided Block-wise Adaptation for Speech Summarization (2024.findings-naacl)
Copied to clipboard
| Challenge: | End-to-end speech summarization on long recordings is challenging because of the high computational cost. |
| Approach: | They propose a new relevance-aware block-wise adaptation method that automatically estimates block relevance based on lexical and semantic similarity between transcript and summary. |
| Outcome: | The proposed method can drop 86.3 % of blocks while maintaining comparable performance. |
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with maintaining accuracy throughout multiple reasoning steps, especially in mathematical reasoning where an error in earlier steps can propagate to subsequent ones and ultimately leading to an incorrect answer. |
| Approach: | They propose an Outcome-supervised Value Model (OVM) that employs outcome supervision for training a value model, which prioritizes steps that lead to accurate conclusions. |
| Outcome: | The proposed model performs better on two multi-step reasoning datasets, GSM8K and Game of 24. |
The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown excellent performance on various NLP tasks. |
| Approach: | They propose a method that integrates multiple demonstration users into one aggregated demonstration to improve sequential recommendation. |
| Outcome: | The proposed method outperforms state-of-the-art LLM-based sequential recommendation methods on three recommendation datasets. |
Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA (2024.findings-naacl)
Copied to clipboard
| Challenge: | a universal question-answering system that can operate on any knowledge graph is presented . previous work that relied on training data to query structured data stores is unrealistic . |
| Approach: | They propose a universal question-answering system that can operate on any knowledge graph . they use an LLM-backed symbolic agent to generate query-program exemplars . |
| Outcome: | The proposed system outperforms state-of-the-art model on domain-specific KGs by 7.08 F1 . the proposed system can be ready to use within a day, the authors show . |
GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) benefit from external knowledge stored in graph structures for various downstream tasks. |
| Approach: | They propose a graph-guided self-attention mechanism that integrates token-level structural information into PLMs without additional alignment or concatenation efforts. |
| Outcome: | The proposed model outperforms baseline models and achieves comparable results to state-of-the-art models on WebNLG datasets. |
Can Public Large Language Models Help Private Cross-device Federated Learning? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that public data can be used to improve privacy-utility trade-offs for large and small language models. |
| Approach: | They propose to use large-scale public data to help differentially private FL training . they propose a distribution matching algorithm with theoretical grounding to sample public data close to private data distribution . |
| Outcome: | The proposed method is efficient and effective for training private models by taking advantage of public data. |
LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view. |
| Approach: | They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions. |
| Outcome: | The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations. |
Planning and Editing What You Retrieve for Enhanced Tool Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for integrating external tools with Large Language Models fall short on effectively shortlisting relevant tools. |
| Approach: | They propose a plan-and-retrieve and edit-and ground paradigms for LLMs that decompose complex queries into actionable tasks. |
| Outcome: | The proposed paradigms significantly improve recall and NDCG in tool retrieval tasks, surpassing current state-of-the-art models. |
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs (2024.findings-naacl)
Copied to clipboard
Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, Abhanshu Sharma
| Challenge: | Visual language models (VLMs) are achieving increasingly strong performance on multimodal tasks. |
| Approach: | They propose to transfer reasoning capabilities from large-language models to VLMs by constructing a 20x larger dataset and a larger dataset to improve general reasoning capabilities. |
| Outcome: | The proposed model outperforms larger models without an upstream OCR system while keeping inference time constant. |
SLiM: Speculative Decoding with Hypothesis Reduction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as an alternative to autoregressive decoding for expediting inference in large language models (LLMs). prevailing assumptions focus solely on latency reduction, neglecting the computational expenses. |
| Approach: | They propose a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. |
| Outcome: | The proposed method reduces the speculation set while validating more effective tokens. |
REMATCH: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing AMR metrics are inefficient and struggle to capture semantic similarity . Existing metrics are not efficient and lack a systematic evaluation benchmark . |
| Approach: | They propose a new AMR similarity metric, rematch, which matches graphs structurally and semantically to each other. |
| Outcome: | The proposed metric is five times faster than the next most efficient metric. |
Modeling the Sacred: Considerations when Using Religious Texts in Natural Language Processing (2024.findings-naacl)
Copied to clipboard
| Challenge: | This paper concerns the use of religious texts in natural language processing (NLP) religious texts are expressions of culturally important values, and machine learning models reproduce cultural values encoded in training data. |
| Approach: | They argue that NLP's use of religious texts raises considerations beyond model biases . authors argue that religious texts are culturally important and are often used by researchers . |
| Outcome: | The proposed method repurposes translations from their original uses and motivations, and raises considerations beyond model biases. |
Testing the Effect of Code Documentation on Large Language Model Code Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive abilities in recent years with regards to code generation and understanding. |
| Approach: | They propose to provide an LLM with "incorrect" documentation that can greatly hinder code understanding, while incomplete or missing documentation does not seem to significantly affect an LRM's ability to understand code. |
| Outcome: | The proposed model can generate and understand code in a language with high documentation quality while lacking documentation does not significantly affect the ability to understand code. |
Aligning Large Language Models with Recommendation Knowledge (2024.findings-naacl)
Copied to clipboard
Yuwei Cao, Nikhil Mehta, Xinyang Yi, Raghunandan Hulikal Keshavan, Lukasz Heldt, Lichan Hong, Ed Chi, Maheswaran Sathiamoorthy
| Challenge: | Large language models (LLMs) excel at natural language reasoning, but cannot model complex user-item interactions inherent in recommendation tasks. |
| Approach: | They propose to equip large language models with recommendation-specific knowledge to address this gap by combining Masked Item Modeling and Bayesian Personalized Ranking (BPR) auxiliary task data samples are generated that encode item correlations and user preferences. |
| Outcome: | Experiments on Amazon Toys & Games, Beauty, and Sports & Outdoors show that the proposed method outperforms conventional and LLM-based baselines by significant margins in retrieval. |
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to pretrain multilingual models are limited by the number of embedding parameters and the complexity of the model. |
| Approach: | They propose a framework that initializes the embeddings of unseen subwords and can adapt a model to multiple languages. |
| Outcome: | The proposed framework can adapt a pre-trained model to multiple languages efficiently and effectively. |
SELF-EXPERTISE: Knowledge-based Instruction Dataset Augmentation for a Legal Expert Language Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | generating instructions and outputs from LLMs can produce unintentionally inaccurate or misleading information. |
| Approach: | They propose to generate an instruction dataset in the legal domain from a seed dataset by extracting knowledge from the outputs of the seed dataset. |
| Outcome: | The proposed method reduces hallucinations in automatic instruction dataset augmentation. |
Re-evaluating the Need for Visual Signals in Unsupervised Grammar Induction (2024.findings-naacl)
Copied to clipboard
Boyi Li, Rodolfo Corona, Karttikeya Mangalam, Catherine Chen, Daniel Flaherty, Serge Belongie, Kilian Weinberger, Jitendra Malik, Trevor Darrell, Dan Klein
| Challenge: | Recent studies show multimodal inputs can improve grammar induction, but weak textual baselines are needed for training. |
| Approach: | They use a fixed grammar family to compare multimodal grammar induction methods . they find multimodal inputs can improve grammar induction by grounding textual inputs to the visual world . |
| Outcome: | The proposed model outperforms weaker baselines on four benchmark datasets. |
EDEntail: An Entailment-based Few-shot Text Classification with Extensional Definition (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification use either class labels or intensional definitions of class labels for label semantics expression. |
| Approach: | They propose a method that employs extensional definition of class labels in hypotheses and then order and format them into a sequence to form hypothese . |
| Outcome: | The proposed method surpasses supervised-learning methods and prompt-based methods on five classification datasets and is comparable to state-of-the-art models. |
What Makes Math Word Problems Challenging for LLMs? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Experiments show that even quite powerful LLMs are still challenged by MWPs. |
| Approach: | They propose to analyze what makes math word problems (MWPs) in English challenging for large language models (LLMs). |
| Outcome: | The proposed model can handle a range of core NLP tasks, but it has emergent abilities, such as ability to solve mathematical puzzles. |
SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Despite advances in artificial intelligence, building social intelligence remains a challenge. |
| Approach: | They propose a task to explain why people laugh in a video and a dataset to do this. |
| Outcome: | The proposed dataset generates plausible explanations for laughter in video and in-the-wild videos. |
T3M: Text Guided 3D Human Motion Synthesis from Speech (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for speech-driven 3D motion synthesisrely on speech audio . existing methods are inaccurate and inflexible, leading to inflexibility and inefficient synthesis results. |
| Approach: | They propose a text-guided 3D human motion synthesis method that uses text input to generate motions from human speech. |
| Outcome: | The proposed method outperforms existing methods in quantitative and qualitative evaluations. |
Deja vu: Contrastive Historical Modeling with Prefix-tuning for Temporal Knowledge Graph Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing text-based methods for Temporal Knowledge Graph Reasoning struggle to balance textual knowledge and temporal information with expensive purpose-built training strategies. |
| Approach: | They propose a Contrastive historical modeling framework with prefix-tuning for TEmporal Reasoning that feeds history-contextualized text into the pseudo-Siamese encoders to strike a textual-temporal balance. |
| Outcome: | The proposed framework achieves superior performance on four transductive and three few-shot inductive TKGR benchmarks. |
Explanation Extraction from Hierarchical Classification Frameworks for Long Legal Documents (2024.findings-naacl)
Copied to clipboard
| Challenge: | Hierarchical classification frameworks are black boxes with no explanation for their predictions. |
| Approach: | They develop an extractive explanation algorithm for hierarchical frameworks for long sequences based on the sensitivity of the trained model to input perturbations. |
| Outcome: | The proposed algorithm achieves a minimum gain of 1 point over the previous benchmark on most of the performance metrics. |
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning. |
| Approach: | They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored. |
| Outcome: | The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks. |
A Tree-of-Thoughts to Broaden Multi-step Reasoning across Languages (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for eliciting Large Language Models (LLMs) to solve complex tasks are limited to English due to the imbalance in the distribution of pre-training data. |
| Approach: | They propose a method for aligning Cross-lingual CoT reasoning across languages . they propose eliciting Large Language Models to solve complex tasks step-by-step . |
| Outcome: | The proposed method outperforms existing prompting methods by reducing interactions and achieving state-of-the-art performance. |
Emergent Abilities in Reduced-Scale Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models can solve new tasks without task-specific fine-tuning. |
| Approach: | They propose to use pre-training data to pre-train 36 language models with billions of parameters to investigate whether emergent properties are tied to model size or can be demonstrated by smaller models. |
| Outcome: | The proposed model performs comparable to models trained on unrestricted language. |
Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies suggest using only a portion of the dialogue context in the annotation process, but the impact of this limitation on label quality remains unexplored. |
| Approach: | They propose to use large language models to summarize the dialogue context to provide a rich and short description of the dialogue and to examine the impact of doing so on the annotator’s performance. |
| Outcome: | The proposed model reduces the context and produces higher quality ratings but introduces ambiguity in usefulness ratings. |
Matching Varying-Length Texts via Topic-Informed and Decoupled Sentence Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to matching text with non-comparable lengths are limited due to truncation issues. |
| Approach: | They propose a model that decouples sentences and embeds them into natural sentences for matching texts of significantly different lengths. |
| Outcome: | The proposed model matches texts of significantly different lengths across three well-studied datasets. |
Instruction Tuning with Human Curriculum (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that human curriculum-inspired strategies can enhance performance of large language models. |
| Approach: | They propose a method for generating instruction-response datasets that emulate human learning . they find that substantial improvements can be achieved through curriculum ordering . |
| Outcome: | The proposed method achieves performance improvements on truthfulQA, MMLU, OpenbookQA, and ARC-hard benchmarks without additional computational costs. |
Natural Language-based State Representation in Deep Reinforcement Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new method for learning policies from images is proposed to reduce image-based observations' complexity and improve interpretability. |
| Approach: | They propose a method that compresses images into a natural language form for state representation. |
| Outcome: | The proposed method allows better interpretability and leverages processing capabilities of large-language models. |
Learning Cross-Architecture Instruction Embeddings for Binary Code Analysis in Low-Resource Architectures (2024.findings-naacl)
Copied to clipboard
| Challenge: | Applying deep learning to binary code analysis has drawn great attention because of its notable performance. |
| Approach: | They propose to learn cross-architecture instruction embeddings where semantically-similar instructions have close embeddements in a shared space. |
| Outcome: | The proposed approach generates high-quality CAIE with good transferability on four ISAs. |
ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing static benchmarks do not guarantee that models can use the provided evidence for answering, which is essential to avoid hallucination when the required knowledge is new or private. |
| Approach: | They propose to automatically perturb existing static one for dynamic evaluation by using a chatGPT framework and a set of open-domain QA datasets. |
| Outcome: | The proposed framework generates new test cases on two open-domain QA datasets and is human-readable and useful to trigger hallucination in LLMs. |
An Effective Automated Speaking Assessment Approach to Mitigating Data Scarcity and Imbalanced Distribution (2024.findings-naacl)
Copied to clipboard
| Challenge: | Automated speaking assessment (ASA) typically involves automatic speech recognition (ASR) and hand-crafted feature extraction from the transcript of a learner’s speech. |
| Approach: | They propose to use metric-based classification and loss re-weighting to model the impact of different SSL-based embedding features on the CEFR score. |
| Outcome: | The proposed model outperforms baselines on the ICNALE benchmark dataset, achieving a significant improvement of more than 10% in CEFR prediction accuracy. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
Subword Attention and Post-Processing for Rare and Unknown Contextualized Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Word embeddings are useful, but struggle on rare and unknown words. |
| Approach: | They propose a rare/unknown embedding architecture that focuses on contextualized representations. |
| Outcome: | The proposed architecture improves performance in most intrinsic and downstream tasks. |
UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI (2024.findings-naacl)
Copied to clipboard
| Challenge: | Identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. |
| Approach: | They propose to use help documents to create step-by-step tutorials overlaid on the phone UI to overcome challenges in retrieval, parsing, and grounding in multilingual-multimodal setting. |
| Outcome: | The proposed dataset contains 4,184 tasks across 8 languages and shows that the end-to-end completion rate drops from 48% in English to 32% for other languages. |
SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large datasets are increasingly available for pre-training source code models, but obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources. |
| Approach: | They propose a systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and investigates model behavior under different fine-tuning methodologies. |
| Outcome: | The proposed approach simulates various OOD scenarios along different dimensions of source code data properties and exposes multiple failure modes attributed to OOD generalization issues. |
Pruning as a Domain-specific LLM Extractor (2024.findings-naacl)
Copied to clipboard
Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, Haifeng Chen
| Challenge: | Large Language Models (LLMs) have exhibited remarkable proficiency across a wide array of NLP tasks. |
| Approach: | They propose a method for pruning large language models using general or task-specific weights to extract a compressed, task-agnostic LLM. |
| Outcome: | The proposed method extracts a compressed, domain-specific, and task- agnostic LLM by identifying LLM weights that are pivotal for general capabilities, like linguistic capability and multi-task solving, and domain- specific knowledge. |
LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback (2024.findings-naacl)
Copied to clipboard
Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, Markus Freitag
| Challenge: | Recent large language models (LLMs) are leveraging human feedback to improve their output quality. however, human feedback is costly to collect, especially at inference time when the model provides new, unseen input. |
| Approach: | They propose an inference-time optimization method to refine large language models' output based on fine-grained feedback to pinpoint defects and guide iterative refinement . |
| Outcome: | The proposed method consistently outperforms baseline approaches on three text generation tasks, including machine translation, long-form question answering, and topical summarization. |
Noisy Multi-Label Text Classification via Instance-Label Pair Correction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Noise is a significant challenge for machine learning models, especially deep learning models. |
| Approach: | They propose a holistic selection metric that identifies noisy pairs while considering global loss information and instance-specific ranking information. |
| Outcome: | The proposed approach significantly improves performance in noisy multi-label text classification tasks. |
Composite Backdoor Attacks Against Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated superior performance on various tasks, but untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. |
| Approach: | They propose a composite backdoor attack that scatters multiple trigger keys in different prompt components. |
| Outcome: | The proposed attack achieves 100% Attack Success Rate (ASR) with a False Triggered Rate (FTR) below 2.06% and negligible model accuracy degradation. |
Adapting Fake News Detection to the Era of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a gap exists in understanding the interplay between machine-paraphrased real news, machine-generated fake news, and human-written real news . false information is easier to generate but harder to detect due to the bias of detectors against machine-generated texts . |
| Approach: | They propose a strategy to adapt fake news detectors to the era of large language models and AI-driven content creation . |
| Outcome: | The proposed detectors perform well on human-written articles but not vice versa . the proposed detector should be trained on datasets with lower machine-generated news ratio than the test set . |
MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large-scale visual-language pretraining models have shown remarkable capabilities in understanding both vision and language. |
| Approach: | They propose a multi-teacher cross-modality alignment distillation technique to integrate the advantages of single-stream and dual-stream models. |
| Outcome: | The proposed model is lightweight and has only 100M running memory and 8.0ms search latency. |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024.findings-naacl)
Copied to clipboard
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky
| Challenge: | Existing methods to rank documents using large language models do not understand these challenging ranking formulations. |
| Approach: | They propose to use Pairwise Ranking Prompting to improve ranking performance . they propose to outperform fine-tuned baseline rankers on benchmark datasets . |
| Outcome: | The proposed technique outperforms supervised baselines on benchmark datasets and outperformed other LLM-based solutions by over 10% on average. |
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing frameworks for multilingual modeling face communication costs and parameter interference conflicts. |
| Approach: | They propose a communication-efficient federated learning framework with low-rank adaptation and language family clustering for Multilingual Modeling (MM) they maintain the weights of the base model, updating the lightweight Low-rank adapt parameters to minimize communication costs. |
| Outcome: | The proposed model outperforms the baseline models in performance and reduces communication overhead. |
Gaussian Process Optimization for Adaptable Multi-Objective Text Generation using Linearly-Weighted Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Multi-objective text generation requires a method to optimize for dynamic requirements of the overall objective. |
| Approach: | They propose a linear combination of objective-specific language models to efficiently adapt the decoding process and optimize for the desired overall objective without retraining one or more language models. |
| Outcome: | The proposed method outperforms other weighting schemes and standard baselines in a few iterations of decoding. |
Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | a significant portion of correct answers remain compromised by hallucinations in large language models. |
| Approach: | They examine whether every generated sentence is grounded in retrieved documents or the model’s pre-training data. |
| Outcome: | The findings highlight the need for more robust mechanisms in large language models to mitigate the generation of ungrounded content. |
TagDebias: Entity and Concept Tagging for Social Bias Mitigation in Pretrained Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to mitigate social bias in pre-trained language models are limited . researchers have found that these models inherit substantial social biases in their pre-training data . |
| Approach: | They propose a method which proposes debiasing a dataset using type tags and fine-tunes PLMs on this debiased dataset. |
| Outcome: | The proposed method improves bias scores on a ranking task . it is based on analyzing type tags and fine-tuning pre-trained models . |
Improving Absent Keyphrase Generation with Diversity Heads (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to generating keyphrases for a given text are limited to extracting only the keyphrase that is directly seen in the document. |
| Approach: | They propose to treat present keyphrase extraction as a sequence labeling problem and treat absent keyphrases together in a text-to-text generation framework during training. |
| Outcome: | The proposed model improves on the state-of-the-art for present keyphrase extraction and five datasets for absent keyphrase generation among the six English datasets. |
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. |
| Approach: | They propose a synthetic task, Multilingual Othello, as a testbed to investigate the factors that contribute to the learning of a language-neutral representation. |
| Outcome: | The proposed approach induces the learning of language-neutral representation and facilitates cross-lingual transfer. |
Discovering and Mitigating Indirect Bias in Attention-Based Model Explanations (2024.findings-naacl)
Copied to clipboard
| Challenge: | Discrimination is the unfair treatment or prejudice directed towards individuals, groups, or certain ideas or beliefs, intentionally or unintentionally. |
| Approach: | They propose an algorithm to detect and mitigate indirect bias in transformer models by leveraging attention explanations. |
| Outcome: | The proposed algorithm shows that it is more accurate than traditional fairness metrics and that it can be used to mitigate bias in transformer models. |
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2024.findings-naacl)
Copied to clipboard
Ziyi Yang, Mahmoud Khademi, Yichong Xu, Reid Pryzant, Yuwei Fang, Chenguang Zhu, Dongdong Chen, Yao Qian, Xuemei Gao, Yi-Ling Chen, Robert Gmyr, Naoyuki Kanda, Noel Codella, Bin Xiao, Yu Shi, Lu Yuan, Takuya Yoshioka, Michael Zeng, Xuedong Huang
| Challenge: | i-Code V2 is one of the first models capable of generating natural language from any combination of Vision, Language, and Speech data. |
| Approach: | They propose to create a model that can generate natural language from any combination of Vision, Language, and Speech data. |
| Outcome: | i-Code V2 matches or outperforms state-of-the-art single- and dual-modality baselines on 7 multimodal tasks. |
Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge-to-text generators often struggle to faithfully generate descriptions for input facts . we propose a decoding-only method to reduce hallucinations . |
| Approach: | They propose a decoding-only method to generate accurate descriptions for input facts . they use a Natural Language Inference model as the model and replace it with a task-specific HVM . |
| Outcome: | The proposed method improves faithfulness with minimal impact on quality and in/out-of-distribution evaluations. |
It’s All Relative! – A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models generate synthetic query-document pairs by prompting with as few as 8 demonstrations. |
| Approach: | They propose to generate queries simultaneously for different labels by prompting with 8 demonstrations. |
| Outcome: | Extensive experimentation shows that synthetic queries generated in such a fashion improve performance. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Hypernetwork-Assisted Parameter-Efficient Fine-Tuning with Meta-Knowledge Distillation for Domain Knowledge Disentanglement (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work on domain adaptation for text summarization fails to account for the huge gap between dialogue and general articles. |
| Approach: | They propose a hypernetwork-assisted encoder-decoder architecture with parameter-efficient fine-tuning to disentangle domain-invariant knowledge from source domains while learning specific knowledge of the target domain. |
| Outcome: | The proposed model can disentangle domain-invariant knowledge from source domains while learning specific knowledge of the target domain. |
MICo: Preventative Detoxification of Large Language Models through Inhibition Control (2024.findings-naacl)
Copied to clipboard
Roy Siegelmann, Ninareh Mehrabi, Palash Goyal, Prasoon Goyal, Lisa Bauer, Jwala Dhamala, Aram Galstyan, Rahul Gupta, Reza Ghanadan
| Challenge: | Large Language Models (LLMs) have a tendency to devolve into toxic degeneration . model may classify prompts as toxic or non-toxic and categorically refuse to respond to those deemed toxic. |
| Approach: | They propose a mechanism for LLM detoxification by labeling acceptable and unacceptable examples and including a corresponding acceptable rewrite with every unacceptable example. |
| Outcome: | The proposed model improves on the baseline model and shows that it detects and rewrites toxic and harmful examples. |
Reinforcement Learning with Token-level Feedback for Controllable Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for controllable text generation are guided by coarse-grained feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. |
| Approach: | They propose a reinforcement learning algorithm which formulates TOken-LEvel rewards for controllable text generation and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm. |
| Outcome: | The proposed algorithm can achieve superior performance on single-attribute and multi-attract control tasks. |
CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown great ability in solving traditional natural language tasks and elementary reasoning tasks with appropriate prompting techniques. |
| Approach: | They propose a collaborative multi-agent, multi-reasoning-path prompting framework that prompts LLMs to play different roles in a problem-solving team and encourages different role-play agents to collaboratively solve the target task. |
| Outcome: | The proposed framework is applied to two college-level science problems over competitive baselines. |
Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies (2024.findings-naacl)
Copied to clipboard
Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, Rahul Gupta
| Challenge: | a recent study documented the harmful limitations of gender binary-centric large language models . data scarcity is a known culprit, but the precise mechanisms through which scarcity affects this behavior remain underexplored. |
| Approach: | They propose to use BPE tokenization to enforce consistent tokenization across gendered pronouns to improve neopronoun proficiency. |
| Outcome: | The proposed methods outperform finetuning with standard BPE, and improve neopronoun proficiency. |
AdaPT: A Set of Guidelines for Hyperbolic Multimodal Multilingual NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | Euclidean space is used for training neural models and performing arithmetic operations, but many data types have complex geometries and cannot be captured in the Euclidesan space. |
| Approach: | They propose a set of guidelines for initialization, parametrization, and training of neural networks that can be generalized over existing neural network training methodologies. |
| Outcome: | The proposed framework outperforms Euclidean methods on three tasks over 12 languages and modalities on a variety of domains. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
ZSEE: A Dataset based on Zeolite Synthesis Event Extraction for Automated Synthesis Platform (2024.findings-naacl)
Copied to clipboard
| Challenge: | Automated synthesis of zeolite holds great significance for attaining economic and environmental benefits. |
| Approach: | They propose an event extraction task to mine structural synthesis actions from experimental narratives for modular automated synthesis. |
| Outcome: | The proposed method can significantly expedite automated synthesis of zeolites owing to its machine readability. |
Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior studies have attempted to enhance faithfulness of abstractive summarization, yet hallucination remains a persistent challenge. |
| Approach: | They propose a decoding strategy that adjusts the generation probability of each token by comparing it with the token’s marginal probability within the domain of the source text. |
| Outcome: | The proposed method significantly improves faithfulness and source relevance on the XSUM dataset. |
Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing toxicity within large language models can negatively impact the user experience, causing performance degradation. |
| Approach: | They propose an adversarial DPO algorithm that improves direct preference optimization (DPO) by incorporating harmful data into the generative model. |
| Outcome: | The proposed training algorithm improves the model’s resilience against harmful conversations while minimizing performance degradation. |
Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prompt engineering is an essential technique for enhancing the abilities of large language models (LLMs) by providing explicit and specific instructions. |
| Approach: | They propose a new approach that uses text embeddings to obtain basis vectors by matrix decomposition and constructs a space for representing all prompts. |
| Outcome: | The proposed approach significantly outperforms state-of-the-art prompt paradigms on ten public reasoning benchmarks. |
DAGCN: Distance-based and Aspect-oriented Graph Convolutional Network for Aspect-based Sentiment Analysis (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in sentiment analysis tend to interference from local factors such as irrelevant words and edges, hindering the precise identification of opinion words. |
| Approach: | They propose a distance-based syntactic weight and Aspect-Fusion Attention to solve this problem. |
| Outcome: | The proposed model outperforms state-of-the-art models on three public datasets and verify its effectiveness. |
Connecting the Dots: Inferring Patent Phrase Similarity with Retrieved Phrase Graphs (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for inferring patent phrase similarity do not perform satisfactorily . et al., 2010: patents are pivotal to innovation, safeguarding novel ideas . |
| Approach: | They propose a graph-augmented approach to amplify patent phrase contextual information . they construct a phrase graph that links to patents cited by or cited in patents for each phrase . |
| Outcome: | The proposed approach significantly improves the representation of patent phrases in a self-supervised fashion. |
Self-Regulated Sample Diversity in Large Language Models (2024.findings-naacl)
Copied to clipboard
Mingyue Liu, Jonathan Frawley, Sarah Wyer, Hubert P. H. Shum, Sara Uckelman, Sue Black, Chris Willcocks
| Challenge: | Existing methods that require expensive setups or maintain static values during inference are inflexible and require expensive training. |
| Approach: | They propose a self-regulating approach that adjusts sample diversity parameters dynamically based on the input prompt. |
| Outcome: | The proposed method significantly improves the quality of responses generically without model retraining or fine-tuning. |
Methods, Applications, and Directions of Learning-to-Rank in NLP Research (2024.findings-naacl)
Copied to clipboard
| Challenge: | Learning-to-rank (LTR) algorithms aim to order items according to some criteria. |
| Approach: | They focus on the formal background of LTR and the most widely-used supervised methods . they also discuss how large language models are changing the LTR landscape . |
| Outcome: | The proposed methods are used in natural language processing and information retrieval tasks. |
When Quantization Affects Confidence of Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that quantization compromises performance and exacerbates biases in Large Language Models. |
| Approach: | They propose an explanation for quantization loss based on confidence levels . they propose a range of efficient compression and acceleration methods including quan-tization . |
| Outcome: | The proposed methods show that quantization decreases confidence regarding true labels and that it exacerbates biases across different scales. |
MedCycle: Unpaired Medical Report Generation via Cycle-Consistency (2024.findings-naacl)
Copied to clipboard
| Challenge: | Generating medical reports for X-ray images presents a significant challenge . previous studies have required a specific labeling schema for images and reports . |
| Approach: | They propose a cycle-consistent mapping function that transforms image embeddings into report embedds and auto-encoding for medical report generation. |
| Outcome: | The proposed approach outperforms state-of-the-art results in unpaired chest X-ray report generation, showing improvements in both language and clinical metrics. |
Beta-LR: Interpretable Logical Reasoning based on Beta Distribution (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to capture logical information from text are limited by the uncertainty of the text. |
| Approach: | They propose a probabilistic embedding method to capture logical information from text . they embed texts into beta distributions on each dimension to eliminate logical uncertainty . |
| Outcome: | The proposed method achieves competitive performances on two datasets. |
Applications of BERT Models Towards Automation of Clinical Coding in Icelandic (2024.findings-naacl)
Copied to clipboard
| Challenge: | Traditionally, clinical coding is manual and laborintensive task prone to human error. |
| Approach: | They analyze 25 years of electronic health records from the Landspitali University Hospital in Icelandic to explore the potential of using NLP for clinical coding. |
| Outcome: | The best-performing model achieves competitive results in micro and macro F1 scores, with label attention contributing significantly to its success. |
“Tell me who you are and I tell you how you argue”: Predicting Stances and Arguments for Stakeholder Groups (2024.findings-naacl)
Copied to clipboard
| Challenge: | Argument mining has focused on the identification, extraction, and formalization of arguments. |
| Approach: | They propose a framework that relies on a recommender-based architecture to predict stances and argumentative main points on societally controversial topics for a given stakeholder. |
| Outcome: | The proposed framework predicts arguments on a debate topic based on BERTScore and debate.org datasets. |
Psychometric Predictive Power of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | despite efforts to align large language models with human preferences, instruction tuning does not always make LLMs human-like from a cognitive modeling perspective. |
| Approach: | They find that instruction tuning does not always make large language models human-like from a cognitive perspective. |
| Outcome: | The proposed prompts improve predictive power but are still inferior to small base models. |
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated impressive capabilities in various NLP tasks, but previous studies have shown they are sensitive to prompt wording and few-shot demonstrations and their order. |
| Approach: | They focus on LLMs robustness on multiple-choice questions . they find a performance gap of 13% to 85% when options are reordered . |
| Outcome: | The proposed model outperforms supervised models on multiple choice questions even outperforming humans. |
PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck (2024.findings-naacl)
Copied to clipboard
| Challenge: | CLIP-based classifiers rely on the prompt containing a class name that is known to the text encoder and perform poorly on new classes or the classes whose names rarely appear on the Internet. |
| Approach: | They propose to use a set of text descriptors to express a class name into a textual descriptable and match the embeddings of the detected parts to their textual ones to compute a logit score. |
| Outcome: | The proposed classifier outperforms CLIP-based classifiers on zero-shot and supervised learning settings by 88.80% and 92.20% accuracy on CUB-200 and Stanford Dogs-120. |
Ethos: Rectifying Language Models in Orthogonal Parameter Space (2024.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) generate toxic, biased content and reveal private training records. |
| Approach: | They propose an efficient approach that rectifies LMs to mitigate toxicity and bias . Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors . |
| Outcome: | The proposed approach mitigates toxicity and bias in outputs and avoids privacy leakage. |
Crafting In-context Examples according to LMs’ Parametric Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | In-context examples can improve the performance of knowledge-rich tasks such as question answering by triggering a language model to surface information stored in its parametric knowledge. |
| Approach: | They propose to construct in-context example sets based on model's parametric knowledge by prompting models with 'unknown' examples. |
| Outcome: | The proposed model can perform better on in-context examples in three multi-answer question answering datasets, and prompting with ‘unknown’ examples decreases the performance. |
ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing research has focused on fully supervised XMC, but real-world scenarios often lack supervision signals, highlighting the importance of zero-shot settings. |
| Approach: | They propose a framework that generates a set of candidate labels through in-context learning and then reranks them. |
| Outcome: | The proposed framework advances state-of-the-art on two diverse public benchmarks. |
CLGSI: A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on contrastive learning, but lack detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive training representation space. |
| Approach: | They propose a framework for multimodal sentiment analysis based on contrastive learning guided by sentiment intensity (CLGSI) it selects positive and negative sample pairs based upon sentiment intensity differences and assigns corresponding weights accordingly. |
| Outcome: | The proposed framework extracts common features between different modalities and then uses them to predict sentiment intensity. |
Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models for yes-no questions are challenging, but they still face challenges. |
| Approach: | They propose an approach grounded on distant supervision and blended training to quickly adapt to a new dialogue domain. |
| Outcome: | The proposed approach improves F1 performance in movie scripts, tennis interviews, and airline customer service domains. |
Enhancing Perception: Refining Explanations of News Claims with LLM Conversations (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new framework for Large Language Models (LLMs) streamlines the task of crafting explanations for fake news . a study compared refinement conversations between human and LLMs to enhance the effectiveness of LLM explanations . |
| Approach: | They propose a framework for Large Language Models to streamline the task of crafting fake news explanations. |
| Outcome: | The proposed framework enhances the process of crafting explanations for fake news claims through conversational refinement. |
How Interpretable are Reasoning Explanations from Prompting Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prompt Engineering has garnered significant attention for enhancing the performance of large language models across a multitude of tasks. |
| Approach: | They propose a simple prompting technique that yields more than 70% improvement in interpretability. |
| Outcome: | The proposed method improves interpretability by 70% across multiple dimensions. |
Plug-in Language Model: Controlling Text Generation with a Simple Regression Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have demonstrated unrivaled capacity in generating text that closely resembles human-written content. |
| Approach: | They propose a plug-in language model that leverages reinforcement learning to adjust latent states to control text generation. |
| Outcome: | The proposed model outperforms existing methods that rely on gradient-based, weighted decoding, or prompt-based methods. |
Signer Diversity-driven Data Augmentation for Signer-Independent Sign Language Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for sign language translation (SLT) rely on signer identity labels, which is often impractical and costly in real-world applications. |
| Approach: | They propose a signer diversity-driven data augmentation method that can generalize to signers not encountered during training. |
| Outcome: | The proposed method achieves state-of-the-art results without relying on signer identity labels. |
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Multilingual modelling can improve machine translation for low-resource languages, partly through shared subword representations. |
| Approach: | They propose to use subword regularisation to promote synergy and BPE to facilitate cross-lingual transfer. |
| Outcome: | The proposed methods promote synergy and prevent interference across different linguistic typologies. |
Multi-Granularity Guided Fusion-in-Decoder (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-domain question answering requires deriving factual responses without explicit evidence . recent approaches combine retrieval of relevant information with response generation . |
| Approach: | They propose a model that concatenates multiple contexts in the decoding phase . they propose MGFiD, which harmonizes passage re-ranking with sentence classification . |
| Outcome: | The proposed model outperforms existing models on Natural Questions and TriviaQA datasets . it aggregates evident sentences into an anchor vector that instructs the decoder . |
Group Fairness in Multilingual Speech Recognition Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study evaluates the performance disparities of ASR models across languages and demographics . a large amount of data is required to mitigate performance disparity, but this is computationally expensive . |
| Approach: | They evaluate the performance disparity of ASR models using a multilingual dataset . they find that model size correlates logarithmically with worst-case performance disparities . |
| Outcome: | The proposed models exhibit significant performance disparities across binary genders for adolescents. |
Rethinking Machine Ethics – Can LLMs Perform Moral Reasoning through the Lens of Moral Theories? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to making moral judgments are mostly bottom-up and lack explainability. |
| Approach: | They propose a top-down framework to steer Large Language Models to perform moral reasoning with well-established moral theories. |
| Outcome: | The proposed framework can integrate various moral theories on moral datasets. |
Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) suffer catastrophic forgetting when tailored to specific domains . authors present a novel approach to manage multi-domain LLM adaptation . |
| Approach: | They propose a strategy to manage multi-domain LLM adaptation using self-distillation and role integration. |
| Outcome: | The proposed model alleviates catastrophic forgetting and inter-domain confusion while maintaining robust general capabilities. |
BERTweet’s TACO Fiesta: Contrasting Flavors On The Path Of Inference And Information-Driven Argument Mining On Twitter (2024.findings-naacl)
Copied to clipboard
| Challenge: | Argument mining is a challenging analytical task in the rich context of Twitter (now X). |
| Approach: | They propose to optimize the embeddings of the BERTweet transformer for argument mining on Twitter and broader generalization across topics. |
| Outcome: | The proposed approach improves classification and generalization across topics using a siamese network and a dataset. |
Testing the limits of logical reasoning in neural and hybrid models (2024.findings-naacl)
Copied to clipboard
| Challenge: | despite the successes of deep learning models, we still need to know more about how and what they learn. |
| Approach: | They create tests to analyze logical reasoning patterns in neural and hybrid models . they find that models can generalize logical thinking only to a limited degree . |
| Outcome: | The proposed models can capture elementary aspects of meaning but only to limited extent . authors say they need to understand how and what they learn . |
METAL: Towards Multilingual Meta-Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models excel on many standard NLP benchmarks. |
| Approach: | They propose a framework for end-to-end evaluation of Large Language Models as evaluators in multilingual scenarios. |
| Outcome: | The proposed framework evaluates LLMs as evaluators in multilingual scenarios. |
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)
Copied to clipboard
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
| Challenge: | Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks. |
| Approach: | They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions. |
| Outcome: | The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam. |
Product Description and QA Assisted Self-Supervised Opinion Summarization (2024.findings-naacl)
Copied to clipboard
Tejpalsingh Siledar, Rupasai Rangaraju, Sankara Muddu, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera, Swaprava Nath, Pushpak Bhattacharyya
| Challenge: | Existing methods to generate opinion summarization without supervised training data are limited due to the lack of additional sources. |
| Approach: | They propose a synthetic dataset creation strategy that leverages reviews and additional sources to generate a pseudo-summary. |
| Outcome: | The proposed approach achieves 14.5% improvement in ROUGE-1 F1 over existing models. |
COMEM: In-Context Retrieval-Augmented Mass-Editing Memory in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for "knowledge editing" in large language models are inadequate . authors propose a method that can be used to update outdated information or correct false information . |
| Approach: | They propose a unified knowledge editing method called in-COntext retrieval-augmented Mass-Editing Memory . it incorporates retrieval augmented IKE, a novel extension of IKE designed for massive editing tasks . |
| Outcome: | The proposed method outperforms existing methods on the zsRE and CounterFact datasets. |
Content-Specific Humorous Image Captioning Using Incongruity Resolution Chain-of-Thought (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for generating humorous captions are generic and do not capture the content of images. |
| Approach: | They propose a framework that generates content-specific resolutions from fine details extracted from an image and integrates logit bias and negative sampling to suppress the output of generic resolutions. |
| Outcome: | The proposed framework generates humorous captions tailored to the content of specific input images. |
Denoising Attention for Query-aware User Modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has proposed to build user models at query time by leveraging the Attention mechanism, which allows weighing the contribution of the user-related information w.r.t. the current query. |
| Approach: | They propose to use the Attention mechanism to build user models at query time by weighing the contribution of the user-related information w.r.t. the Attention variant adopts a robust normalization scheme and introduces . filtering mechanism to better discern among the user related data those helpful for personalization. |
| Outcome: | The proposed approach improves MAP, MRR, and NDCG above 15% w.r.t. other Attention variants at the state-of-the-art. |
A Lightweight Mixture-of-Experts Neural Machine Translation Model with Stage-wise Training Strategy (2024.findings-naacl)
Copied to clipboard
| Challenge: | Using mixture-of-experts (MoE) to deal with language heterogeneity is a challenge in neural machine translation (NMT). |
| Approach: | They propose a lightweight MoE-based NMT model that is trained via an elaborate stage-wise training strategy. |
| Outcome: | The proposed model achieves stable improvements in translation tasks by introducing fewer extra parameters compared to baseline models. |
BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate LMs rely on objective function and are therefore limited to masked or causal LM types. |
| Approach: | They propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. |
| Outcome: | The proposed framework can probe for knowledge across different LM types. |
Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing frameworks for fast and accurate intent classification for task-oriented dialogue systems do not provide a clear definition of the true intent. |
| Approach: | They propose to augment the framework for out-of-scope detection by disambiguating between a small number of likely intents. |
| Outcome: | The proposed framework generates small clarification questions and is capable of out-of-scope detection. |
Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models in Court Decisions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Despite high re-identification rates on Wikipedia, even the best LLMs struggled with court decisions. |
| Approach: | They construct an anonymized Wikipedia dataset to investigate re-identification risks . they also introduce new metrics to measure performance . |
| Outcome: | The proposed model can be used to identify individuals in court decisions, but it fails in the vast majority of cases. |
X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment (2024.findings-naacl)
Copied to clipboard
DongJae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, SeungWoo Song, HanGyeol Yoo, SangMin Kim, KyungTae Lim
| Challenge: | constructing multilingual data for large multimodal models presents its own set of challenges due to language diversity and complexity. |
| Approach: | They propose to use GPT4-V to construct multimodal training datasets using a text-only version of GPT4. |
| Outcome: | The proposed method performs well in Korean and English, surpassing existing methods. |
Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing retrieval-augmented language models assume query relevance and irrelevance as dichotomy . existing models are highly brittle to the presence of conflicting information in both the fine-tuning and in-context few-shot learning scenarios. |
| Approach: | They propose methods for handling knowledge conflicts by fine-tuning a discriminator or prompting it to elicit its discriminative capability. |
| Outcome: | The proposed approaches significantly enhance model robustness on open-domain QA. |
Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfake (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has focused on audio deepfake detection (ADD) due to its ability to impersonate and share false, often malicious information. |
| Approach: | They propose to use multilingual speech Pre-Trained models for Audio deepfake detection (ADD) they propose to combine models with existing models to achieve better ADD detection . |
| Outcome: | The proposed models gain knowledge about diverse pitches, accents, and tones, during theirpre-training phase and are more robust to variations. |
Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts (2024.findings-naacl)
Copied to clipboard
Chenghao Yang, Tuhin Chakrabarty, Karli Hochstatter, Melissa Slavin, Nabila El-Bassel, Smaranda Muresan
| Challenge: | Experimental results show that identifying the phases of opioid use disorder is highly contextual and challenging. |
| Approach: | They analyze 2500 opioid-related posts from various subreddits labeled with six different phases of opioid use . they annotate span-level extractive explanations and critically evaluate state-of-the-art models in a supervised, few-shot, or zero-shot setting. |
| Outcome: | The proposed models improve classification accuracy and quality of the extracted explanations. |
Self-Adaptive Sampling for Accurate Video Question Answering on Image Text Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Image–text models (ITMs) are the prevalent architecture to solve video question–answering tasks, which requires only a few input frames to save huge computational cost compared to video–language models. |
| Approach: | They propose a sampling method based on question–frame correlation that is efficient for the few-frame situations. |
| Outcome: | The proposed method can boost the performance of image–text pretrained models and have a wide application scenario in terms of model architectures and dataset types. |
Towards an On-device Agent for Text Rewriting (2024.findings-naacl)
Copied to clipboard
Yun Zhu, Yinxiao Liu, Felix Stahlberg, Shankar Kumar, Yu-Hui Chen, Liangchen Luo, Lei Shu, Renjie Liu, Jindong Chen, Lei Meng
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting, however creating a smaller yet potent language model presents two formidable challenges: costly data collection and absence of emergent capabilities. |
| Approach: | They propose a new instruction tuning method to develop a mo-bile text rewriting model that leverages LLM-generated data and heuristic reinforcement learning, eliminating the need for human data collection. |
| Outcome: | The proposed model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size using public benchmark EditEval and our new benchmark. |
Tailoring Vaccine Messaging with Common-Ground Opinions (2024.findings-naacl)
Copied to clipboard
Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra
| Challenge: | Vaccine interventions aim to answer concerns expressed about vaccination. |
| Approach: | They propose a dataset to evaluate how well responses are tailored to a common-ground opinion . they find that GPT-4-Turbo performs significantly better than others . |
| Outcome: | The proposed dataset outperforms fine tuned LLMs on the task of tailoring vaccine responses to common-ground opinions. |
Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification (2024.findings-naacl)
Copied to clipboard
| Challenge: | a novel neuro-symbolic architecture for relation classification combines rule-based methods with deep learning techniques. |
| Approach: | They propose a neuro-symbolic architecture for relation classification that combines rule-based methods with deep learning techniques. |
| Outcome: | The proposed approach outperforms state-of-the-art models in three out of four settings . human interventions boost the performance on the relation org:parents by as much as 26% relative improvement . |
Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for continual prompt tuning are limited by the ever-growing parameter scale of modern language models (e.g., GPT-4 that may have 1.76 trillion parameters). |
| Approach: | They propose a method for continual prompt tuning that enables the lifelong learning of a pre-trained language model by adding a task-specific prompt to a queue of older tasks. |
| Outcome: | The proposed method outperforms the state-of-the-art methods substantially on continual prompt tuning benchmarks. |
In-Context Example Ordering Guided by Label Distributions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has shown that ICL is sensitive to different natural language instructions and different orderings of in-context examples. |
| Approach: | They propose two principles for in-context example ordering guided by model’s probability predictions. |
| Outcome: | The proposed model outperforms baseline models on 13 text classification datasets and nine autoregressive LLMs with 700M to 13B parameters. |
Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing pretrained embeddings and LLM embeddables fail to discern subtle financial narrative shifts, resulting in a lack of insight for investors and regulators. |
| Approach: | They propose a financial domain-specific NLP task to measure nuanced semantic similarity between pairs of financial narratives. |
| Outcome: | The proposed method outperforms existing methods trained on classic STS tasks and generic LLM embeddings on a human-annotated dataset. |
Laying Anchors: Semantically Priming Numerals in Language Modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Numeracy is the comprehension of numbers, and numerals are important for comprehension. |
| Approach: | They propose methods to semantically prime numerals by generating anchors governed by the distribution of numeral in any corpus. |
| Outcome: | The proposed methods perform better on numeracy tasks for both in-domain and out-domain numerals. |
UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)
Copied to clipboard
Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu
| Challenge: | Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data. |
| Approach: | They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts. |
| Outcome: | The proposed model breaks through performance upper bounds of experts without additional annotated data. |
LatticeGen: Hiding Generated Text in a Lattice for Privacy-Aware Large Language Model Generation on Cloud (2024.findings-naacl)
Copied to clipboard
Mengke Zhang, Tianxing He, Tianle Wang, Lu Mi, Niloofar Mireshghallah, Binyi Chen, Hao Wang, Yulia Tsvetkov
| Challenge: | Currently, the server controls the generated text, but users can't keep it private . prompted generation is a common interaction paradigm for large language models on cloud . |
| Approach: | They propose a protocol where the server handles most of the computation while the client controls the sampling operation. |
| Outcome: | The proposed protocol protects both prompt and generation under strong attacks. |
HateModerate: Testing Hate Speech Detectors against Content Moderation Policies (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on hate speech detection have failed to answer this question. |
| Approach: | They propose a dataset for testing the behaviors of automated content moderators against content policies. |
| Outcome: | The proposed dataset includes hateful and non-hateful examples matching the 41 community standards guideline policies of Facebook. |
Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other (2024.findings-naacl)
Copied to clipboard
| Challenge: | Emergent Large Language Models (LLMs) use extraordinary performance and powerful deduction capacity to discern from traditional language models. |
| Approach: | They propose a method that uses weights to compensate quantization error and learnable singular value incremental (LSI) LSI is a technique that helps weights compensate each other conditioned on activation. |
| Outcome: | The proposed method achieves state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios. |
Contrastive Preference Learning for Neural Machine Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing discrepancies between token-level objective and overall sequence-level quality of a model are causing exposure bias and other issues in NMT. |
| Approach: | They propose a contrastive preference model that integrates an indicator function to fine-tune a pre-trained model in Neural Machine Translation. |
| Outcome: | The proposed model outperforms the traditional Plackett-Luce model on three language pairs and also outperFORMs token-level and sequence-level baseline models. |
SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing reference-free reasoning evaluation metrics rely on human-annotated reasoning chains as references, but require fine-tuning with human-derived chains before evaluation. |
| Approach: | They propose to use GPT-4 to automatically evaluate reasoning chain quality by leveraging the Socratic method. |
| Outcome: | Empirical results show that the proposed approach significantly improves existing reference-free reasoning evaluation metrics. |
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis (2024.findings-naacl)
Copied to clipboard
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, Lei Li
| Challenge: | Existing studies show that large language models (LLMs) can handle multilingual machine translation (MMT) However, the multilingual translation ability of LLMs remains under-explored. |
| Approach: | They evaluate eight popular LLMs including ChatGPT and GPT-4 to determine their performance in multilingual machine translation. |
| Outcome: | The proposed model can generate moderate translation even on zero-resource languages and cross-lingual exemplars can provide better task guidance for low-resourced translation than exemplar in the same language pairs. |
Unleashing the Power of LLMs in Court View Generation by Stimulating Internal Knowledge and Incorporating External Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made remarkable strides in language generation, but they encounter difficulties in the knowledge-intensive legal domain. |
| Approach: | They propose to decompose court views into different parts, stimulate internal knowledge, and incorporate external information to unleash the power of LLMs in the task. |
| Outcome: | The proposed method generates more accurate and reliable court views on two real-world datasets LAIC2021 and CJO2022. |
Prompting Vision-Language Models For Aspect-Controlled Generation of Referring Expressions (2024.findings-naacl)
Copied to clipboard
Danfeng Guo, Sanchit Agarwal, Arpit Gupta, Jiun-Yu Kao, Emre Barut, Tagyoung Chung, Jing Huang, Mohit Bansal
| Challenge: | Referring Expression Generation (REG) is the task of generating a descriptive caption that uniquely identifies a given target in the scene. |
| Approach: | They propose an Aspect-Controlled REG task which requires generating a referring expression conditioned on the input aspect(s) by changing the input input such as color, location, action etc. |
| Outcome: | The proposed model beats all prior works in the CIDEr score and achieves comparable performance to training with 100% of real data. |
Task-Agnostic Detector for Insertion-Based Backdoor Attacks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for textual backdoor detection are task-specific and less effective beyond sentence classification. |
| Approach: | They propose a task-agnostic method for backdoor detection that leverages final layer logits and an efficient pooling technique. |
| Outcome: | TABDet can jointly learn from diverse task-specific models, demonstrating superior detection efficacy over traditional methods. |
Uncertainty Estimation on Sequential Labeling via Uncertainty Transmission (2024.findings-naacl)
Copied to clipboard
| Challenge: | Named entity recognition tasks are often suboptimal for NER . previous work focused on UE-NER, which estimates uncertainty scores for ner . |
| Approach: | They propose to use a Sequential Labeling Posterior Network to estimate uncertainty for NER . they propose to consider wrong-span cases and to evaluate the specificity of wrong-pan cases. |
| Outcome: | The proposed system improves on three datasets and AUPR on MIT-Restaurant datasets. |
Exploring Language Model’s Code Generation Ability with Auxiliary Functions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Auxiliary function is a useful component to improve language model’s code generation ability, but a systematic exploration of how they affect has yet to be done. |
| Approach: | They construct a human-crafted evaluation set which contains examples of two functions where one function assists the other to examine their ability in a multifaceted way. |
| Outcome: | The proposed model is underutilized to call the auxiliary function, suggesting future directions to enhance their implementation by eliciting the supplementary function call ability encoded in the models. |
Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing open-source LLMs exhibit limited effectiveness in processing Vietnamese . lack of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation exacerbates these issues. |
| Approach: | They propose to fine tune LLMs specifically for Vietnamese and develop a framework for evaluation . they find that larger models introduce more biases and uncalibrated outputs . |
| Outcome: | The proposed framework finetunes LLMs specifically for Vietnamese and provides a framework for evaluation . |
GoT: Effective Graph-of-Thought Reasoning in Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have been advancing at an unprecedented pace. |
| Approach: | They propose a graph-based approach which models human thought processes as a chain and as 'graphs' by representing thought units as nodes and connections between them as edges, they capture the non-sequential nature of human thinking and allows for a more realistic modeling of thought processes. |
| Outcome: | The proposed model improves on a text-only reasoning task and a multimodal reasoning task. |
Enhancing the General Agent Capabilities of Low-Paramter LLMs through Tuning and Multi-Branch Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-source pre-trained Large Language Models exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. |
| Approach: | They propose a method to construct agent-specific data using GPT-4 and supervised fine-tuning . they find that supervised tunning can significantly reduce hallucination outputs and formatting errors in agent tasks . |
| Outcome: | The proposed method improves on five agent tasks of AgentBench. |
MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) that integrate with external Python interpreters are not able to demonstrate the calculation process, which compromises user-friendliness and understanding of problem-solving steps. |
| Approach: | They propose to use LLaMA-2 to refine LLti-perspective augmentation methods to improve performance. |
| Outcome: | The proposed model achieves 88.3% on GSM8K and 34.5% on MATH. |
Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to generate source code summaries are coarse-grained and noise-filled . however, they do not capture contextual code semantics and are often outdated in continuous software iteration. |
| Approach: | They propose a fine-grained Token-level retrieval-augmented mechanism on the decoder side to enhance performance of neural models. |
| Outcome: | The proposed method produces more low-frequency tokens and is interpretable. |
UNO-DST: Leveraging Unlabelled Data in Zero-Shot Dialogue State Tracking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for zero-shot dialogue state tracking (DST) ignore unlabelled data in the target domain. |
| Approach: | They propose to transform zero-shot dialogue state tracking into few-shot DST by utilising unlabelled data via joint and self-training methods. |
| Outcome: | The proposed method improves joint goal accuracy by 8% on general language models in zero-shot scenarios, and can be used in many domains. |
Evaluating Step-by-Step Reasoning through Symbolic Verification (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. |
| Approach: | They propose to use symbolic examples to iteratively reason over symbolic examples and to recover Prolog’s backward chaining algorithm to iterate over KBs. |
| Outcome: | The proposed model performs better on length generalization benchmarks than CoT on explanations and chain-of-thoughts (CoT) tasks. |
Multi-Review Fusion-in-Context (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current methods for generating text are opaque and difficult to control and interpret due to their opaque nature. |
| Approach: | They propose a modular approach with separate components for each step . they formalize Fusion-in-Context as a standalone task, whose input consists of source texts with highlighted spans of targeted content. |
| Outcome: | The proposed approach is based on a curated dataset of 1000 instances in the reviews domain and a novel evaluation framework for assessing the faithfulness and coverage of highlights. |
Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to extract examples from memory are limited, but the upstream retrieval step is still unexplored. |
| Approach: | They propose to use a standard autoregressive model, edit-based model and a large language model with in-context learning to investigate the effect of retrieval methods on translation scores. |
| Outcome: | The proposed architectures improve translation scores and increase diversity of examples. |
Extending Input Contexts of Language Models through Training on Segmented Sequences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Effectively training languages models on long sequences poses many technical challenges. |
| Approach: | They propose a method for extending positional embeddings by sub-sampling segments from long inputs while maintaining their original position. |
| Outcome: | The proposed method extends the input con-text size of pretrained models without any changes in the model's memory and memory costs. |
Reason from Fallacy: Enhancing Large Language Models’ Logical Reasoning through Logical Fallacy Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but struggle with some more complex reasoning tasks including logical reasoning. |
| Approach: | They propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW to evaluate LLMs’ capability of logical fallacy understanding. |
| Outcome: | The proposed dataset can be used to evaluate LLMs’ LFU capability and to fine-tune LLM models to obtain significantly enhanced performance on logical reasoning. |
Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models (2024.findings-naacl)
Copied to clipboard
Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, Andrew Lan
| Challenge: | Multiple-choice questions (MCQs) are easy to administer and grade . but crafting high-quality distractors remains labor-intensive and limited scalability . |
| Approach: | They propose to automate the generation of distractors in math MCQs by using large language models to generate distractors. |
| Outcome: | The proposed methods can generate valid distractors, but they are less adept at anticipating common errors or misconceptions among real students. |
Aspect-based Sentiment Analysis with Context Denoising (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to ABSA use text encoders to locate important context features or remove them from input. |
| Approach: | They propose to improve ABSA with context denoising to remove noise from text . they use diffusion networks to perform denoizing process to gradually eliminate noise . paper shows that aspect-based sentiment analysis is effective for fine-grained analysis . |
| Outcome: | The proposed approach improves ABSA on five widely used ABSA datasets. |
IruMozhi: Automatically classifying diglossia in Tamil (2024.findings-naacl)
Copied to clipboard
| Challenge: | Literary Tamil is highly diglossic, with two very different registers in everyday use . Spoken Tamil is under-studied in modern NLP systems compared to Literary Tamil written in the Tamil script . |
| Approach: | They present a human-translated dataset of parallel text in Literary and Spoken Tamil. |
| Outcome: | The proposed model trains classifiers on the task of identifying which Tamil variety a text belongs to. |
RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations (2024.findings-naacl)
Copied to clipboard
Haolan Zhan, Zhuang Li, Xiaoxi Kang, Tao Feng, Yuncheng Hua, Lizhen Qu, Yi Ying, Mei Rianto Chandra, Kelly Rosalin, Jureynolds Jureynolds, Suraj Sharma, Shilin Qu, Linhao Luo, Ingrid Zukerman, Lay-Ki Soon, Zhaleh Semnani Azad, Reza Haf
| Challenge: | Norm violations occur when individuals fail to conform to culturally accepted behaviors, which may lead to potential conflicts. |
| Approach: | They propose to use a large corpus of 9,258 multi-turn dialogues annotated with social norms to equip AI systems with a remediation ability. |
| Outcome: | The proposed system can understand and remediate norm violations step by step. |
Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Data augmentation techniques generate low-quality texts with incorrect labels . a new technique is needed to winnow out texts with inaccurate labels based on provenance inspection . |
| Approach: | They develop a data inspection technique that uses provenance inspection and assistive labeling to winnow out texts with incorrect labels. |
| Outcome: | a new human-in-the-loop data inspection technique can winnow out texts with incorrect labels . the technique can reduce human inspection effort by combining provenance inspection and assistive labeling . |
COMMIT: Code-Mixing English-Centric Large Language Model for Multilingual Instruction Tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | COMMIT improves the exact match score of low-resource language QA by 32x. |
| Approach: | They propose to specialize instruction tuning to deviate from English-centric instruction tuning . they propose to perform cross-lingual alignment to overcome data imbalance . |
| Outcome: | The proposed method improves the exact match score of low-resource language QA by 32x. |
DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to extract word embeddings from training datasets are not efficient for training other models. |
| Approach: | They propose a method to distill a training dataset into a textual model by combining a small number of informative synthetic samples. |
| Outcome: | The proposed method outperforms existing methods on training datasets and language models. |
MindAgent: Emergent Gaming Interaction (2024.findings-naacl)
Copied to clipboard
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Yusuke Noda, Zane Durante, Zilong Zheng, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao, Hoi Vo
| Challenge: | Large foundation models (LFMs) can perform complex scheduling in a multi-agent system and can coordinate agents to complete complex tasks that require extensive collaboration. |
| Approach: | They propose a gaming-based infrastructure that evaluates LFMs' planning and coordination capabilities in the context of gaming interaction. |
| Outcome: | The proposed infrastructure can be deployed in a customized VR version of Cuisineworld and adapted in the “Minecraft” domain. |
BotChat: Evaluating LLMs’ Capabilities of Having Multi-Turn Dialogues (2024.findings-naacl)
Copied to clipboard
Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Modern Large Language Models (LLMs) facilitate high-quality, multi-turn dialogues with humans, but human-based evaluation of such a capability requires substantial manual effort. |
| Approach: | They propose to evaluate LLMs' ability to emulate human-like, multi-turn conversations using an LLM-centric approach. |
| Outcome: | The proposed model emulates human-like, multi-turn conversations using an LLM-centric approach. |
Learning Mutually Informed Representations for Characters and Subwords (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models rely on subword tokenization to process text as a sequence of subwords. |
| Approach: | They propose a character-subword language model that integrates character and subword modalities into one model. |
| Outcome: | The proposed model outperforms its backbone language models on English sequence labeling and classification tasks. |
A Novel Two-step Fine-tuning Framework for Transfer Learning in Low-Resource Neural Machine Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing transfer learning methods for neural machine translation use a well-trained translation model to initialize a child model with corresponding datasets. |
| Approach: | They propose a two-step fine-tuning framework for transfer learning in low-resource neural machine translation that adjusts the parent model to fit the child language by using the child source data. |
| Outcome: | The proposed framework improves on five low-resource translations on high-resolution languages. |
Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current approaches to obtain cross-lingual sentence embeddings rely on pre-trained language models that implicitly align the contextual representations of similar units of sentences in different languages. |
| Approach: | They propose a framework that explicitly aligns words between English and eight low-resource languages by using off-the-shelf word alignment models. |
| Outcome: | The proposed framework improves on the bitext retrieval task and in high-resource languages. |
C3LPGCN:Integrating Contrastive Learning and Cooperative Learning with Prompt into Graph Convolutional Network for Aspect-based Sentiment Analysis (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that graph convolutional networks (GCNs) can model syntactic information but incorrect syntaktic structure may introduce additional noise. |
| Approach: | They propose a graph convolutional network which integrates Contrastive Learning and Cooperative Learning with Prompt into GCN to alleviate the noise when modeling syntactic information. |
| Outcome: | The proposed model outperforms state-of-the-art models on three datasets and significantly outperformed existing models. |
Visual Enhanced Entity-Level Interaction Network for Multimodal Summarization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to generate concise summarizations rely on coarse-grained textual and visual information, but they are underutilized. |
| Approach: | They propose a Visual Enhanced Entity-Level Interaction Network to address underutilization of multimodal inputs at a fine-grained level. |
| Outcome: | The proposed model outperforms existing models on two MMS datasets and proposes new metrics to measure factual consistency of entities in the output. |
Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have explored multiple aspects that affect the performance of large language models (LLMs) such as input-output mapping, extensive data resources, and the ability to train on labeled examples. |
| Approach: | They propose a framework that injects knowledge into LLMs during continual self-supervised pre-training and judiciously selects examples with high knowledge relevance. |
| Outcome: | The proposed framework outperforms baseline models and improves by more than 13% and 7% on text classification and question-answering tasks. |
Time Machine GPT (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models are often trained on extensive, temporally indiscriminate text corpora . conventional methods for creating temporal adapted models depend on pre-training static models on time-specific data. |
| Approach: | They propose a series of point-in-time LLMs called TimeMachineGPT to be nonprognosticative . time-series forecasting and event prediction aim to infer a future state from past data . authors propose linguistically-based models that can be used to predict future events . |
| Outcome: | The proposed model is nonprognosticative and ensures it remains uninformed about future factual information and linguistic changes. |
An End-to-End Submodular Framework for Data-Efficient In-Context Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in natural language tasks leverage the emergent In-Context Learning ability of pretrained Large Language Models (LLMs). |
| Approach: | They propose a framework for exemplar selection for in-context learning that uses a pool-based active learning approach to select Diverse and informative exemplars from the target tasks’ unlabeled pool. |
| Outcome: | The proposed framework outperforms existing methods for data annotation and similarity-based methods for test query-specific exemplar retrieval on 7 different NLP datasets and 5 LLMs of varying complexities. |
Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to adapt pretrained Large Language Models to new lower-resource languages are limited to English. |
| Approach: | They propose to combine cross-lingual instruction-tuning with additional monolingual pretraining to adapt LLMs to new lower-resource languages. |
| Outcome: | The proposed model is the first open-source instruction-following LLM for Estonian . the proposed model improves commonsense reasoning and multi-turn conversation capabilities . |
Simulating Opinion Dynamics with Networks of LLM-based Agents (2024.findings-naacl)
Copied to clipboard
Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, Timothy Rogers
| Challenge: | Existing approaches to simulating opinion dynamics often over-simplify human behavior . authors propose refining LLMs with real-world discourse to better simulate evolution of beliefs . |
| Approach: | They propose to use large language models to simulate opinion dynamics in groups of simulated agents . they found that LLM agents produce more accurate information than ABMs . |
| Outcome: | The proposed model can be used to better simulate opinion dynamics in real-world discourses. |
Probing the Category of Verbal Aspect in Transformer Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a particular challenge is posed by ”alternative contexts” where either the perfective or the imperfective aspect is suitable grammatically and semantically. |
| Approach: | They investigate how pretrained language models encode the grammatical category of verbal aspect in Russian. |
| Outcome: | The proposed model has high predictive uncertainty about aspect in alternative contexts, the authors show . |
A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study aims to assess linguistic diversity of multilingual data sets against a reference language sample . linguistic diversity is typically measured as the number of languages included in the data set . but such measures do not consider structural properties of the included languages . |
| Approach: | They propose to measure linguistic diversity against a reference language sample to maximise linguistic diversity. |
| Outcome: | The proposed measure can be used to identify the types of languages that are not represented in a data set. |
Beyond Read-Only: Crafting a Comprehensive Chinese Text-to-SQL Dataset for Database Manipulation and Query (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current research focuses mainly on read operations and ignores other aspects of database operations such as create, update, and delete operations. |
| Approach: | They propose a large-scale cross-domain single-table CRUD operations Chinese Text-to-SQL dataset . the dataset contains 10,000 question/SQl pairs involving 625 tables from different domains . |
| Outcome: | The proposed method achieves 67.08% and 83.8% exact set matching accuracy under read and delete operations, but only 49.6% and 61.8% under create and update operations. |
Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new method to normalize orthographic variations of historical documents is needed for digital humanities and diachronic studies. |
| Approach: | They propose to normalize orthographic wordforms found in Middle French archives . authors say it improves accuracy and accuracy over a strong baseline . |
| Outcome: | The proposed methods normalize orthographic variations of historical documents without modernizing them. |
Anti-LM Decoding for Zero-shot In-context Machine Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot learning with large language models are poorly calibrated for zero-shoot tasks. |
| Approach: | They propose a contrastive decoding objective with a decay factor to address in-context bias . they conduct experiments on 3 model types and sizes, 3 language directions, and beam search . |
| Outcome: | The proposed method outperforms state-of-the-art decoding objectives with 20 BLEU points improvement from the default objective in some settings. |
Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are not effective for weight-poisoning backdoor attacks. |
| Approach: | They propose a parameter-efficient fine-tuning (PEFT) method that updates only a limited set of model parameters and provides a robust defense against weight-poisoning backdoor attacks. |
| Outcome: | The proposed method identifies poisoned samples through confidence and is robust against weight-poisoning backdoor attacks. |
Select and Summarize: Scene Saliency for Movie Script Summarization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models for summarizing long-form narrative texts are computationally and memory limited. |
| Approach: | They propose a scene saliency dataset that consists of human-annotated salient scenes for 100 movies. |
| Outcome: | The proposed model outperforms state-of-the-art models and reflects the information content of a movie more accurately than a model that takes the whole movie script as input. |
Don’t be a Fool: Pooling Strategies in Offensive Language Detection from User-Intended Adversarial Attacks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Offensive language detection is important for filtering out abusive expressions, authors argue . authors propose user-intended adversarial attacks that insert special symbols or leverage distinctive features of the Korean language. |
| Approach: | They propose user-intended adversarial attacks that insert special symbols or leverage the distinctive features of the Korean language. |
| Outcome: | The proposed models are more robust to performance degradation even when the attack rate is increased, compared to models trained on noisy texts. |
Z-GMOT: Zero-shot Generic Multiple Object Tracking (2024.findings-naacl)
Copied to clipboard
Kim Tran, Anh Duy Le Dinh, Tien-Phat Nguyen, Thinh Phan, Pha Nguyen, Khoa Luu, Donald Adjeroh, Gianfranco Doretto, Ngan Le
| Challenge: | Existing approaches to Multi-Object Tracking (MOT) rely on initial bounding boxes and struggle with unseen objects. |
| Approach: | They propose a cutting-edge multi-object tracking solution that can track unseen objects . they propose iGLIP and MA-SORT, which integrate motion and appearance matching strategies . |
| Outcome: | The proposed solution can track objects from never-seen categories without initial bounding boxes or predefined categories. |
NLP for Counterspeech against Hate: A Survey and How-To Guide (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on the challenges of analysing, collecting, classifying, and automatically generating counterspeech, to reduce the huge burden of manually producing it. |
| Approach: | They propose a guide for doing research on counterspeech, with detailed examples and best practices that can be learnt from the NLP community. |
| Outcome: | The proposed strategies can reduce online and offline violence while preserving the freedom of speech of the users. |
PRODIGy: a PROfile-based DIalogue Generation dataset (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing profiles-based dialogue datasets lack explicit profile representations or are difficult to collect. |
| Approach: | They propose a dataset that brings together multiple profiles for each speaker, and then integrates them together to provide a more comprehensive profile dimension set for generative language models. |
| Outcome: | The PRODIGy dataset provides a more comprehensive profile dimension set for each speaker. |
WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that small, context-dependent shifts in word distributions can be used to apply and detect watermarks, but little work has analyzed the impact of these perturbations on the quality of generated texts. |
| Approach: | They propose a framework that allows for analysis of the impact of watermark settings on the quality of generated texts. |
| Outcome: | The proposed framework provides easy visualization of the quality-detection trade-off of watermark settings. |
Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated increasing power, but they also have vulnerabilities. |
| Approach: | They propose a black-box attack that targets the cognitive structure and processes of large language models (LLMs) they propose defending cognitive overload attacks from three perspectives. |
| Outcome: | The proposed attack is a black-box attack with no need for knowledge of model architecture or access to model weights. |
PAELLA: Parameter-Efficient Lightweight Language-Agnostic Captioning Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that only generate English captions are expensive due to the trend of scaling both data and model size. |
| Approach: | They propose a parameter-efficient lightweight language-agnostic captioning model that uses retrieval enhancement to train parameters between a visual model and a multilingual language model. |
| Outcome: | The proposed model outperforms models with more parameters and data and shows strong zero-shot abilities in low-resource languages. |
OSCaR: Object State Captioning and State Change Representation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to extrapolate and comprehend changes in object states are limited . relying on a small set of symbolic words to represent changes has restricted expressiveness of language. |
| Approach: | They propose a dataset and benchmark to evaluate multimodal large language models . they investigate causal relations between a concrete action and the change . |
| Outcome: | The proposed method achieves near parity with GPT-4V ratings across helpfulness, accuracy, reasoning, and other key metrics. |
SumCSE: Summary as a transformation for Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Sentence embedding models are typically trained using contrastive learning (CL) using human annotations directly or by repurposing other annotated datasets. |
| Approach: | They propose to use generative language models to generate CL data using annotated data. |
| Outcome: | The proposed method outperforms the previous best unsupervised method by 1.8 points and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark. |
The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study examines the effects of training language models on synthetic data generated by their predecessors. |
| Approach: | They propose to use recursive finetuning techniques to assess linguistic diversity of models. |
| Outcome: | The proposed metrics show a decrease in diversity of model outputs through successive iterations, especially for tasks demanding high levels of creativity. |
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can generate content that aligns with their assigned personality traits, but there is limited research on whether they consistently reflect specific personality traits. |
| Approach: | They propose to study the behavior of LLM-based agents which they refer to as LLM personas and simulate them to measure their personality traits. |
| Outcome: | The proposed model is based on the Big Five personality model and has been validated by human evaluations and automatic evaluations. |
FIRE: A Dataset for Financial Relation Extraction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) and Relation Extraction (RE) datasets require extensive linguistic and domain knowledge, making dataset creation costly and labor-intensive. |
| Approach: | They introduce a sentence-level dataset of named entities and relations within the financial sector that encapsulates 13 named entity types along with 18 relation types. |
| Outcome: | The proposed dataset encapsulates 13 named entity types along with 18 relation types and was labeled by a single annotator to minimize labeling noise. |
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response (2024.findings-naacl)
Copied to clipboard
Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, Emmanouil Benetos
| Challenge: | Large Language Models have shown immense potential in multimodal applications, but convergence between textual and musical domains remains unexplored. |
| Approach: | They propose a system that aligns music representations with a frozen LLM . they train the system on an extensive music caption dataset and fine-tune it with instructional data . |
| Outcome: | The proposed system bridges the gap between music audio and textual contexts by combining music captions with a frozen model . it performs well in generating music caption and composing music-related Q&A pairs . the proposed system is available for free download at http://www.musilingo.com/ . |
Investigating Acceleration of LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with ‘LITE’ (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have remarkable performance across a wide variety of tasks, however, their large size makes their inference slow and computationally expensive. |
| Approach: | They propose to perform 'dynamic confidence-based early exiting' at token level from the intermediate layers which improves the computational efficiency of text generation without sacrificing the quality of the generation. |
| Outcome: | The proposed model achieves significant cost and quality improvements while maintaining the quality of the generation. |
Instruction-following Evaluation through Verbalizer Manipulation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on common instructions that align well with what the model learned during training, but proficiency in responding to these instructions does not necessarily imply strong ability in instruction following. |
| Approach: | They propose a new instruction-following evaluation protocol called verbalizer manipulation that instructs the model to verbalize the task label with words aligning with model priors to different extents. |
| Outcome: | The proposed protocol can be integrated with any classification benchmark to examine the model’s reliance on priors and its ability to override them to accurately follow the instructions. |
WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work to control software has used reinforcement learning (RL), requiring many demonstrations and trials to learn simple interaction tasks. |
| Approach: | They propose a Large Language Model to automatically perform web software tasks using click, scroll, and text in- put operations using filtered Document Object Models as observations. |
| Outcome: | The proposed method performs better on the MiniWob++ benchmark with only one in-context example. |
CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)
Copied to clipboard
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
| Challenge: | Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases. |
| Approach: | They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. |
| Outcome: | Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art. |
Prompting Few-shot Multi-hop Question Generation via Comprehending Type-aware Semantics (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches for multi-hop question generation rely on large annotated data . supervised approaches rely only on large labeled data, making it hard to perform tasks. |
| Approach: | They propose a type-aware semantics extraction-based chain-of-thought method for multi-hop question generation for documents . they first extract question types and essential semantic phrases from the given documents and the answer . |
| Outcome: | The proposed approach extracts question types and essential semantic phrases from documents and the answer. |
When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose guidelines for when to implement self-reflection in Large Language Models. |
| Outcome: | The proposed approach improves the reasoning capabilities of Large Language Models under a more stringent evaluation setting, and reduces tendency toward majority voting. |
CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | a low-resource dataset is limited in training data, so generating task-specific data is challenging. |
| Approach: | They propose a data augmentation technique that prompts off-the-shelf instruction-following Large Language Models to generate augmentations. |
| Outcome: | The proposed technique outperforms baselines on 11 datasets spanning 3 tasks and 3 low-resource settings. |
Synonym relations affect object detection learned on vision-language data (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models that accept textual input are not robust to variations in how input is provided. |
| Approach: | They propose two approaches to improve vision-language object detectors' performance . they use back-translation and class embedding enrichment to improve their models . |
| Outcome: | The proposed approaches improve performance on synonyms from mAP@0.3=33.87% to 37.93%. |
CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Neural Text-to-Speech systems are a promising approach for high-fidelity speech synthesis . but the efficiency of multi-step sampling in Diffusion Models presents challenges . |
| Approach: | They propose a novel architecture grounded in consistency models to improve model convergence. |
| Outcome: | The proposed architecture achieves top-quality speech synthesis in fewer steps without adversarial training or pre-trained model dependencies. |
RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained language models exhibit poor generalization and robustness in adversarial settings. |
| Approach: | They propose a self-supervised sentence embedding framework that improves generalization and robustness against adversarial attacks. |
| Outcome: | The proposed framework reduces the success rate of adversarial attacks by almost half . it also improves semantic text similarity tasks and various transfer tasks . |
Characterizing Human and Zero-Shot GPT-3.5 Object-Similarity Judgments (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have yielded few-shot, human-comparable performance on a range of tasks, but studies of LLM annotation accuracy and behavior are sparse. |
| Approach: | They characterize OpenAI’s GPT-3.5’s judgment on a behavioral task for implicit object categorization and give similarities and differences between them. |
| Outcome: | The proposed model augments human responses with LLMs for domains where data is sparse or compute resources are low. |
Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods that rely on limited demos and out-of-demonstration (OOD) queries fail when faced with out- of-demotion queries. |
| Approach: | They propose a query-aware prompting method that elicits the inherent generalizability of large language models by query-based demo generation. |
| Outcome: | The proposed method outperforms state-of-the-art methods in the OOD setting and two public math benchmarks. |
Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Event temporal reasoning aims at identifying the temporal relations between two or more events from narratives. |
| Approach: | They propose to detect knowledge conflicts in event temporal reasoning using bias indicators such as event relation prior bias, tense bias, narrative bias, and dependency bias. |
| Outcome: | The proposed method can be applied to Pre-trained Language Models and Large Language Model (LLMs) as additional training data or demonstrations for In- Context Learning. |
MCECR: A Novel Dataset for Multilingual Cross-Document Event Coreference Resolution (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing datasets for event coreference resolution focus on within-document event coreference and English text, lacking cross-document ECR datasets beyond English. |
| Approach: | They propose a multiligual dataset that manually annotates documents for event mentions and coreference in five languages. |
| Outcome: | The proposed dataset annotates documents for event mentions and coreference in five languages . the dataset fetches related news articles from the google search engine to increase the number of non-singleton clusters . |
Sentiment Analysis in the Era of Large Language Models: A Reality Check (2024.findings-naacl)
Copied to clipboard
| Challenge: | Sentiment analysis (SA) has been a long-standing research area in natural language processing. |
| Approach: | They propose a benchmark to evaluate LLMs' SA abilities and propose 'sentiEval' benchmark to be used for a more comprehensive evaluation. |
| Outcome: | The proposed benchmark outperforms small language models on 26 datasets on 13 tasks and compared them with LLMs trained on domain-specific datasets. |
Tokenizer Choice For LLM Training: Negligible or Crucial? (2024.findings-naacl)
Copied to clipboard
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, Nicolas Flores-Herr
| Challenge: | Recent success of large language models has been driven by curating the training dataset composition, scaling of model architectures and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. |
| Approach: | They conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale. |
| Outcome: | The proposed model can significantly impact the model's downstream performance and training costs. |
Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue (2024.findings-naacl)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) improves capabilities of dialogue systems . but they lack communication skills, which make them more like information seeking tools . |
| Approach: | They propose to empower LLMs with communication skills through inner monologues . they use a benchmark to evaluate the dialogue generation ability of the model . |
| Outcome: | The proposed model outperforms the baselines in the evaluation of communication skills. |
The Impact of Differential Privacy on Group Disparity Mitigation (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study evaluated the impact of differential privacy on fairness across four tasks. |
| Approach: | They evaluate the impact of differential privacy on fairness across four diverse tasks . they train (,)-differentially private models with empirical risk minimization . |
| Outcome: | The proposed model shows that differential privacy increases performance differences between groups . the model also reduces performance differences in the robust setting . |
Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Traditional Automatic Video Dubbing (AVD) pipelines use isometric-NMT algorithms to regulate the length of the output text. |
| Approach: | They propose an isometric-NMT system that regulates the length of the output text . they propose a phoneme Count Compliance score to measure length compliance . |
| Outcome: | The proposed approach improves phoneme count compliance scores by 36% compared to state-of-the-art models in English-Hindi language pairs. |
Read between the lines - Functionality Extraction From READMEs (2024.findings-naacl)
Copied to clipboard
| Challenge: | While text summarization is a well-known NLP task, functionality extraction from Git README files is based on a novel and useful variant of it. |
| Approach: | They propose a novel task called functionality extraction from Git README files that detects all the functionalities supported by the corresponding application software. |
| Outcome: | The proposed model outperforms baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. |
AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph (2024.findings-naacl)
Copied to clipboard
Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, Xin Liu, Yangqiu Song
| Challenge: | Existing language models only touch nouns or verbs within simplified events or specific domains. |
| Approach: | They propose an entailment graph that collects abstract knowledge for 3 components of diverse events to comprehensively evaluate the abstraction ability of language models. |
| Outcome: | The proposed benchmark improves LLMs across two previous abstraction tasks. |
Few-TK: A Dataset for Few-shot Scientific Typed Keyphrase Recognition (2024.findings-naacl)
Copied to clipboard
| Challenge: | Named Entities are a common form of Information Extraction (IE) tasks for scientific texts. |
| Approach: | They propose a rechristening of Named Entities as Typed Keyphrases (TK) they advocate for exploring this task in the few-shot domain due to the scarcity of labeled scientific IE data. |
| Outcome: | The proposed dataset includes scientific Typed Keyphrase annotations on abstracts of 500 research papers. |
Language Models can be Deductive Solvers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. |
| Approach: | They propose a novel language model that internalizes and emulates the reasoning processes of logical solvers and avoids parsing errors by learning strict adherence to solver syntax and grammar. |
| Outcome: | The proposed model outperforms state-of-the-art solver-augmented language models and few-shot prompting methods on public deductive reasoning benchmarks. |
Interpreting User Requests in the Context of Natural Language Standing Instructions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to LLM-based dialogue modeling provide additional context for users to make requests. |
| Approach: | They propose an approach to LLM-based dialogue modeling where persistent user constraints and preferences are provided as additional context for such interfaces. |
| Outcome: | The proposed model achieves a maximum of 46% exact match on the prediction of 2.4K English dialogues with a language-to-program dataset. |
Secure Your Model: An Effective Key Prompt Protection Mechanism for Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent years have seen an unprecedented surge in the development and application of large language models (LLMs) however, the development of LLMs is a complex endeavor, requiring substantial investments in terms of financial and computational resources. |
| Approach: | They propose a mechanism wherein a unique key prompt is embedded within the LLM to protect it from unauthorized access and potential theft. |
| Outcome: | The proposed protection can protect the model without significantly impacting its original function. |
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Do Prompt Positions Really Matter? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prompt-based learning models have a high level of interest due to their ability to perform zero-shot and fewshot tasks. |
| Approach: | They conduct the most comprehensive analysis to date of prompt position for diverse natural language processing tasks. |
| Outcome: | The proposed model is more robust than previous models and is consistent even in instruction-tuned models. |
Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning (2024.findings-naacl)
Copied to clipboard
Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, James Glass
| Challenge: | Existing methods for surfacing symbolic reasoning capabilities are limited to narrow tasks . arithmetic computations are unnatural to perform in pure language space, and hence present difficulties for LLMs. |
| Approach: | They propose a natural language embedded program framework for solving symbolic reasoning tasks. |
| Outcome: | The proposed framework improves on strong baselines across math and symbolic reasoning, text classification, question answering, and instruction following tasks. |
A Study on Scaling Up Multilingual News Framing Analysis (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on media framing have focused on English only data, leaving a gap in research concerning multilingual contexts. |
| Approach: | They propose to use crowd-sourced datasets to automate framing analysis by automating translation and annotation. |
| Outcome: | The proposed system improves on existing models in Bengali and Portuguese . the proposed system can train on a crowd-sourced dataset in 12 languages . |
ViGLUE: A Vietnamese General Language Understanding Benchmark and Analysis of Vietnamese Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language understanding have been suggested, but there is a lack of such a benchmark in Vietnamese due to the difficulty in accessing datasets or the scarcity of task-specific datasets. |
| Approach: | They propose to use a benchmark to evaluate Vietnamese language models in a variety of tasks and areas to explore the relationship between specific tasks and the number of shots. |
| Outcome: | The proposed benchmark contains twelve tasks and encompasses over ten areas and subjects, enabling it to evaluate models comprehensively over a broad spectrum of aspects. |
Exploring the Trade-off Between Model Performance and Explanation Plausibility of Text Classifiers Using Human Rationales (2024.findings-naacl)
Copied to clipboard
| Challenge: | Saliency post-hoc explainability methods are important tools for understanding complex NLP models, but they may not align with human intuition, making the explanations not plausible. |
| Approach: | They propose a method for incorporating rationales into text classification models by augmenting the standard cross-entropy loss with a novel loss function inspired by contrastive learning. |
| Outcome: | The proposed approach enhances the plausibility of post-hoc explanations while preserving their faithfulness. |
Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) methods are important in low-resource language (LRL) Neural Machine Translation (NMT) but their practical effectiveness varies significantly across different languages. |
| Approach: | They evaluated the performance of 8 parameters-efficient fine-tuning methods with 15 architectures using the SacreBLEU score. |
| Outcome: | The Houlsby+Inversion adapter outperforms the baseline architectures in both in-domain and out-domain tests and the Houlson+Inverter achieves the best performance overall. |
ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)
Copied to clipboard
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot
| Challenge: | Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. |
| Approach: | They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them. |
| Outcome: | The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft. |
Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations (2024.findings-naacl)
Copied to clipboard
| Challenge: | supervised systems have not replaced dedicated supervised models for machine translation tasks. |
| Approach: | They propose to guide LLMs to post-edit MT with feedback from MQM annotations . they then fine-tune the LLM to improve its ability to exploit the feedback . |
| Outcome: | The proposed model improves TER, BLEU and COMET scores on Chinese-English, English-German and English-Russian data. |
Non-contrastive sentence representations via self-supervision (2024.findings-naacl)
Copied to clipboard
| Challenge: | Text embeddings are an important tool for a variety of NLP tasks. |
| Approach: | They compare sample contrastive methods with the standard baseline for contrastive sentence embeddings, SimCSE, and a class of self-supervised non-contrastive loss functions and methods. |
| Outcome: | The proposed methods outperform the standard baseline for contrastive sentence embeddings, SimCSE, on downstream tasks without auxiliary loss functions. |
Semantically-Prompted Language Models Improve Visual Descriptions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Language-vision models have made significant progress in zeroshot vision tasks, but lack expressive visual descriptions. |
| Approach: | They propose a new method for generating visual descriptions with pre-trained language models and semantic knowledge bases. |
| Outcome: | The proposed method improves visual descriptions and achieves strong results on image-classification datasets. |
GenTKG: Generative Forecasting on Temporal Knowledge Graph with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for temporal relational forecasting are limited and require limited training data. |
| Approach: | They propose a retrieval-augmented generation framework that uses temporal logical rule-based retrieval and parameter-efficient instruction tuning to solve temporal knowledge forecasting challenges. |
| Outcome: | The proposed framework outperforms conventional methods in the temporal knowledge graph domain with low computation resources. |
A Transformer with Stack Attention (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent research suggests that transformer-based language models fail to learn basic algorithmic patterns. |
| Approach: | They propose to augment transformer-based language models with a differentiable stack-based attention mechanism that adds a level of interpretability to the model. |
| Outcome: | The proposed model can model some, but not all, deterministic context-freelanguages. |
InstructEval: Systematic Evaluation of Instruction Selection Methods (2024.findings-naacl)
Copied to clipboard
| Challenge: | In-context learning (ICL) performs tasks by prompting a large language model using an instruction and a small set of annotated examples. |
| Approach: | They develop an ICL evaluation suite to evaluate the performance of popular instruction selection methods. |
| Outcome: | The proposed evaluation suite compares instruction selection methods over five metrics relevant to ICL. |
RecMind: Large Language Model Powered Agent For Recommendation (2024.findings-naacl)
Copied to clipboard
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Yanbin Lu, Xiaojiang Huang, Yingzhen Yang
| Challenge: | Existing recommendations systems are limited in generalizing to new tasks due to model scale and data size constraints. |
| Approach: | They propose an LLM-powered autonomous recommender agent, RecMind, which is capable of leveraging external knowledge to provide zero-shot personalized recommendations. |
| Outcome: | The proposed model outperforms existing zero/few-shot LLM-based recommendation baseline methods in various tasks and achieves comparable performance to a fully trained recommendation model P5. |
GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to generate data using LLMs are limited by sampling from the center of original content distribution. |
| Approach: | They propose a task-agnostic data generation and knowledge distillation framework for LLMs that employs an iterative out-of-distribution-guided feedback mechanism to generate data. |
| Outcome: | The proposed framework outperforms prior arts and the LLM on 10 different classification tasks and noisey generated data. |
How Lexical is Bilingual Lexicon Induction? (2024.findings-naacl)
Copied to clipboard
| Challenge: | lexical variation and low-resource settings make it difficult to learn in low-level settings. |
| Approach: | They propose to incorporate additional lexical information into the retrieve-and-rank approach to improve lexicon induction. |
| Outcome: | The proposed approach improves on XLING by an average of 2% across all language pairs. |
Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability (2024.findings-naacl)
Copied to clipboard
| Challenge: | ChatGPT is a powerful NLP tool but its language identification abilities are unclear. |
| Approach: | They compile a benchmark comprising 670 languages representing 23 language families spoken in five continents and compare their language identification abilities to ChatGPT's (both GPT-3.5 and GPT-4) performance. |
| Outcome: | The proposed model performs poorly on African languages, while GPT-3.5 and GPT-4 perform poorly on English, Afrikaans, Arabic, Indonesian, Italian, Mandarin Chinese, and several more. |
Targeted Augmentation for Low-Resource Event Extraction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for low-resource information extraction struggle to strike a balance between weak augmentation and drastic augmentation. |
| Approach: | They propose a data augmentation paradigm that uses back validation and targeted augmentation to produce augmented examples with enhanced diversity, polarity, accuracy, and coherence. |
| Outcome: | The proposed paradigm produces augmented examples with enhanced diversity, polarity, accuracy, and coherence. |
Asking More Informative Questions for Grounded Retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to question generation for interactive retrieval have constrained answer spaces, limiting the amount of information a model can gain in a single turn. |
| Approach: | They propose a method that incorporates presupposition handling into question selection and belief updates. |
| Outcome: | The proposed method increases accuracy over the past state-of-the-art by 14% while resulting in 48% more efficient games in human evaluations. |
Efficient Citer: Tuning Large Language Models for Enhanced Answer Quality and Verification (2024.findings-naacl)
Copied to clipboard
Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing models with explicit citations lack the ability to verify information generated by these models. |
| Approach: | They construct a citation training dataset and fine-tune two models to address the challenge of explicit citations efficiently. |
| Outcome: | The proposed models surpass ChatGPT and exhibit exceptional out-of-domain generalization in both human and automatic evaluation. |
Addressing Healthcare-related Racial and LGBTQ+ Biases in Pretrained Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) propagate social stigmas and stereotypes, a critical concern given their widespread use. |
| Approach: | They adapt two intrinsic bias benchmarks to quantify racial and LGBTQ+ biases in prevalent PLMs and empirically evaluate the effectiveness of various debiasing methods in mitigating these biase. |
| Outcome: | The proposed methods reduce biases without compromising performance in downstream tasks. |
ATG: Benchmarking Automated Theorem Generation for Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing generative language models (LMs) can generate new or reusable theorems, but their ability to generate new theorels is under-explored. |
| Approach: | They propose to use Metamath library to generate new theorems that can be saved as reusable knowledge for future theoretical proving. |
| Outcome: | The proposed benchmark evaluates whether an agent can generate valuable (and possibly brand new) theorems that are applicable for downstream theoretic proving as reusable knowledge. |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation (ERC) is a task that aims to identify the emotions behind each utterance in a conversation. |
| Approach: | They propose an Emotion-Anchored Contrastive Learning framework that generates more distinguishable utterance representations for similar emotions. |
| Outcome: | The proposed framework achieves state-of-the-art on similar emotions and performs well on similar ones. |
SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | SUQL is a conversational language that supports the generality of hybrid data access for large knowledge corpora. |
| Approach: | They propose a conversational agent that supports the full generality of hybrid data access for large knowledge corpora using SUQL. |
| Outcome: | The proposed language can handle hybrid data sources. |
On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study evaluates how Large Language Models interact with a SQL interpreter . the model is limited in context and is stochastic, making it less suited for tasks requiring high precision and extensive computations. |
| Approach: | They propose and evaluate two interaction strategies to evaluate how LLMs interact with a SQL interpreter. |
| Outcome: | The proposed framework improves the accuracy and reliability of the evaluations. |
CARE: Extracting Experimental Findings From Clinical Literature (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing annotation schemas and datasets fail to capture real-world complexity and nuance of experimental findings. |
| Approach: | They propose a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes. |
| Outcome: | The proposed schema captures fine-grained findings as n-ary relations between entities and attributes. |
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)
Copied to clipboard
Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao
| Challenge: | Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive . |
| Approach: | They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training . |
| Outcome: | The proposed framework achieves superior performance compared with baselines. |
SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods have significantly boosted the performance of Knowledge Base Question Generation (KBQG) through pre-trained language models thanks to the richly endowed semantic knowledge. |
| Approach: | They propose a framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG by combining skeleton heuristic guidance with a soft prompting approach. |
| Outcome: | The proposed framework incorporates "skeleton heuristics" which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions. |
Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer (2024.findings-naacl)
Copied to clipboard
| Challenge: | Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedically knowledge base graph, ignoring the inter-concept interactions and a concept’s local neighborhood. |
| Approach: | They propose a Graph-Augmented Multi-Objective Transformer which captures both inter-concept and intra-conception interactions from the multilingual UMLS graph. |
| Outcome: | The proposed model captures inter- and intra-concept interactions from the multilingual UMLS graph using pre-trained language models and graph neural networks. |
Cross-Lingual Summarization with Pseudo-Label Regularization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization use only a single reference, resulting in an underrepresented hypothesis space. |
| Approach: | They propose to use pseudo-labels to regularize cross-lingual summarization training by combining a single reference and a network to perform the model training. |
| Outcome: | The proposed approach significantly improves over gold reference training in XLS with 8 languages from different families. |
On the Way to Gentle AI Counselor: Politeness Cause Elicitation and Intensity Tagging in Code-mixed Hinglish Conversations for Social Good (2024.findings-naacl)
Copied to clipboard
| Challenge: | Politeness is a multifaceted concept influenced by individual perceptions of what is considered polite or impolite. |
| Approach: | They propose a task to identify the underlying reasons behind the use of politeness and gauge the degree of politity conveyed. |
| Outcome: | The proposed method is compared against state-of-the-art datasets and their results show it is superior. |
Leveraging Summarization for Unsupervised Dialogue Topic Segmentation (2024.findings-naacl)
Copied to clipboard
Aleksei Artemiev, Daniil Parinov, Alexey Grishanov, Ivan Borisov, Alexey Vasilev, Daniil Muravetskii, Aleksey Rezvykh, Aleksei Goncharov, Andrey Savchenko
| Challenge: | Existing methods to segment textual data are difficult to handle for noisy spoken dialogues. |
| Approach: | They propose to leverage dialogue summaries for unsupervised topic segmentation . they show that the new approach outperforms state-of-the-art methods in unsupervised segmentation and requires less setup . |
| Outcome: | The proposed approach outperforms state-of-the-art methods in unsupervised topic segmentation and requires less setup. |
LLaMA-Rider: Spurring Large Language Models to Explore the Open World (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have used Large Language Models to help decision-making and planning in environments, but their capacity to acquire environmental knowledge and adapt in an open world remains uncertain. |
| Approach: | They propose an approach to spur LLMs to explore the open world, gather experiences, and learn to improve their task-solving capabilities by using a feedback-revision mechanism. |
| Outcome: | The proposed model enhances the efficiency of the LLM in exploring the open world and improves its ability to accomplish more tasks through fine-tuning with merely 1.3k instances of collected data, showing minimal training costs compared to baseline using reinforcement learning. |
Contrastive Learning as a Polarizer: Mitigating Gender Bias by Fair and Biased sentences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have highlighted social biases inherent in training data can lead models to learn and propagate them. |
| Approach: | They propose a contrastive learning method that uses anchor points to push further negatives and pull closer positives within the representation space. |
| Outcome: | The proposed method achieves state-of-the-art in the ICAT score on the StereoSet, a benchmark for measuring bias in models. |
PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have recognized hallucination as a notable concern in large autoregressive language models (LLMs). |
| Approach: | They propose a polygraph for large language models that detects "hallucination" they demonstrate that hallucination can be detected by tractable probabilistic models . |
| Outcome: | The proposed model outperforms state-of-the-art methods on open-source LLMs by 20% on TruthfulQA benchmarks. |
Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing question answering systems rely on pre-selected and annotated evidence documents, thus making them inadequate for addressing novel questions. |
| Approach: | They propose to use the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents to answer health questions from three diverse datasets. |
| Outcome: | The proposed approach improves the macro F1 score by 10% by utilizing the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents. |
DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity. |
| Approach: | They propose a method to analyze encoder-decoder Transformers by using the decoder module Model Output encoder to cross-attend representations of intermediate encoder activations instead of using the default output. |
| Outcome: | The proposed method maps uninterpretable representations to human-interpreted sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models. |