Findings of the Association for Computational Linguistics: ACL 2024
Controllable Data Augmentation for Few-Shot Text Mining with Chain-of-Thought Attribute Manipulation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for data augmentation generate new examples wildly without proper control, which hinders the usefulness of the proposed approach. |
| Approach: | They propose a chain-of-thought attribute manipulation approach that generates new data from existing examples by tweaking in the user-provided attribute. |
| Outcome: | The proposed approach generates new data from existing examples by tweaking in the user-provided, task-specific attribute, e.g., sentiment polarity or topic in movie reviews. |
Match More, Extract Better! Hybrid Matching Model for Open Domain Web Keyphrase Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for keyphrase extraction use noisy information to filter the salient phrases from the document. |
| Approach: | They propose a hybrid matching model that combines representation-focused and interaction-based matching modules into a unified framework for improving keyphrase extraction. |
| Outcome: | The proposed model outperforms state-of-the-art keyphrase extraction models on the OpenKP dataset. |
AFPQ: Asymmetric Floating Point Quantization for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-bit weight quantization can save memory and accelerate inference. |
| Approach: | They propose asymmetric FP quantization which sets separate scales for positive and negative values. |
| Outcome: | The proposed method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance. |
End-to-End Emotion Semantic Parsing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on detecting emotion from text, but recent studies have shown that emotions may be experienced and caused by semantic constituents. |
| Approach: | They propose a task called emotion semantic parsing which parses emotion and semantic constituents into an abstract semantic tree structure. |
| Outcome: | The proposed model performs better than existing models and is more cognitive and structurally robust. |
Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that HESIT effectively alleviates catastrophic forgetting by exemplar selection, and achieves state-of-the-art performance on the largest CL benchmark of ToDs in terms of all metrics. |
| Approach: | They propose a method to overcome catastrophic forgetting in task-oriented dialogue systems by tracing their hyper-gradients and a retraining strategy that uses influential exemplars for periodic retrains. |
| Outcome: | The proposed method achieves state-of-the-art on the largest CL benchmark of ToDs in terms of all metrics. |
Unveiling Imitation Learning: Exploring the impact of Data Falsity to Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on improving open-source language models through imitation learning. |
| Approach: | They propose to use false pairs to control the factuality ratio of a dataset to study noise. |
| Outcome: | The proposed model is based on a Falsity-Controllable dataset with false pairs and false pairs to control the factuality ratio. |
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations? (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are more proficient at code generation, but they still generate incorrect programs. |
| Approach: | They define a group of models that have a high log-probability and weak correctness checks. |
| Outcome: | The proposed model samples fail to understand counterfeits through three clear failure modes . counterfeits are confusing to the model as they are to other models, the authors say . |
CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support (2024.findings-acl)
Copied to clipboard
Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Wang, Aakanksha Naik
| Challenge: | Literature review requires researchers to synthesize a large amount of information. |
| Approach: | They propose to use LLMs to generate hierarchical organizations from a set of studies . they use a human-in-the-loop process to correct errors in LLM-generated hierarchies . |
| Outcome: | The proposed model improves assignment of studies to categories by 12.6 F1 points. |
Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation (2024.findings-acl)
Copied to clipboard
Hao Li, Yuping Wu, Viktor Schlegel, Riza Batista-Navarro, Tharindu Madusanka, Iqra Zahid, Jiayan Zeng, Xiaochi Wang, Xinran He, Yizhi Li, Goran Nenadic
| Challenge: | Recent advances in large language models (LLMs) have made it difficult to build an automated debate system that helps people to synthesise persuasive arguments. |
| Approach: | They propose to use an argument mining dataset to capture the end-to-end process of preparing an argumentative essay for a debate. |
| Outcome: | The proposed dataset shows that it performs better on individual tasks than on human-centred evaluations. |
A Grounded Preference Model for LLM Alignment (2024.findings-acl)
Copied to clipboard
Tahira Naseem, Guangxuan Xu, Sarathkrishna Swaminathan, Asaf Yehudai, Subhajit Chaudhury, Radu Florian, Ramón Astudillo, Asim Munawar
| Challenge: | Large Language Models (LLMs) suffer from factual inconsistency and hallucination despite recent advances . training a preference model requires substantial human annotation, which is expensive and labor-intensive. |
| Approach: | They propose to generate synthetic grounded preference data and train a Grounded Preference Model to assess the overall quality of grounded responses. |
| Outcome: | The proposed model can generate much better grounded responses as judged by GPT4 and achieves the TRUE faithfulness Benchmark. |
Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs (2024.findings-acl)
Copied to clipboard
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, Jiawei Han
| Challenge: | Existing studies suggest augmenting LLMs with external text corpora to alleviate hallucination problems. |
| Approach: | They propose to augment large language models with text units retrieved from external knowledge corpora to alleviate the issue. |
| Outcome: | The proposed framework outperforms baselines on GRBench with three LLMs and shows that iterative reasoning outperformed the baselines. |
Text2DB: Integration-Aware Information Extraction with Large Language Model Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Current methods for information extraction (IE) focus on integrating IE output with the database . a long-overlooked question is what counts as "relevant knowledge" |
| Approach: | They propose a task that emphasizes integration of IE output and the database . they introduce a benchmark and an LLM agent framework for this task . |
| Outcome: | The proposed task integrates IE output and the target database (or knowledge base) it meets common demands such as data infilling, row population, and column addition . |
How Important is a Language Model for Low-resource ASR? (2024.findings-acl)
Copied to clipboard
| Challenge: | Using an n-gram language model in ASR may seem obvious, but its absence in most implementations suggests otherwise. |
| Approach: | They examine whether using an n-gram language model in ASR can improve accuracy in low-resource languages. |
| Outcome: | The proposed model is absent in most implementations, but it does improve accuracy in English and Mandarin. |
MediSwift: Efficient Sparse Pre-trained Biomedical Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models are typically trained on general source data forvarious domains, but domain-specific pre-training is expensive and requires computational costs. |
| Approach: | They propose a suite of biomedicalLMs that leverage sparse pre-training on domain-specific biomedically text data. |
| Outcome: | The proposed model outperforms existing LLMs on biomedical tasks by 22.5x . |
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) are trained on orders of magnitude more language data than human language learners receive, but without supervision from other sensory modalities that play a crucial role in human learning. |
| Approach: | They propose a grounded language learning procedure that leverages visual supervision to improve textual representations. |
| Outcome: | The proposed procedure outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes and improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. |
P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Contemporary approaches to generate tabular data are limited due to the lack of external knowledge. |
| Approach: | They propose to use proximal policy optimization to apply GANs and fine-tune Large Language Models to enhance the probability distribution of tabular features. |
| Outcome: | The proposed method improves accuracy of GANs and LLMs over state-of-the-art over three real-world datasets. |
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated state-of-the-art (SOTA) performance across a wide spectrum of tasks. |
| Approach: | They propose a framework that leverages three signal types to improve efficiency within resource-constrained, imperfect teacher scenarios. |
| Outcome: | The proposed framework improves on four complex reasoning tasks by 20.79% compared to fine-tuning without any signals across datasets. |
Small Models are Valuable Plug-ins for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are difficult to fine-tune due to their huge weights and limited context length. |
| Approach: | They propose an approach which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. |
| Outcome: | The proposed approach overcomes the challenges of poor performance and instability of In-Context Learning (ICL) while reducing the complexity of in-context learning. |
Are self-explanations from Large Language Models faithful? (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction-tuned Large Language Models excel at many tasks and will explain their reasoning, so-called self-explanations. |
| Approach: | They propose to employ self-consistency checks to measure faithfulness to LLMs to determine if they are model-dependent and if their reasoning is convincing and wrong. |
| Outcome: | The proposed measures show that self-explanations are explanation, model, and task-dependent and should not be trusted in general. |
ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction (2024.findings-acl)
Copied to clipboard
Henry Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip Yu, Cornelia Caragea
| Challenge: | Existing datasets for attribute value extraction focus on explicit attribute values while neglecting the implicit ones. |
| Approach: | They present a multimodal dataset for implicit attribute value extraction that includes AVE and multimodality. |
| Outcome: | The proposed dataset includes 68k training and 1.6k testing data across five domains. |
Prompt Engineering a Prompt Engineer (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies indicate that large language models can be meta-prompted to perform automatic prompt engineering, but their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. |
| Approach: | They propose to infuse three key components into a meta-prompt to guide reasoning . they find prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K . |
| Outcome: | The proposed method outperforms “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K and outperfies baselines on counterfactual tasks by 6.9%. |
ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations (2024.findings-acl)
Copied to clipboard
Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar, Utkarsh Tyagi, S Sakshi, Sanjoy Chowdhury, Dinesh Manocha
| Challenge: | Neural image classifiers often rely on non-predictive features that are spuriously correlated with the class labels in training data. |
| Approach: | They propose a language-guided data augmented with images without spurious correlations that can be used to augment training datasets for robust learning. |
| Outcome: | The proposed model improves the worst-group classification accuracy of prior methods by 1% - 38%. |
Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed an explosion of Large Language Models (LLMs), with impressive performance on various NLP tasks. |
| Approach: | They propose to use image-based representations to compare LLMs' performance on table-related tasks such as question-answering and fact-checking to determine their effectiveness. |
| Outcome: | The proposed model performs better on image-based representations than on text-based models. |
Biasly: An Expert-Annotated Dataset for Subtle Misogyny Detection and Mitigation (2024.findings-acl)
Copied to clipboard
Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Smith, Tamara Kneese, Carolyne Pelletier, Ioana Baldini, Yue Dong
| Challenge: | the Biasly dataset captures misogyny in movies in ways unique within the literature. |
| Approach: | The Biasly dataset captures misogyny in North American film by combining annotations of movie subtitles with common NLP algorithms. |
| Outcome: | The Biasly dataset captures misogyny expressions in North American film . it contains annotations of movie subtitles and text generation for rewrites . |
BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing hybrid question answering systems use a "prompt-and-pray" paradigm . context size limitations limit ability of many transformer-based LLMs to fit into a given prompt . |
| Approach: | They propose a superset of SQLite to act as a unified dialect for orchestrating reasoning across unstructured and structured data. |
| Outcome: | The proposed framework scales to massive datasets and improves performance while using 35% fewer tokens. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model (2024.findings-acl)
Copied to clipboard
Haogeng Liu, Quanzeng You, Yiqi Wang, Xiaotian Han, Bohan Zhai, Yongfei Liu, Wentao Chen, Yiren Jian, Yunzhe Tao, Jianbo Yuan, Ran He, Hongxia Yang
| Challenge: | InfiMM is a multimodal large language model that adapts to complex vision-language tasks. |
| Approach: | They present a Multimodal Large Language Model that adapts to intricate vision-language tasks using large-scale training data and comprehensive training strategies. |
| Outcome: | Empirical evaluations across a variety of benchmarks underscore InfiMM’s remarkable capability in multimodal understanding. |
Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics and benchmarks to attribute large language models to structured knowledge are lacking. |
| Approach: | They propose a task of Knowledge-aware Language Model Attribution that improves upon three core concerns with conventional attributed LMs. |
| Outcome: | The proposed model improves upon core concerns with conventional attributed LMs. |
Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been shown to be effective as automatic evaluators with simple prompting and in-context learning. |
| Approach: | They assemble 16 Large Language Models and evaluate their outputs by preference ranking . they introduce a cognitive bias benchmark to measure six different cognitive biases in LLM evaluation outputs. |
| Outcome: | The proposed model is biased on the CoBBLer benchmark, indicating that machine preferences are misaligned with humans. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |
Muffin: Mitigating Unhelpfulness in Emotional Support Conversations with Multifaceted AI Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that emotional support conversation models generate unhelpful responses that can hinder their effectiveness. |
| Approach: | They propose a model-agnostic framework called Mitigating unhelpfulness with multifaceted AI feedback for emot io nal support (Muffin) it uses a multifaceted feedback module to assess helpfulness model responses across various facets of emotional support and contrasts helpful and unhelpful responses generated by the model. |
| Outcome: | The proposed framework reduces the likelihood of unhelpful responses by comparing helpful and unhelpfully responses generated by previous models to improve response fluency and relevance. |
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024.findings-acl)
Copied to clipboard
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, Mark Gerstein
| Challenge: | Large language models face unique challenges such as domain-specific terminologies and reasoning over specialized knowledge. |
| Approach: | They propose a multi-disciplinary collaboration framework that leverages LLM-based agents in a role-playing setting. |
| Outcome: | The proposed framework excels at mining and harnessing medical expertise within LLMs, as well as extending its reasoning abilities. |
Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods rely on syntactically mapping natural languages to complete formal languages like Python and SQL. |
| Approach: | They propose to deconstruct reasoning-independent semantic information into generic symbolic representations, thereby efficiently capturing more generalized reasoning knowledge. |
| Outcome: | The proposed method improves in-context reasoning accuracy, learning efficiency, out-of-domain generalization, and output stability compared to the Chain-of thought technique. |
DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models (2024.findings-acl)
Copied to clipboard
Baohang Zhou, Zezhong Wang, Lingzhi Wang, Hongru Wang, Ying Zhang, Kehui Song, Xuhui Sui, Kam-Fai Wong
| Challenge: | Existing methods to detect pretraining data from large language models are unrealistic to them. |
| Approach: | They propose to detect pre-training data from LLM in a black-box way by using GPT-2 as reference model and feed it with sequence probabilities to detect whether it was used to train it. |
| Outcome: | The proposed framework outperforms existing methods on the benchmark datasets and shows that it is effective on different popular LLMs. |
PACIT: Unlocking the Power of Examples for Better In-Context Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context instruction tuning is a method that fine tunes with supervised instruction data to enhance the instruction following ability of large language models. |
| Approach: | They propose an in-context instruction tuning method that incorporates positive and negative examples into the prompt for better performance. |
| Outcome: | The proposed method outperforms baseline instruction tuning methods on in-domain and out-domain tasks up to 9.16 and 3.14 average ROUGE-L scores. |
Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR). |
| Approach: | They propose a multimodal LLM to receive source speech as extra input and reformat it as a cloze test with logits calibration to remove input information redundancy and simplify GER with clear instructions. |
| Outcome: | The proposed model improves on 9 popular ASR datasets and is faster than vanilla GER. |
Towards Better Graph-based Cross-document Relation Extraction via Non-bridge Entity Enhancement and Prediction Debiasing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on relation extraction ignore non-bridge entities, leading to bias during inference. |
| Approach: | They propose a graph-based cross-document Relation Extraction model with non-bridge entity enhancement and prediction debiasing that integrates non-cross entities with target entities and bridge entities. |
| Outcome: | The proposed model outperforms baseline models on open and closed datasets. |
Large Language Models can Share Images, Too! (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a zero-shot prompting, large language models can be used to share images in a multi-tasking environment. |
| Approach: | They introduce a dataset that includes enriched annotations and a framework to evaluate LLMs. |
| Outcome: | The proposed framework unlocks image-sharing capability of LLMs in zero-shot prompting, with ChatGPT achieving the best performance. |
CodeM: Less Data Yields More Versatility via Ability Matrix (2024.findings-acl)
Copied to clipboard
Daoguang Zan, Ailun Yu, Wei Liu, Bo Shen, Shaoxin Lin, Yongshun Gong, Yafen Yao, Yan Liu, Bei Guan, Weihua Luo, Yongji Wang, Qianxiang Wang, Lizhen Cui
| Challenge: | Recent efforts to train code large language models have been booming recently . however, this will incur significant costs in constructing data and training model considering the countless downstream scenarios. |
| Approach: | They propose a data construction strategy which decouples code LLMs’ abilities into two dimensions and constructs a lightweight training corpus that only covers a subset of target scenarios. |
| Outcome: | The proposed model can train a multilingual multitasking model using less data and training data. |
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning (2024.findings-acl)
Copied to clipboard
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji
| Challenge: | LVLMs are known for producing text that is factually inconsistent with visual input . factuality of generated captions for structured visuals has not been studied as much . |
| Approach: | They propose a typology of factual errors in captions generated by large vision-language models . they propose CHOCOLATE, a visual entailment model that outperforms current models based on this analysis . |
| Outcome: | The proposed model outperforms current models in evaluating caption factuality. |
BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated efficacy in knowledge-intensive tasks such as open-domain QA, but inconsistencies between retrieval knowledge and the necessary knowledge for LLMs, leading to a decline in LLM’s answer quality. |
| Approach: | They propose a retrieval-augmented large language model that refines retrieval documents into Key Supporting Evidence (KSE) through knowledge synthesis, supervised fine-tuning, and preference alignment. |
| Outcome: | The proposed model outperforms existing methods in five datasets and boosts answer quality by 7% while reducing input content length by 80%. |
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual grounding rely on the assumption that the given expression must be literal . this impedes the practical deployment of agents in real-world scenarios. |
| Approach: | They propose a visual grounding task that uses intention expressions to locate foreground entities . they build a large-scale IVG dataset with free-form intention expression to promote VG . |
| Outcome: | The proposed method is based on a large-scale intention-driven visual-language (V-L) dataset with free-form intention expressions. |
Incremental Sequence Labeling: A Tale of Two Shifts (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to incremental sequence labeling have focused on the E2O and O2E issues, but neglect the O2e issue. |
| Approach: | They propose a framework for incremental sequence labeling without semantic shifts that mitigate catastrophic forgetting in models by using knowledge distillation to maintain the model’s discriminative ability for old entities. |
| Outcome: | The proposed framework mitigates catastrophic forgetting in models while maintaining discriminative ability for old entities while minimizing the model’s bias towards new entities. |
How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have validated that large language models (LLMs) are capable of solving some KBQA problems, but there has been little discussion on the differences in LLMs’ proficiency in formal languages used in semantic parsing. |
| Approach: | They propose to evaluate the understanding and generation ability of large language models (LLMs) to deal with differently structured logical forms by examining the inter-conversion of natural and formal language through in-context learning of LLMs. |
| Outcome: | The proposed model can understand formal languages as well as humans, but generating correct logical forms remains a challenge. |
MELOV: Multimodal Entity Linking with Optimized Visual Features in Latent Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal entity linking focus on textual contexts but lack in social media vision modality. |
| Approach: | They propose a latent space vision feature optimization framework MELOV to address these challenges . they exploit variational autoencoder to mine shared information and generate text-based visual features . |
| Outcome: | The proposed framework is superior to existing methods on three benchmark datasets. |
Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that large-scale models can generate unsupervised DG without expensive distractor annotations. |
| Approach: | They propose a dual task training framework that integrates pseudo distractors from LLMs and answer information as the objective target with a two-stage training process. |
| Outcome: | The proposed method surpasses GPT-3.5-turbo zero-shot performance with 200 fewer model parameters. |
Conversational Question Answering with Language Models Generated Reformulations over Knowledge Graph (2024.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art ConvQA methods struggle with inexplicit question-answer pairs, which can degrade ConvQ performance. |
| Approach: | They propose a reinforcement learning based model, CoRnNet, which utilizes question reformulations generated by large language models to improve ConvQA performance. |
| Outcome: | The proposed model outperforms state-of-the-art ConvQA models by using question reformulations generated by large language models (LLMs). |
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are leading progress in code generation, but they are underutilized in the literature. |
| Approach: | They propose a debugging framework that allows LLMs to refine their generated programs with the runtime execution information. |
| Outcome: | The proposed framework improves the baseline performance by 9.8% across the HumanEval, MBPP, and TransCoder benchmarks. |
Effective In-Context Example Selection through Data Compression (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning has been validated in large language models, but the mechanism and selection strategy for in-cont example selection lacks systematic and in-depth research. |
| Approach: | They propose a data compression approach to select in-context examples using large language models. |
| Outcome: | The proposed method shows a significant improvement of 5.90% across five real-world datasets using four language models. |
Are U a Joke Master? Pun Generation via Multi-Stage Curriculum Learning towards a Humor LLM (2024.findings-acl)
Copied to clipboard
Yang Chen, Chong Yang, Tu Hu, Xinhao Chen, Man Lan, Li Cai, Xinlin Zhuang, Xuan Lin, Xin Lu, Aimin Zhou
| Challenge: | Existing research has demonstrated that the ability of large language models (LLMs) to generate humorous sentences is limited to producing 25 unique jokes. |
| Approach: | They propose a multi-stage curriculum preference learning framework to optimize both pun structure preferences and humor preferences by a Chinese Pun dataset. |
| Outcome: | The proposed method significantly outperforms baseline models on Chinese and English benchmark datasets. |
Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Domain-specific question answering (QA) requires a comprehensive understanding of a specific domain to answer specialized questions. |
| Approach: | They propose a new alignment objective to align the LLM preference with different human preferences uniformly to optimize LLM performance in real-world, domain-specific QA settings. |
| Outcome: | The proposed pipeline is superior for real-scenario domain-specific question answering with LLMs. |
MARIO: MAth Reasoning with code Interpreter Output - A Reproducible Pipeline (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack mathematical reasoning, a hurdle on the path to true artificial general intelligence. |
| Approach: | They propose a protocol for fine-tuning large language models with a Python code interpreter to enhance the text analysis of the LLMs. |
| Outcome: | The proposed protocol improves the performance of a 7B-parameter LLM on the GSM8K and MATH datasets while allowing for an outlier-free value model-based inference method. |
DiffusPoll: Conditional Text Diffusion Model for Poll Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media platforms manipulate public opinion through sheer numbers and cause biases, authors say . they say new paradigm for poll generation can generate high-quality samples while preserving diversity . |
| Approach: | They propose a non-autoregressive diffusion model that uses masks to generate polls . they use attribute tags to enhance the quality of polls and to diversify poll options . |
| Outcome: | The proposed model matches the Transformer model while offering greater diversity and quality. |
Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown excellent capabilities in language understanding, text generation and many other tasks, but struggle in complex multi-step reasoning problems such as mathematical reasoning. |
| Approach: | They propose to fine tune an open-llama-3B model to perform well on multi-step reasoning tasks via synthetic data. |
| Outcome: | The proposed model can reach a zero-shot pass@1 at 0.44 on the in-domain dataset and demonstrates certain generalization capabilities on the out-of-domain data. |
Implanting LLM’s Knowledge via Reading Comprehension Tree for Toxicity Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for toxic content detection are small language model (SLM) based and large language model(LLM) -based. |
| Approach: | They propose to implant LLM's knowledge into SLM based methods to stick to both types of models' strengths by constructing a reading comprehension tree to transfer knowledge between two models. |
| Outcome: | The proposed method can stick to both types of models' strengths . it is compared with existing methods on real-world and machine-generated datasets. |
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024.findings-acl)
Copied to clipboard
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang
| Challenge: | Existing approaches to compress prompts only leverage unidirectional context, causing suboptimal results. |
| Approach: | They propose a task-agnostic prompt compression method that takes tokens from context . they use a Transformer encoder to capture all essential information needed for prompt compression . |
| Outcome: | The proposed method is 3x-6x faster than existing prompt compression methods and faster than baselines. |
EconNLI: Evaluating Large Language Models on Economics Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used for writing economic analysis reports or providing financial advice, but their ability to understand economic knowledge and reason about potential results of economic events or actions lacks systematic evaluation. |
| Approach: | They propose a dataset to evaluate LLMs’ knowledge and reasoning abilities in the economic domain. |
| Outcome: | The proposed dataset evaluates LLMs’ knowledge and reasoning abilities in the economic domain. |
Better Late Than Never: Model-Agnostic Hallucination Post-Processing Framework Towards Clinical Text Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating concise and coherent summaries may include unintended text with hallucinations, causing computational costs. |
| Approach: | They propose a model-agnostic framework to post-process medical hallucinations . MEDAL integrates with any medical summarization model, requiring no additional computational overhead . |
| Outcome: | MEDAL can post-process medical hallucinations without additional computational overhead. |
Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to identify key neurons for interpretability of multi-modal large language models are unclear. |
| Approach: | They propose a method to identify key neurons for interpretability by multi-modal large language models. |
| Outcome: | The proposed method improves conventional works upon efficiency and applied range by removing needs of costly gradient computation. |
Realistic Evaluation of Toxicity in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | a large amount of data exposes large language models to toxicity and bias . prompt engineering can be easily bypassed with minimal prompt engineering. |
| Approach: | They propose a dataset that uses manually crafted prompts to nullify protective layers of large language models. |
| Outcome: | The proposed dataset shows that prompts can nullify protective layers of large language models. |
Controllable Text Generation with Residual Memory Transformer (2024.findings-acl)
Copied to clipboard
| Challenge: | Large-scale Causal Language Models (CLMs) have been successful in text generation, but there is still a challenge to control the generation process. |
| Approach: | They propose a non-intrusive, lightweight control plugin to control the generation process of a CLM at arbitrary time steps. |
| Outcome: | The proposed plugin can handle any type of control conditions and cooperate with the base CLM through a residual learning paradigm. |
Prompt-Based Length Controlled Generation with Multiple Control Types (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing length control methods focus on a simple control type of “equal to” a target length. |
| Approach: | They propose a prompt-based method to achieve length controlled generation under different control types with high accuracy by using reinforcement learning and sample filtering with the reward signal given by rule-based reward models. |
| Outcome: | The proposed method significantly improves the accuracy of prompt-based length control on popular summarization datasets like CNNDM and NYT under multiple control types. |
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain (2024.findings-acl)
Copied to clipboard
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
| Challenge: | a new multimodal decision-making benchmark evaluates the integrated capabilities of multimodal large language models. |
| Approach: | They propose a multimodal decision-making benchmark for evaluating MLLMs . they propose an automatic evaluation protocol to assess 10 prevalent ML models . |
| Outcome: | The proposed benchmark improves performance of multimodal large language models in three scenarios . the model is required to integrate multiple capabilities to make accurate decisions . |
Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset (2024.findings-acl)
Copied to clipboard
Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee
| Challenge: | Existing datasets for conversational recommender systems lack specific user preferences and explanations for recommendations . current datasets lack specific preferences, hindering high-quality recommendations despite advances in large language models . |
| Approach: | They propose to synthesize a conversational recommendation dataset with persona- and knowledge-augmented LLM simulators to address these challenges. |
| Outcome: | The proposed dataset outperforms baselines in human and automatic evaluations. |
CoLLaVO: Crayon Large Language and Vision mOdel (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) and instruction tuning have been used to drive the evolution of Vision Language Model (VLM) towards a versatile general-purpose model. |
| Approach: | They propose a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting. |
| Outcome: | The proposed model outperforms closed-source models on vision language tasks and achieves a significant leap in numerous benchmarks. |
Modelling Variability in Human Annotator Simulation (2024.findings-acl)
Copied to clipboard
| Challenge: | Human annotator simulation (HAS) is a cost-effective alternative to human evaluation tasks. |
| Approach: | They propose a framework to model human annotation variability via meta-learning . conditional softmax flow model leverages diverse human annotations via meta learning . results demonstrate that method can predict aggregated behaviours of human annotators . |
| Outcome: | The proposed method achieves state-of-the-art performance on two real-world human evaluation tasks: emotion recognition and toxic speech detection. |
BEnQA: A Question Answering Benchmark for Bengali and English (2024.findings-acl)
Copied to clipboard
| Challenge: | a dataset of parallel Bengali and English exam questions is used to compare LLMs in low-resource languages. |
| Approach: | They introduce BEnQA, a dataset comprising parallel Bengali and English exam questions . they benchmark several Large Language Models with their parallel dataset and observe performance disparity . |
| Outcome: | The proposed dataset consists of 5K questions covering several subjects in science . the authors find that the models perform poorly in Bengali and English . |
MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Language Models (LLMs) have gained increasing prominence in artificial intelligence, especially Large Language Model (LLm) due to the well-recognized reporting bias, the recording of commonsense information is significantly less than its existence in reality. |
| Approach: | They propose a Multi-mOdal REtrieval framework to leverage both text and images to enhance commonsense ability of language models. |
| Outcome: | The proposed framework can leverage both text and images to enhance commonsense ability of language models. |
Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models (2024.findings-acl)
Copied to clipboard
Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
| Challenge: | Existing methods to expand internal memory boundaries of language models by providing external context can often conflict, leading to knowledge conflicts. |
| Approach: | They propose a method that prunes conflicting attention heads without updating model parameters. |
| Outcome: | The proposed method can flexibly control eight LMs to use internal memory or external context without updating model parameters. |
BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning (2024.findings-acl)
Copied to clipboard
Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, Rui Yan
| Challenge: | BioT5+ is an extension of the BioT5, but lacked a nuanced understanding of molecular structures. |
| Approach: | They propose a new bio-entity modeling framework, BioT5+, which integrates IUPAC names and molecule data. |
| Outcome: | The proposed model bridges the gap between molecular representations and textual descriptions and improves the grounded reasoning of bio-text and bio-sequences. |
SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in parameter-efficient fine-tuning (PEFT) techniques allow for adjustments to only a minor fraction of the parameters of large language models. |
| Approach: | They propose a SImple BOoster to enhance parameter-efficient fine-tuning techniques by injecting an initial residual into the model. |
| Outcome: | The proposed model improves performance on 22 benchmark datasets and can be extended to a range of state-of-the-art techniques. |
GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) and multi-modal models (MMs) have demonstrated remarkable capabilities in problem-solving, but their proficiency in tackling geometry math problems has not been thoroughly evaluated. |
| Approach: | They propose a benchmark to evaluate the performance of large language models and multi-modal models in solving geometry math problems. |
| Outcome: | The proposed model achieves 55.67% accuracy on main subset but only 6.00% accuracy on hard subset. |
Boosting Textural NER with Synthetic Image and Instructive Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a key task reliant on textual data. |
| Approach: | They propose a method to transform NER into a multimodal task by using images from the internet as auxiliaries. |
| Outcome: | The proposed method surpasses all text-only baselines and improves F1 score by 1.4% to 2.3% on prominent MNER datasets. |
Neurons in Large Language Models: Dead, N-gram, Positional (2024.findings-acl)
Copied to clipboard
| Challenge: | We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. |
| Approach: | They analyze a family of large language models that rely on whether an FFN neuron is activated or not . they use a single GPU to analyze the OPT network and use it to analyze data . |
| Outcome: | The proposed model is lightweight and can be run on a single GPU. |
LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Grounded Multimodal Named Entity Recognition (GMNER) lack a strong correlation between image-text pairs and is ungroundable. |
| Approach: | They propose a framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models as a connecting bridge. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks. |
Learning Job Title Representation from Job Description Aggregation Network (2024.findings-acl)
Copied to clipboard
Napat Laosaengpha, Thanit Tativannarat, Chawan Piansaddhayanon, Attapol Rutherford, Ekapol Chuangsuwanich
| Challenge: | Existing methods for learning job title representation neglect the rich content within the job description. |
| Approach: | They propose a framework for learning job titles through their respective job description and utilize a Job Description Aggregator component to handle the lengthy description and bidirectional contrastive loss. |
| Outcome: | The proposed framework outperforms the skill-based approach on in-domain and out-of-domain settings and achieving a superior performance. |
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024.findings-acl)
Copied to clipboard
Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth
| Challenge: | Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. |
| Approach: | They propose to use flowcharts as visual contexts to assess the capabilities of visual question-answering multimodal language models in reasoning. |
| Outcome: | The proposed benchmarks evaluate models' ability to follow visual information without pre-existing knowledge on a suite of open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. |
Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Named Entity Recognition (CNER) use knowledge distillation to retain old knowledge, but they are too expensive and fail to integrate with existing state-of-the-art models. |
| Approach: | They propose a weight tuning and weightfusion strategy to learn new entity types while mitigating catastrophic forgetting of old models. |
| Outcome: | The proposed strategies improve the performance of existing models and are model-agnostic. |
Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have highlighted various neural metrics that align well with human evaluations. |
| Approach: | They propose a black-box adversarial framework that generates strong disagreements between human and victim evaluators. |
| Outcome: | The proposed framework can significantly improve the performance of human and victim evaluators. |
Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they often fail to maintain crucial invariances for specific tasks. |
| Approach: | They propose to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model. |
| Outcome: | The proposed model outperforms larger, biased teacher models by reducing the invariances of the student model by incorporating encoders. |
Uncovering Limitations of Large Language Models in Information Seeking from Tables (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Information Seeking (TabIS) are lacking in reliable evaluation. |
| Approach: | They propose a benchmark to evaluate the table information seeking abilities of large language models . they use a single-choice question format instead of a text-based evaluation . |
| Outcome: | The proposed benchmark is more reliable than existing models and is available online. |
An Ensemble-of-Experts Framework for Rehearsal-free Continual Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for continual relation extraction (CRE) are rehearsal-based and need to store samples and thus may encounter privacy and security issues. |
| Approach: | They propose an Ensemble-of-Experts framework for rehearsal-free continual relation extraction that discriminates between experts and augments analogous relations across tasks. |
| Outcome: | The proposed method outperforms existing rehearsal-free methods and is even better than existing methods. |
Temporal Validity Change Prediction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarking tasks require models to identify temporal validity duration of a single statement . however, many data sources contain additional context, which may alter the duration of the original statement if the context is present . |
| Approach: | They propose a task benchmarking the ability of machine learning to detect context statements that induce such change. |
| Outcome: | The proposed task uses a dataset of temporal target statements and crowdsource corresponding context statements to benchmark them. |
RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) can be fine-tuned for downstream text processing tasks. |
| Approach: | They propose to use paraphrases to enrich the input text of a few-shot model with a Maximum-Marginal Likelihood objective to improve performance. |
| Outcome: | The proposed methods improve performance beyond what can be achieved with parameter-efficient fine-tuning alone. |
Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings (2024.findings-acl)
Copied to clipboard
| Challenge: | Concept embeddings are a useful and efficient mechanism for injecting commonsense knowledge into downstream tasks. |
| Approach: | They propose to model commonalities in concepts by capturing a more diverse range of commonsense properties. |
| Outcome: | The proposed model captures a more diverse range of commonsense properties and improves ontology completion and ultra-fine entity typing tasks. |
Revisiting Multimodal Transformers for Tabular Data with Text Fields (2024.findings-acl)
Copied to clipboard
| Challenge: | Tabular data with text fields can be used in financial risk assessment and diagnosis prediction. |
| Approach: | They propose a tabular/text dual-stream Transformer network with numerical embedding schemes and an overall attention module to estimate whether a prediction is uncertain. |
| Outcome: | The proposed model can estimate whether a prediction is uncertain or not based on two well-informed modality streams . |
An Empirical Study on the Characteristics of Bias upon Context Length Variation for Bangla (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models exhibit various social biases due to widespread usage. |
| Approach: | They extend existing methods for measuring gender bias in Bangla by examining context length variation. |
| Outcome: | The proposed method relies on context length variation, highlighting the need for nuanced considerations in Bangla bias analysis. |
ConTempo: A Unified Temporally Contrastive Framework for Temporal Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Temporal relation extraction (TRE) is a task of classifying temporal relations between events conveyed in narratives. |
| Approach: | They propose a Temporally Contrastive learning model that increases the model’s awareness of the meaning of temporal relations by leveraging their symmetric or antisymmetric properties. |
| Outcome: | The proposed model improves the model's representation of meaning of temporal relations and its ability to integrate with the underlying temporal calculus. |
CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems (2024.findings-acl)
Copied to clipboard
Abbas Ghaddar, David Alfonso-Hermelo, Philippe Langlais, Mehdi Rezagholizadeh, Boxing Chen, Prasanna Parthasarathi
| Challenge: | CHARP is a testbed for knowledge-grounded dialogue evaluation of models trained on FaithDial data. |
| Approach: | They propose a testbed for evaluating models trained on FaithDial with annotation artifacts that may bias models towards completely ignoring the conversation history. |
| Outcome: | The proposed model fails to accurately evaluate the conversational history and lacks hallucination detection. |
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | CriticBench is a benchmark designed to assess LLMs’ abilities to critique and refine their reasoning across a variety of tasks. |
| Approach: | They propose a benchmark to assess LLMs' ability to critique and correct reasoning across a variety of tasks. |
| Outcome: | The proposed benchmark examines the performance of 17 large language models in generation, critique, and correction reasoning. |
DAFNet: Dynamic Auxiliary Fusion for Sequential Model Editing in Large Language Models (2024.findings-acl)
Copied to clipboard
Taolin Zhang, Qizhou Chen, Dongyang Li, Chengyu Wang, Xiaofeng He, Longtao Huang, Hui Xue’, Jun Huang
| Challenge: | Large language models (LLMs) have shown impressive results, but still suffer from hallucination, i.e., the generation of false information. |
| Approach: | They propose a task of sequential model editing that aims to rectify mistakes continuously. |
| Outcome: | The proposed method significantly outperforms baselines in single-turn and sequential editing. |
Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects - A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | scholarly attention has turned to the development of text summarization methods that are more closely tailored and controlled to align with specific objectives and user needs. |
| Approach: | They formalize a controllable text summarization task and categorize controllability attributes according to their shared characteristics and objectives. |
| Outcome: | The proposed method is tailored to meet the specific intent and needs of users. |
Benchmarking Large Language Models on Communicative Medical Coaching: A Dataset and a Novel System (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing applications of natural language processing (NLP) focus on patient-centered services, but the potential of NLP to benefit inexperienced doctors remains unexplored. |
| Approach: | They propose a human-AI cooperative framework to assist medical learners in practicing communication skills during patient consultations. |
| Outcome: | The proposed framework enables medical learners to practice communication skills during patient consultations while a coach agent provides immediate, structured feedback. |
Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation (2024.findings-acl)
Copied to clipboard
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang
| Challenge: | Recent advances in Large Language Models (LLMs) have greatly advanced problem solving in diverse domains such as mathematical reasoning and knowledge reasoning. |
| Approach: | They propose a thought prompting approach called 'Everything of Thoughts' it leverages pretrained reinforcement learning and Monte Carlo Tree Search to incorporate external domain knowledge and planning capability into thoughts. |
| Outcome: | The proposed approach outperforms existing approaches on game of 24, 8-Puzzle, and Pocket Cube. |
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing (2024.findings-acl)
Copied to clipboard
| Challenge: | SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes is a hybrid question-answering pipeline . |
| Approach: | They propose a hybrid question-answering pipeline that leverages knowledge from multiple knowledge sources. |
| Outcome: | The proposed approach achieves state-of-the-art on the Compmix dataset with 56.5% exact match rate. |
Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges (2024.findings-acl)
Copied to clipboard
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, Shafiq Joty
| Challenge: | Data augmentation (DA) is a key technique for enhancing model performance by diversifying training examples without the need for additional data collection. |
| Approach: | They examine various strategies that utilize LLMs for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for diverse forms of further training. |
| Outcome: | The proposed approach addresses the primary open challenges faced by LLMs in the field of large language models and aims to serve as a comprehensive guide for researchers and practitioners. |
k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent watermarked generation algorithms inject detectable signatures during language generation to facilitate post-hoc detection. |
| Approach: | They propose a watermark which assigns signatures to each watermarked sentence according to locality-sensitive hashing (LSH) they propose k-SemStamp, which uses kmeans clustering to partition the semantic space with awareness of inherent semantic structure. |
| Outcome: | The proposed watermark improves its robustness and sampling efficiency while preserving the generation quality, making it more effective for machine-generated text detection. |
ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work reveals that vision and language models struggle to comprehend fine grained distinctions in images. |
| Approach: | They propose a dataset to assess multimodal models' ability to match objects with their colors. |
| Outcome: | The proposed model performs well in visual questionanswering, text-to-image generation and word-order understanding tasks. |
Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. |
| Approach: | They propose to revisit the optimization by prompting approach for small-scale LLMs . they suggest future prompting engineering to consider both model capabilities and computational costs . |
| Outcome: | The proposed approach shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. |
CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained Language Models suffer in inference latency due to their large size. |
| Approach: | They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers. |
| Outcome: | The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy. |
UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment, but excessive focus on safety alignment can lead to unintended hallucinations. |
| Approach: | They propose a "safety-priming" method to generate synthetic safety data and overcome safety bottlenecks. |
| Outcome: | The proposed framework generates synthetic safety data and overcomes safety bottlenecks. |
A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-way parallel, machine generated content dominates the translations in lower resource languages . a limited investigation suggests this selection bias is the result of low quality content generated in English and translated into many lower resource language via MT. |
| Approach: | They show that multi-way parallel, machine generated content dominates translations in many languages . they also find evidence of a selection bias in the type of content which is translated into many languages. |
| Outcome: | The results suggest that the low quality of multi-way translations on the web was likely created using machine translation. |
RankMean: Module-Level Importance Score for Merging Fine-tuned LLM Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Traditionally, developing new language models involves fine-tuning pre-trained LMs . model merging is a cost-effective alternative to fine-timing LM models for multiple tasks . |
| Approach: | They propose an algorithm for merging fine-tuned language models without additional training. |
| Outcome: | The proposed algorithm outperforms baseline methods on multiple benchmarks. |
VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on object hallucinations, focusing on object outputs . current evaluation methods struggle to address subtle semantic distinctions between outputs and reference data . |
| Approach: | They propose a multi-dimensional benchmark covering objects, attributes, and relations . they propose metric that generalizes CHAIR metric and incorporates faithfulness and coverage . |
| Outcome: | The proposed evaluation framework is more comprehensive and better correlated with humans than existing evaluation methods. |
Cyclical Contrastive Learning Based on Geodesic for Zero-shot Cross-lingual Spoken Language Understanding (2024.findings-acl)
Copied to clipboard
| Challenge: | zero-shot cross-lingual SLU is a challenging task in low-resource languages . a lack of labeled training data makes it difficult to align representations of similar sentences . |
| Approach: | They propose a framework that uses cyclical contrastive learning to achieve consistency between languages . they propose to use geodesic to measure the similarity to construct positive and negative pairs . |
| Outcome: | The proposed framework achieves state-of-the-art performance on multiATIS++ and MTOP datasets. |
Towards Safer Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Approach: | They propose a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Outcome: | The proposed approach eliminates harmful knowledge while preserving utility on normal prompts. |
The Impact of Reasoning Step Length on Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Long reasoning steps in LLMs improve reasoning abilities, but the correlation between their effectiveness and the length of reasoning steps remains largely unknown. |
| Approach: | They conducted experiments that expand and compress the rationale reasoning steps within CoT demonstrations while keeping all other factors constant. |
| Outcome: | The results show that lengthening the reasoning steps in prompts significantly enhances LLMs’ reasoning abilities across multiple datasets. |
Towards Understanding Task-agnostic Debiasing Through the Lenses of Intrinsic Bias and Forgetfulness (2024.findings-acl)
Copied to clipboard
Guangliang Liu, Milad Afshari, Xitong Zhang, Zhiyu Xue, Avrajit Ghosh, Bidhan Bashyal, Rongrong Wang, Kristen Johnson
| Challenge: | Debiasing Pretrained Language Models (PLMs) are task-agnostic and can be generalizable, but its impact on language modeling ability and the risk of relearning social biases remain as the two most significant challenges. |
| Approach: | They propose a framework which can Propagate Socially-fair Debiasing to Downstream Fine-tuning to alleviate the forgetting issue of PLMs by regularizing debiased attention heads based on the PLM’s bias levels from stages of pretraining and debiase. |
| Outcome: | The proposed framework can Propagate Socially-fair Debiasing to Downstream Fine-tuning, indicating that the ineffectiveness of debiase can be alleviated by overcoming the forgetting issue through regularizing successfully debiased attention heads based on the PLMs’ bias levels from stages of pretraining and debiases. |
SKGSum: Structured Knowledge-Guided Document Summarization (2024.findings-acl)
Copied to clipboard
Qiqi Wang, Ruofan Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, Jiamou Liu, Xianda Zheng, Zijian Huang
| Challenge: | Existing summarization methods ignore the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections. |
| Approach: | They propose a method that uses automatically extracted summary points to generate summaries. |
| Outcome: | The proposed methods improve quality and BERTScore of summaries and broaden the types of documents that can be effectively summarized. |
Chinese Spoken Named Entity Recognition in Real-world Scenarios: Dataset and Approaches (2024.findings-acl)
Copied to clipboard
| Challenge: | Current Chinese Spoken NER datasets are laboratory-controlled and are limited in topics. |
| Approach: | They propose to use Chinese Spoken NER datasets to extract entities from speech to help voice assistants better grasp the intent behind user's questions and instructions. |
| Outcome: | The proposed methods improve on self-training-asr and mapping then distilling, and even compared with GPT4.0, they achieve better results. |
DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality of machine-generated texts have a relatively low correlation with human performance. |
| Approach: | They propose an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil’s Advocate. |
| Outcome: | The proposed evaluation framework outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. |
Can Large Multimodal Models Uncover Deep Semantics Behind Images? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on visual deep semantics focus primarily on superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantic. |
| Approach: | They propose a benchmark to assess Large Multimodal Models’ (LMMs) capacities of visual deep semantics. |
| Outcome: | The proposed benchmark demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans. |
Harvesting Events from Multiple Sources: Towards a Cross-Document Event Extraction Paradigm (2024.findings-acl)
Copied to clipboard
| Challenge: | Document-level event extraction aims to extract structured information from unstructured text. |
| Approach: | They propose a cross-document event extraction pipeline that integrates event information from multiple documents and provides a comprehensive perspective on events. |
| Outcome: | The proposed pipeline achieves about 72% F1 in end-to-end cross-document event extraction, setting up a benchmark for future research. |
A Graph per Persona: Reasoning about Subjective Natural Language Descriptions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) perform poorly in reasoning about subjective knowledge, showing strong biases and lack interpretability requirements. |
| Approach: | They propose a novel approach for reasoning about subjective knowledge that integrates potential and implicit meanings and explicitly models the relational nature of the information. |
| Outcome: | The proposed model outperforms several prominent large language models on the OpinionQA dataset, showing its unique advantages and complementary nature. |
MolTC: Towards Molecular Relational Modeling In Language Models (2024.findings-acl)
Copied to clipboard
Junfeng Fang, Shuai Zhang, Chang Wu, Zhengyi Yang, Zhiyuan Liu, Sihang Li, Kun Wang, Wenjie Du, Xiang Wang
| Challenge: | Molecular Relational Learning (MRL) is a promising way to understand interactions between molecular pairs. |
| Approach: | They propose a novel LLM-based multi-modal framework for molecular interaction modeling following Chain-of-Thought (CoT) theory which integrates graphical information of two molecules in pair. |
| Outcome: | The proposed framework integrates graphical information of two molecules in pair. |
KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for keyphrase extraction and generation rely on exact matching with human references. |
| Approach: | They propose a framework for evaluation that includes four critical aspects: reference agreement, faithfulness, diversity, utility and semantic-based metrics. |
| Outcome: | The proposed evaluation framework correlates better with human preferences than previously proposed metrics. |
Learning Low-dimensional Multi-domain Knowledge Graph Embedding via Dual Archimedean Spirals (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding methods make domain constraints on embeddable domains, leading to poor performance. |
| Approach: | They propose a low-dimensional KGE model for multi-domain knowledge graphs that embeds domains and domains by regularization function. |
| Outcome: | The proposed model can distinguish entities from domains by encoding the same relation on the same archimedean spiral. |
LoRA Meets Dropout under a Unified Framework (2024.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficientfinetuning (PEFT) has gained popularity as a lightweight approach for model customization. |
| Approach: | They propose a parameter-efficient dropout method that is overfitting-prone and parameter-freezed. |
| Outcome: | The proposed method is superior to existing methods and compares with transformer-specific methods. |
Enhancing Text-to-SQL Parsing through Question Rewriting and Execution-Guided Refinement (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt engineering methods exploit database content and execution feedback to improve text-to-sql performance. |
| Approach: | They propose a framework for large language model-based text-to-sql task that exploits database content and execution feedback to improve execution accuracy. |
| Outcome: | The proposed framework improves execution accuracy and usability by 12.41% and 5.38% on four widely used benchmarks. |
The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models often ignore external knowledge to generate accurate answers . despite correct groundings, they can rely on wrong grounding or biases to hallucinate . |
| Approach: | They propose a framework that integrates human and human user clarifications to improve knowledge alignment. |
| Outcome: | The proposed framework improves model performance and mitigates hallucination by producing user-centered clarifications. |
ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models (2024.findings-acl)
Copied to clipboard
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, Yifan Zhu, Anh Tuan Luu
| Challenge: | Existing KBQA methods address inefficient knowledge retrieval and semantic parsing errors. |
| Approach: | They propose a generatethen-retrieve KBQA framework that generates logical form and replaces entities and relations with an unsupervised retrieval method to improve both generation and retrieval more directly. |
| Outcome: | Experimental results show that ChatKBQA achieves new state-of-the-art performance on standard KBQA datasets, WebQSP, and CWQ. |
Achilles-Bench: A Challenging Benchmark for Low-Resource Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing low-resource datasets that challenge neural networks cause over-estimated performance, despite promising yet saturated results in high-res settings. |
| Approach: | They propose a benchmark Achilles-Bench to better evaluate the learning ability of neural networks in low-resource settings. |
| Outcome: | The proposed benchmarks show that even pre-trained language models show performance drops on NLP tasks. |
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that INTERVENOR surpasses baseline models, exhibiting improvements of approximately 18% and 4.3% over GPT-3.5 in code generation and code translation tasks. |
| Approach: | They propose a system that prompts Large Language Models to play distinct roles during the code repair process, functioning as both a Code Learner and a code teacher. |
| Outcome: | The proposed system surpasses baseline models in code generation and code translation tasks and improves on syntax errors and assertion errors. |
SocialBench: Sociality Evaluation of Role-Playing Conversational Agents (2024.findings-acl)
Copied to clipboard
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang
| Challenge: | Existing studies on role-playing agents have focused on enhancing their conversational capability, role-specific knowledge and style, but there has been a gap in assessing their social intelligence. |
| Approach: | They propose a benchmark to evaluate the sociality of role-playing agents using LLMs. |
| Outcome: | The proposed benchmark is constructed from various sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. |
From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for large language models yield numerical scores that ignore user experience. |
| Approach: | They propose a metric that suggests revision edits that mimic the human writing process . their results show that the metric offers more insightful feedback and distinguishes between texts . |
| Outcome: | The proposed metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. |
Context-Aware Tracking and Dynamic Introduction for Incomplete Utterance Rewriting in Extended Multi-Turn Dialogues (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reconstruct utterance with omitted information and pronouns are limited to brief multi-turn dialogues. |
| Approach: | They propose a method to reconstruct utterance with omitted information and pronouns to be standalone and complete based on context. |
| Outcome: | The proposed method improves existing models and achieves state-of-the-art on three benchmarks. |
EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of emotional intelligence in large language models (LLMs) focus on basic sentiment analysis tasks, such as emotion recognition, which is not enough to evaluate LLMs’ overall emotional intelligence. |
| Approach: | They propose a framework for evaluating the emotional intelligence of large language models (LLMs) that includes four distinct tasks: Key Event Recognition, Mixed Event Recognition and Implicit Emotional Recognition. |
| Outcome: | The proposed framework includes four distinct tasks: Key Event Recognition, Mixed Event Recognition and Implicit Emotional Recognition. |
Plum: Prompt Learning using Metaheuristics (2024.findings-acl)
Copied to clipboard
Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, KaShun Shum, Jipeng Zhang, Renjie Pi, Tong Zhang
| Challenge: | Recent advances in prompt learning have led to a need for general prompt optimization methods. |
| Approach: | They propose a branch of discrete non-convex optimization methods with over 100 options as a promising approach to prompt learning. |
| Outcome: | The proposed methods can be used to discover more human-understandable prompts that were previously unknown in reasoning and image generation tasks. |
HOTVCOM: Generating Buzzworthy Comments for Videos (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research focuses on generating descriptive comments in English . hot-comments are important for video marketing and branding, authors say . |
| Approach: | They propose a framework to generate hot-comments on a Chinese video dataset . they use a combination of visual, auditory, and textual data to generate them . |
| Outcome: | The proposed framework shows that it generates hot-comments on both the new and existing datasets. |
Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing games such as "Who is undercover" are subjective and difficult to evaluate . |
| Approach: | They propose a game called BrainKing that evaluates LLMs' problem-solving capability under incomplete information scenarios. |
| Outcome: | The proposed game requires LLMs to identify target entities with limited yes-or-no questions and potential misleading answers. |
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
Into the Unknown: Generating Geospatial Descriptions for New Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Similar to vision-and-language navigation tasks, the Rendezvous (RVS) task requires reasoning over allocentric spatial relationships using non-sequential navigation instructions and maps. |
| Approach: | They propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. |
| Outcome: | The proposed method improves accuracy on unseen and seen environments by 45.83% on the Rendezvous (RVS) task. |
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Despite being the cornerstone of BPE, the importance of compression in the tokenization process is still unclear. |
| Approach: | They argue for the theoretical importance of compression in the tokenization process . they also demonstrate the empirical importance of compressing tokenizers for downstream success of pre-trained language models. |
| Outcome: | The proposed method can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. |
Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation (2024.findings-acl)
Copied to clipboard
Jungseob Lee, Hyeonseok Moon, Seungjun Lee, Chanjun Park, Sugyeong Eo, Hyunwoong Ko, Jaehyung Seo, Seungyoon Lee, Heuiseok Lim
| Challenge: | Byte Pair Encoding (BPE) is an effective approach in machine translation across several languages, but it is prone to over-segmentation in Korean, an agglutinative and morphologically rich language. |
| Approach: | They propose a new method that incorporates long words into the Korean vocabulary by strategically preserving morphological information and reducing semantic confusion. |
| Outcome: | The proposed method outperforms BPE and surpasses state-of-the-art morpheme-aware tokenization methods. |
Multilingual Instruction Tuning With Just a Pinch of Multilinguality (2024.findings-acl)
Copied to clipboard
| Challenge: | Using multilingual instruction tuning, large language models can be used to follow instructions in multiple languages . a multilingual model can be tuned on a wide range of languages, yet most datasets are limited to English . |
| Approach: | They investigate how multilinguality during instruction tuning affects instruction-following across languages . they find that only 40 multilingual examples improve multilingual instruction- follow . |
| Outcome: | The results show that multilingual models perform better on multilingual mixtures compared to monolingual models . the results suggest that building multilingual instruction-tuned models can be done with only 2-4 languages . |
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we introduce a new embedding model for semantic retrieval of more than 100 working languages. |
| Approach: | They propose a new embedding model that supports multi-lingual, cross-lingual and long-document retrieval . they propose integrating relevance scores from different retrieval functionalities into the teacher signal . |
| Outcome: | The proposed model exhibits superior performance on multilingual, cross-lingual, and long-document retrieval benchmarks. |
Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback (2024.findings-acl)
Copied to clipboard
Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, Xuanhua Shi
| Challenge: | Large Language Models (LLMs) generate code for given contexts, such as incomplete code, class, data structure, or project-specific information. |
| Approach: | They propose a compiler feedback-based code generation approach that leverages static analysis to identify mismatches between the generated code and the project's context. |
| Outcome: | The proposed model outperforms retrieval-based code generation baselines and significantly outperfies the existing large language models. |
An Element is Worth a Thousand Words: Enhancing Legal Case Retrieval by Incorporating Legal Elements (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for legal case retrieval lack the definition of relevance for legal cases . however, the definition goes beyond the common semantic relevance of ad-hoc retrieval. |
| Approach: | They propose a legal element dataset that incorporates legal elements into a semi-automatic method . they propose two models to enhance legal search using legal elements . |
| Outcome: | The proposed models outperform existing methods in enhancing legal search using legal elements. |
SoMeLVLM: A Large Vision Language Model for Social Media Processing (2024.findings-acl)
Copied to clipboard
Xinnong Zhang, Haoyu Kuang, Xinyi Mou, Hanjia Lyu, Kun Wu, Siming Chen, Jiebo Luo, Xuanjing Huang, Zhongyu Wei
| Challenge: | Genereal domain large models lack nuanced multimodal understanding of social media . general domain models focus more on text than other modalities, which is not consistent with real-world user habits. |
| Approach: | They propose a Large Vision Language Model for Social Media Processing that combines five key capabilities to understand and generate real social media behavior. |
| Outcome: | The proposed model achieves state-of-the-art performance in multiple social media tasks. |
KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are striving to grasp commonsense reasoning, but they are lacking in Korean commons- ense benchmarks. |
| Approach: | They present a fine-grained benchmark dataset focused on Korean commonsense reasoning that includes multiple-choice questions across seven error categories. |
| Outcome: | The proposed datasets show that LLMs struggle with Korean commonsense reasoning . human accuracy benchmarked at approximately 85%, while GPT-4’s performance lags at about 74%, and other LLM models demonstrate an average accuracy of around 42%. |
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurelie Lozano, Payel Das, Georgios Kollias
| Challenge: | Transformer-based Language Models have become ubiquitous in natural language processing due to impressive performance on various tasks. |
| Approach: | They explore how sparsity affects network topology by exploiting mechanisms seen in biological networks . they show that model-agnostic sparsities are performant across diverse NLP tasks . |
| Outcome: | The proposed model-agnostic sparsity approaches are performant and efficient across NLP tasks. |
Ranking Large Language Models without Ground Truth (2024.findings-acl)
Copied to clipboard
| Challenge: | Evaluation and ranking of large language models has become a problem with the proliferation of these models and their impact. |
| Approach: | They propose to rank large language models without access to ground truth or reference responses . they propose to use triplets of models to evaluate the other two, correctly identifying the worst model in the triplet with high probability. |
| Outcome: | The proposed method reliably recovers true rankings without reference data on generative tasks. |
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can enhance response richness and coherence, but there is a pressing need to bolster the model’s capacity for diagnostic logic to ensure patient safety. |
| Approach: | They propose an approach termed preference learning from process feedback (PLPF) that integrates the doctor’s diagnostic logic into LLMs. |
| Outcome: | The proposed approach improves the diagnostic accuracy of the baseline model in medical conversations by 17.6%, surpassing the performance of traditional approaches. |
LM-Cocktail: Resilient Tuning of Language Models via Model Merging (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are continually fine-tuned to better support downstream applications. however, this operation may result in significant performance degeneration on general perspectives. |
| Approach: | They propose a method which enables pre-trained language models to stay resilient in general perspectives. |
| Outcome: | The proposed model achieves strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. |
Episodic Memory Retrieval from LLMs: A Neuromorphic Mechanism to Generate Commonsense Counterfactuals for Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved satisfactory performance in counterfactual generation, however, there are misalignments between LLMs and humans which hinder LLM from handling complex tasks like relation extraction. |
| Approach: | They propose to mimic the episodic memory retrieval mechanism of human hippocampus to align LLMs’ generation process with that of humans. |
| Outcome: | The proposed framework improves over existing methods in terms of quality of counterfactuals. |
SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages (2024.findings-acl)
Copied to clipboard
Nedjma Ousidhoum, Shamsuddeen Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Ahmad, Sanchit Ahuja, Alham Aji, Vladimir Araujo, Abinew Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine Kock, Genet Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Yimam, Saif Mohammad
| Challenge: | SemRel datasets are annotated by native speakers across 13 languages . they are used to characterise the relationship between two units of text . |
| Approach: | They propose to use a semantic relatedness dataset to measure the degree of semantic textual relatedness between sentences in Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish, and Telugu. |
| Outcome: | The proposed datasets are annotated by native speakers across 13 languages and represent the semantic relatedness of 13 languages. |
Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to address overcorrection in Chinese grammatical error correction (CGEC) are difficult to adapt to decoder-only large language models (LLMs). |
| Approach: | They propose an alignment-enhanced corrector for the overcorrection problem that applies to both Seq2Seq models and decoder-only large language models (LLMs). |
| Outcome: | The proposed corrector alleviates the overcorrection problem in Chinese grammatical error correction (CGEC) using generative models and decoder-only large language models. |
VISPool: Enhancing Transformer Encoders with Vector Visibility Graph Neural Networks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing graph-based graph construction methods rely on static graphs and are not scalable with increasing document and word counts. |
| Approach: | They propose a dynamic graph construction method based on vector visibility graphs (VVGs) they propose scalable model architecture that integrates VVG convolutional networks into transformer pipelines. |
| Outcome: | The proposed model outperforms baseline models on the GLUE benchmark datasets. |
The Emotion Dynamics of Literary Novels (2024.findings-acl)
Copied to clipboard
| Challenge: | a new study examines the emotional journeys of characters in novels . previous studies have considered a novel as representing a single story arc . |
| Approach: | They analyze the emotion arcs of English literary novels using Utterance Emotion Dynamics . they find that narration and dialogue largely express disparate emotions through the course of a novel . |
| Outcome: | The analysis of English literary novels shows that narration and dialogue express disparate emotions . the commonalities or differences in the emotional arcs are more accurately captured by individual characters . |
Accurate and Nuanced Open-QA Evaluation Through Textual Entailment (2024.findings-acl)
Copied to clipboard
| Challenge: | Open-domain question answering (Open-QA) evaluations are criticized for the ambiguity in questions and the lack of semantic understanding in evaluators. |
| Approach: | They propose to examine the entailment relations of answers to identify more informative and more general system answers. |
| Outcome: | The proposed evaluations offer a much closer evaluation to human judgment on NaturalQuestions and TriviaQA while being learning-free. |
Dictionary-Aided Translation for Handling Multi-Word Expressions in Low-Resource Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-word expressions (MWEs) are a challenging task in natural language processing . they are defined as combinations of at least two words with distinct lexical, morphological, syntactic, semantic or statistical characteristics. |
| Approach: | They propose a method leveraging an available out-of-context lexicon to improve translations . they propose to use a dictionary-aided translation to better translate multi-word expressions based on human annotations. |
| Outcome: | The proposed method improves translations comparable to those of a human speaker. |
LANS: A Layout-Aware Neural Solver for Plane Geometry Problem (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing neural solvers take GPS as vision-language task but lack layout awareness . Existing models are criticized for complex rules and poor adaptability . |
| Approach: | They propose a layout-aware neural solver called LANS that integrates two modules to solve GPS. |
| Outcome: | The proposed solver outperforms existing neural and symbolic solvers on two datasets. |
Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, Yulia Tsvetkov
| Challenge: | Existing tasks and datasets assess LLM knowledge abilities mostly by focusing on atomic (e.g., open-domain QA) or linear (e-hop QA). |
| Approach: | They propose a geometric knowledge reasoning benchmark consisting of incomplete knowledge networks bounded by structured factual constraints where LLMs are tasked with inferring the missing facts to meet all constraints. |
| Outcome: | The proposed methods outperform baseline methods and are more robust towards problems in the hard subset. |
DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models are limited by challenges in factuality and hallucinations to be directly employed off-the-shelf for judging the veracity of news articles. |
| Approach: | They propose to integrate large language models into the news pipeline by generating news reactions and generating proxy tasks. |
| Outcome: | The proposed model outperforms state-of-the-art baselines by 16.8% in macro f1-score on seven datasets with three LLMs. |
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts (2024.findings-acl)
Copied to clipboard
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, Daniel Khashabi
| Challenge: | Recent studies show that malicious prompt instructions could solicit objectionable content from LLMs. |
| Approach: | They compare how state-of-the-art LLMs respond to malicious prompts in different languages . they find that LLM's generate unsafe responses more often when a prompt is written in a lower-resource language . |
| Outcome: | The proposed model can generate unsafe responses more often when a malicious prompt is written in a lower-resource language, and less irrelevant responses when written in lower-source languages. |
Self-Specialization: Uncovering Latent Expertise within Large Language Models (2024.findings-acl)
Copied to clipboard
Junmo Kang, Hongyin Luo, Yada Zhu, Jacob Hansen, James Glass, David Cox, Alan Ritter, Rogerio Feris, Leonid Karlinsky
| Challenge: | Recent studies have demonstrated the effectiveness of self-alignment in which a large language model is aligned to follow general instructions using instructional data generated from the model itself. |
| Approach: | They propose to use human-written seeds to align large language models to follow general instructions to achieve cross-task generalization. |
| Outcome: | The proposed model outperforms base models and models that are generally instruction-tuned or have been adapted to the target domain by a large margin. |
FUSE: Measure-Theoretic Compact Fuzzy Set Representation for Taxonomy Expansion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work models taxonomy concepts as vectors or geometric objects, but fuzzy sets are efficient for concept modeling. |
| Approach: | They propose a set representation learning task based on fuzzy set approximation . they demonstrate remarkable improvements in taxonomy expansion using FUSE . |
| Outcome: | The proposed framework improves taxonomy expansion performance by 23% over baselines. |
Chain of Logic: Rule-Based Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Logic models are prone to hallucinations and are not able to perform basic tasks like drafting and drafting documents. |
| Approach: | They propose a new prompting method which elicits rule-based reasoning through decomposition and recomposition. |
| Outcome: | The proposed method outperforms other prompting methods including chain of thought and self-ask on eight rule-based reasoning tasks. |
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing factuality metrics cannot evaluate paragraphs with ambiguous entities, authors show . |
| Approach: | They propose a new metric to evaluate the factuality of long-form generations from large language models. |
| Outcome: | The proposed metric can assess the factuality of people biographies with entity ambiguity better than FActScore. |
Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment (2024.findings-acl)
Copied to clipboard
| Challenge: | et al. argued that sentence co-occurrence probabilities should reflect entailment . but it is unclear whether probabilities predicted by neural LMs encode enanglement based on their theory . |
| Approach: | They propose a test that decodes entailment relations between natural sentences . they argue that the test that predicts a flipped test does not account for redundancy . |
| Outcome: | The proposed test can decode entailment relations between natural sentences, but not perfectly. |
Simulated Misinformation Susceptibility (SMISTS): Enhancing Misinformation Research with Large Language Model Simulations (2024.findings-acl)
Copied to clipboard
| Challenge: | Psychological inoculations have shown efficacy in curbing its spread and mitigating its adverse effects at early stages, but their design and optimization typically requires substantial human and financial resources due to the need for repeated experimental trials. |
| Approach: | They propose to use large language models to simulate participant responses in misinformation studies to mitigate caricatures and stereotypes in the simulations. |
| Outcome: | The proposed method mitigates caricatures and stereotypes in LLM simulations and enhances response diversity. |
Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on social intelligence in NLP does not provide a coherent subfield for researchers to analyze and identify research gaps and future directions. |
| Approach: | They build a social AI taxonomy and a data library of 480 NLP datasets to analyze existing datasets and evaluate language models’ performance in different social intelligence aspects. |
| Outcome: | The proposed infrastructure analyzes existing dataset efforts and evaluates language models’ performance in different social intelligence aspects. |
Selective Prefix Tuning for Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are time-consuming and memory-inefficient. |
| Approach: | They propose a method that inserts learnable vectors into each Transformer layer . they propose SL to encourage diversity in prefix tokens . |
| Outcome: | Extensive experiments validate the effectiveness of Prefix Tuning in sentence and token classification tasks. |
MODABS: Multi-Objective Learning for Dynamic Aspect-Based Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating content specific summarization assume a fixed set of known aspects. |
| Approach: | They propose a dynamic aspect-based summarization framework that optimizes aspect number prediction and minimizes disparity between generated and reference summaries. |
| Outcome: | The proposed method outperforms baselines on three diverse datasets on different aspects of the input text. |
Non-compositional Expression Generation and its Continual Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models are limited in their ability to generate non-compositional expressions. |
| Approach: | They propose a mask-infilling task to examine non-compositional expressions in English . they compare large pre-trained language models and continual learning methods . |
| Outcome: | The proposed task aims to investigate the ability of pre-trained language models to generate non-compositional expressions in English and their continual learning. |
Medical Dialogue System: A Survey of Categories, Methods, Evaluation and Challenges (2024.findings-acl)
Copied to clipboard
Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, Xiaofan Zhang, Shaoting Zhang
| Challenge: | Existing medical dialogue systems have significant potential to simplify diagnostic procedure and reduce the cost of collecting information from patients. |
| Approach: | They analyze 325 papers from well-known computer science, natural language processing conferences and journals to find out the major challenges of medical dialog systems. |
| Outcome: | The proposed systems have been surveyed in the medical community but have not been evaluated from a technical perspective. |
Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior research on evaluating large language models focused on answer accuracy, neglecting the correctness of the generated CoT. |
| Approach: | They propose a discriminative and generative CoT evaluation paradigm to assess LLMs’ knowledge of reasoning and the accuracy of the generated CoT. |
| Outcome: | The proposed evaluation paradigm assesses LLMs’ knowledge of reasoning and the accuracy of the generated CoT. |
Comprehensive Abstractive Comment Summarization with Dynamic Clustering and Chain of Thought (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on news comment summarization has focused on extractive methods within constraints. |
| Approach: | They propose an enhanced fast clustering algorithm that maintains a dynamic similarity threshold to ensure high density of each comment cluster being built. |
| Outcome: | The proposed method improves the baseline methods and the test suite on real-world news comments. |
Self-Supervised Position Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for debiasing large language models require external bias knowledge or annotated non-biased samples, which is lacking for position debiases. |
| Approach: | They propose a self-supervised position debiasing framework that leverages unsupervised responses from pre-trained LLMs for debiazing without external bias knowledge. |
| Outcome: | The proposed framework outperforms existing methods in mitigating three types of position biases on eight datasets and five tasks. |
HyperCL: A Contrastive Learning Framework for Hyper-Relational Knowledge Graph Embedding with Hierarchical Ontology (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies neglect the ontology of knowledge Graph (KG) embeddings and suffer from the dominance issue of facts over ontologies. |
| Approach: | They propose a framework for hyper-relational KG embeddings that captures the hierarchical ontology and a concept-aware contrastive loss to alleviate the dominance issue. |
| Outcome: | The proposed framework improves on three real-world datasets and shows that it can integrate with other embedding methods and improve link prediction performance. |
Encoding Hierarchical Schema via Concept Flow for Multifaceted Ideology Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on ideology detection focus on one generic facet and ignore label semantics and explanatory descriptions of ideologies. |
| Approach: | They propose a concept semantics-enhanced framework for multifaceted ideology detection . it enables concepts to flow across levels of the schema tree and enriches concept representations with multi-granularity semantics. |
| Outcome: | The proposed framework achieves state-of-the-art in the cross-topic scenario and on the benchmark dataset. |
Character-Level Chinese Dependency Parsing via Modeling Latent Intra-Word Structure (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing word-level dependency parsing methods in Chinese lack explicit word boundaries due to the lack of word boundaries. |
| Approach: | They propose to model latent internal structures within Chinese words by constrained Eisner algorithm . they propose to guarantee a single root for intra-word structures and establish inter-word dependencies . |
| Outcome: | The proposed model outperforms existing models on Chinese treebanks and shows that it can predict plausible intra-word structures. |
AlignRE: An Encoding and Semantic Alignment Approach for Zero-Shot Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing prototype-based methods for ZSRE ignore abundant side information and suffer from a significant encoding gap between prototypes and sentences. |
| Approach: | They propose a framework to encode schema alignment to enhance prototype-based ZSRE methods. |
| Outcome: | The proposed method outperforms existing methods on FewRel and Wiki-ZSL datasets and exhibits substantially faster performance and reduces the need for extensive manual labor in prototype construction. |
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models may not follow human instructions and produce toxic, hallucinated, or biased content. |
| Approach: | They propose a disperse-then-merge framework that dispersers instruction-following data into portions and trains multiple sub-models using different data portions. |
| Outcome: | The proposed framework outperforms data curation and training regularization on standard knowledge and reasoning benchmarks. |
Efficient Knowledge Infusion via KG-LLM Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge infusion face knowledge mismatch and poor information compliance of LLMs with knowledge graphs. |
| Approach: | They propose a three-stage alignment strategy to enhance the LLM's capability to utilize information from knowledge graphs. |
| Outcome: | The proposed method outperforms baselines on biomedical question-answering datasets and outperformed existing methods. |
Towards Precise Localization of Critical Errors in Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing critical error detection focuses on identifying sentence-level errors, leaving the precise localization of such errors unaddressed. |
| Approach: | They propose a task to detect critical errors at a fine-grained level in machine translation sentences. |
| Outcome: | The proposed method outperforms existing methods and LLMs in English to Korean translations. |
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaption (LoRA) is a low-level pruning method that can be expensive and slow to deploy. |
| Approach: | They propose a low-rank adaption pruning framework that provides an accurate structured pruned model in a memory-efficient manner. |
| Outcome: | The proposed pruning framework reduces perplexity and memory usage by 52.6% on LLaMA and T5 models while reducing memory usage. |
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to generate draft tokens in large language models are expensive and resource-intensive. |
| Approach: | They propose an approach to generate draft tokens using a segment of the LLM and a self-distillation method to enhance the quality of draft token. |
| Outcome: | The proposed approach generates draft tokens using a segment of the LLM and a self-distillation method to improve quality and speed up generation. |
Towards Better Utilization of Multi-Reference Training Data for Chinese Grammatical Error Correction (2024.findings-acl)
Copied to clipboard
| Challenge: | a high proportion of Chinese training data is multi-referenced for the grammatical error correction task . however, there are many ways to correct an erroneous input sentence . a systematic study on multi-referencing training data has been proposed . |
| Approach: | They propose two new approaches and a simple two-stage training strategy to better utilize multi-reference training data. |
| Outcome: | The proposed methods show that Chinese training data contain multiple references. |
AgentTuning: Enabling Generalized Agent Abilities for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Open large language models (LLMs) with great performance in various tasks are far inferior to commercial models such as ChatGPT and GPT-4 when acting as agents to tackle complex tasks in the real world. |
| Approach: | They propose a method to enhance the agent capabilities of LLMs while maintaining their general abilities. |
| Outcome: | The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent tasks, demonstrating generalized agent capabilities. |
Transition-based Opinion Generation for Aspect-based Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to aspect-based sentiment analysis often overlook the importance of explicitly modeling structure among sentiment elements. |
| Approach: | They propose to integrate general pre-trained sequence-to-sequence language models with a structure-aware transition-based approach to model sentiment structure. |
| Outcome: | The proposed model improves the state-of-the-art performance on several benchmark datasets. |
Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing dynamic topic models lack the ability to reveal the evolution of topics . Existing models suffer from repetitive topic and unassociated topic issues . |
| Approach: | They propose a new evolution-tracking contrastive learning method that builds the similarity relations among dynamic topics and an unassociated word exclusion method to avoid unassociated topics. |
| Outcome: | The proposed model outperforms state-of-the-art models on downstream tasks and is robust to evolution intensities. |
A Chinese Dataset for Evaluating the Safeguards in Large Language Models (2024.findings-acl)
Copied to clipboard
Yuxia Wang, Zenan Zhai, Haonan Li, Xudong Han, Shom Lin, Zhenxuan Zhang, Angela Zhao, Preslav Nakov, Timothy Baldwin
| Challenge: | a recent study has shown that large language models can produce harmful responses, exposing users to unexpected risks. |
| Approach: | They propose a dataset for the safety evaluation of Chinese LLMs in Mandarin Chinese . they extend the dataset to better identify false negative and false positive examples . |
| Outcome: | The proposed dataset is for the safety evaluation of Chinese LLMs, and is based on a Chinese dataset. |
LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work has focused on the performance of Large Language Models (LLMs) but the finance sector is relying on time-series data for complex forecasting tasks. |
| Approach: | They propose a framework that employs Sequential Knowledge-Guided Prompting to identify factors that influence stock movements using LLMs. |
| Outcome: | The proposed framework outperforms existing methods and is effective in time-series forecasting. |
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
SP3: Enhancing Structured Pruning via PCA Projection (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a widely used technique for reducing the size of pre-trained language models, but current methods overlook the potential of compressing the hidden dimension d in PLMs. |
| Approach: | They propose a structured pruning approach that projectes features into a space defined by principal components before masking the hidden dimension d in pre-trained language models. |
| Outcome: | Experiments on benchmarks show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy. |
GENDEX: Generative Data Augmentation Strategy Leveraging External Data for Abstractive Dialogue Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to summarize text data are limited by the lack of data. |
| Approach: | They propose a method that uses external data to generate synthetic dialogues from short texts containing people and their interpersonal interactions. |
| Outcome: | The proposed method shows robust performance, generalizability, and scalability regardless of complexity of dialogues. |
Concept-Best-Matching: Evaluating Compositionality In Emergent Communication (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods do not expose compositionality of emergent communication . compositionality is a trait that enables the construction of complex meanings from the meaning of parts. |
| Approach: | They propose to find best-match between emergent words and natural language concepts to assess compositionality of emergentic communication. |
| Outcome: | The proposed algorithm provides a global score and translation-map between emergent words and natural language concepts. |
A Tale of Two Revisions: Summarizing Changes Across Document Versions (2024.findings-acl)
Copied to clipboard
| Challenge: | Document revision is a crucial aspect of the writing process, especially in collaborative environments where multiple authors contribute simultaneously. |
| Approach: | They propose a task of providing thematic summary of changes between document versions, organizing individual edits based on shared themes, and propose three strategies to tackle this task. |
| Outcome: | The proposed model improves its capacity to handle the task and also enables it to be used in a curated dataset. |
Refine, Align, and Aggregate: Multi-view Linguistic Features Enhancement for Aspect Sentiment Triplet Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Aspect Sentiment Triplet Extraction (ASTE) aims to extract the triplets of aspect terms, their associated sentiment and opinion terms. |
| Approach: | They propose to use multi-view linguistic features enhancement to explore the prior indication effect in the “Refine, Align, and Aggregate” learning process to enhance aspect-opinion relations. |
| Outcome: | The proposed model achieves state-of-the-art on several benchmark datasets and is robust to state- of-the art constraints. |
Pro-Woman, Anti-Man? Identifying Gender Bias in Stance Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Gender bias has been widely observed in NLP models, which can perpetuate harmful stereotypes and discrimination. |
| Approach: | They construct a dataset to measure gender bias in stance detection using 36k samples . they find that all models are gender-biased and prone to classify sentences that contain male nouns as Against and those with female noun as Favor . |
| Outcome: | The proposed dataset shows that all models are gender-biased and prone to classify sentences that contain male nouns as Against and those with female noun as Favor. |
Likelihood-based Mitigation of Evaluation Bias in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. |
| Approach: | They propose to use LLMs to evaluate sentences with higher likelihoods and lower likelihoods to mitigate the likelihood bias. |
| Outcome: | The proposed method overrates sentences with higher likelihoods while underrating sentences with lower likelihoods. |
The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks to evaluate LLMs' capabilities are inadequate for assessing their musical capabilities. |
| Approach: | They propose to use a large-scale music benchmark specifically designed to evaluate the music-related capabilities of large language models (LLMs). |
| Outcome: | The proposed framework evaluates 16 large language models in the domain of music. |
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce memory usage for large language models neglect inter-layer dependency between layers and huge memory consumption in pre-computation. |
| Approach: | They propose a method that compresses the KV cache by layer-wise retaining crucial context. |
| Outcome: | The proposed method reduces memory usage by layer-wise retaining crucial context . it can improve 2.2x throughput compared to Accelerate with over 54% memory reduction . |
From Role-Play to Drama-Interaction: An LLM Solution (2024.findings-acl)
Copied to clipboard
| Challenge: | aristotle defined drama as a form of storytelling that involves a predefined storyline, emotions and thoughts. |
| Approach: | They propose to use LLMs to create an immersive mode of storytelling . they propose to create a backbone drama LLM to drive the playing process . |
| Outcome: | The proposed model can be used to drive the playing process, the authors say . it can be compared with existing models and can be evaluated on multiple scenarios. |
TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be used to simulate human behaviors, but point-in-time role-playing is a key component of fandom role-players. |
| Approach: | They propose a benchmark to evaluate point-in-time character hallucination in role-playing LLMs. |
| Outcome: | The proposed method reduces point-in-time character hallucinations effectively by decomposing reasoning steps and using narrative experts. |
Red Teaming Visual Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | VLMs (Vision-Language Models) can be induced to generate harmful or inaccurate content through specific test cases. |
| Approach: | They propose a red teaming dataset which encompasses 12 subtasks under 4 primary aspects (faithfulness, privacy, safety, fairness) this dataset is the first to benchmark current VLMs in terms of these 4 aspects . |
| Outcome: | The proposed dataset shows that 10 open-source VLMs struggle with red teaming in different degrees and have up to 31% performance gap with GPT-4V. |
Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models generate inconsistent and sometimes contradictory outputs when presented with a prompt that has equivalent semantics but is expressed differently from the original prompt. |
| Approach: | They propose to refine a Large Language Model (LLM) with prompt-output pairs with equivalent semantics to achieve semantic consistency. |
| Outcome: | The proposed method improves the semantic consistency and task performance of LLMs. |
Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for grounding pretrained language models as task planners are challenging due to their intricate entanglement with domain knowledge. |
| Approach: | They propose a framework that leverages the hierarchical nature of semantic skills to ground them in different domains. |
| Outcome: | The proposed framework is effective in 300 cross-domain EIF scenarios. |
LIRE: listwise reward enhancement for preference alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | prevailing approaches to preference alignment focus on pairwise comparisons, with limited exploration into multi-response scenarios. |
| Approach: | They propose a listwise reward enhancement approach that integrates offline rewards of multiple responses into a streamlined listwise framework. |
| Outcome: | The proposed approach outperforms existing methods on dialogue and summarization tasks with good transferability to out-of-distribution data. |
See It All: Contextualized Late Aggregation for 3D Dense Captioning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches to 3D dense captioning struggle with contradicting objectives . SIA generates captions with different region of interest and aggregates them afterwards . |
| Approach: | They propose a transformer pipeline that engages in 3D dense captioning with a new paradigm . SIA decodes two sets of queries—context query and instance query—and then aggregates them afterwards . |
| Outcome: | The proposed pipeline generates captions with different region of interest and aggregates them afterwards to enhance local-global sensitivity. |
DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to answer questions over Knowledge Graphs (KGQA) are not available for KGQA. |
| Approach: | They propose a framework to improve the neural-symbolic reasoning capabilities of language agents powered by Large Language Models (LLMs) they show that DARA can be efficiently trained with a small number of high-quality reasoning trajectories. |
| Outcome: | The proposed framework outperforms in-context learning-based agents with GPT-4 and alternative fine-tuned agents across different benchmarks. |
GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods of acceleration require fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users. |
| Approach: | They propose a Guidance-based Knowledge Transfer framework that leverages a larger LLM as a 'teacher' and a smaller 'student' model to finalize responses. |
| Outcome: | The proposed framework achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement 14.00% along with 7.73 times speed up in CSQA. |
Compositional Generalization with Grounded Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for combining language models with knowledge graphs struggle with generalization to sequences of unseen lengths and novel combinations of seen base components. |
| Approach: | They propose a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and avoids grounding models in information already encoded in their weights. |
| Outcome: | The proposed method fails to generalize to unseen lengths and to novel combinations of seen base components. |
Rethinking Negative Instances for Generative Named Entity Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models are constrained by a pre-defined label set and require extensive human annotations, which limits their flexibility and adaptability to unseen tasks. |
| Approach: | They propose a Generative NER system that shows improved zero-shot performance across unseen entity domains by introducing contextual information and delineating label boundaries. |
| Outcome: | The proposed model outperforms state-of-the-art methods in zero-shot evaluation. |
WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods focus on single editing, failing to meet the requirements for lifelong editing. |
| Approach: | They propose an approach that selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models. |
| Outcome: | The proposed method improves on GPT2-XL and GPT-J in lifelong editing compared to state-of-the-art methods . |
DINER: Debiasing Aspect-based Sentiment Analysis with Multi-variable Causal Inference (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to debiase ABSA focus on single-variable causal inference . aspect-based sentiment analysis models are prone to learn spurious correlations from annotation biases . |
| Approach: | They propose a framework based on multivariable causal inference for debiasing ABSA . they propose to model different types of biases based upon different causal intervention methods . |
| Outcome: | The proposed framework tackles different types of biases based on different intervention methods. |
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that supervised training is still necessary for complex reasoning tasks. |
| Approach: | They propose a method to integrate uncertainty-based active learning and LoRA to effectively integrate the two methods. |
| Outcome: | The proposed approach outperforms baseline models on three reasoning tasks. |
How Much Does Nonverbal Communication Conform to Entropy Rate Constancy?: A Case Study on Listener Gaze in Interaction (2024.findings-acl)
Copied to clipboard
| Challenge: | Whether the Entropy Rate Constancy principle applies to nonverbal communication signals is still under investigation. |
| Approach: | They perform empirical analyses of video-recorded dialogue data and investigate whether listener gaze adheres to the Entropy Rate Constancy principle. |
| Outcome: | The results show that the ERC principle holds for listener gaze, and that linguistic factors syntactic complexity and turn transition potential are weakly correlated with local entropy of listener gaze. |
Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) can be used as translation evaluators. |
| Approach: | They propose to use both coarse-grained and fine-grounded prompts to discern the utility of source versus reference data in machine translation evaluation tasks. |
| Outcome: | The proposed model can be used to evaluate translations in multiple languages. |
Chain-of-Verification Reduces Hallucination in Large Language Models (2024.findings-acl)
Copied to clipboard
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston
| Challenge: | Large language models can generate plausible but incorrect factual information, termed hallucinations, but they can still fail on lesser known facts. |
| Approach: | They develop a method that allows language models to deliberate on the responses they give in order to correct their errors. |
| Outcome: | The proposed method decreases hallucinations across a variety of tasks, including list-based questions, closed book MultiSpanQA and longform text generation. |
Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a novel approach, we can evaluate an agent’s bargaining abilities as an asymmetric incomplete information game. |
| Approach: | They propose an approach that integrates a deterministic Offer Generator and an LLM Narrator to create natural language sentences for generated offers. |
| Outcome: | The proposed approach improves the buyer’s deal rates from 26.67% to 88.88% and brings a ten times multiplication of profits on all baselines, even a model that has not been aligned. |
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024.findings-acl)
Copied to clipboard
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, Mengfei Yang
| Challenge: | Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs). |
| Approach: | They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories. |
| Outcome: | The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks. |
LPNL: Scalable Link Prediction with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on graph learning with large language models have focused on the link prediction task on large graphs. |
| Approach: | They propose a framework for scalable link prediction on large-scale heterogeneous graphs based on large language models. |
| Outcome: | The proposed framework outperforms baselines in link prediction tasks on large graphs. |
Aligning Speech Segments Beyond Pure Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing speech-to-speech parallel data is scarce and expensive to create from scratch. |
| Approach: | They propose an algorithm which automatically aligns pairs of speech segments aligned in meaning and expressivity. |
| Outcome: | The proposed algorithm outperforms semantic-focused approaches on content translation quality. |
Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives (2024.findings-acl)
Copied to clipboard
Thong Nguyen, Yi Bin, Junbin Xiao, Leigang Qu, Yicong Li, Jay Zhangjie Wu, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu
| Challenge: | Existing video-language understanding systems with human-like senses can mimic both our linguistic medium and visual environment with temporal dynamics. |
| Approach: | They propose to develop video-language understanding systems with human-like senses . they summarize their methods and highlight challenges associated with them . |
| Outcome: | The proposed models perform well in a variety of tasks and domains. |
Generative Input: Towards Next-Generation Input Methods Paradigm (2024.findings-acl)
Copied to clipboard
| Challenge: | generative models have been used for various NLP tasks but their application in the field of input methods remains under-explored. |
| Approach: | They propose a novel Generative Input paradigm that uses prompts to handle all input scenarios and other intelligent auxiliary input functions, optimizing the model with user feedback. |
| Outcome: | The proposed paradigm achieves state-of-the-art in the Full-mode Key-sequence to Characters task and surpasses GPT-4 in the other input methods. |
A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) is an effective solution to supplement necessary knowledge to large language models. |
| Approach: | They propose a "generate-then-read" pipeline to replace retrieval stage with generation from the LLM itself. |
| Outcome: | The proposed framework outperforms single models in the base and chat versions and addresses safety and helpfulness post-adaptation challenges. |
Functional Overlap Reranking for Neural Code Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Code Large Language Models (CodeLLMs) have ushered in a new era in code generation, but selecting the best code solutions remains a challenge. |
| Approach: | They propose a new reranking strategy that quantifies the functional overlap between solution clusters to provide a better ranking strategy for code solutions. |
| Outcome: | Empirical results show that the proposed method surpasses state-of-the-art methods on the pass@1 score. |
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for training large language models require additional annotations to adjust to shifted distributions. |
| Approach: | They propose an algorithm that allows LLMs and reward models to update alternatively via a min-max game to improve their alignment. |
| Outcome: | The proposed framework improves existing alignment baselines in terms of LLM helpfulness and harmlessness. |
Pinpointing Diffusion Grid Noise to Enhance Aspect Sentiment Quad Prediction (2024.findings-acl)
Copied to clipboard
| Challenge: | Current studies on aspect-based sentiment analysis focus on essential content for model generation, ignoring the incorporation of various noise during training. |
| Approach: | They propose a grid noise-diffusion pinpoint network (GDP) model that incorporates three new modules to tackle generation instability. |
| Outcome: | The proposed model reduces the generation instability of model learning and outputs by incorporating Consistency Likelihood Learning and GDP-FOR. |
Continual Contrastive Spoken Language Understanding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech processing require extensive offline training . however, these models struggle to retain their previously acquired knowledge when learning new tasks continuously. |
| Approach: | They propose a method that relies on experience replay and contrastive learning to preserve the learned representations by pulling closer samples from the same class and pushing away the others. |
| Outcome: | The proposed method preserves the learned representations by pulling closer samples from the same class and pushing away the others. |
LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge Graph (KG) inductive reasoning is widely adopted in various applications. |
| Approach: | They propose a framework for low-resource inductive reasoning using Large Language Models to generate a graph-structural prompt for pre-trained KGs. |
| Outcome: | The proposed framework outperforms previous methods in three-shot, one-shot and zero-shot reasoning tasks. |
Unsupervised Parsing by Searching for Frequent Word Sequences among Sentences with Equivalent Predicate-Argument Structures (2024.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised constituency parsing focuses on identifying word sequences that form a syntactic unit (i.e., constituents) in target sentences. |
| Approach: | They propose a frequency-based parser that computes the span-overlap score as the word sequence’s frequency in the PAS-equivalent sentence set and identifies the constituent structure by finding a constituent tree with the maximum span- overlap score. |
| Outcome: | The proposed method outperforms existing unsupervised parsers in eight out of ten languages and is more accurate than previous methods. |
Data-Centric Explainable Debiasing for Improving Fairness in Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing data-centric debiasing strategies mainly leverage explicit bias words for counterfactual data augmentation to balance the training data. |
| Approach: | They propose a method which uses an explainability method to search for implicit bias words to assist in debiasing PLMs. |
| Outcome: | Extensive results show that the proposed method achieves state-of-the-art debiasing performance and strong generalization while maintaining predictive abilities. |
Knowledge-Driven Cross-Document Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to extract relationships between entities are based on sentence-level tasks, but they do not consider domain knowledge, which are assumed to be known to the reader when documents are authored. |
| Approach: | They propose to embed domain knowledge of entities with input text for cross-document RE by embedding domain knowledge with the document. |
| Outcome: | The proposed framework offers interpretability by producing explanatory text for predicted relations between entities and improves performance over baseline methods. |
Injecting Salesperson’s Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research in dialogue systems focuses on task-oriented (TOD) and open-domain (chit-chat) dialogues. |
| Approach: | They propose to use chit-chat to simulate task-oriented dialogues to train sales agents. |
| Outcome: | The proposed model improves coherence and reduces aggression, improving model learning for sales-customer interactions. |
KG-Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are criticized for lack of expertise and knowledge conflict . KG-Adapter is a parameter-level KG integration method for decoder-only LLMs . |
| Approach: | They propose a parameter-level KG integration method based on parameter-efficient fine-tuning . they use KG-Adapter to integrate knowledge graphs with LLMs and perform joint reasoning . |
| Outcome: | The proposed method outperforms the current state-of-the-art method on four datasets for two different tasks. |
Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios (2024.findings-acl)
Copied to clipboard
Lei Lin, Jiayi Fu, Pengli Liu, Qingyang Li, Yan Gong, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai
| Challenge: | chain-of-thought (CoT) prompting has been shown to be effective on complex reasoning tasks, but the naive greedy decoding used in CoT prompting causes the repetitiveness and local optimality. |
| Approach: | They propose a generalizable ensemble-optimization method that uses a set of reasoning paths to prompt a language model one more time to determine the optimal answer. |
| Outcome: | The proposed method can be generalized to almost all scenarios where the type of input questions and answer format of reasoning paths may be unknown. |
Evaluating LLMs’ Mathematical Reasoning in Financial Document Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in natural language understanding, but their capability for complex mathematical reasoning with a hybrid of structured tables and unstructured text remain uncertain. |
| Approach: | They propose a prompting technique tailored to semi-structured documents that matches or outperforms baselines performance while providing a nuanced understanding of LLMs' abilities. |
| Outcome: | The proposed prompting technique outperforms baseline prompting techniques while providing a nuanced understanding of LLMs' abilities. |
Improving In-Context Learning with Prediction Feedback for Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved promising results in sentiment analysis through the in-context learning paradigm. |
| Approach: | They propose a framework that incorporates prior predictions and feedback to improve sentiment understanding by incorporating prior feedback and leveraging a feedback-driven prompt. |
| Outcome: | The proposed framework improves on nine sentiment analysis datasets with an average improvement of 5.95% over conventional methods. |
Can Large Language Models Mine Interpretable Financial Factors More Effectively? A Neural-Symbolic Factor Mining Agent Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing factor mining models are inefficient and inefficient, resulting in a significant challenge to extract interpretable factors. |
| Approach: | They propose a model that integrates the strengths of both neural and symbolic models for factor mining. |
| Outcome: | The proposed model surpasses the SOTA RankIC and RankICIR in predicting S&P 500 returns on real-world stock market data. |
Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraint (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing decoding methods for large language models (LLMs) are specialized in resolving knowledge conflicts and could inadvertently deteriorate performance in absence of conflicts. |
| Approach: | They propose an adaptive decoding method to discern whether knowledge conflicts occur and resolve them by a contextual information-entropy constraint decoding technique. |
| Outcome: | The proposed method improves the model’s faithfulness to conflicting context and maintains high performance among non-conflicting contexts. |
SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | SALAD-Bench is a safety benchmark specifically designed for LLMs . it provides a robust source for evaluating both attack and defense algorithms . |
| Approach: | They propose a hierarchical safety benchmark specifically designed for LLMs . it uses a taxonomy of questions spanning three levels and a robust taxonomies based on a QA pair . |
| Outcome: | The proposed safety benchmark shows that LLMs are resilient against emerging threats and the effectiveness of contemporary defense methods. |
Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation (2024.findings-acl)
Copied to clipboard
| Challenge: | Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. |
| Approach: | They propose a Fact Extractor that leverages large language models to extract factual statements from radiology reports. |
| Outcome: | The proposed framework outperforms current state-of-the-art methods in sentence ranking, natural language inference, and label extraction tasks. |
GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. |
| Approach: | They propose a prompt-based parameter-efficient fine-tuning approach that leverages insights into ICL’s information flow dynamics and hardwires the desired information flow into the GNN. |
| Outcome: | The proposed approach surpasses prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. |
M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answering (2024.findings-acl)
Copied to clipboard
Anand Subramanian, Viktor Schlegel, Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Vijay Prakash Dwivedi, Stefan Winkler
| Challenge: | Existing studies on adapting large language models to perform a variety of tasks in high-stakes domains such as healthcare lack understanding of the extent and contributing factors that allow them to recall relevant knowledge and combine it with presented information. |
| Approach: | They propose to use multiple choice and abstractive question answering to investigate the extent and contributing factors that allow LLMs to recall relevant knowledge and combine it with presented information in the clinical and biomedical domain. |
| Outcome: | The proposed models perform better on 22 datasets in three generalist and three specialist biomedical sub-domains, and show that they can generalise to unseen sub- domains. |
MovieSum: An Abstractive Summarization Dataset for Movie Screenplays (2024.findings-acl)
Copied to clipboard
| Challenge: | Movie screenplay summarization requires an understanding of long input contexts and elements unique to movies. |
| Approach: | They propose a dataset for movie screenplay summarization that includes movie screenplayers accompanied by their Wikipedia plot summaries. |
| Outcome: | The proposed dataset includes 2200 movie screenplays accompanied by their Wikipedia plot summaries. |
Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality (2024.findings-acl)
Copied to clipboard
Jiahuan Pei, Irene Viola, Haochen Huang, Junxiao Wang, Moonisa Ahsan, Fanghua Ye, Jiang Yiming, Yao Sai, Di Wang, Zhumin Chen, Pengjie Ren, Pablo Cesar
| Challenge: | a fine-grained, comprehensive understanding of multimodal environments remains under-explored. |
| Approach: | They propose an automated workflow for integrating AI agents into extended reality (XR) they propose a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent . |
| Outcome: | The proposed workflow integrates AI agents seamlessly into extended reality (XR) applications for fine-grained training. |
Perceptions of Language Technology Failures from South Asian English Speakers (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have identified performance disparities between Standard American English and other English dialects, but the degree to which these discrepancies affect user experience is not well understood. |
| Approach: | They aim to reduce performance gap for South Asian Englishes by surveying their interactions with language technology and comparing their results to a control survey. |
| Outcome: | The proposed model reduces the performance gap for South Asian Englishes (SAsE) speakers are more likely to recall failures with language technology and to reference specific issues with written language technology than SAmE speakers. |
A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies do not provide insights into the internal mechanisms driving the observed abilities. |
| Approach: | They propose to implement a depth-bounded recurrent mechanism that operates in parallel and stores intermediate results in selected token positions. |
| Outcome: | The proposed model implements a depth-bounded recurrent mechanism that operates in parallel and stores intermediate results in selected token positions. |
Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking (2024.findings-acl)
Copied to clipboard
Zefeng Zhang, Jiawei Sheng, Zhang Chuang, Liangyunzhi Liangyunzhi, Wenyuan Zhang, Siqi Wang, Tingwen Liu
| Challenge: | Existing methods to link ambiguous mentions to entities in multimodal knowledge graphs rely on partial correlations. |
| Approach: | They propose a framework that leverages multi-element correlations to bridge modality gap and enable fine-grained semantic matching by exploiting correlation between multimodal features and entities. |
| Outcome: | The proposed framework outperforms state-of-the-art models and confirms the effectiveness of the proposed method. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
A Survey on Modelling Morality for Text Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on modelling morality in text has garnered increasing attention due to its complexity and complexity. |
| Approach: | They provide a systematic review of recent work on modelling morality in text . they discuss challenges and research gaps in the area of NLP . |
| Outcome: | The authors present their work on the modelling of morality in text, which has garnered increasing attention in recent years. |
Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024.findings-acl)
Copied to clipboard
Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang
| Challenge: | Existing data selection methods for instruction-following large language models rely on unreliable scores or use downstream tasks for selection. |
| Approach: | They propose a method that utilizes the VLM itself as a filter to select high-quality instruction-tuning data. |
| Outcome: | The proposed method can reach better results compared to full data settings with merely about 15% samples and can achieve superior performance against competitive baselines. |
DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)
Copied to clipboard
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun
| Challenge: | Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored. |
| Approach: | They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python. |
| Outcome: | The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python. |
POP-CEE: Position-oriented Prompt-tuning Model for Causal Emotion Entailment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for emotion analysis in conversations ignore the specific semantic associations between emotions and cause utterances. |
| Approach: | They propose a position-oriented prompt-tuning model to solve the CEE task in an end-to-end manner. |
| Outcome: | The proposed model achieves state-of-the-art performance on a benchmark dataset. |
Context Length Extension via Generalized Extrapolation Scale (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on extrapolating positional embedding (RoPE) has limited results in the application of long context language models. |
| Approach: | They propose a set of parameterized extrapolation functions applied to each layer and attention head to adaptively adjust its extrapolations scales. |
| Outcome: | The proposed model achieves stable extrapolation on 64k contexts by training on 16k length text. |
Selectively Answering Visual Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are capable of visual question answering (VQA) with unprecedented accuracy. |
| Approach: | They propose a calibration score that can be used to quantify uncertainty in visual question answering models. |
| Outcome: | The proposed calibration score is better calibrated than in text-only models for in-context learning. |
Wav2SQL: Direct Generalizable Speech-To-SQL Parsing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for speech-driven SQL parsing are based on a cascaded approach, resulting in data scarcity and inconsistent performance. |
| Approach: | They propose a direct generalizable speech-to-SQL parsing model which avoids error compounding across cascaded systems. |
| Outcome: | The proposed model avoids error compounding and achieves state-of-the-art results by 4.7% improvement over baseline. |
E2-LLM: Efficient and Extreme Length Extension of Large Language Models (2024.findings-acl)
Copied to clipboard
Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, JiakaiWang JiakaiWang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, Bo Zheng
| Challenge: | Existing techniques for extending context capabilities in LLMs require additional training procedures and access to datasets with long context (e.g., sequences of 32K tokens). |
| Approach: | They propose a solution to extend context capabilities in Large Language Models by training a single process over a sequence of 4K tokens. |
| Outcome: | The proposed solution significantly reduces the cost of continual-pretraining or fine-tuning over short sequences and improves robustness to diverse relative positions. |
Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality (2024.findings-acl)
Copied to clipboard
| Challenge: | Typicality effects shape the way we use language, says a researcher . mentioning unusual properties of events is not a typical feature of bananas, he says . |
| Approach: | They explore whether occupations and gender display the same patterns of mentioning as do bananas and color . they find that gender mentioning is correlated with femaleness of occupation in particular . |
| Outcome: | a new study shows that people tend to mention unusual properties of events when they talk about them . the findings contradict the idea that gender mentioning occurs when special events are pointed out . |
Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments (2024.findings-acl)
Copied to clipboard
Sitao Cheng, Ziyuan Zhuang, Yong Xu, Fangkai Yang, Chaoyun Zhang, Xiaoting Qin, Xiang Huang, Ling Chen, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
| Challenge: | Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graphs and tables. |
| Approach: | They propose a framework that allows LLMs to efficiently and faithfully reason over structured environments. |
| Outcome: | The proposed framework surpasses state-of-the-art fine-tuned methods on three KGQA and two TableQA datasets and surpasse CWQ and WTQ methods. |
Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts (2024.findings-acl)
Copied to clipboard
| Challenge: | Prediction with Explanation is the largest expert-annotated dataset for legal judgment prediction and explanation in the Indian context . |
| Approach: | They propose to use an annotated legal judgment prediction corpus to improve models' accuracy . they employ transformer-based models tailored for both general and Indian legal contexts . |
| Outcome: | The proposed system improves the accuracy and explanatory depth of models for legal judgments. |
RulE: Knowledge Graph Reasoning with Rule Embedding (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge graph reasoning is an important problem for knowledge graphs. |
| Approach: | They propose a framework that leverages logical rules to enhance KG reasoning by learning rule embeddings from existing triplets and first-order rules. |
| Outcome: | The proposed framework outperforms existing embedding-based and rule-based methods on multiple benchmarks. |
Multi-Objective Linguistic Control of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Large language models prefer to generate verbose responses due to the length bias, which may increase unnecessary reading complexity. |
| Approach: | They propose to use off-the-shelf data to fine tune multiple linguistic complexities of LLM outputs to improve multi-complexity controllability and improve the quality of the responses. |
| Outcome: | The proposed method improves multi-complexity controllability significantly and retains or enhances the quality of the responses as a side benefit. |
Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Controllable text generation is increasingly tailored to individual preferences. |
| Approach: | They propose to evaluate the attribute intensity of text generated by large language models on five different attributes for error, variation of the generated sentence's intensities and relevance to the generation questions. |
| Outcome: | The proposed methods are based on Elo rating system and GPT4 and are able to be trained without training. |
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios (2024.findings-acl)
Copied to clipboard
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, Qun Liu
| Challenge: | Existing benchmarks focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. |
| Approach: | They propose a benchmark to evaluate LLMs’ ability in tool utilization within real-world scenarios. |
| Outcome: | The proposed benchmark improves LLMs’ ability in tool utilization within real-world scenarios and eliminates the restriction of pre-defined toolset. |
Do Androids Know They’re Only Dreaming of Electric Sheep? (2024.findings-acl)
Copied to clipboard
| Challenge: | Detecting hallucinations in grounded generation tasks is commonly framed as a textual entailment problem. |
| Approach: | They develop probes that are narrowly trained to predict hallucination in a transformer language model. |
| Outcome: | The probes can detect hallucinations at many transformer layers outperforming baselines and human annotators on two out of three generation tasks. |
URG: A Unified Ranking and Generation Method for Ensembling Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to rank and generate large language models have limited performance due to time-intensive nature of ranking process and lack of error propagation. |
| Approach: | They propose a framework that jointly ranks the outputs of Large Language Models and generates fine-grained fusion results. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) performance on ranking and generation tasks. |
Multi-Modal Retrieval For Large Language Model Based Speech Recognition (2024.findings-acl)
Copied to clipboard
Aditya Gourav, Jari Kolehmainen, Prashanth Shivakumar, Yile Gu, Grant Strimel, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko
| Challenge: | kNN-LM and cross-attention techniques are used to extend text based retrieval to other modalities . wide adoption of large language models has driven new application areas leveraging this technology . |
| Approach: | They propose to use kNN-LM and cross-attention techniques to extend text retrieval methods to other modalities. |
| Outcome: | The proposed methods outperform text-based retrieval and improve word error rate on a speech recognition dataset. |
LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is an effective yet efficient solution for fine-tuning large language models. |
| Approach: | They propose a low-rank Adaptation framework that retrieves and composes multiple LoRAs according to input prompts. |
| Outcome: | Experimental results show that LoraRetriever outperforms baselines in terms of performance and versatility. |
ELAD: Explanation-Guided Large Language Models Active Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hindered by their memory inefficiency, computational demands, and the high costs of API inferences. |
| Approach: | They propose an Explanation-Guided LLMs Active Distillation framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. |
| Outcome: | The proposed framework significantly improves the efficiency of LLMs knowledge distillation. |
Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ (2024.findings-acl)
Copied to clipboard
| Challenge: | a global majority of non-English speakers are underrepresented by large language models . however, most open LLMs are limited in their language coverage . |
| Approach: | They propose a silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. |
| Outcome: | The proposed model can answer questions in 27.4k questions across 137 languages. |
Semantics or spelling? Probing contextual word embeddings with orthographic noise (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) are used to generate contextual word embeddings . linguistics research has focused on semantic information in hidden states . |
| Approach: | They investigate whether a single character swap in the input word will not affect the resulting representation . they find that PLM-derived contextual word embeddings are highly sensitive to noise . |
| Outcome: | The results show that the PLM-derived representations are highly sensitive to noise . the fewer tokens used to represent a word at input, the more sensitive their CWE is . |
The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) (2024.findings-acl)
Copied to clipboard
Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, Jiliang Tang
| Challenge: | Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model generation with proprietary and private data, where data privacy is . a privacy issue that is currently under-explored, is posed by RAG. |
| Approach: | They propose to use retrieval-augmented generation (RAG) to facilitate language model generation with proprietary and private data where data privacy is a pivotal concern. |
| Outcome: | The proposed attack methods demonstrate that RAG can mitigate the old risks, i.e., leakage of the LLMs’ training data. |
EmpathicStories++: A Multimodal Dataset for Empathy Towards Personal Experiences (2024.findings-acl)
Copied to clipboard
Jocelyn Shen, Yubin Kim, Mohit Hulse, Wazeer Zulfikar, Sharifa Alghowinem, Cynthia Breazeal, Hae Park
| Challenge: | Existing datasets for empathy modeling are limited in the ways they are not captured in the wild. |
| Approach: | They propose a multimodal dataset for empathy during personal experience sharing that contains 53 hours of video, audio, and text data of 41 participants. |
| Outcome: | The EmpathicStories++ dataset contains 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. |
MRL Parsing Without Tears: The Case of Hebrew (2024.findings-acl)
Copied to clipboard
| Challenge: | a new approach to parsing morphologically rich languages (MRLs) is needed to overcome the deficiencies of current approaches. |
| Approach: | They propose a "flipped pipeline" where multiple layers are predicted independently on whole-token basis and then synthesized. |
| Outcome: | The proposed model achieves near-SOTA performance on Hebrew NLP tasks. |
SyntaxShap: Syntax-aware Explainability Method for Text Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for explaining sequence-to-sequence tasks are not suitable for textual data. |
| Approach: | They propose a local, model-agnostic explainability method that takes into account the syntax in text data and extends Shapley values to account for parsing-based syntactic dependencies. |
| Outcome: | The proposed method is compared to state-of-the-art explainability methods for text generation tasks using various metrics including faithfulness, coherency, and semantic alignment of the explanations to the model. |
Automated Detection and Analysis of Data Practices Using A Real-World Corpus (2024.findings-acl)
Copied to clipboard
| Challenge: | a crowd-sourced annotation tool matches data practices with policy excerpts . the complexity of privacy policies often deter users from reading them . |
| Approach: | They propose an automated approach to identify and visualize data practices within privacy policies at different levels of detail. |
| Outcome: | The proposed approach matches data practices with policy excerpts at different levels of detail. |
Enhancing Hyperbolic Knowledge Graph Embeddings via Lorentz Transformations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge graph embedding rely on tangent approximation and are not fully hyperbolic. |
| Approach: | They propose a fully hyperbolic KGE method that represents entities as points in the Lorentz model and represents relations as the intrinsic transformation. |
| Outcome: | The proposed method captures various types of relations including hierarchical structures. |
Tell Me What’s Next: Textual Foresight for Generic UI Representations (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities. |
| Approach: | They propose a pretraining objective for learning UI screen representations using captioning. |
| Outcome: | The proposed approach outperforms state-of-the-art on generation tasks with 28x fewer images. |
Probing the Uniquely Identifiable Linguistic Patterns of Conversational AI Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in deep learning and natural language processing have led to the proliferation of conversational AI agents. |
| Approach: | They construct linguistic profiles for five CAAs and use authorship attribution techniques to identify uniquely identifieable linguistic patterns for each model. |
| Outcome: | The proposed model identifies unique identifiers (UILPs) for each CAA using authorship attribution techniques. |
The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used to label data across many domains and for myriad tasks. |
| Approach: | They ask large language models to label data using a series of decisions by practitioners . they find that even the smallest perturbations can change the LLM's answer . |
| Outcome: | The proposed model can be used to quickly get a response for arbitrary tasks. |
X-Shot: A Unified System to Handle Frequent, Few-shot and Zero-shot Learning Simultaneously in Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on few-shot and zero-shot learning, but label occurrences vary widely . authors propose a new classification challenge that can be used to manage labels across the full frequency spectrum . |
| Approach: | They propose a new classification challenge that allows for label co-occurrences without predefined limits. |
| Outcome: | The proposed system can handle freq-shot, few-shot and zero-shot labels without limits. |
SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized text classification, but current paradigms rely on output of final layer . implicit internal structures that contribute to LLMs' impressive performance are neglected, forgoing potential performance gains. |
| Approach: | They propose a model-agnostic framework that sparsifies internal neurons of intermediate layers of LLMs for text classification. |
| Outcome: | The proposed framework significantly improves text classification accuracy, efficiency and interpretability. |
Decomposing Co-occurrence Matrices into Interpretable Components as Formal Concepts (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that word vectors capture relational meanings, but the interpretability of their dimensions remains an open issue. |
| Approach: | They employ the mathematical methodology of Formal Concept Analysis to examine word embeddings using a count-based co-occurrence matrix. |
| Outcome: | The proposed model shows that the formal concepts identified align with interpretable categories, as shown in the category completion task. |
Two-Pronged Human Evaluation of ChatGPT Self-Correction in Radiology Report Simplification (2024.findings-acl)
Copied to clipboard
| Challenge: | Radiology reports are highly technical documents aimed primarily at doctor-doctor communication. |
| Approach: | They propose a new evaluation protocol that employs radiologists and laypeople to produce high-quality simplifications. |
| Outcome: | The proposed evaluation protocol combines radiologists and laypeople to produce high-quality simplifications. |
Planning First, Question Second: An LLM-Guided Method for Controllable Question Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current CQG methods focus on difficulty control, neglecting the control of question content and assessed abilities, which are also crucial in educational QG. |
| Approach: | They propose an LLM-guided method PFQS which utilizes Llama 2 to generate an answer plan and then generates questions based on it. |
| Outcome: | The proposed method outperforms state-of-the-art methods and achieves better consistency with requirements in a zero-shot setting. |
RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated excellent performance in numerous tasks but the parameterized knowledge stored within LLMs may be incomplete and hard to incorporate up-to-date knowledge. |
| Approach: | They propose a framework that iteratively decomposes tasks and processes them in three submodules to enhance the model’s problem-solving capabilities. |
| Outcome: | The proposed method outperforms existing benchmarks on GPT3.5, Llama2 and other large language models significantly enhancing factual reasoning capabilities and reducing hallucinations. |
MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing IR techniques contain deficiencies, posing a performance bottleneck . combining diverse approaches to retrieve information is a viable strategy . |
| Approach: | They propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems. |
| Outcome: | The proposed method outperforms existing methods on two Retrieval Question Answering tasks. |
Chain-of-Question: A Progressive Question Decomposition Approach for Complex Knowledge Base Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to answer complex questions rely on decomposition of complex questions into sub-questions . Existing approaches to decompose complex questions are limited by the original question . |
| Approach: | They propose a question decomposition approach to decompose semantically clear questions . they use the decomposed sub-questions to select relevant patterns as auxiliary information . |
| Outcome: | The proposed method achieves state-of-the-art performance on multiple datasets. |
Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) identifies sentiment information related to specific aspects . previous studies have proposed using fixed examples for instruction tuning . |
| Approach: | They propose an instruction learning method with retrieval-based example ranking for ABSA tasks. |
| Outcome: | The proposed method is superior to existing models on three ABSA subtasks. |
Unveiling the Truth and Facilitating Change: Towards Agent-based Large-scale Social Movement Simulation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for simulating social movements encounter challenges in capturing behavior of participants. |
| Approach: | They propose a hybrid framework for social media user simulation wherein users are categorized into two types: core and ordinary users. |
| Outcome: | The proposed framework is able to simulate the behavior of social media users across real-world datasets and demonstrate its effectiveness and flexibility. |
Incorporating Syntax and Lexical Knowledge to Multilingual Sentiment Classification on Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | generative approach to multilingual sentiment classification is based on syntactic and lexical knowledge and requires retraining and tuning. |
| Approach: | They propose to use a sentiment extractor supported by syntactic and lexical resources to enhance multilingual sentiment classification without retraining LLMs. |
| Outcome: | The proposed approach reduces the multilingual sentiment classification error by 33 points and performs well even for nongenerative tasks such as topic classification and sentiment polarity judgment. |
Locating and Extracting Relational Concepts in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge recall models lack interpretability for relational concepts . a hidden state expresses causal effects of relational concept in input prompts . |
| Approach: | They propose to use causal mediation analysis to find hidden states that express relational concepts in LLMs. |
| Outcome: | The proposed representations exhibit high credibility and can be flexibly transplanted into other recall processes. |
Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Large Language Models (RALMs) do not consistently outperform the original retrieval-free Language Model (LM). |
| Approach: | They propose a trainable framework that can adaptively retrieve from different knowledge sources and effectively decrease unpredictable reader errors. |
| Outcome: | The proposed framework significantly improves performance over the RALM with a single retriever by significantly reducing inconsistent behaviors. |
SenticVec: Toward Robust and Human-Centric Neurosymbolic Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to tackle learning challenges such as knowledge forgetting and extensive computing resources are not effective. |
| Approach: | They propose a novel neurosymbolic method for sentiment analysis that places emphasis on human subjectivity within varying domain annotations. |
| Outcome: | The proposed method is lightweight, robust across domains and languages, efficient few-shot training, and rapid convergence. |
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on pre-trained LLMs to better understand and improve their trustworthiness. |
| Approach: | They apply linear probing to LLMs to explore five key dimensions of trustworthiness: reliability, privacy, toxicity, fairness, and robustness. |
| Outcome: | The proposed model can distinguish concepts in each trustworthiness dimension, suggesting that it can be trained in early pre-training. |
Language Models can Evaluate Themselves via Probability Discrepancy (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on superficial text differences and fail to align with human judgment. |
| Approach: | They propose a new method to evaluate the performance of Large Language Models (LLMs) by calculating probability discrepancies between original response generation and revised versions of LLMs. |
| Outcome: | The proposed method eliminates the need for training an additional evaluation model or relying on external proprietary models such as GPT-4 as a judger. |
Evaluating the Validity of Word-level Adversarial Attacks with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing adversarial examples can generate invalid adversarials due to significant changes in semantic meanings compared to their originals. |
| Approach: | They propose to use a large language model to evaluate adversarial examples by semantic constraints. |
| Outcome: | The proposed method can generate valid adversarial examples even when they are not equipped with semantic constraints. |
On the Language Encoder of Contrastive Cross-modal Models (2024.findings-acl)
Copied to clipboard
Mengjie Zhao, Junya Ono, Zhi Zhong, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Wei-Hsiang Liao, Takashi Shibuya, Hiromi Wakaki, Yuki Mitsufuji
| Challenge: | Pretrained audio-language models such as AudioCLIP and AudioCLAP have shown promising results on vision-language (VL) tasks. |
| Approach: | They extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. |
| Outcome: | The proposed model improves on visual-language (VL) and audio-language tasks when the amount of training data is large. |
Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on how large language model agents collaborate with humans in equal roles emphasize the importance of coordination and communication. |
| Approach: | They propose to use chain-of-thought prompts to evaluate different collaboration perspectives, from independent to more complex, dependent tasks. |
| Outcome: | The proposed model significantly improves the evaluation metric. |
Anchor-based Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) require massive GPU memory due to their size and parameter count. |
| Approach: | They propose to use anchor-based self-attention network and anchor-basic inference strategy to compress sequence information into an anchor token, reducing the keys/values cache and enhancing inference efficiency. |
| Outcome: | The proposed model reduces the key/value cache and improves inference efficiency by 99% while maintaining similar accuracy levels. |
MLeVLM: Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, Yu Huang
| Challenge: | Existing MVQA models ignore multi-level progressive capabilities due to unspecific data and plain architecture. |
| Approach: | They propose a multi-level visual language model for medical visual question answering (MVQA) which covers multi- level questions and answers as well as reasoning processes from visual clues to semantic cognition. |
| Outcome: | The proposed model outperforms existing medical multimodal large language models on a multi-level instruction dataset and a feature alignment module. |
Disentangling Length from Quality in Direct Preference Optimization (2024.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) exploits biases in human preferences, such as verbosity, and is under-explored for Direct Alignment Algorithms such as DPO. |
| Approach: | They propose to exploit verbosity biases in RLHF by using direct preference optimization to fine-tune models. |
| Outcome: | The proposed approach achieves 20% improvement in win rates when controlling for length on summarization and dialogue datasets, despite the GPT4 judge’s verbosity bias. |
MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing (2024.findings-acl)
Copied to clipboard
Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, Bozhong Tian
| Challenge: | Current benchmarks focus on coarse-grained knowledge, leaving the intricacies of fine-grounded knowledge unexplored. |
| Approach: | They propose a benchmark and dataset specifically designed for FG multimodal entity knowledge editing. |
| Outcome: | The proposed benchmark underscoring the complexity of FG knowledge editing in MLLMs. |
Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domain (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models like GPT-4 have demonstrated astonishing zero-shot capabilities in general domain tasks, but they often generate content with hallucinations in specific domains such as Chinese law. |
| Approach: | They propose a framework for adapting large language models (LLMs) to Chinese legal domains by reformulating generation as an adapt-retrieve-revise process. |
| Outcome: | The proposed framework outperforms existing models in the Chinese legal domain by +33.6 points in the zero-shot setting. |
MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on harms of memes in closed environments, such as hate speech and cyber-bullying. |
| Approach: | They propose a multimodal question-answering framework that solicits accurate responses to structured questions while providing coherent explanations. |
| Outcome: | The proposed framework outperforms existing frameworks in predicting answer prediction accuracy and text generation lead over a baseline. |
Improving Attributed Text Generation of Large Language Models via Preference Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models have been widely adopted in natural language processing, yet they produce unreliable content. |
| Approach: | They propose to model the attribution task as preference learning and introduce an automatic preference optimization framework that synthesizes attribution preference data. |
| Outcome: | The proposed method achieves state-of-the-art citation F1 with higher answer quality than existing methods. |
KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models for Korean have overlooked character representation principles . a new framework for Korean PLMs uses the invention principles of Hangeul to represent letters . |
| Approach: | They propose a framework that uses Hangeul's invention principles to represent character in Korean . their method outperforms the state-of-the-art Korean PLM by 2.11% in five tasks . |
| Outcome: | The proposed method outperforms the state-of-the-art Korean PLM by 2.11% in five Korean natural language understanding tasks. |
Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactic Language Models (SLMs) have difficulty with inference efficiency due to explicit generation of syntactical structures. |
| Approach: | They propose a method to "plant" trees into attention weights of unidirectional Transformer LMs to implicitly reflect syntactic structures of natural language. |
| Outcome: | The proposed method outperforms SLMs on the SyntaxGym benchmark. |
Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing jailbreak attacks primarily utilize scenario camouflage techniques, however their explicit mention of malicious intent will be easily recognized and defended by LLMs. |
| Approach: | They propose an indirect jailbreak attack approach, Puzzler, which can bypass LLM’s defensive strategies and obtain malicious response by implicitly providing LLMs with some clues about the original malicious query. |
| Outcome: | The proposed approach can bypass the LLM’s defensive strategies and obtain malicious response by implicitly providing LLMs with some clues about the original malicious query. |
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes (2024.findings-acl)
Copied to clipboard
Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, Edward Choi
| Challenge: | Clinical notes are an extensive repository of information specific to individual patients. |
| Approach: | They create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature and train a clinical large language model, Asclepius. |
| Outcome: | The proposed model outperforms several other models and is supported by detailed evaluations conducted by GPT-4 and medical professionals. |
Extending Context Window of Large Language Models via Semantic Compression (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models rely on a quadratic computation to generate long texts . current models impose limitations on the length of text inputs . |
| Approach: | They propose a semantic compression method that extends the context window of large language models . the method reduces the semantic redundancy of long inputs before passing them to the LLMs . |
| Outcome: | The proposed method extends the context window of large language models across tasks . it exhibits consistent fluency in text generation while reducing associated computational overhead. |
Plausible Extractive Rationalization through Semi-Supervised Entailment Signal (2024.findings-acl)
Copied to clipboard
| Challenge: | Abstract: Large language models are gaining widespread adoption in natural language processing tasks. |
| Approach: | They propose a semi-supervised approach to optimize for plausibility of extracted rationales by using a pre-trained natural language inference model and a supervised NLI predictor. |
| Outcome: | The proposed model outperforms unsupervised models by > 100% on a ERASER dataset. |
Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering (2024.findings-acl)
Copied to clipboard
ChaeHun Park, Koanho Lee, Hyesu Lim, Jaeseok Kim, Junmo Park, Yu-Jung Heo, Du-Seong Chang, Jaegul Choo
| Challenge: | Recent studies have employed machine translation systems for cross-lingual VQA tasks . however, translated texts contain unique characteristics distinct from human-written ones, referred to as translation artifacts. |
| Approach: | They propose a machine translation system that can train models in multiple languages . they propose augmentation strategies that reduce translation artifacts in translated texts . |
| Outcome: | The proposed approach reduces translation artifacts in models across languages and languages. |
Scented-EAE: Stage-Customized Entity Type Embedding for Event Argument Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for incorporating entities into EAE rely on prompts or NER . weak semantic associations due to missing role-entity correspondence cues . one-sided semantic understanding relying solely on argument role semantics a problem . |
| Approach: | They propose an EAE model with stage-customized entity type embedding to explore the role of entity types. |
| Outcome: | The proposed model achieves state-of-the-art performance on mainstream benchmarks and robustness in low-resource settings. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
SDA: Semantic Discrepancy Alignment for Text-conditioned Image Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for textconditioned image retrieval are limited due to the scale of training and the complexity of attributes in modification texts. |
| Approach: | They propose a general boosting framework to address these issues by employing semantic discrepancy alignment by formulating distance consistency and neighbor consistency between the image and text domains. |
| Outcome: | The proposed framework improves retrieval performance on three prominent datasets with state-of-the-art results. |
Se2: Sequential Example Selection for In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work has explored the selection of examples for in-context learning, neglecting the internal relationships between examples and exist an inconsistency between training and inference. |
| Approach: | They propose a sequential-aware method that leverages the LLM’s feedback on varying context, aiding in capturing inter-relationships and sequential information among examples. |
| Outcome: | Experiments on 23 NLP tasks show that Se2 surpasses baselines and achieves 42% relative improvement over random selection. |
Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models generate tokens sequentially and are memory-bound, resulting in a memory-based inference stage that is memory-limited. |
| Approach: | They propose an approach to accelerate the inference speed of large language models with billions of parameters by integrating semi-autoregressive inference and speculative decoding capabilities. |
| Outcome: | The proposed approach has demonstrated inference speedups of 2.7x-4.0x on humanEval-X while maintaining output quality. |
StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current evaluations for large language models use a single-item assessment paradigm . current evaluations struggle to discern whether a model possesses the required capabilities or merely memorizes/guesses the answers to specific questions. |
| Approach: | They propose a framework to evaluate large language models using atomic test objectives. |
| Outcome: | The proposed evaluation framework resists data contamination and reduces interference of potential biases, and sheds light on the design of future principled and trustworthy LLM evaluation protocols. |
Mitigating Privacy Seesaw in Large Language Models: Augmented Privacy Neuron Editing via Activation Patching (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing privacy protection frameworks for large language models are inadequate . however, training data for LLMs often contains sensitive or unauthorized information . |
| Approach: | They propose a framework that augments privacy data by automatically synthesizing new private data, which deactivates the first trigger to privacy seesaw issue. |
| Outcome: | The proposed framework can mitigate Privacy Seesaw in large language models by synthesizing new private data and adapting activation patching to privacy neuron editing for switching off the second trigger. |
Which Information Matters? Dissecting Human-written Multi-document Summaries with Partial Information Decomposition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to characterize human-written summaries do not account for the nature of high-quality summary. |
| Approach: | They propose to characterize human-written summaries using partial information decomposition . they propose to decompose mutual information provided by all source documents into union, redundancy, synergy, and unique information . |
| Outcome: | The proposed approach decomposes the mutual information provided by all source documents into union, redundancy, synergy, and unique information. |
BadActs: A Universal Backdoor Defense in the Activation Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Backdoor attacks pose an increasingly severe security threat to Deep Neural Networks . existing methods focused on the word space are ineffective against feature-space triggers - a recent study has shown . |
| Approach: | They propose a backdoor defense that purifies backdoor samples in the activation space . they aim to eliminate backdoor triggers while preserving the integrity of clean data . |
| Outcome: | The proposed method achieves state-of-the-art against backdoor attacks on clean data. |
ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining (2024.findings-acl)
Copied to clipboard
Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua
| Challenge: | Molecular-text modeling is an emerging research field that aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge. |
| Approach: | They propose a new method for reaction-text modeling that uses three types of input contexts to incrementally pretrain LMs. |
| Outcome: | The proposed method improves experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. |
Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical Visual Question Answering (Med-VQA) aims to provide accurate answers to questions regarding medical images, a task particularly challenging for open-ended questions. |
| Approach: | They propose a multi-modal concept alignment pre-training approach for generative Med-VQA that leverages a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System. |
| Outcome: | The proposed approach significantly outperforms existing methods on a set of benchmark datasets and shows high efficiency and knowledge-image alignment capability. |
Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques (2024.findings-acl)
Copied to clipboard
Siva Rajesh Kasa, Aniket Goel, Karan Gupta, Sumegh Roychowdhury, Pattisapu Priyatam, Anish Bhanushali, Prasanna Srinivasa Murthy
| Challenge: | Ordinal classification (OC) is a key task in natural language processing with applications in various domains such as sentiment analysis, rating prediction, and more. |
| Approach: | They propose to tackle ordinal classification (OC) through the implicit semantics of the labels . they propose to use a classical explicit approach and an implicit approach that organically engages the semantics. |
| Outcome: | The proposed methods are based on pre-trained language models and offer strategic recommendations based upon specific settings. |
Evaluating Large Language Models on Wikipedia-Style Survey Generation (2024.findings-acl)
Copied to clipboard
Fan Gao, Hang Jiang, Rui Yang, Qingcheng Zeng, Jinghui Lu, Moritz Blum, Tianwei She, Yuang Jiang, Irene Li
| Challenge: | Recent studies have shown that large language models can perform well in general tasks, but their effectiveness and limitations in domainspecific tasks remain unclear. |
| Approach: | They examine the proficiency of Large Language Models (LLMs) in generating succinct survey articles specific to the niche field of NLP in computer science. |
| Outcome: | The LLMs perform better in generating succinct survey articles specific to the niche field of NLP in computer science, compared to human-authored surveys, but they exhibit bias in evaluation. |
The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse (2024.findings-acl)
Copied to clipboard
| Challenge: | Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. |
| Approach: | They propose to use perplexity as a surrogate metric to determine whether an edited model's performance is affected by a single edit. |
| Outcome: | The proposed method shows that even a single edit can cause model collapse, manifesting as significant performance degradation in various benchmark tasks. |
Can We Continually Edit Language Models? On the Knowledge Attenuation in Sequential Model Editing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for model editing suffer from knowledge attenuation due to redundant parameters interference and update weight disentanglement. |
| Approach: | They propose a method to mitigate the problem of knowledge attenuation in sequential editing by analyzing redundant parameters interference and update weight disentanglement. |
| Outcome: | The proposed method mitigates the knowledge attenuation issue and improves on existing benchmarks. |
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) driven by In-Context Learning (ICL) have improved performance of text-to-SQL. |
| Approach: | They propose a strategy to mitigate hallucinations in large language models driven by In-Context Learning (ICL) they propose TA-SQL, a text-to-Sql framework that encourages LLMs to take advantage of similar tasks rather than starting from scratch. |
| Outcome: | The proposed framework improves the performance of the GPT-4 model by 21.23% on BIRD dev. |
Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image with translations in target language. |
| Approach: | They propose an end-to-end IIMT model with four modules that translate images . they propose a two-stage training framework to assist the model in learning alignment across languages . |
| Outcome: | The proposed model outperforms cascaded models with only 70.9% of parameters and is highly accurate. |
StatBot.Swiss: Bilingual Open Data Exploration in Natural Language (2024.findings-acl)
Copied to clipboard
Farhad Nooralahzadeh, Yi Zhang, Ellery Smith, Sabine Maennel, Cyril Matthey-Doret, Raphaël De Fondeville, Kurt Stockinger
| Challenge: | StatBot.Swiss dataset is the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. |
| Approach: | They propose to use a bilingual dataset to evaluate LLMs in Text-to-SQL systems. |
| Outcome: | The proposed dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for English and German. |
Subtle Signatures, Strong Shields: Advancing Robust and Imperceptible Watermarking in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have led to an increase in AI-generated text on the Internet, presenting a crucial challenge to differentiate AI-created content from human-written text. |
| Approach: | They propose a novel approach to embed watermarks into LLMs that leverages token prior probabilities to improve detectability and maintain watermark imperceptibility. |
| Outcome: | The proposed method improves detectability and imperceptibility of watermarks by partitioning tokens into two distinct groups based on prior probabilities and employing tailored strategies for each group. |
Thinking about how to extract: Energizing LLMs’ emergence capabilities for document-level event argument extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for document-level event argument extraction (D-EAE) lack key feature forgetting and cross-event argument confusion. |
| Approach: | They propose a document-level event argument extraction method based on guided summarization and reasoning that leverages the emergence capabilities of large language models to highlight key event information. |
| Outcome: | The proposed method outperforms baseline models by 1.3% F1 and 1.6% F1 on WIKIEVENTS and RAMS. |
Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) methods require a substantial quantity of high-quality annotation for training models. |
| Approach: | They propose a method to reduce the number of incorrect pseudo labels in self-training . they propose 'uncertainty-aware teacher learning' and 'student-student collaboration' |
| Outcome: | The proposed method is superior to state-of-the-art DS-NER denoising methods. |
Predicting Narratives of Climate Obstruction in Social Media Advertising (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media advertising allows entities to construct narratives that align with their commercial interests and sway public perception. |
| Approach: | They propose to classify climate-related narratives into seven categories based on existing definitions and data. |
| Outcome: | The proposed method outperforms other methods and can reduce human annotation costs. |
SSS: Editing Factual Knowledge in Language Models towards Semantic Sparse Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to modify LMs suffer from sub-optimal locality, where irrelevant neighborhood examples can be adversely influenced. |
| Approach: | They propose to use a model editing method to modify specific examples in LMs to improve locality and reasoning capability by directing the hidden state of edit example towards spaces where semantics are sparse. |
| Outcome: | The proposed method improves locality and reasoning capability on two datasets. |
GeoHard: Towards Measuring Class-wise Hardness through Modelling Class Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in measuring hardness-wise properties of data guide language models in sample selection within low-resource scenarios. |
| Approach: | They propose to use class-wise hardness to measure class-specific properties of data in the semantic embedding space by modeling class geometry in the . semantic embeddining space. |
| Outcome: | The proposed method surpasses instance-level metrics by over 59 percent on Pearson‘s correlation on measuring class-wise hardness. |
Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Using zero-shot or few-shot prompting, Large Language Models have been widely adopted in downstream applications. |
| Approach: | They propose to quantify the impact of option order and token usage on LLMs and propose mitigation strategies to enhance model performance. |
| Outcome: | The proposed mitigation strategies improve model performance and reduce the impact of token and order sensitivity on LLMs. |
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic (2024.findings-acl)
Copied to clipboard
Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, Timothy Baldwin
| Challenge: | evaluating language models in Arabic remains challenging due to limited datasets . focus has shift to reasoning and knowledge-intensive tasks due to lack of relevant datasets. |
| Approach: | They propose to use ArabicMMLU to evaluate models' understanding of Arabic . they use 40 tasks and 14,575 multiple-choice questions from school exams in different countries . |
| Outcome: | The ArabicMMLU is the first multi-task language understanding benchmark for the Arabic language . it is based on 40 tasks and 14,575 multiple-choice questions in modern standard Arabic . the models are based in different countries across North Africa, the Levant, and the Gulf regions . |
On the Relationship Between RNN Hidden-State Vectors and Semantic Structures (2024.findings-acl)
Copied to clipboard
| Challenge: | Using hidden-state vectors of recurrent neural networks (RNNs) we examine the assumption that hidden- state vectors tend to form clusters of semantically similar vectors, which we dub the clustering hypothesis. |
| Approach: | They propose to use recurrent neural networks (RNNs) that model processes with internal states to test their hypothesis. |
| Outcome: | The proposed model is based on a set of RNNs that were trained to recognize regular languages and a context-free language. |
XMC-Agent : Dynamic Navigation over Scalable Hierarchical Index for Incremental Extreme Multi-label Classification (2024.findings-acl)
Copied to clipboard
Yanjiang Liu, Tianyun Zhong, Yaojie Lu, Hongyu Lin, Ben He, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Existing methods for XMC struggle with the growing set of labels due to their static label assumptions, and embedding-based methods struggle with complex mapping relationships due to late interaction paradigm. |
| Approach: | They propose a large language model (LLM) powered agent framework for extreme multi-label classification, XMC-Agent, which can effectively learn, manage and predict the extremely large and dynamically increasing set of labels. |
| Outcome: | The proposed framework can learn, manage and predict the extremely large and dynamically growing set of labels and achieves state-of-the-art performance on three standard datasets. |
Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized natural language processing (NLP) there is an urgent need for new benchmarks to keep pace with the development of LLMs. |
| Approach: | They propose a benchmark to assess the capability of large language models (LLMs) they use a dataset to provide both knowledge assessment and application assessment . |
| Outcome: | The proposed benchmark provides datasets tailored for knowledge assessment and application assessment. |
Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods do not provide fine-grained supervision for complex reasoning tasks. |
| Approach: | They propose a reinforcement learning method that incorporates a generative model as the reward model and a token-level supervision model for RL training. |
| Outcome: | Experiments on 8 tasks show the proposed method is effective . |
Definition generation for lexical semantic change detection (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have attempted to bridge the gap between lexical semantic change detection and sense-based LSCD methods. |
| Approach: | They propose a sense distribution based LSCD method which uses contextualized word definitions as 'senses' they argue that the method preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-sense. |
| Outcome: | The proposed method outperforms previous sense-based methods on five datasets and three languages and preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. |
MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector (2024.findings-acl)
Copied to clipboard
Marta Costa-jussà, Mariano Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alexandre Mourachko, Christophe Ropers, Carleigh Wood
| Challenge: | Existing studies on text-based toxicity detection for other languages are limited, especially for languages other than English. |
| Approach: | They propose a multilingual audio-based toxicity classifier which covers 14 different linguistic families and a dataset of 20,000 audio utterances for English and Spanish. |
| Outcome: | The new classifier improves F1-Score by an average of 100% when compared to existing wordlist-based classifiers. |
Phased Instruction Fine-Tuning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance pre-trained language models' ability to follow instructions are limited due to the simultaneous handling of varying instruction complexities. |
| Approach: | They propose a phased instruction fine-tuning method that posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. |
| Outcome: | The proposed method surpasses the one-off instruction fine-tuning method in win rate and validates the hypothesis of progressive alignment. |
TOREE: Evaluating Topic Relevance of Student Essays for Chinese Primary and Middle School Education (2024.findings-acl)
Copied to clipboard
Xinlin Zhuang, Hongyi Wu, Xinshu Shen, Peimin Yu, Gaowei Yi, Xinhao Chen, Tu Hu, Yang Chen, Yupei Ren, Yadong Zhang, Youqi Song, Binxuan Liu, Man Lan
| Challenge: | Existing research on Automatic Essay Scoring (AES) for Chinese essays has overlooked topic relevance and lacks detailed feedback. |
| Approach: | They propose to use TOREE to assess topic relevance in Chinese primary and middle school students’ essays to improve automatic and human evaluations. |
| Outcome: | The proposed method significantly improves both automatic and human evaluations across four diverse LLMs. |
Predicting the Unpredictable: Uncertainty-Aware Reasoning over Temporal Knowledge Graphs via Diffusion Process (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Temporal Knowledge Graph reasoning capture indeterminacy in future events, but they are limited in capturing it. |
| Approach: | They propose a Temporal Knowledge Graph reasoning process that denoises historical events and introduces Gaussian noise to corrupt target facts. |
| Outcome: | Empirical results show that DiffuTKG outperforms state-of-the-art methods on four real-world datasets. |
Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on adversarial attacks against text-to-image models have not explored the reasons for their effectiveness. |
| Approach: | They propose an adversarial attack objective - entity swapping using adversarials and two gradient-based attack algorithms. |
| Outcome: | The proposed attack objective - entity swapping - is easier to replace “human” with “robot” in the prompt “a human dancing in the rain.” |
Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Controlled Text Generation (CTG) aims to produce texts that exhibit specific desired attributes. |
| Approach: | They propose a pluggable CTG framework for Large Language Models to control text . they use attribute scorers to evaluate attributes of sentences and construct dynamic attribute graphs . |
| Outcome: | The proposed framework achieves a peak improvement of 19.29% over baseline methods in two tasks. |
Coconut: Contextualized Commonsense Unified Transformers for Graph-Based Commonsense Augmentation of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained language models lack commonsense knowledge . |
| Approach: | They propose a contextualized knowledge prompting scheme to guide the contextualization of structured commonsense knowledge based on large language models. |
| Outcome: | The proposed approach outperforms the state-of-the-art technique by an average of 5.8%. |
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. |
| Approach: | They propose a method that allows users to edit factual associations without catastrophic forgetting. |
| Outcome: | The proposed method achieves 10% increase in magnitude metrics while requiring minimal parameter modifications. |
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024.findings-acl)
Copied to clipboard
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, Richard Dufour
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. |
| Approach: | They propose an open-source LLM tailored for the biomedical domain that utilizes Mistral as its foundation model and pre-trained on PubMed Central. |
| Outcome: | The proposed model outperforms existing models on a benchmark comprising 10 established medical question-answering tasks in English and is competitive with proprietary models. |
All Languages Matter: On the Multilingual Safety of LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. |
| Approach: | They propose a prompting method to improve multilingual safety of ChatGPT by enhancing cross-lingual generalization of safety alignment. |
| Outcome: | The proposed method can significantly reduce the ratio of unsafe responses by 42% for non-English queries. |
LJPCheck: Functional Tests for Legal Judgment Prediction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LJP models fail to evaluate specific aspects of their performance, such as legal fairness and judicial fairness. |
| Approach: | They propose a suite of functional tests for LJP models to comprehend LJp models’ behaviors and offer diagnostic insights. |
| Outcome: | Extensive tests reveal weaknesses in LJP models and provide diagnostic insights. |
CMDL: A Large-Scale Chinese Multi-Defendant Legal Judgment Prediction Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Legal Judgment Prediction (LJP) has attracted significant attention in recent years. |
| Approach: | They propose a large-scale Chinese Multi-Defendant LJP dataset . they propose case-level evaluation metrics dedicated for the multi-defendant scenario . |
| Outcome: | The proposed methods show weaknesses when applied to cases involving multiple defendants. |
Model Editing by Standard Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | specialized methods for model editing are not as effective due to poor performance . standard fine-tuning alone can yield competitive model editing performance if it is modified . |
| Approach: | They propose to optimize conditional likelihood rather than the full likelihood . they also train on random or similar unedited facts to encourage locality . |
| Outcome: | The proposed model editing method outperforms specialized models in terms of edit score. |
Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning (2024.findings-acl)
Copied to clipboard
Qiming Bao, Alex Peng, Zhenyun Deng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Neset Tan, Nathan Young, Yang Chen, Yonghua Zhu, Paul Denny, Michael Witbrock, Jiamou Liu
| Challenge: | Empirical evidence shows that our proposed method improves performance across seven downstream tasks. |
| Approach: | They propose a logic-driven data augmentation approach that converts text into AMR graphs and converts them back into text to create augmented data. |
| Outcome: | The proposed method leads on the ReClor leaderboard and improves on seven downstream tasks. |
CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024.findings-acl)
Copied to clipboard
| Challenge: | Comprising 3,402 crafted examples, our dataset is designed for both model finetuning and standalone evaluation. |
| Approach: | They propose a dataset that provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. |
| Outcome: | The proposed dataset includes 3,402 hand-written examples and 3,121 unrefined examples. |
ViHateT5: Enhancing Hate Speech Detection in Vietnamese With a Unified Text-to-Text Transformer Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for HSD in Vietnamese focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia. |
| Approach: | They propose a T5-based HSD model pre-trained on a domain-specific dataset . their results highlight the significance of label distribution in pre-training data on model efficacy. |
| Outcome: | The proposed model can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. |
Bias in News Summarization: Measures, Pitfalls and Corpora (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained large language models can reproduce harmful social biases in constrained settings, such as summarization. |
| Approach: | They propose a method to generate input documents with carefully controlled demographic attributes and then apply it to a controlled setting. |
| Outcome: | The proposed method allows to generate input documents with carefully controlled demographic attributes while working with real-world input documents. |
When to Trust LLMs: Aligning Confidence with Response Quality (2024.findings-acl)
Copied to clipboard
Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, Bolin Ding
| Challenge: | Existing methods express reliability by confidence level, but lack objective guidance . Existing approaches express reliability but lack guidance on when to trust LLMs . |
| Approach: | They propose a reward-based approach to align confidence with quality to ensure reliability . they propose 'conqORD' to help model to verbalize greater confidence for higher quality responses . |
| Outcome: | Experiments show that CONQORD significantly improves confidence and response accuracy . the proposed approach can be used to determine reliability of large language models . |
Zero-shot Cross-lingual Alignment for Embedding Initialization (2024.findings-acl)
Copied to clipboard
| Challenge: | CrossInit initializes embeddings into similar geometrical structures across languages in unsupervised manner. |
| Approach: | They propose a method that initializes embeddings into similar geometrical structures across languages in an unsupervised manner. |
| Outcome: | The proposed method demostrates similar patterns in low-resource and dissimilar languages. |
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases. |
| Approach: | They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations. |
| Outcome: | The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores. |
It takes two to borrow: a donor and a recipient. Who’s who? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for identifying the direction of borrowing are limited. |
| Approach: | They propose strong benchmarks for automatic borrowing direction detection by using a borrowings dataset from the recent RoBoCoP database for five Romance languages. |
| Outcome: | The proposed model improves the accuracy of the proposed task and proposes additional directions for future work. |
Advancing Post-OCR Correction: A Comparative Study of Synthetic Data (2024.findings-acl)
Copied to clipboard
| Challenge: | Optical Character Recognition (OCR) is a useful tool for preserving cultural heritage and supporting downstream tasks. |
| Approach: | They propose an algorithm that leverages computer vision feature detection algorithms to calculate glyph similarity for constructing post-OCR synthetic data. |
| Outcome: | The proposed method significantly reduces CERs in low-resource languages without manually annotating data. |
GeoAgent: To Empower LLMs using Geospatial Tools for Address Standardization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to address address standardization are lacking in the current field. |
| Approach: | They propose a framework that incorporates spatial knowledge into address texts and achieves efficient address standardization. |
| Outcome: | The proposed framework incorporates spatial knowledge into address texts and achieves efficient address standardization. |
HQP: A Human-Annotated Dataset for Detecting Online Propaganda (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for detecting online propaganda use weak labels that can be noisy and incorrect. |
| Approach: | They propose a dataset for detecting online propaganda with high-quality labels . they show that state-of-the-art language models fail in detecting propaganda when trained with weak labels compared to prompt-based learning . |
| Outcome: | The proposed dataset is the first large-scale dataset for detecting online propaganda that was created through human annotation. |
Teaching Language Models to Self-Improve by Learning from Language Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) generate content that can be untruthful or harmful. |
| Approach: | They propose a method that leverages model feedback for alignment . they use a base language model to generate initial responses, critiqued and refined . |
| Outcome: | The proposed method outperforms strong baselines across diverse tasks and model sizes. |
Exploring Spatial Schema Intuitions in Large Language and Vision Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in varied NLP tasks, but lack a direct connection between sensory perception and physical action. |
| Approach: | They examine whether large language models capture implicit human intuitions about building blocks of language . they employ spatial cognitive foundations developed through early sensorimotor experiences . |
| Outcome: | The proposed model captures implicit human intuitions about building blocks of language without a tangible connection to embodied experiences. |
Efficient Detection of LLM-generated Texts with a Bayesian Surrogate Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can be used to produce text that is coherent, well-written, and persuasive . some individuals have misused LLMs for nefarious purposes, such as creating fake news articles or engaging in cheating . |
| Approach: | They propose to incorporate a Bayesian surrogate model to improve query efficiency . they propose to select typical samples based on Bayes' uncertainty and interpolate scores . |
| Outcome: | The proposed method significantly outperforms existing approaches under a low query budget. |
Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Reddit (2024.findings-acl)
Copied to clipboard
| Challenge: | Our study aims to develop a multi-level, multi-label classification model to analyze online user-generated texts about substance use experiences. |
| Approach: | They propose a taxonomy to assess the nature of posts, including intended connections (Inquisition or Disclosure), subjects (e.g., Recovery, Dependency), and specific objectives (eg. relapse, quality, safety). |
| Outcome: | The proposed model outperforms other models on annotated data and shows that topics such as Safety, Combination of Substances, and Mental Health see more disclosure, while discussions about physiological Effects focus on harm reduction. |
Unveiling the Art of Heading Design: A Harmonious Blend of Summarization, Neology, and Algorithm (2024.findings-acl)
Copied to clipboard
| Challenge: | Creating an appealing heading is crucial for attracting readers and marketing work or products. |
| Approach: | They propose a benchmark to measure the quality of heading generation using summarization, neology, and algorithm metrics. |
| Outcome: | The proposed benchmark compared 6,653 abstracts with corresponding descriptions and acronyms and found that it excels across summarization, neology, and algorithm aspects. |
Understanding Fine-grained Distortions in Reports of Scientific Findings (2024.findings-acl)
Copied to clipboard
| Challenge: | a fine-grained understanding of how scientific findings are reported is crucial, says a new study . a recent study found that tweets distort scientific findings more often than news reports . |
| Approach: | They propose to annotate 1,600 scientific findings from academic papers paired with corresponding tweets . they also establish baselines for automatically detecting these characteristics . |
| Outcome: | The proposed method outperforms few-shot prompting in detecting distortions in unpaired data. |
MM-SOC: Benchmarking Multimodal Large Language Models in Social Media Platforms (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. |
| Approach: | They propose a benchmark to evaluate MLLMs' understanding of multimodal social media content and a large-scale YouTube tagging dataset to evaluate their performance. |
| Outcome: | The proposed model performs better in a zero-shot setting, suggesting potential improvements. |
Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized zero-shot task performance, mitigating the need for task-specific annotations while enhancing task generalizability. |
| Approach: | They propose an approach that optimizes the zero-shot prompts for individual task instances following an innovative manner of "LLMs in the loop" their results show that PRomPTed outperforms naive zero- shot approaches and a strong baseline which refines the task output instead of the input prompt. |
| Outcome: | The proposed approach outperforms naive approaches and a strong baseline which refines the task output instead of the input prompt. |
Benchmarking Retrieval-Augmented Generation for Medicine (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have state-of-the-art performance on a wide range of medical question answering tasks, but they still face challenges with hallucinations and outdated knowledge. |
| Approach: | They propose a benchmark to evaluate medical RAG systems using large-scale experiments with over 1.8 trillion prompt tokens. |
| Outcome: | The proposed benchmark improves accuracy of six different LLMs by up to 18% over chain-of-thought prompting. |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack the ability to deal with temporal knowledge. |
| Approach: | They propose a temporal question-answering dataset Complex-TR that focuses on multi-answered and multi-hop temporal reasoning and propose augmentation strategy to improve LLMs' performance. |
| Outcome: | The proposed dataset improves LLMs’ performance on temporal QA benchmarks by significant margins. |
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models demonstrate remarkable ability for learning to solve new tasks from a few examples. |
| Approach: | They propose to use templates to aggregate model predictions across multiple templates to improve model performance. |
| Outcome: | The proposed model ensembles boost model predictions while being robust to the choice of random set of templates. |
Knowledge Graph-Enhanced Large Language Models via Path Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown unprecedented performance in various real-world applications, but they are known to generate factually inaccurate outputs. |
| Approach: | They propose a framework to integrate external knowledge extracted from Knowledge Graphs (KGs) they propose to generate scores for knowledge paths with input texts via latent semantic matching. |
| Outcome: | Experiments on real-world datasets validate the effectiveness of a framework to extract knowledge from Knowledge Graphs (KGs) incorporating external knowledge has become a promising strategy to improve the factual accuracy of LLM-generated outputs. |
OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for detecting hallucinations and omissions in Machine Translation systems focus on analyzing the model’s internal states or relying on external tools. |
| Approach: | They propose an Optimal Transport-based word aligner specifically designed to enhance the detection of hallucinations and omissions in Machine Translation systems. |
| Outcome: | The proposed method is competitive with state-of-the-art methods across 18 language pairs on the HalOmi benchmark and shows promising features. |
ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | TKGF is a technique that requires experience during testing and relying on a single short-term history. |
| Approach: | They propose a framework that integrates dynamic causal rule mining and dual history augmented generation to enhance event prediction. |
| Outcome: | The proposed framework shows significant performance improvements across diverse datasets and significantly improves Hit@k. |
Speech-based Slot Filling using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown an unprecedented ability across various language tasks. |
| Approach: | They propose to use prompts and LoRA fine-tuning to improve slot filling robustness . they propose a linearised knowledge injection scheme to integrate dynamic external knowledge into LLMs. |
| Outcome: | The proposed model improves slot filling with noisy ASR transcriptions with 6.7% and 17.6% absolute SLU-F1 improvements compared to a fully fine-tuned Flan-T5-XL model. |
Too Big to Fail: Larger Language Models are Disproportionately Resilient to Induction of Dementia-Related Linguistic Anomalies (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that the attention mechanism in transformer-based NLMs may present an analogue to the notions of cognitive and brain reserve. |
| Approach: | They propose a bidirectional ablation method that masks attention heads to display degradation of similar magnitude to masking in smaller models. |
| Outcome: | The proposed method exhibits properties attributed to the concepts of cognitive and brain reserve in human brain studies. |
HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in various natural language tasks in English, but their performance in low-resource languages like Hebrew remains unclear. |
| Approach: | They propose a benchmark dataset specifically designed for Hebrew abstractive text summarization that combines 10,000 article-summary pairs from Hebrew news websites. |
| Outcome: | The proposed dataset shows that it presents distinct difficulties even for state-of-the-art LLMs. |
TRAM: Benchmarking Temporal Reasoning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in natural language processing, temporal reasoning is still a challenge . despite advancements in NLP, current language models have yet to reach human level in this domain . |
| Approach: | They propose a temporal reasoning benchmark that measures time-related temporal aspects of events . they evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios . |
| Outcome: | The proposed model outperforms human models in a few-shot and zero-shot scenarios . the best-performing model lags significantly behind human models, the authors say . |
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Known-unknown questions are characterized by high uncertainty due to the absence of definitive answers. |
| Approach: | They introduce a dataset with known-unknown questions and establish a categorization framework to clarify the origins of uncertainty in such queries. |
| Outcome: | The proposed model improved in distinguishing between known and unknown queries within open-ended question-answering scenarios. |
Exploring Defeasibility in Causal Reasoning (2024.findings-acl)
Copied to clipboard
Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, Boi Faltings
| Challenge: | Existing studies ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defensible settings. |
| Approach: | They propose a metric that measures causal strength based on token-level causal relationships. |
| Outcome: | The proposed metric improves on existing metrics by 69.7% . supporters and defeaters are more effective than opponents, the authors show . |
Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in large language models, task-specific data is not available for many use cases . a new method to improve automated dataset generation uses publicly available datasets . |
| Approach: | They propose a method to make better use of existing datasets to improve automatic dataset generation. |
| Outcome: | The proposed method outperforms existing methods on language-based tasks . it significantly increases diversity and difficulty of generated data on many tasks compared to other methods . |
Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning is a popular paradigm in natural language processing, but its performance can be significantly influenced by the order of in-concept demonstration examples. |
| Approach: | They propose an unsupervised fine-tuning method to reduce the sensitivity of causal language models to the order of in-context demonstration examples. |
| Outcome: | The proposed method reduces the sensitivity of CausalLMs to the order of in-context examples and exhibits robust generalizability. |
Perspective Taking through Generating Responses to Conflict Situations (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models struggle to understand and explain the beliefs of others, despite improving performance on a wide variety of tasks. |
| Approach: | They propose to modify the social-chem-101 corpus to allow for perspective-taking, the process of conceptualizing the point of view of another person. |
| Outcome: | The proposed models outperform the recent models conditioned on self-disclosures with high similarity to the conflict situation. |
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
The Power of Summary-Source Alignments (2024.findings-acl)
Copied to clipboard
Ori Ernst, Ori Shapira, Aviv Slobodkin, Sharon Adar, Mohit Bansal, Jacob Goldberger, Ran Levy, Ido Dagan
| Challenge: | Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection, followed by text generation. |
| Approach: | They propose to extend the summary-source alignment framework by applying it at the more fine-grained proposition span level and annotating alignment manually in a multi-document setup. |
| Outcome: | The proposed framework can yield several datasets for at least six different tasks. |
An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
Gantavya Bhatt, Yifang Chen, Arnav Das, Jifan Zhang, Sang Truong, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Du, Kevin Jamieson, Jordan Ash, Robert Nowak
| Challenge: | Supervised finetuning (SFT) on instruction datasets has shown immense potential in improving the zero-shot generalization capabilities observed in large language models (LLMs). |
| Approach: | They propose to use experimental design to minimize the computational cost of active learning by identifying useful subsets of samples to annotate from an unlabeled pool. |
| Outcome: | The proposed methods save 50% of the annotation cost compared to random sampling on generative tasks. |
Learning Multimodal Contrast with Cross-modal Memory and Reinforced Contrast Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a memory module, we learn multimodal contrast using encoding-decoding paradigm . multimodal information are used in many applications, including news feeding, social media, etc. |
| Approach: | They propose an LLM-based approach for learning multimodal contrast following the encoding-decoding paradigm . they use a memory module with reinforced contrast recognition to enhance learning . |
| Outcome: | The proposed approach outperforms baseline and state-of-the-art studies on four English and Chinese benchmark datasets. |
Text Simplification via Adaptive Teaching (2024.findings-acl)
Copied to clipboard
| Challenge: | Text simplification is the process of rewriting a text using simpler vocabulary and grammatical structure in order to make it more accessible and understandable for a larger audience. |
| Approach: | They propose a model for text simplification based on adaptive teaching using a teacher network and a text generation network. |
| Outcome: | The proposed model outperforms the current state-of-the-art model on the Wiki-Doc and D-Wikipedia datasets and performs well on human evaluations in terms of text simplicity, correctness, and fluency. |
A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts (2024.findings-acl)
Copied to clipboard
| Challenge: | a multi-level, multi-label text classification dataset is used to classify over 3000 documents . authors use a classical bag-of-words (BoW) naive Bayes model and three modern LLMs . |
| Approach: | They propose to apply large language models to a multi-level, multi-label text classification dataset . the dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian . |
| Outcome: | The proposed dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. |
It is Simple Sometimes: A Study On Improving Aspect-Based Sentiment Analysis Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on developing ad hoc models of varying complexity for ABSA subtasks. |
| Approach: | They propose an instruction-based model with task descriptions followed by in-context examples on ABSA subtasks. |
| Outcome: | The proposed method outperforms state-of-the-art methods on most domains and achieves competitive results on biomedical domain datasets. |
Whose Emotions and Moral Sentiments do Language Models Reflect? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has focused on positional alignment, which measures how closely the models mimic the opinions and stances of different social groups. |
| Approach: | They define the problem of affective alignment, which measures how LMs’ emotional and moral tone represents those of different groups. |
| Outcome: | The results show that the models represent the perspectives of some social groups better than others, suggesting a systemic bias within LMs. |
LLM can Achieve Self-Regulation via Hyperparameter Aware Generation (2024.findings-acl)
Copied to clipboard
Siyin Wang, Shimin Li, Tianxiang Sun, Jinlan Fu, Qinyuan Cheng, Jiasheng Ye, Junjie Ye, Xipeng Qiu, Xuanjing Huang
| Challenge: | Existing decoding strategies and hyperparameters may not be optimal for each sample. |
| Approach: | They propose a model that auto-regulates decoding strategies and hyperparameters . this approach eliminates the need for extensive manual tuning, they argue . |
| Outcome: | The proposed model eliminates the need for extensive manual tuning, offering a more autonomous, self-regulate model behavior. |
Forward-Backward Reasoning in Large Language Models for Mathematical Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on six standard mathematical data sets and three LLMs show that FOBAR achieves state-of-the-art performance. |
| Approach: | They propose to combine forward and backward reasoning to verify candidate answers . they propose to use a template to mask a number and ask the LLM to answer a backward question . |
| Outcome: | Experiments on mathematical data show that proposed backward reasoning outperforms Self-Consistency. |
Towards Uncertainty-Aware Language Agent (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Language Agents neglect the notion of uncertainty during interactions with external worlds. |
| Approach: | They propose a framework that orchestrates the interaction between the agent and the external world using uncertainty quantification. |
| Outcome: | The proposed framework improves performance on 3 representative tasks and lowers reliance on external world. |
Detection and Positive Reconstruction of Cognitive Distortion Sentences: Mandarin Dataset and Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have investigated the application of NLP models in English for each stage of this process. |
| Approach: | They propose a Positive Reconstruction Framework based on broaden-and-build theory to address and reframe negative thoughts through a positive reinterpretation. |
| Outcome: | The proposed framework is based on broaden-and-build theory and can detect cognitive distortions and suggest a positive reframe in Mandarin. |
PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models are pre-trained on unstructured data, which leads to poor performance when dealing with structured data. |
| Approach: | They propose a framework to train large language models to act as verifier modules and to apply iterative corrections offline. |
| Outcome: | The proposed framework improves graph-based generative capability of large language models by iterating corrective instructions on three graph-derived datasets. |
Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Temporal knowledge graph question answering (TKGQA) is one of the most challenging QA tasks due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. |
| Approach: | They propose a generative temporal knowledge graph question answering framework which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. |
| Outcome: | The proposed framework exploits LLM’s intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. |
VISREAS: Complex Visual Reasoning with Unanswerable Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Logic2Vision is a visual question-answering dataset that validates question authenticity with the corresponding image and then reasoning over it. |
| Approach: | They propose a compositional visual question-answering dataset, VisReas, that consists of answerable and unanswerable visual queries . they use visual genome scene graphs to generate the query and the reasoning steps to generate it. |
| Outcome: | The proposed model outperforms generative models and the existing classification models and outperformed existing models. |
A Unified Generative Framework for Bilingual Euphemism Detection and Identification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing euphemism datasets are only domain-specific or language-specific. |
| Approach: | They propose a unified model to jointly conduct bilingual euphemism detection and identification tasks. |
| Outcome: | The proposed model is effective and provides a new reference standard for euphemism detection and identification. |
StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing (2024.findings-acl)
Copied to clipboard
Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton Hengel, Ming-Hsuan Yang, Chenggang Yan, Qingming Huang
| Challenge: | Existing methods for movie dubbing break phonemes in scripts, resulting in incomplete phoneme pronunciation and poor identity stability. |
| Approach: | They propose a method that switches dubbing learning from frame level to phoneme level . it uses a multimodal style adaptor to learn pronunciation style from audio . |
| Outcome: | The proposed method improves on two benchmarks, V2C and Grid, and is available on github. |
ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks. |
| Approach: | They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy. |
| Outcome: | The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks. |
Reasoning Like a Doctor: Improving Medical Dialogue Systems via Diagnostic Reasoning Process Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical dialogue systems have attracted significant attention for their potential to act as medical assistants. |
| Approach: | They propose a framework that emulates clinicians' diagnostic reasoning processes and aligns with clinician preferences through thought process modeling. |
| Outcome: | The proposed framework generates appropriate responses that relies on abductive and deductive diagnostic reasoning analyses and aligns with clinician preferences through thought process modeling. |
ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models (2024.findings-acl)
Copied to clipboard
Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, ZhiqiBai ZhiqiBai, Haibin Chen, Tiezheng Ge, Wanli Ouyang, Wenbo Su, Bo Zheng
| Challenge: | ConceptMath evaluates concept-wise mathematical reasoning of Large Language Models (LLMs) Existing benchmarks that evaluate general mathematical reasoning with an average accuracy fail to probe the fine-grained failure modes of mathematical reasoning on specific datasets. |
| Approach: | They introduce a bilingual, fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models. |
| Outcome: | The proposed benchmarks evaluate concept-wise mathematical reasoning of Large Language Models with concept-based accuracies. |
REInstruct: Building Instruction Data from Unlabeled Corpus (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for annotating instruction data are expensive and difficult to scale. |
| Approach: | They propose a method to automatically build instruction data from an unlabeled corpus without heavy reliance on proprietary LLMs and human annotation. |
| Outcome: | The proposed method outperforms existing methods on AlpacaEval leaderboard and other open-source methods. |
Learning to Maximize Mutual Information for Chain-of-Thought Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation is a technique of transferring knowledge from large, complex models to smaller ones. |
| Approach: | They propose a method utilizing chain-of-thought distillation to transfer knowledge from large, complex models to smaller ones by maximizing mutual information of the representation features of the two tasks. |
| Outcome: | The proposed method outperforms the state-of-the-art knowledge distillation method on four datasets. |
PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) is an effective method for adapting pre-trained language models to various tasks efficiently. |
| Approach: | They propose a parameter-efficient fine-tuning framework that captures transferable knowledge as a weighted combination of adapters trained on source tasks. |
| Outcome: | The proposed method yields stable improvements over full fine-tuning and knowledge transferring methods on a broad range of tasks over 17 datasets. |
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024.findings-acl)
Copied to clipboard
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Recent advances in large language models have showcased significant improvements in mathematics, but traditional benchmarks like GSM8k offer a unidimensional perspective. |
| Approach: | MathBench is a benchmark that rigorously assesses the mathematical capabilities of large language models. |
| Outcome: | MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. |
Identifying Semantic Induction Heads to Understand In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance, but lack of transparency in their inference logic raises concerns about their trustworthiness. |
| Approach: | They conduct a detailed analysis of the operations of attention heads to understand their in-context learning of LLMs. |
| Outcome: | The proposed analysis of attention heads reveals that they increase the output logits of object tokens and recall objects . the proposed model is a novel approach to understand the in-context learning of large language models. |
Chinese Spelling Corrector Is Just a Language Learner (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that self-supervised learning can improve Chinese spelling correction by removing errors from training data. |
| Approach: | They propose a method that decodes Chinese spelling correction models using noise . they say it outperforms current methods that rely on annotated errors . |
| Outcome: | The proposed method outperforms the confusion set in specific domains because there are no errors in the training data. |
Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Object hallucination has been an Achilles’ heel which hinders the broader applications of large vision-language models (LVLMs). |
| Approach: | They propose a logical closed loop-based framework for Object Hallucination Detection and Mitigation that uses logical consistency probing to raise questions with logical correlations to determine hallucinations. |
| Outcome: | The proposed method can be applied to all existing LVLMs and is effective and general. |
RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for assessing retrieval of relevant information are understudied . previous studies have neglected to evaluate ARAG methods . |
| Approach: | They propose a benchmark to evaluate existing ARAG methods that use threshold tuning to adjust retrieval for queries instead of indiscriminate retrieval. |
| Outcome: | The proposed method can be used to evaluate existing ARAG methods without calibration or training. |
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | ***LLaST*** is a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Approach: | They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Outcome: | The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. |
Plan, Generate and Complicate: Improving Low-resource Dialogue State Tracking via Easy-to-Difficult Zero-shot Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing data augmentation methods rely on pre-defined user goals and neglect the importance of data complexity in this task. |
| Approach: | They propose an Easy-to-Difficult Zero-shot Data Augmentation framework for low-resource dialogue state tracking that utilizes large language models to automatically catch the relationships of different domains and then generate dialogue data. |
| Outcome: | The proposed framework outperforms existing methods on MultiWOZ and other similar datasets. |
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a reward model (RM) to improve the effectiveness of large language models, there are two challenges in training. |
| Approach: | They propose a reward model (RM) that is a proxy of human preferences and assigns scores to the outputs of the large language model (LLM) a human annotation consistency rate of 60% to 75% is causing training data to contain a lot of noise. |
| Outcome: | The proposed model outperforms state-of-the-art ensemble methods and mitigates the overoptimization problem. |
LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language adaptation methods overlook the benefits of cross-lingual supervision. |
| Approach: | They propose a language adaptation tuning method that augments target corpus with English entity names and trains the model using left-to-right language modeling. |
| Outcome: | The proposed method shows significant performance gains across non-English languages. |
Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies decompose complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. |
| Approach: | They propose to use code comments as natural logic pivot between natural language and code language to boost the code generation ability of code LLMs. |
| Outcome: | The proposed method significantly improves the code pass rate on humanEval and MBPP, while the robustness of the logical comment decoding strategy is higher than the Chain-of-thoughts prompting. |
Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration (2024.findings-acl)
Copied to clipboard
Sunhao Dai, Weihao Liu, Yuqi Zhou, Liang Pang, Rongju Ruan, Gang Wang, Zhenhua Dong, Jun Xu, Ji-Rong Wen
| Challenge: | Large Language Models (LLMs) have led to an influx of AI-generated content on the internet, transforming corpus of Information Retrieval (IR) systems from human-written to a coexistence with LLM-generated contents. |
| Approach: | They propose a benchmark named Cocktail that compares IR models with LLMs to find relevant documents and passages from a corpus. |
| Outcome: | The proposed benchmark aims to evaluate IR models in the mixed-sourced data landscape of the LLM era. |
Continual Dialogue State Tracking via Reason-of-Select Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research on dialogue systems has focused on domain-specific offline systems lacking adaptation abilities. |
| Approach: | They propose a Reason-of-Select distillation method that enhances smaller models with a novel "meta-reasoning" capability. |
| Outcome: | Experiments show that the proposed method significantly improves the performance and generalization capabilities of existing models. |
Spotting AI’s Touch: Identifying LLM-Paraphrased Spans in Text (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work focuses on detecting (partially) AI-generated texts, but paraphrasing is commonly employed in various application scenarios for text refinement and diversity. |
| Approach: | They propose a framework for paraphrased text span detection that takes in the full text and assigns each sentence with a score indicating the paraphrasing degree. |
| Outcome: | The proposed framework can detect paraphrased text spans within a text . it takes in the full text and assigns each sentence with a score indicating the paraphrasing degree. |
SoFA: Shielded On-the-fly Alignment via Priority Rule Following (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing alignment methods fail to adapt to the diversity of preferences and regulatory standards. |
| Approach: | They propose a method for prioritizing rules over user instructions to minimize misalignments in Large Language Models. |
| Outcome: | The proposed approach minimizes misalignments and adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately. |
Do Zombies Understand? A Choose-Your-Own-Adventure Exploration of Machine Cognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in LLMs have sparked a debate on whether they understand text. |
| Approach: | They propose two working definitions for understanding which explicitly acknowledge the question of consciousness and draw connections with a rich literature in philosophy, psychology and neuroscience. |
| Outcome: | The proposed models achieve impressive results on various benchmarks, seeming to generalize to unseen tasks and domains. |
Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on how to model emotionality in stories has been limited to dictionary-based methods . |
| Approach: | They propose to introduce continuous valence and arousal labels for an existing dataset of children’s stories originally annotated with discrete emotion categories. |
| Outcome: | The proposed model achieves a Concordance Correlation Coefficient (CCC) of .8221 for valence and .7125 for arousal on the test set, demonstrating the efficacy of the proposed model. |
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024.findings-acl)
Copied to clipboard
Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, Ge Li
| Challenge: | Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. |
| Approach: | They propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter . they propose a low-rank modulation module to refine per-image features from frozen CLIP backbone . |
| Outcome: | Experiments on four TVR datasets show that the proposed method performs better than other methods. |
Benchmarking and Improving Long-Text Translation with Large Language Models (2024.findings-acl)
Copied to clipboard
Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek Wong, Shuming Shi, Zhaopeng Tu
| Challenge: | Recent studies have illuminated the promising capabilities of large language models (LLMs) in handling long texts. |
| Approach: | They construct a benchmark dataset specifically designed for the finetuning and evaluation of large language models (LLMs) they compare LLMs with MT models and find they exhibit shortcomings in long-text domains . |
| Outcome: | The proposed model performs better in long-text translation, and its performance diminishes as document size increases. |
Personalized Topic Selection Model for Topic-Grounded Dialogue (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing topic-grounded dialogue systems tend to predict user-uninteresting and contextually irrelevant topics due to noise within side information sources. |
| Approach: | They propose a personalized topic selection model for topic-grounded dialogue that selectively aggregates side information to generate engaging responses. |
| Outcome: | The proposed model outperforms state-of-the-art models on multiple evaluation metrics. |
Debiasing In-Context Learning by Instructing LLMs How to Follow Demonstrations (2024.findings-acl)
Copied to clipboard
Lvxue Li, Jiaqi Chen, Xinyu Lu, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | In-context learning (ICL) has gained considerable attention due to its data efficiency and task adaptability. |
| Approach: | They propose to de-biase demonstration bias in in-context learning by focusing on semantic ambiguity induced by demonstrations and reducing the semantic hazard. |
| Outcome: | The proposed methods significantly improve performance on six datasets. |
Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | Creating effective task-oriented dialog systems is challenging due to the scarcity of training data. |
| Approach: | They empirically evaluate eight DA methods that have shown promising results in task-oriented dialog systems and other NLP systems. |
| Outcome: | The proposed methods have been successful in other NLP systems but not in the ToDSs. |
MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions for sign language production are limited due to phonological differences and data scarcity. |
| Approach: | They propose a unified framework for continuous sign language production that generates sign predictions step by step from text or speech embeddings. |
| Outcome: | The proposed model achieves competitive performance on how2sign and PHOENIX14T datasets. |
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
Xueliang Zhao, Xinting Huang, Tingchen Fu, Qintong Li, Shansan Gong, Lemao Liu, Wei Bi, Lingpeng Kong
| Challenge: | Multimodal reasoning is a key capability for large vision-language models . however, the vanilla Chain-of-Thought method fails to address critical steps in multi-step reasoning tasks. |
| Approach: | They propose a bi-modal Behavioral Alignment method to augment multimodal reasoning . they use domain-specific language to integrate multimodal information into a precise alternative form . |
| Outcome: | The proposed method significantly improves GPT-4V(ision) on geometry problem solving, chess positional advantage prediction and molecular property prediction. |
PartialFormer: Modeling Part Instead of Whole for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing feed-forward neural networks have significant computational and parametric overhead. |
| Approach: | They propose a parameter-efficient Transformer architecture that utilizes multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. |
| Outcome: | The proposed architecture reduces computational and parameter overhead while maintaining essential hidden dimensions. |
Self-Consistent Reasoning-based Aspect-Sentiment Quad Prediction with Extract-Then-Assign Strategy (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have developed powerful generative methods for aspect sentiment quad prediction (ASQP) but they still suffer from imprecise predictions and limited interpretability due to data scarcity and inadequate modeling of the quadruplet composition process. |
| Approach: | They propose a self-consistent reasoning-based aspect sentiment quadruple prediction framework which generates reasonings and corresponding quadruples in sequence. |
| Outcome: | The proposed model significantly improves its ability to handle complex reasoning tasks and correctly predict quadruplets through consistency voting, resulting in enhanced interpretability and accuracy in aspect sentiment quadr uplp prediction. |
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Prompt with Actor-Critic Editing (PACE) for LLMs improves performance of different human-written prompts, resulting in significant performance discrepancies. |
| Approach: | They propose to use LLMs as actors and critics to enable automatic prompt editing by taking feedback from both actors performing prompt and criticizing response into account. |
| Outcome: | The proposed model improves the performance of human-written prompts by 98% and compares to high-quality human-writing prompts. |
Penetrative AI: Making LLMs Comprehend the Physical World (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of tasks. |
| Approach: | They explore how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that they call "Penetrative AI". |
| Outcome: | The proposed approach extends LLMs' capabilities to interact with and reason about the physical world through IoT sensors and actuators. |
The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis (2024.findings-acl)
Copied to clipboard
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba Alabi, Xiaoyu Shen, Dietrich Klakow, Marius Mosbach
| Challenge: | In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations without updating the model parameters. |
| Approach: | They conduct multidimensional analysis of multilingual in-context learning using 5 models from different model families and 9 datasets covering classification and generation tasks. |
| Outcome: | The results show that demonstrations vary significantly across models, tasks, and languages. |
Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spell Checking (CSC) is a widely used technology for speech to text and optical character recognition. |
| Approach: | They propose to use Chinese rich semantic information to introduce large language models as the foundation model. |
| Outcome: | The proposed framework performs better on few-shot CSC task than existing methods. |
An Empirical Study of In-context Learning in LLMs for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on optimizing translation quality, with limited attention to understanding specific aspects of ICL that influence the said quality. |
| Approach: | They conduct the first of its kind, exhaustive study of in-context learning for machine translation (MT) they establish that ICL is primarily example-driven and not instruction-driven . |
| Outcome: | The proposed model is based on examples and not instruction-driven learning. |
“My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models (2024.findings-acl)
Copied to clipboard
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, Barbara Plank
| Challenge: | Multiple choice questions are one of the most popular evaluation formats for understanding the capabilities of autoregressive large language models (LLMs). |
| Approach: | They evaluated how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. |
| Outcome: | The proposed evaluation methods are misaligned on all dimensions, reaching mismatch rates over 60%. |
ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to integrate large language models and knowledge graphs with LLMs often ignore the rich cognitive potential inherent in KGs. |
| Approach: | They propose an observation-driven agent framework that integrates KG reasoning abilities via global observation and integrates it into the action and reflection modules. |
| Outcome: | The proposed framework improves on several datasets and achieves 12.87% and 8.9% accuracy improvements. |
A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated capabilities for generating content that could be deemed harmful. |
| Approach: | They conduct a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. |
| Outcome: | The proposed techniques underperform existing white-box attacks and include special tokens significantly affects the likelihood of successful attacks. |
A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning (2024.findings-acl)
Copied to clipboard
Panagiotis Kaliosis, John Pavlopoulos, Foivos Charalampakos, Georgios Moschovis, Ion Androutsopoulos
| Challenge: | Diagnostic Captioning (DC) systems receive one or more medical images of a patient, such as X-Rays or Magnetic Resonance Images (MRIs). |
| Approach: | They propose a data-driven guided decoding method that incorporates medical information into the beam search of the diagnostic text generation process. |
| Outcome: | The proposed method improves on two medical datasets and can be used in few- and zero-shot learning scenarios. |
Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Aligned Large Language Models exhibit remarkable versatility, capable of handling diverse real-world tasks. |
| Approach: | They propose a coarse to fine framework to fine-tune aligned Large Language Models to achieve a balance between speciality and versatility. |
| Outcome: | The proposed framework outperforms baseline methods across diverse tasks and model scales. |
A Two-Agent Game for Zero-shot Relation Triplet Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for relation triplet extraction rely on labeled data and are limited in their applicability. |
| Approach: | They propose a two-agent game approach to deliberate and debate unseen relations by two agents, a generator and an extractor. |
| Outcome: | The proposed method outperforms baseline methods by 6%-16% in F1 scores. |
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training . |
| Approach: | They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters. |
| Outcome: | The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT. |
Building Bridges: A Dataset for Evaluating Gender-Fair Machine Translation into German (2024.findings-acl)
Copied to clipboard
| Challenge: | a societal movement towards using gender-fair language exists, but gender-free German is barely supported in machine translation. |
| Approach: | They propose to use a community-created gender-fair language dictionary to study gender-neutral German . they also use encyclopedic text and parliamentary speeches to translate the words in isolation . |
| Outcome: | The proposed study shows that most systems produce mainly masculine forms and rarely gender-neutral variants. |
Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can improve summary quality by mirroring a human-like iterative process of critique and refinement starting from the initial draft. |
| Approach: | They propose to use Prompt Chaining and Stepwise Prompting to perform iterative refinement . they propose to combine the two methods to produce a more favorable outcome . |
| Outcome: | The proposed methods can improve summary quality by mirroring a human-like iterative process . the results show that the prompt chaining method produces a more favorable outcome . |
Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs. |
| Approach: | They propose a knowledge retrieval method that leverages external sources to enhance visual entity information and a prioritization scheme that handles noisy retrieval results. |
| Outcome: | The proposed method shows improvements of 3.0% to 6.5% across all evaluation metrics compared to baselines. |
A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Word-in-Context (WiC) datasets are used to detect temporal semantic changes of words. |
| Approach: | They propose a supervised two-staged SCD method that uses existing Word-in-Context (WiC) datasets to predict temporal semantic changes of words. |
| Outcome: | The proposed method achieves strong performance in multiple languages and significant improvements on WiC benchmarks. |
What Have We Achieved on Non-autoregressive Translation? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that non-autoregressive (NAT) methods underperform autoregressive methods (AT) however, their evaluation using BLEU has been shown to weakly correlate with human annotations. |
| Approach: | They propose to evaluate four representative NAT methods using BLEU to narrow the performance gap between autoregressive and autoregressive translations. |
| Outcome: | The proposed methods underperform NAT and autoregressive methods under more reliable evaluation metrics. |
From Zero to Hero: Cold-Start Anomaly Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing anomaly detection methods require previous observations to be effective . contaminated observations are often not observed, making them ineffective . |
| Approach: | They propose a method that adapts a zero-shot anomaly detector to contaminated observations . they propose an evaluation suite consisting of evaluation protocols and metrics . |
| Outcome: | The proposed method adapts the zero-shot anomaly detector to contaminated observations. |
Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for narrative understanding fail to represent complexity and uncertainty of relationships in real-life social scenarios. |
| Approach: | They propose a benchmark for extracting and analysing intricate character relation graphs from detective narratives using large-scale large-language models. |
| Outcome: | The proposed dataset extracts and analyses character relation graphs from detective narratives using advanced Large Language Models like GPT-3.5, GPT-4, and Llama2 . |
DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context knowledge editing has shown respectable abilities on knowledge editing in terms of generalization and specificity. |
| Approach: | They propose a novel extension of in-context knowledge editing (IKE) that allows for massive edits to be injected into large language models. |
| Outcome: | The proposed method shows state-of-the-art perfomrances and comparable performance with MIKE. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
Hierarchy-aware Biased Bound Margin Loss Function for Hierarchical Text Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Hierarchical text classification (HTC) is a challenging problem with two key issues: utilizing structural information and mitigating label imbalance. |
| Approach: | They propose a hierarchy-aware biased bound margin loss function for unit-based HTC models that integrates learnable bounds, biases, and a margin to address static thresholding and mitigate label imbalance adaptively. |
| Outcome: | Experimental results show that the proposed model outperforms the global approach and is more robust to label imbalances. |
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts (2024.findings-acl)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation can be used to process long contexts in Open-Domain Question-Answering tasks. |
| Approach: | They propose a method to cover longer contexts in Open-Domain Question-Answering tasks by using a small encoder language model and cross-attention with origin inputs. |
| Outcome: | The proposed method can cover longer contexts while keeping the computing requirements close to the baseline. |
CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Contaminated or adulterated food poses a substantial risk to human health. |
| Approach: | They present a dataset of 7,546 text messages describing public food recalls. |
| Outcome: | The proposed model outperforms RoBERTa and XLM-R on classes with low support while reducing energy consumption. |
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing quantization methods are compromising performance of large language models (LLMs) despite their high computational intensity, LLMs are still demanding intensive computation. |
| Approach: | They propose to generate the KV cache of pivot tokens losslessly from the full-precision model. |
| Outcome: | The proposed method generates the KV cache of pivot tokens losslessly from the full-precision model with no extra inference overhead. |
Learning Adverbs with Spectral Mixture Kernels (2024.findings-acl)
Copied to clipboard
| Challenge: | In order for robots to collaborate with humans, it is important to share and understand their experiences through language. |
| Approach: | They propose a hierarchical Dirichlet Process-Spectral Mixture Latent Dirichlets Allocation model which learns the relationship between human motions and adverbs by capturing frequency kernels that represent motion characteristics and shared topics of a given aadverts. |
| Outcome: | The proposed model outperforms representative neural network models in terms of perplexity score and predicts more appropriate adverbs. |
E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, Min Yang
| Challenge: | despite the rapid development of Large Language Models, there is no dedicated benchmark for evaluating LLMs in Chinese K-12 education. |
| Approach: | They propose to develop a benchmark specifically tailored for Chinese K-12 education. |
| Outcome: | EVAL is the first evaluation benchmark specifically tailored for Chinese K-12 education. |
ChartAssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts are an effective tool for understanding data patterns, but their combination of graphical elements and textual components poses challenges for general-purpose multimodal models. |
| Approach: | They propose a chart-based vision-language model for universal chart comprehension and reasoning that leverages a dataset of chart-related tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art charts with zero-shot setting on various chart tasks. |
Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can teach small language models to solve complex reasoning tasks by Chain-of-thought Distillation (CoTD) e.g., mathematical question answering. |
| Approach: | They propose a method that distills two student models to solve a multi-hop question . they use chain-of-thought distillation to generate step-by-step reasoning paths . |
| Outcome: | The proposed method surpasses existing methods on knowledge-intensive multi-hop questions. |
ALaRM: Align Language Models via Hierarchical Rewards Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Current alignment approaches struggle with inconsistency and sparsity of human supervision signals. |
| Approach: | They propose a framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF) it integrates holistic rewards with aspect-specific rewards to enhance alignment of large language models with human preferences. |
| Outcome: | The proposed framework improves the alignment of large language models with human preferences by integrating holistic rewards with aspect-specific rewards. |
LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing prompting methods oversimplify time-series forecasting (TSF) time-Series data are ubiquitous across various domains, including public health, finance and energy. |
| Approach: | They propose a method for prompting off-the-shelf Large Language Models (LLMs) they decompose TSF into short-term and long-term forecasting sub-tasks, tailoring prompts to each . |
| Outcome: | The proposed approach decomposes TSF into short-term and long-term forecasting sub-tasks, tailoring prompts to each. |
Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | a new text classification framework for large language models addresses the problem of boundary ambiguity and inherent biases in LLMs. |
| Approach: | They propose a two-stage classification framework for large language models to mitigate bottlenecks . their approach uses pairwise comparisons to efficiently narrow down options . |
| Outcome: | The proposed framework reduces the number of options and improves on four datasets. |
UOR: Universal Backdoor Attacks on Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to attack pre-trained language models rely on manual selection of triggers and backdoor representations. |
| Approach: | They propose a backdoor attack method that turns manual selection into automatic optimization . they propose to use poisoned contrastive learning to learn more uniform backdoor representations . |
| Outcome: | The proposed method achieves better attack performance on text classification tasks compared to manual methods. |
Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences (2024.findings-acl)
Copied to clipboard
| Challenge: | incorporating cognitive capacities increases predictive power of surprisal and entropy measures on reading data, whereas high performance in the psychometric tests is associated with lower sensitivity to predictability effects. |
| Approach: | They examine the predictive power (PP) of surprisal and entropy estimated from generative language models (LMs) on reading data from individuals who also completed a wide range of psychometric tests. |
| Outcome: | The LMs' predictive power is based on cognitive capacities and high performance in psychometric tests is associated with lower sensitivity to predictability effects. |
The State of Relation Extraction Data Quality: Is Bigger Always Better? (2024.findings-acl)
Copied to clipboard
| Challenge: | Relation extraction (RE) methods extract tuples of relationships from text . many datasets with frequent label errors have been used . |
| Approach: | They review recent surveys and a sample of recent RE methods papers . they find that real-time evaluations of RE methods are possible . |
| Outcome: | a sample of 38 datasets currently being used shows that many have frequent label errors . a small number of relations in specific domains can more realistically evaluate methods . |
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science. |
| Approach: | They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks. |
| Outcome: | The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times. |
LLMCrit: Teaching Large Language Models to Use Criteria (2024.findings-acl)
Copied to clipboard
| Challenge: | Current research on using criteria to provide feedback on tasks is limited . a general framework that can be used to teach large language models to use criteria is lacking . |
| Approach: | They propose a framework that enables large language models to use criteria for feedback . criteria are extracted from guidelines and construct in-context demonstrations for each criterion . |
| Outcome: | The proposed framework can be used to provide natural language feedback on tasks. |
Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often unbalanced towards English because of the imbalance in the distribution of pre-training data. |
| Approach: | They propose to empower Instruction-tuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. |
| Outcome: | The proposed model outperforms the existing model on multilingual question answering benchmarks and adapted versions of MMLU and BBH on six different languages. |
Ranking Entities along Conceptual Space Dimensions with LLMs: An Analysis of Fine-Tuning Strategies (2024.findings-acl)
Copied to clipboard
| Challenge: | Conceptual spaces represent entities in terms of their primitive semantic features. |
| Approach: | They argue that conceptual spaces should be used alongside knowledge graphs in many settings to model entities in terms of their primitive semantic features. |
| Outcome: | The proposed model can rank entities according to a given conceptual space dimension but ground truth rankings for conceptual space dimensions are rare. |
Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for non-parametric domain adaptation lack kNN retrieval at each timestep, leading to substantial time overhead. |
| Approach: | They propose a kNN-MT-based model that uses a domain-specific translation knowledge store to interpolate the prediction distribution of the model. |
| Outcome: | The proposed model significantly extends kNN-MT with dynamic retrieval on widely-used datasets. |
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources. |
| Approach: | They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism. |
| Outcome: | The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%. |
Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Several datasets have been developed for building conversational question answering systems. |
| Approach: | They propose a robust dialog synthesising method that learns segmentation instead of using sentence boundaries. |
| Outcome: | The proposed method achieves superior quality when compared to WikiDialog . it also improves performance across OR-QuAC benchmarks . |
Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to distill knowledge from vast datasets during the pretraining phase. |
| Approach: | They propose to use in-context concept definitions as prompts for zero-shot sentence labeling tasks. |
| Outcome: | The proposed model outperforms the leading open-source models and their proprietary APIs in most cases. |
Alignment-Based Decoding Policy for Low-Latency and Anticipation-Free Neural Japanese Input Method Editors (2024.findings-acl)
Copied to clipboard
| Challenge: | Japanese input method editors (IMEs) allow users to input Japanese text using a limited set of characters such as the kana syllabary. |
| Approach: | They propose a simple decoding policy to enable simultaneous kana-kanji conversion in Japanese IMEs inspired by simultaneous machine translation. |
| Outcome: | The proposed approach achieves a better quality-latency trade-off than baselines while being more practical due to its ability to directly handle streaming input. |
ECoK: Emotional Commonsense Knowledge Graph for Mining Emotional Gold (2024.findings-acl)
Copied to clipboard
Zhunheng Wang, Xiaoyi Liu, Mengting Hu, Rui Ying, Ming Jiang, Jianfeng Wu, Yalan Xie, Hang Gao, Renhong Cheng
| Challenge: | Existing knowledge graphs focus on the representation and reasoning of general factual knowledge, while there are significant deficiencies in the understanding and reasoning for emotional knowledge. |
| Approach: | They propose a commonsense knowledge graph that can be used to represent emotional knowledge by combining theories from psychology, cognitive science, and linguistics. |
| Outcome: | The proposed model surpasses GPT-4-Turbo in the emotion-related tasks. |
Deterministic Reversible Data Augmentation for Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent neural machine translation models have improved translation quality but they also introduce small perturbations like misspelling and paraphrasing. |
| Approach: | They propose a method that generates multi-granularity subword representations with reversible operations and deterministic segmentations. |
| Outcome: | The proposed method outperforms strong baselines on several translation tasks with a clear margin and exhibits good robustness in noisy, low-resource, and cross-domain datasets. |
Latent Learningscape Guided In-context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to select demonstrations based on surface-level semantic similarities fall short of identifying the most fitting ones. |
| Approach: | They propose a method that characterizes latent learningscape features of demonstrations and uses them to create more effective prompts. |
| Outcome: | The proposed method outperforms leading models in arithmetic, commonsense, and symbolic reasoning tasks showing an average increase in scores by 7.4 percentage points. |
SMR: State Memory Replay for Long Sequence Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing state space models (SSMs) address non-uniform sampling, but their recursive structures impede efficient SSM computation via convolution. |
| Approach: | They propose a plug-and-play mechanism to solve the Non-Stable State problem by adjusting input sequences with early memories. |
| Outcome: | The proposed method overcomes the non-uniform sample processing problem . it can achieve Sampling Step Adaptation (SSA) by adjusting input sequences with early memories. |
Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate fluent text with minimal task-specific supervision, but their ability to generate rationales for knowledge-intensive tasks (KITs) remains under-explored. |
| Approach: | They propose to generate retrieval-augmented rationalization of KIT model predictions via external knowledge guidance within a few-shot setting. |
| Outcome: | The proposed rationales were compared with crowd-sourced rationale models on factuality, sufficiency, and convincingness. |
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation approaches for large language models (LLMs) rely on existing tasks and benchmarks, raising concerns about test set contamination and the genuine comprehension abilities of LLMs. |
| Approach: | They propose to evaluate LLMs by designing new tasks, automatically generating evaluation datasets for the tasks, and conducting detailed error analyses to scrutinize LLM's adaptability to new tasks. |
| Outcome: | The proposed method examines LLMs’ adaptability to new tasks, their sensitivity to prompt variations, and their error tendencies. |
Linear Cross-Lingual Mapping of Sentence Embeddings (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that a sentence has less ambiguity than a single word . if the word semantics is changed in translation, then a better translation is possible. |
| Approach: | They propose a linear cross-lingual mapping to improve multilingual embeddings . they also consider deviation from orthogonality conditions as a measure of deficiency . |
| Outcome: | The proposed method improves the multilingual embeddings by allowing for a linear cross-lingual mapping. |
ULTRA: Unleash LLMs’ Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinement (2024.findings-acl)
Copied to clipboard
| Challenge: | Structural extraction of events within discourse is critical for event-centric understanding . document-level EAE focuses on arguments that are scattered across an entire document . ULTRA is a hierarchical framework that extracts event arguments more cost-effectively . |
| Approach: | They propose a hierarchical framework that extracts event arguments more cost-effectively . ULTRA sequentially reads text chunks of a document to generate a candidate argument set . they propose to use a supervised model to find the exact boundary of an argument . |
| Outcome: | The proposed framework outperforms strong models and ChatGPT by 9.8% when evaluated by Exact Match (EM). |
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent multilingual models support limited number of human languages due to lack of training data for low resource languages. |
| Approach: | They propose a multilingual multilingual LLM that scales to 100 languages . they use a human feedback dataset and a data set to perform multilingual instruction tuning . |
| Outcome: | The proposed model outperforms its peers on five multilingual benchmarks. |
BASS: Batched Attention-optimized Speculative Sampling (2024.findings-acl)
Copied to clipboard
Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras
| Challenge: | Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. |
| Approach: | They propose a batched speculative decoding system that generates sequences at an average speed of 5.8ms per token and a batch size of 8 at a 2.15 speed-up over optimized regular decoding. |
| Outcome: | The proposed system achieves state-of-the-art latency and speed-up over optimized regular decoding. |
Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games (2024.findings-acl)
Copied to clipboard
| Challenge: | In this study, we explore the application of Large Language Models (LLMs) in Jubensha, a Chinese detective role-playing game and a novel area in Artificial Intelligence (AI) driven gaming. |
| Approach: | They propose to use large language models to foster AI agent development in Jubensha, a Chinese detective role-playing game. |
| Outcome: | The proposed framework enables AI agents to engage in Jubensha games autonomously. |
It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimenting with 9 large language models across 3 datasets, emphasizing the context yields superior results compared to question emphasis. |
| Approach: | They ask: How does the order of inputs affect model performance? |
| Outcome: | Experiments with 9 large language models show that emphasizing the question and context improves model performance. |
Large Language Models Relearn Removed Concepts (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent study has shown that models can re-acquire pruned concepts after editing . however, it remains unclear whether models can recover such concepts after retraining . |
| Approach: | They evaluate concept relearning in large language models by tracking concept saliency and similarity in pruned neurons during retraining for named entity recognition tasks. |
| Outcome: | The results show that models can re-acquire pruned concepts after pruning . they also show that they can blend old and new concepts in individual neurons . |
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders their adaptability to prompt-guided Large Language Models (LLMs). |
| Approach: | They propose a framework for unified task embedding that harmonizes task embeds from various models within a single vector space. |
| Outcome: | The proposed framework harmonizes task embeddings from various models within a single vector space. |
TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for Task-Oriented Dialogs (TOD) lack consideration for adaptive response styles and neglect to simulate interactions with app contexts like calendars or alarms. |
| Approach: | They propose to generate an annotated task-oriented dialog dataset and an automatic pipeline to generate it. |
| Outcome: | The proposed dataset provides a variety of system response styles and provides verbose or non-verbal responses. |
Machine-Generated Text Localization (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work focused on identifying only part of a document as machine or human written . a key challenge is that short spans of text provide little information indicating if it is machine generated due to its short length . |
| Approach: | They propose a method that localizes the portions of a document that were machine generated. |
| Outcome: | The proposed method can detect changes in style or content to boost performance. |
BenchIE^FL: A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark (2024.findings-acl)
Copied to clipboard
| Challenge: | Numerous OIE systems claim ever-increasing performance, limiting their effectiveness. |
| Approach: | They propose a benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. |
| Outcome: | The proposed benchmark aims to improve the performance of OIE extractors while avoiding errors and omissions. |
CausalCite: A Causal Formulation of Paper Citations (2024.findings-acl)
Copied to clipboard
Ishan Agrawal, Zhijing Jin, Ehsan Mokhtarian, Siyuan Guo, Yuen Chen, Mrinmaya Sachan, Bernhard Schölkopf
| Challenge: | citation counts are often criticized for failing to accurately reflect the true impact of a paper. |
| Approach: | They propose a method to measure the impact of a paper on follow-up papers by comparing similar papers by cosine similarity. |
| Outcome: | The proposed method is based on a new causal inference method, TextMatch. |
Question Translation Training for Better Multilingual Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models have shown compelling performance on reasoning tasks but they tend to perform much worse in languages other than English. |
| Approach: | They propose to train a model to translate reasoning questions into English by fine tuning on X-English parallel question data. |
| Outcome: | The proposed approach improves on LLaMA2-13B on the MGSM and MSVAMP multilingual reasoning benchmarks. |
Improving LLM Generations via Fine-Grained Self-Endorsement (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable capabilities but can still fail frequently on knowledge-intensive tasks. |
| Approach: | They propose a self-endorsement framework that leverages fine-grained fact-level comparisons across multiple sampled responses. |
| Outcome: | The proposed framework can improve factuality of generations with simple prompts across scales of LLMs. |
Multi-Label Classification for Implicit Discourse Relation Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior research in discourse relation recognition has treated these instances as separate examples during training, with a gold-standard prediction matching one of the labels considered correct at test time. |
| Approach: | They propose to use multiple labels to annotate an example when multiple relations are believed to hold simultaneously. |
| Outcome: | The proposed frameworks don't depress performance for single-label prediction. |
StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing CodeLLM benchmarks rely on a single expert-written prompt per problem . a growing body of work shows their utility to professional programmers . |
| Approach: | They propose a natural-language-to-code benchmark of prompts written by non-experts . student prompts are written by 80 students who have only completed one introductory Python course . |
| Outcome: | The proposed model is better discriminator of student prompt descriptions than existing benchmarks. |
ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution (2024.findings-acl)
Copied to clipboard
| Challenge: | Lexical Substitution fails to consider substitutes of equal or higher proficiency than the target word. |
| Approach: | They propose a task to find appropriate substitutes for a given word in a context sentence but not those that are of equal or higher proficiency than the target. |
| Outcome: | The proposed model outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex. |
Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing decoding algorithms to generate diverse outputs are based on beam search or random sampling, thus their output quality is capped by these underlying decoding methods. |
| Approach: | They propose to add a diversity penalty to MBR decoding and a clustering problem to create diversity-promoting decoding algorithms by enforcing diversity objectives. |
| Outcome: | The proposed method achieves a better trade-off than the diverse beam search and sampling algorithms overall. |
GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | despite improvements in NMT quality and adoption, perpetuation of gender bias remains a concern. |
| Approach: | They propose a translation gender rewriting solution built with GPT-4 and use it to evaluate it. |
| Outcome: | The proposed translation gender rewriting solution is open source and can be used in English. |
Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Minimum Bayes-Risk (MBR) decoding is a powerful alternative to beam search for text generation tasks. |
| Approach: | They propose a method that uses a hyperparameter-free method to run MBR decoding efficiently. |
| Outcome: | The proposed method reduces the amount of computation but requires hyperparameter tuning to be effective. |
Simplifying Translations for Children: Iterative Simplification Considering Age of Acquisition with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) systems do not take into account the complexity of the words used to compose the translations. |
| Approach: | They propose a method that replaces high Age of Acquisitions words in translations with simpler words to match the user’s level. |
| Outcome: | The proposed method replaces high-AoA words with lower-Aa words while maintaining high BLEU and COMET scores. |
Bi-Chainer: Automated Large Language Models Reasoning with Bidirectional Chaining (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing unidirectional chaining methods suffer from low prediction accuracy and efficiency. |
| Approach: | They propose a bidirectional chaining method which dynamically switches to depth-first reasoning in the opposite reasoning direction when it encounters multiple branching options within the current direction. |
| Outcome: | The proposed method achieves sizable accuracy boots over unidirectional chaining frameworks on four challenging logical reasoning datasets. |
Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on the evaluation of large language models (LLMs) has shown unprecedented performance on diverse language generation tasks. |
| Approach: | They investigate the controllability of large language models on scientific summarization tasks by controlling stylistic and content coverage factors. |
| Outcome: | The proposed model outperforms humans on the MuP review generation task in terms of similarity to reference summaries and human preferences. |
Knowledge Context Modeling with Pre-trained Language Models for Contrastive Knowledge Graph Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Text-based knowledge graph completion methods neglect knowledge contexts in inferring process. |
| Approach: | They propose a framework which models the knowledge context as additional prompts with pre-trained language models for knowledge graph completion. |
| Outcome: | The proposed framework achieves state-of-the-art on FB15k-237, WN18RR and Wikidata5M datasets. |
Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E-Commerce Products (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for attribute value extraction struggle for parameter efficiency and reliability due to data contamination and catastrophic forgetting. |
| Approach: | They propose to decouple product type and attribute to promote de-contamination and parameter efficiency while scaling up. |
| Outcome: | The proposed model achieves state-of-the-art performance with affordable parameter size, least historical knowledge forgetting, and greatest robustness against noises. |
Exploring Domain Robust Lightweight Reward Models based on Router Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have relied on the large reward model for fine-tuning, but the use of a single reward model across domains may not always be optimal. |
| Approach: | They propose to use router mechanisms to train small language models in a domain-specific manner . they use internal routers, external routers and router adapters to create a single reward model . |
| Outcome: | The proposed approach reduces parameter size while minimizing parameter size. |
Generalized Category Discovery with Large Language Models in the Loop (2024.findings-acl)
Copied to clipboard
Wenbin An, Wenkai Shi, Feng Tian, Haonan Lin, QianYing Wang, Yaqiang Wu, Mingxiang Cai, Luyan Wang, Yan Chen, Haiping Zhu, Ping Chen
| Challenge: | Generalized Category Discovery (GCD) is a crucial task that aims to recognize both known and novel categories from a set of unlabeled data. |
| Approach: | They propose a framework that introduces Large Language Models into the training loop to generate category names without human effort. |
| Outcome: | The proposed framework outperforms SOTA models on three benchmark datasets and generates accurate category names for the discovered clusters. |
VAEGPT-Sim: Improving Sentence Representation with Limited Corpus Using Gradually-Denoising VAE (2024.findings-acl)
Copied to clipboard
| Challenge: | Text embedding requires a highly efficient method for training domain-specific models on limited corpora. |
| Approach: | They propose a model that combines a denoising variational autoencoder with a target-specific discriminator to generate synonymous sentences that closely resemble human language. |
| Outcome: | The proposed model surpasses ConSERT by 2.8 points in small-dataset training on STS benchmarks. |
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent evaluations of Large Language Models (LLMs) focus on their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. |
| Approach: | They propose a PowerPoint Task Completion benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions. |
| Outcome: | The proposed system outperforms open-source and closed LLMs with 75.1% accuracy in single-turn dialogue testing but only achieves 6% session accuracy. |
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the confidence calibration of LLMs have not explored the effects of different prompting strategies on LLM performance. |
| Approach: | They propose Fact-and-Reflection prompting which improves LLM confidence calibration . they propose to use human cognition to elicit known "facts" and ask model to "reflect" over them . |
| Outcome: | The proposed method lowers the expected calibration error by 23.5% on multi-purpose QA tasks. |
DB-LLM: Accurate Dual-Binarization for Efficient LLMs (2024.findings-acl)
Copied to clipboard
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, Dacheng Tao
| Challenge: | Existing methods for ultra-low bit quantization cause severe accuracy drops . a novel Dual-Binarization method is proposed for efficient Large Language Models . |
| Approach: | They propose a Dual-Binarization method that takes 2-bit-width and binarization into account . they propose DB-LLM, which uses a 2-bit binarized weighted model to represent weights efficiently . |
| Outcome: | The proposed method surpasses the current State-of-the-Art in ultra-low bit quantization and achieves 20% reduction in computational consumption compared to the SOTA method under the same bit-width. |
TempCompass: Do Video LLMs Really Understand Videos? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks on video large language models lack a comprehensive feedback on temporal perception ability . current models cannot distinguish between different temporal aspects and are limited in task formats . |
| Approach: | They propose a benchmark to evaluate temporal perception ability of video large language models . they construct conflicting videos that share the same static content but differ in a specific temporal aspect . |
| Outcome: | The proposed benchmarks show that video large language models exhibit poor temporal perception ability. |
“Get Their Hands Dirty, Not Mine”: On Researcher-Annotator Collaboration and the Agency of Annotators (2024.findings-acl)
Copied to clipboard
| Challenge: | Annotation quality is often framed as post-hoc cleanup of annotator-caused issues . authors argue that this narrative limits the scope of improving annotation . |
| Approach: | They propose to consider annotation as a procedural collaboration . they propose to capture the nuance and describe the full procedure to resolve issues . |
| Outcome: | The proposed study examines whether and why annotation quality is often framed as post-hoc cleanup of annotator-caused issues. |
Teaching Large Language Models an Unseen Language on the Fly (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to support numerous low-resource languages . Existing models lack sufficient training data for effective parameter updating . |
| Approach: | They propose a framework for adapting LLMs to unseen languages by in-context learning. |
| Outcome: | The proposed framework improves Chinese-to-Zhuang translation performance and Zhuan-to Chinese translation performance. |
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows that large language models (LLMs) perform poorly at segment level. |
| Approach: | They propose a new prompting method that emulates the commonly accepted human evaluation framework . they will release their code and scripts to facilitate the community . |
| Outcome: | The proposed method is based on the human evaluation framework MQM and produces explainable and reliable MT evaluations at both the system and segment level. |
GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets focus on primary perception abilities and commonsense knowledge, or have low level of text comprehension difficulty, which are insufficient to reflect comprehensive capabilities of large vision-language models. |
| Approach: | They propose a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO) which sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning. |
| Outcome: | The proposed model derives from native Chinese context and sets human-level requirements for its abilities, including perception, understanding, knowledge and reasoning. |
DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation (2024.findings-acl)
Copied to clipboard
| Challenge: | a novel method to align Large Language Models to "chat" with prompt-as-input Text-to-Image Synthesis models is proposed . a user-specified instruction can be used to create a high quality image . |
| Approach: | They propose a method to align Large Language Models to "chat" with prompt-as-input Text-to-Image Synthesis models for interactive image creation. |
| Outcome: | The proposed method can exhibit superior performance than baseline models and strong competitors based on automatic and human evaluations. |
Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extending the maximum context lengths of language models are lacking a strong baseline for in-context few-shot classification and on more challenging Chain-of-Thought reasoning, such as HotpotQA, deteriorate question miscomprehension and false inference. |
| Approach: | They propose to harness window-wise attention and positional embedding techniques to extend the maximum context lengths of language models. |
| Outcome: | The proposed method is able to extend the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. |
Rationales for Answers to Simple Math Word Problems Confuse Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that large language models have advanced mathematical problem-solving abilities in grade school math word problems. |
| Approach: | They propose to combine fine-tuning and prompt-based methods to improve performance . they propose to use a hybrid algorithm to fine- tune LLMs on specific tasks . |
| Outcome: | The proposed methods improve performance on the proposed reasoning process evaluation benchmarks. |
ResLoRA: Identity Residual Mapping in Low-Rank Adaption (2024.findings-acl)
Copied to clipboard
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Low-rank adaptation (LoRA) is one of the most popular parameter-efficient fine-tuning methods. |
| Approach: | They propose a low-rank adaptation method that adds residual paths during training and merges them together during inference to achieve better results. |
| Outcome: | The proposed method achieves 2.5x faster convergence speed and improves performance by 14.3% on NLG, NLU, and text-to-image tasks. |
Towards Objectively Benchmarking Social Intelligence of Language Agents at the Action Level (2024.findings-acl)
Copied to clipboard
| Challenge: | evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. |
| Approach: | They propose a social task in sandbox simulation benchmark that assesses language agents objectively at the action level by scrutinizing goal achievements within the multi-agent simulation. |
| Outcome: | The proposed social task-in-sandbox simulation is a language-level benchmark . the proposed benchmark effectively discriminates between distinct language agents . |
Semantic Role Labeling from Chinese Speech via End-to-End Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Semantic role labeling (SRL) has traditionally focused on text input. |
| Approach: | They propose an end-to-end approach for SRL from speech integrating ASR and SRL in a joint-learning framework, focusing on the Chinese language. |
| Outcome: | The proposed model improves on the Chinese Proposition Bank 1.0 dataset and the existing model with improved performance. |
MEEL: Multi-Modal Event Evolution Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models fail to grasp the principles governing event evolution in various scenarios. |
| Approach: | They propose a multi-modal event evolution learning approach to grasp event evolution . they propose an instruction encapsulation process that transforms evolving graphs into instruction-tuning data . |
| Outcome: | The proposed model grasps the event evolution mechanism yielding advanced MMER ability. |
LLM-REDIAL: A Large-Scale Dataset for Conversational Recommender Systems Created from User Behaviors with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing CRS datasets suffer from data inextensibility and semantic inconsistency . |
| Approach: | They introduce the LLM-REDIAL dataset to facilitate the research in CRS by leveraging large language models to generate high-quality dialogues. |
| Outcome: | The proposed dataset is the largest multi-domain CRS dataset which consists of 47.6k multi-turn dialogues with 482.6k utterances across 4 domains. |
Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in language generation models can be used to assist users in a variety of tasks, but there are risks associated with introducing LLM biases into consequential decisions. |
| Approach: | They propose to use a template-generated dataset to measure subtler correlated decisions that LLMs make between social groups and unrelated positive and negative attributes. |
| Outcome: | The proposed model can be used to evaluate progress in more generalized biases and extend the benchmark with minimal human annotation. |
EVIT: Event-Oriented Instruction Tuning for Event Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in event reasoning . however, smaller instruction-tuned models do not consistently demonstrate exceptional proficiency . |
| Approach: | They propose an event-oriented instruction tuning technique to train a large language model . they propose a structure named event quadruple which contains the structure and semantics of events . |
| Outcome: | The proposed model achieves competitive performances on event reasoning tasks. |
InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing sentences do not consider the length constraints in extractive summarization because of their limited model abilities. |
| Approach: | They propose an approach that incorporates length constraints without model modifications into sentences . they use traditional sentence compression datasets to transform them into instruction format . |
| Outcome: | The proposed method can consider the length constraint through instructions without model modifications. |
SymTax: Symbiotic Relationship and Taxonomy Fusion for Effective Citation Recommendation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing recommendations focus on local context or global context but fail to consider actual human citation behaviour. |
| Approach: | They propose a recommendation architecture that considers both local and global contexts . they use hyperbolic separation to compute query-candidate similarity . |
| Outcome: | The proposed framework performs better on a large dataset with 8.27 million citation contexts . it learns to embed the infused taxonomies in the hyperbolic space and computes similarity . |
Assessing News Thumbnail Representativeness: Counterfactual text can enhance the cross-modal matching ability (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for visual and language processing struggle to match news actors’ visual and textual appearances. |
| Approach: | They propose a method that generates counterfactual news thumbnail images and text pairs to assess whether a news thumbnail image represents the actors discussed in the news text. |
| Outcome: | The proposed method can boost the performance for assessing news thumbnail representativeness, supporting the hypothesis. |
Towards Better Question Generation in QA-based Event Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | True. True. EE aims to extract event-related information from unstructured texts. |
| Approach: | They propose a reinforcement learning method that evaluates the quality of a question and provides clear guidance to QA models. |
| Outcome: | The proposed method generates generalizable, high-quality, and context-dependent questions and provides clear guidance to QA models. |
Budget-Constrained Tool Learning with Planning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for budget-constrained tool learning have been overlooked . et al., 2023b) compared tool learning with other methods to improve performance . |
| Approach: | They propose a method for budget-constrained tool learning by creating a preferable plan under the budget constraint before utilizing the tools. |
| Outcome: | The proposed method reduces the cost of tool learning and reaches competitive Pass Rate. |
TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models with instruction-following capabilities have revolutionized the field of artificial intelligence. |
| Approach: | They propose an annotation-free framework for empowering large language models with instruction-following capabilities. |
| Outcome: | The proposed framework generates multi-turn multimodal instruction-response conversations from a language model. |
The Critique of Critique (2024.findings-acl)
Copied to clipboard
| Challenge: | MetaCritique builds specific quantification criteria to evaluate the quality of critique . a systematic method to evaluate critique is lacking. |
| Approach: | They propose a critique of critique, termed MetaCritique, which builds specific quantification criteria and aggregates each AIU's judgment for the overall score. |
| Outcome: | The proposed method can achieve near-human performance across 16 datasets. |
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current vital challenges for autonomous agents lie in two aspects: dependence on strong (M)LLMs and insufficient GUI environment modeling. |
| Approach: | They propose a comprehensive cognitive LLM agent with two novel approaches to improve GUI automation performance. |
| Outcome: | The proposed agent achieves state-of-the-art performance on AITW and META-GUI benchmarks. |
FRVA: Fact-Retrieval and Verification Augmented Entailment Tree Generation for Explainable Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating a entailment tree exhibit the reasoning chains from knowledge facts to predicted answers, but they have large fact search spaces and error accumulation problems resulting in the generation of invalid steps. |
| Approach: | They propose a Fact-Retrieval and Verification Augmented bidirectional entailment tree generation method that contains two systems. |
| Outcome: | The proposed method outperforms existing models and achieves state-of-the-art performance in fact selection and structural correctness. |
P4: Plug-and-Play Discrete Prompting for Large Language Models Personalization (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive capabilities in following instructions, but manually prompting them to exhibit certain personalities may result in sub-optimal performance. |
| Approach: | They propose a plug-and-play prompting method to manipulate Large Language Models with distinct human-like personality traits by appending discrete personalized suffixes to query or dialog histories and focusing exclusively on influential tokens. |
| Outcome: | The proposed method outperforms other prompting methods and model editing methods on four models ranging from 1.1B to 13B and achieves 79.9% accuracy in customizing LLMs’ personalities. |
Large Language Models Can Learn Representation in Natural Language (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are unable to complete complex tasks involving multiple entities, such as tool APIs. |
| Approach: | They propose a method which uses natural language representations to refine entity descriptions for improved retrieval and LLM utilization. |
| Outcome: | The proposed method improves GPT-4's performance on classification tasks and API call tasks. |
CTC-based Non-autoregressive Textless Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing direct speech-to-speech translation models require text supervision during training, which is not feasible for numerous unwritten languages. |
| Approach: | They propose a non-autoregressive (NAR) model that generates discrete units from the source speech and employs a unit-based vocoder to synthesize the target. |
| Outcome: | The proposed model achieves translation quality comparable to the autoregressive model while preserving up to 26.81 decoding speedup. |
RRNorm: A Novel Framework for Chinese Disease Diagnoses Normalization via LLM-Driven Terminology Component Recognition and Reconstruction (2024.findings-acl)
Copied to clipboard
| Challenge: | Clinical Terminology Normalization (CTN) aims at finding standard terms from a given termbase for mentions extracted from clinical texts. |
| Approach: | They propose a method that leverages reasoning capability of large language models to recognize components of terms and automate decomposition. |
| Outcome: | The proposed strategy achieves state-of-the-art on the experimental dataset. |
Unexpected Phenomenon: LLMs’ Spurious Associations in Information Extraction (2024.findings-acl)
Copied to clipboard
Weiyan Zhang, Wanpeng Lu, Jiacheng Wang, Yating Wang, Lihan Chen, Haiyun Jiang, Jingping Liu, Tong Ruan
| Challenge: | Information extraction (IE) tasks require a limited number of example instructions to achieve effective performance. |
| Approach: | They propose two strategies to find spurious associations in large language models (LLMs) they use forward label extension and backward label validation to leverage extended labels to improve model performance. |
| Outcome: | The proposed methods improve performance on Chinese and English datasets and 9.55%, 11.42%, and 21.27% in F1 scores on SciERC, ACE05, and DuEE datasets. |
AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual chain-of-thought integrate reasoning knowledge from different languages, but they still rely on manual language specification and weight allocation. |
| Approach: | They propose an automatic cross-lingual alignment planning framework that integrates reasoning knowledge from different languages. |
| Outcome: | The proposed framework surpasses existing methods that require manual effort to integrate languages. |
LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LT strategies cannot indicate the desired target language on zero-shot translation, i.e., the off-target issue. |
| Approach: | They propose a language converter strategy that embeds the target language into the top encoder layers to mitigate confusion in the encoder and ensures stable language indication for the decoder. |
| Outcome: | The proposed language converter strategy significantly mitigates off-target issue on multiUN, TED, and OPUS-100 datasets. |
Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantitative reasoning with data is a critical skill to analyze data, yet the assessment of such ability remains limited. |
| Approach: | They propose a quantitative reasoning with data benchmark to evaluate Large Language Models' ability in statistical and causal reasoning with real-world data. |
| Outcome: | The proposed model GPT-4 achieves an accuracy of 58%, while open-source model Deepseek-coder-instruct gets the highest accuracy of 37%. |
On the Vulnerability of Safety Alignment in Open-Access LLMs (2024.findings-acl)
Copied to clipboard
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, Fangzhao Wu
| Challenge: | Large language models (LLMs) are susceptible to malicious exploitation, but are often rejected and limited harmfulness is limited. |
| Approach: | They propose two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). |
| Outcome: | The proposed methods can significantly enhance the success rate and harmfulness of jailbreak attacks, but they face high rejection rates and limited harmfulness. |
PEK: A Parameter-Efficient Framework for Knowledge-Grounded Dialogue Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have shown great dialogue generation capability in different scenarios, but the huge VRAM consumption when fine-tuning them is one of their drawbacks. |
| Approach: | They propose a parameter-efficient framework for knowledge-enhanced dialogue generation that leverages external knowledge documents and knowledge graphs to enhance its generation capabilities. |
| Outcome: | The proposed framework outperforms baseline methods on multiple evaluation metrics on Wizard of Wikipedia and CMU_DoG datasets. |
Evidence Retrieval is almost All You Need for Fact Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evidence retrieval methods adopt a trivial retrieval strategy, resulting in task-irrelevant evidence and undesirable performance. |
| Approach: | They propose a framework for evidence retrieval and joint fact verification that integrates two modules. |
| Outcome: | The proposed framework improves evidence retrieval and claims verification on a FEVER dataset. |
Outdated Issue Aware Decoding for Factual Knowledge Editing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods retain outdated responses for reasoning questions . naively retraining LLMs can be computationally intensive and can lead to catastrophic forgetting . |
| Approach: | They propose a simple yet effective decoding strategy to enhance edited models on reasoning questions. |
| Outcome: | The proposed method outDates ISsue aware deCOding (DISCO) to improve models on reasoning questions. |
Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have studied dialect-related fairness for aspects like hate speech, but other aspects of biased language remain unexplored. |
| Approach: | They propose a multitask learning approach that models dialect language as an auxiliary task to incorporate syntactic and lexical variations. |
| Outcome: | The proposed approach improves dialect learning and detects biases more reliably. |
DP-MLM: Differentially Private Text Rewriting Using Masked Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text privatization using Differential Privacy rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. |
| Approach: | They propose a method for differentially private text rewriting using masked language models to rewrite a text one token at a time. |
| Outcome: | The proposed method preserves utility at lower levels, compared to previous methods relying on autoregressive models with a decoder. |
Question-Instructed Visual Descriptions for Zero-Shot Video Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for video QA rely on complex architectures, expensive pipelines or closed models like GPTs. |
| Approach: | They propose a single instruction-aware open vision-language model to tackle videoQA using frame descriptions. |
| Outcome: | The proposed framework achieves higher performance than current state-of-the-art models on videoQA benchmarks. |
EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on fact verification lack a high-quality dataset for explainability . existing systems lack evidence retrieval and veracity prediction, limiting the ability to verify a claim . |
| Approach: | They propose a dataset for multi-hop explainable fact verification that summarises and modifies Wikipedia documents. |
| Outcome: | The proposed dataset aims to improve the accuracy of multi-hop explainable fact verification systems. |
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on prompt engineering or framework scheduling of one/multiple LLMs. |
| Approach: | They propose to integrate LLMs as agents into their training corpus by decomposition and redesigning the training corpu . they propose to use LLM-FLAN to effectively fine-tune LANguage models for Agents by reducing hallucinations. |
| Outcome: | The proposed model outperforms prior best models by 3.5% across agent evaluation datasets. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs. |
| Approach: | They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories. |
| Outcome: | The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models. |
Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural Machine Translation models are extremely data-hungry and require a large dataset to maintain data quality. |
| Approach: | They propose a new data pruning technique that leverages early model training dynamics to identify the most relevant data points for model performance. |
| Outcome: | The proposed technique outperforms the benchmarks on indo-European languages while pruning up to 50% of training data. |
What Are You Token About? Differentiable Perturbed Top-k Token Selection for Scientific Document Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets suffer from a deficiency in source heterogeneity, hindering effective model training and generalizability. |
| Approach: | They propose a dataset that includes technical and lay summaries from multiple journals . they propose 'prinepert' transformer-based model that prunes irrelevant tokens in end-to-end learning. |
| Outcome: | The proposed model achieves a 2x speed-up compared to a state-of-the-art linear transformer, remaining comparable in effectiveness. |
Description Boosting for Zero-Shot Entity and Relation Classification (2024.findings-acl)
Copied to clipboard
Gabriele Picco, Leopold Fuchs, Marcos Martínez Galindo, Alberto Purpura, Vanessa López, Hoang Thanh Lam
| Challenge: | Named Entity Recognition and Relation Extraction (RE) methods are expensive and require domain experts for data acquisition and labeling. |
| Approach: | They propose a strategy for generating variations of an initial description, a heuristic for ranking them and an ensemble method capable of boosting the predictions of zero-shot models. |
| Outcome: | The proposed method outperforms existing approaches and achieves new SOTA results on four different entity and relation classification datasets. |
Domain-Aware k-Nearest-Neighbor Knowledge Distillation for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to transfer knowledge from kNN datastore into new models are expensive and arbitrarily transfer knowledge. |
| Approach: | They propose a domain-aware method which filters out domain-relevant neighborhood knowledge for learning in the distillation process. |
| Outcome: | The proposed method achieves state-of-the-art on four domain translation tasks. |
Beyond Single-Event Extraction: Towards Efficient Document-Level Multi-Event Argument Extraction (2024.findings-acl)
Copied to clipboard
Wanlong Liu, Li Zhou, DingYi Zeng, Yichen Xiao, Shaohuan Cheng, Chen Zhang, Grandee Lee, Malu Zhang, Wenyu Chen
| Challenge: | mainstream event argument extraction methods process each event in isolation, resulting in inefficient inference and ignoring correlations among multiple events. |
| Approach: | They propose a multi-event argument argument extraction model which extracts arguments from all events simultaneously. |
| Outcome: | The proposed model performs better on four public datasets while saving time. |
Revisiting Interpolation Augmentation for Speech-to-Text Generation (2024.findings-acl)
Copied to clipboard
Chen Xu, Jie Wang, Xiaoqian Liu, Qian Dong, Chunliang Zhang, Tong Xiao, JingBo Zhu, Dapeng Man, Wu Yang
| Challenge: | Existing approaches to speech-to-text generation tasks are limited by the lack of extensive labeled datasets. |
| Approach: | They propose to use interpolation augmentation to construct virtual training samples by transforming inputs and labels to enhance generalization in other domains. |
| Outcome: | The proposed approach significantly improves performance across diverse tasks, architectures, and data scales, offering a promising avenue for more robust S2T systems in resource-constrained settings. |
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be prohibitive in terms of feasibility, time, and resources. |
| Approach: | They propose a method for training large language models by enabling "self-talk" they propose supervised fine-tuning of LLMs to improve quality of dialogues . |
| Outcome: | The proposed method generates training data via "self-talk" of LLMs that can be refined and utilized for supervised fine-tuning. |
Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods allow efficient adaptation of Large Language Models (LLMs) to various downstream tasks, but their effectiveness diminishes when downstream tasks require accurate learning of specific knowledge. |
| Approach: | They propose a method that fine-tunes a limited number of model parameters while keeping the majority of original parameters fixed. |
| Outcome: | The proposed method is able to perform on open-source large language models and validates the semantic challenge in PEFT. |
Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Document collections of various domains share some underlying collection-wide structure . structure can be useful in various use cases across different domains, such as legal, medical, or financial . |
| Approach: | They propose to identify the typical structure of document within a collection by using header paraphrases to ground topics to respective document locations. |
| Outcome: | The proposed method extracts meaningful collection-wide structure from documents in three domains in English and Hebrew. |
Enhancing Cross Text-Molecule Learning by Self-Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets are limited due to the difficulty of collecting precise molecule-description pairs. Existing approaches to enhance large language models include a data augmentation framework and a new dataset called SAPubChem-41. |
| Approach: | They propose a framework that interweaves model fine-tuning and data augmentation to overcome the scarcity of high-quality labeled data. |
| Outcome: | The proposed framework interweaves model fine-tuning and data augmentation to overcome the scarcity of high-quality labeled data. |
RePALM: Popular Quote Tweet Generation via Auto-Response Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that the wording of tweets can significantly impact popularity, reflected by user replies, retweets, and likes. |
| Approach: | They propose a novel approach to generate popular quote tweets by leveraging augmented auto-responses from readers to align language generation with popularity. |
| Outcome: | The proposed model outperforms existing models that do not incorporate response augmentation and can generate popular quote tweets with augmented auto-responses. |
On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information. |
| Approach: | They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM. |
| Outcome: | The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient. |
Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST! (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models are unable to identify and interpret semantically underspecified sentences when explicitly prompted. |
| Approach: | They propose a Dataset of semantically underspecified Sentences grouped by type to study whether pre-trained language models correctly identify and interpret underspecify sentences. |
| Outcome: | The proposed dataset shows that pre-trained language models are reasonably able to identify and interpret underspecified sentences when explicitly prompted. |
Visual Hallucinations of Multi-modal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs’ performance under VH. |
| Approach: | They propose a tool called VHTest to generate a diverse set of VH instances from existing image datasets and a text-to-image generative model to generate VH images based on the text descriptions. |
| Outcome: | The proposed tool finds VH instances in existing image datasets and generates images based on the text descriptions. |
SumSurvey: An Abstractive Dataset of Scientific Survey Papers for Long Document Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | a growing need for long document summarization datasets with 16k input is causing problems. |
| Approach: | They propose to use a dataset to analyze salient information in long document summarizations. |
| Outcome: | The proposed dataset outperforms existing models and LLMs in the distribution form of salient information and the distribution of salinal information is an indicator of quality. |
Pushing the Limits of Low-Resource NER Using LLM Artificial Data Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an important task, but it requires a large amount of labeled data to perform well. |
| Approach: | They propose to use open-source Large Language Models to generate NER data with only a few labeled examples, reducing the cost of human annotations. |
| Outcome: | The proposed method significantly improves the baseline on diverse low-resource NER datasets and can be used to augment datasets with class-imbalance problems. |
Understanding and Patching Compositional Reasoning in LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | LLMs have marked a revolutonary shift, yet they falter when faced with compositional reasoning tasks. |
| Approach: | They propose a lightweight method to patch compositional reasoning errors via editing the located MHSA modules in LLMs. |
| Outcome: | The proposed method can be used to patch compositional reasoning errors using MHSA modules located within the layers of the LLMs. |
Bilingual Rhetorical Structure Parsing with Large Parallel Annotations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large RST corpora are inconsistent in annotation guidelines, genre representation, source selection, and relation definitions. |
| Approach: | They propose a parallel Russian annotation for a large and diverse English GUM RST corpus. |
| Outcome: | The proposed RST parser achieves state-of-the-art results on English and Russian corpus . it demonstrates effectiveness in monolingual and bilingual settings, transferring even with limited second-language annotation. |
Book2Dial: Generating Teacher Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots (2024.findings-acl)
Copied to clipboard
| Challenge: | Educational chatbots are a promising tool for assisting student learning, but high-quality data is difficult to obtain due to privacy concerns. |
| Approach: | They propose a framework for generating synthetic teacher-student interactions grounded in a set of textbooks and propose to open-source their results. |
| Outcome: | The proposed framework captures a key aspect of learning interactions where curious students with partial knowledge ask teachers questions about the material in the textbook. |
SELP: A Semantically-Driven Approach for Separated and Accurate Class Prototypes in Few-Shot Text Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for few-shot text classification focus on enhancing separation of prototypes without taking semantic relationships into account. |
| Approach: | They propose to utilize semantically enhanced labels to calibrate class Prototypes . they propose a center loss method to enhance intra-class compactness . |
| Outcome: | The proposed method outperforms baseline methods on eight few-shot text classification datasets. |
Automated Focused Feedback Generation for Scientific Writing Assistance (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work has focused on improving surface form and style rather than manuscript content. |
| Approach: | They propose to use a scientific writing focused feedback tool to generate specific, actionable and coherent comments which identify weaknesses in a paper and/or propose revisions to it. |
| Outcome: | The proposed tool outperforms existing approaches in specificity, reading comprehension and overall helpfulness of the generated reviews. |
FastGAS: Fast Graph-based Annotation Selection for In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to select unlabeled examples for annotation require a long time due to their complexity, hindering their practical viability. |
| Approach: | They propose a graph-based selection method to efficiently identify high-quality instances while minimizing computational overhead. |
| Outcome: | The proposed method significantly reduces selection time and improves performance on different tasks. |
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a feasible solution for end-side LLM deployment . however, achieving a high compression ratio for scaled-up LLMs remains a challenge . |
| Approach: | They propose a task-agnostic structured pruning approach coupled with a compact Transformer architecture to prune LLMs into an intra-module low-rank architecture. |
| Outcome: | The proposed approach reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules while preserving inter-module activations sensitive to perturbations. |
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization is a common method for vocabulary building in NMT systems . but, it has its disadvantages, such as a vocabulary cannot be modified once it is learned . |
| Approach: | They propose a method that learns contextualized information of varying scales . they propose byte-based tokenization to solve these problems with few embedding parameters . |
| Outcome: | Experiments show that the proposed method outperforms subword-based methods in multilingual and out-of-domain scenarios. |
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) generate factually correct text and estimate truth values of individual claims, but they do not reflect a coherent, manipulable model of the world. |
| Approach: | They propose a method that uses language models to identify implications of (and contradictions within) the text they generate. |
| Outcome: | The proposed method improves LM factuality by 3-26% across the CREAK, MQuAKE, and Reversal Curse datasets. |
Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on speech-to-singing voice conversion (STS) are limited by the scarcity of paired speech-song data and the suboptimal quality of outputs. |
| Approach: | They propose a self-supervised singing voice pre-training model that transforms a speech-to-singing voice into a paired singing voice. |
| Outcome: | The proposed model improves both STS and singing voice synthesis tasks by combining spoken language and a self-supervised singing voice pre-training model. |
Evaluating Large Language Model Biases in Persona-Steered Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent wave of powerful new large language models has raised concerns that their expressed opinions may be biased towards certain political, national or moral viewpoints. |
| Approach: | They define an incongruous persona as a persona with multiple traits where one trait makes its other traits less likely in human survey data. |
| Outcome: | The results show that LLMs are less steerable towards incongruous personas than congruous ones . the models that are fine-tuned with RLHF are more steerable, especially towards stances associated with political liberals and women . |
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images. |
| Approach: | They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary. |
| Outcome: | The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model. |
CR-UTP: Certified Robustness against Universal Text Perturbations on Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing certified robustness methods for certifying input-specific text perturbations have shown promise in certifyling UTPs, but masking only adversarial words can eliminate the attack. |
| Approach: | They propose a method to certify a language model’s robustness against UTPs by using random smoothing. |
| Outcome: | The proposed method achieves high certified accuracy under extensive masking and achieves state-of-the-art results in multiple settings. |
Recovering document annotations for sentence-level bitext (2024.findings-acl)
Copied to clipboard
| Challenge: | In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. |
| Approach: | They propose a document-level filtering technique that discards document- level metadata. |
| Outcome: | The proposed method improves translation without degradation of sentence-level translation. |
MetaPro 2.0: Computational Metaphor Processing on the Effectiveness of Anomalous Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for metaphor interpretation are slow due to lack of annotated datasets and effective pre-trained language models. |
| Approach: | They propose a large annotated dataset and a PLM for the metaphor interpretation task. |
| Outcome: | The proposed method improves on metaphor identification and interpretation with comparable baselines on the new dataset. |
Boosting LLM Agents with Recursive Contemplation for Effective Deception Handling (2024.findings-acl)
Copied to clipboard
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, Gao Huang
| Challenge: | Recent advances in large language models (LLMs) have led to significant success in using LLMs as agents. |
| Approach: | They propose a cognitive framework that incorporates first-order and second-order perspective transitions into LLMs to enhance their ability to identify and counteract deceptive information. |
| Outcome: | The proposed framework enhances LLMs’ ability to identify and counteract deceptive information without extra fine-tuning and data. |
Direct Preference Optimization with an Offset (2024.findings-acl)
Copied to clipboard
| Challenge: | Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one. |
| Approach: | They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning. |
| Outcome: | The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value . |
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation (2024.findings-acl)
Copied to clipboard
Xize Cheng, Rongjie Huang, Linjun Li, Zehan Wang, Tao Jin, Aoxiong Yin, Chen Feiyang, Xinyu Duan, Baoxing Huai, Zhou Zhao
| Challenge: | Existing methods for talking head translation rely on cascading, resulting in delays and cascadic errors. |
| Approach: | They propose a model for talking head translation, TransFace, which can translate audio-visual speech into audio-visual speech in other languages. |
| Outcome: | The proposed model can translate audio-visual speech into audio-visual speech in other languages. |
More than Minorities and Majorities: Understanding Multilateral Bias in Language Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on bias dataset construction and mitigation focus on one demographic group . in real-world applications, there are more than two demographic groups at risk of the same bias. |
| Approach: | They propose to analyze and reduce biases across multiple demographic groups using a multi-demographic bias dataset. |
| Outcome: | The proposed method can mitigate biases among multiple demographic groups effectively, the authors show . |
Fair Federated Learning with Biased Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing literature ignores the inherent group unfairness within CLIP and its ethical implications on FL applications. |
| Approach: | They propose a fairness-aware adaptation framework for CLIP in federated learning . they propose to leverage biased pre-trained VLMs to build fair FL frameworks . |
| Outcome: | The proposed framework addresses unique bias in FL, triggered by data heterogeneity . it trains a fair FL model with fairness-aware deep visual prompting (DVP) Extensive results on human face attribute recognition (FAR) applications show it outperforms state-of-the-art FL models . |
SpeechGuard: Exploring the Adversarial Robustness of Multi-modal Large Language Models (2024.findings-acl)
Copied to clipboard
Raghuveer Peri, Sai Muralidhar Jayanthi, Srikanth Ronanki, Anshu Bhatia, Karel Mundnich, Saket Dingliwal, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Srikanth Vishnubhotla, Daniel Garcia-Romero, Sundararajan Srinivasan, Kyu Han, Katrin Kirchhoff
| Challenge: | Integrated Speech and Large Language Models (SLMs) that follow speech instructions and generate relevant text responses have gained popularity lately. |
| Approach: | They propose algorithms that can generate adversarial examples to jailbreak SLMs without human involvement. |
| Outcome: | The proposed algorithms achieve state-of-the-art on spoken question-answering task scoring over 80% on both safety and helpfulness metrics. |
ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent-proposed evaluation metrics for large language models have a preference-bias . however, such metrics often lack interpretability and only offer a single score . |
| Approach: | They propose a metric that leverages the power of large language models to perform two sub-tasks: decomposing summaries into atomic content units and validating them against the source document. |
| Outcome: | The proposed metric improves faithfulness scores on three summarization evaluation benchmarks by 3% compared to the next-best metric. |
An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks. |
| Approach: | They propose to employ four PEFT methods to fine-tune the LLM component of open-source MLLMs. |
| Outcome: | The proposed method is the best performing on seven datasets, while fine-tuning the connector layers leads to improved performance in most MLLMs. |
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on whether large language models are capable of planning or executing plans. |
| Approach: | They propose an abductive reasoning task using wikiHow to test the effectiveness of small models over large models. |
| Outcome: | The proposed task demonstrates the effectiveness of small models over large models in most scenarios. |
TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. |
| Approach: | They propose a language model for the low-resource language Turkish that is capable of both natural language understanding and generation tasks. |
| Outcome: | The proposed model outperforms multilingual models in understanding and generation tasks and competes with monolingual models for understanding tasks. |
MELD-ST: An Emotion-aware Speech Translation Dataset (2024.findings-acl)
Copied to clipboard
Sirou Chen, Sakiko Yahata, Shuichiro Shimizu, Zhengdong Yang, Yihang Li, Chenhui Chu, Sadao Kurohashi
| Challenge: | Emotion plays a crucial role in human conversation. |
| Approach: | They present a MELD-ST dataset for the emotion-aware speech translation task . they show that fine-tuning with emotion labels can enhance translation performance . |
| Outcome: | The proposed dataset shows that fine tuning with emotion labels can improve translation performance in some settings. |
Designing Informative Metrics for Few-Shot Example Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have shown remarkable few-shot learning capabilities when provided with properly formatted examples. |
| Approach: | They propose a complexity-based prompt selection approach for sequence tagging tasks that uses certain metrics to align the syntactico-semantic complexity of test sentences and examples. |
| Outcome: | The proposed approach achieves state-of-the-art performance on few-shot NER, with 5% improvement in F1 score. |
Chain-of-Quizzes: Pedagogy-inspired Example Selection in In-Context-Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful tool for enhancing large language models (LLMs) by mimicking the human learning process. |
| Approach: | They propose a Chain-of-Quizzes framework that uses LLMs to answer a quiz to sift 'good' examples, combine them iteratively with the increasing complexity, and utilize a final exam to gauge the combined example chains. |
| Outcome: | The proposed framework outperforms baseline models on diverse reasoning datasets and shows that it is scalable and can be used in future research. |
It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. |
| Approach: | They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting. |
| Outcome: | The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets. |
From Discrimination to Generation: Low-Resource Intent Detection with Language Model Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies fine-tune discriminative models on specific defined intent classes, preventing them from being directly adopted to new intent domains. |
| Approach: | They propose to use a pre-trained generative intent model to detect new intents from different domains with no parameter updates. |
| Outcome: | The proposed model outperforms baselines that need further fine-tuning or domain-specific samples. |
Efficient Continual Pre-training for Building Domain Specific Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are typically trained entirely on domain corpus to excel at handling domain-specific tasks. |
| Approach: | They propose a continual pre-training strategy to build domain-specific LLMs over existing open-domain LLM. |
| Outcome: | The proposed model outperforms existing LLMs with 10% of corpus size and cost without any degradation on open-domain tasks. |
Distantly-Supervised Joint Extraction with Noise-Robust Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to identifying entity pairs and relations with a single model are noisy . Existing methods only consider one source of noise or make decisions using external knowledge . |
| Approach: | They propose a framework that aligns entity mentions with corresponding tags for joint extraction . they propose DENRL, which employs a lightweight transformer backbone for joint tagging . |
| Outcome: | The proposed framework outperforms baseline models on two benchmark datasets with better interpretability. |
LLM Factoscope: Uncovering LLMs’ Factual Discernment through Measuring Inner States (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) produce outputs that deviate from factual reality, especially in sensitive applications such as medical consultation and legal advice. |
| Approach: | They propose a Siamese network-based model that leverages LLMs’ inner states for factual detection. |
| Outcome: | The proposed model achieves over 96% accuracy on a custom-collected factual detection dataset. |
DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured data processing is a complex and complex process. |
| Approach: | They propose a framework that captures heterogeneity of structured data using large language models . they propose group positional encoding, hierarchical attention bias and optimal transport alignment layer . |
| Outcome: | The proposed framework outperforms baseline methods and few-shot GPT-4 on a medical lab report dataset. |
imapScore: Medical Fact Evaluation Made Easy (2024.findings-acl)
Copied to clipboard
| Challenge: | Automated evaluation of natural language generation tasks fails to focus on medical QA because of the diversity in medical terminology. |
| Approach: | They propose a new data structure, imap, to capture key information in questions and answers. |
| Outcome: | The proposed model outperforms state-of-the-art metrics in correlation with human scores. |
Making Harmful Behaviors Unlearnable for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are often customized by fine-tuning for the requirements of different domains. |
| Approach: | They propose a controllable training framework to make undesired behaviors unlearnable during the fine-tuning process. |
| Outcome: | The proposed framework makes undesired behaviors unlearnable during the fine-tuning process while preserving the ability to learn other information. |
Debiasing Large Language Models with Structured Knowledge (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce biases in pre-training models are hampered by their performance. |
| Approach: | They propose a method that utilizes structured knowledge to mitigate bias in LLMs . their method obviates the need for training from scratch, thus offering enhanced scalability . |
| Outcome: | The proposed method outperforms state-of-the-art (SOTA) baselines in the debiasing ability. |
Contrastive Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Current LLMs exhibit limited robustness to unseen instructions, generating inconsistent outputs when the same instruction is phrased with slightly varied forms or language styles. |
| Approach: | They propose a method which maximizes the similarity between the hidden representations of semantically equivalent instruction-instance pairs while minimizing the similarities between semantically different ones. |
| Outcome: | Experiments on the PromptBench benchmark show that Contrastive Instruction Tuning improves LLMs’ robustness to unseen instructions with variations across character, word, sentence, and semantic levels by +2.5% in accuracy. |
Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for document retrieval rely on static document identifiers . experimental results show that generative retrieval is outperforms dense retrieval in document retrievals. |
| Approach: | They propose a bootstrapped pre-training method that dynamically adjusts document identifiers during pre-train to accommodate the continuing memorization of the corpus. |
| Outcome: | The proposed method significantly outperforms existing pre-training generative retrieval baselines and performs well even in zero-shot settings. |
Refining and Synthesis: A Simple yet Effective Data Augmentation Framework for Cross-Domain Aspect-based Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Aspect-based Sentiment Analysis (ABSA) data augmentation has attracted increasing attention in recent years due to data sparsity. |
| Approach: | They propose a framework to augment ABSA data using pseudo labels for target domain . they refine generated labeled data using a natural language inference filter . |
| Outcome: | The proposed framework outperforms 7 strong baselines on 4 kinds of ABSA tasks. |
Codec-SUPERB: An In-Depth Analysis of Sound Codec Models (2024.findings-acl)
Copied to clipboard
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander Liu, Hung-yi Lee
| Challenge: | Researchers have developed a sound codec that can be used as tokenizers for preserving audio data and minimizing data transmission latency. |
| Approach: | They propose to use codec-SUPERB to assess codec models across representative sound applications and signal-level metrics rooted in sound domain knowledge. |
| Outcome: | The proposed codec-SUPERB model is evaluated on selected experimental settings. |
CACL: Community-Aware Heterogeneous Graph Contrastive Learning for Social Media Bot Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for social media bot detection neglect community structure and poor model generalization due to the relatively small scale of the dataset. |
| Approach: | They propose a framework that constructs social networks as heterogeneous graphs and uses community-aware modules to mine hard positive and hard negative samples for supervised graph contrastive learning. |
| Outcome: | The proposed framework outperforms baselines on three social media bot benchmarks. |
Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evidence that humans make numerous inferences to understand discourse and text is not fully understood. |
| Approach: | They propose to use textual inference datasets with multi-sentence premises to solve the entailment verification problem. |
| Outcome: | The proposed model outperforms GPT-3.5 and rivals GPL-4 in EV tasks. |
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts provide visual representations of data and are used for analyzing information, addressing queries, and conveying insights to others. |
| Approach: | They propose a chart-specific vision-language Instruction-following dataset with 191K instructions and a pipeline model that extracts chart data tables and inputs them into a LLM. |
| Outcome: | The proposed model can solve a wide range of chart-related tasks, achieving state-of-the-art results on four tasks. |
Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models do not differentiate between semantic and linguistic features, resulting in the entanglement of knowledge and linguistics within the model. |
| Approach: | They propose to exploit both semantic and linguistic features to enhance multilingual translation by disentangling encoder representations and integrating low-level linguistic encoders. |
| Outcome: | The proposed model improves zero-shot translation while maintaining performance in supervised translation on multilingual datasets. |
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining. |
| Approach: | They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes. |
SharedCon: Implicit Hate Speech Detection using Shared Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies suggest that classifying hateful posts in a binary manner may not address nuanced task of detecting implicit hate speech. |
| Approach: | They propose a contrastive learning approach that leverages shared semantics among data to detect implicit hate speech. |
| Outcome: | The proposed approach is based on a clustering-based contrastive learning approach with human-written implications or machine-generated augmented data. |
Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction tuning language models can be expensive and expensive to train . current methods require extensive training on large datasets, resulting in high training costs. |
| Approach: | They propose a novel approach to selecting training data based on the learning percentage of the samples. |
| Outcome: | The proposed model performs better on models ranging from 1B to 13B in size compared to training on the entire dataset. |
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work has embodied LLMs as agents, but external content introduces the risk of indirect prompt injection (IPI) attacks. |
| Approach: | They propose a benchmark to assess the vulnerability of tool-integrated LLM agents to IPI attacks. |
| Outcome: | The proposed benchmark assesses the vulnerability of tool-integrated LLM agents to IPI attacks. |
Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing CodePre-trained models struggle to generalize due to superficial mapping from source code to labels instead of understanding the root causes of code vulnerabilities. |
| Approach: | They propose a framework that integrates multi-task learning with Large Language Models to effectively mine deep-seated vulnerability features. |
| Outcome: | The proposed framework surpasses seven state-of-the-art models in effectiveness, generalization, and robustness. |
PPTSER: A Plug-and-Play Tag-guided Method for Few-shot Semantic Entity Recognition on Visually-rich Documents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visually-rich document information extraction are limited . Xu et al., 2020: visually rich document information is a vital aspect of document understanding . |
| Approach: | They propose a plug-and-play Tag-guided method for few-shot Semantic Entity Recognition (PPTSER) on visually-rich documents. |
| Outcome: | The proposed method outperforms fine-tuning and few-shot methods on visual-rich documents. |
LLM Performance Predictors are good initializers for Architecture Search (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have diverse applications, encompassing both open-ended tasks (e.g., brainstorming and chat) and closed-ended ones (eg. question answering). |
| Approach: | They construct PP prompts for Large Language Models (LLMs) that estimate the performance of specific deep neural network architectures on downstream tasks. |
| Outcome: | The proposed model achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors in machine translation tasks. |
MODDP: A Multi-modal Open-domain Chinese Dataset for Dialogue Discourse Parsing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmark datasets for discourse parsing are domain-specific and contain only textual modality . this makes it difficult to accurately understand the dialogue without multi-modal clues . |
| Approach: | They propose a multi-modal Chinese discourse parsing dataset based on open-domain dialogues . they propose to integrate multi-modality into the original textual unimodal DDP model . |
| Outcome: | The proposed dataset improves on the existing unimodal model by adding multimodalities to the model. |
Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for language analysis are inadequate for specialized domains like psychology. |
| Approach: | They have enriched a Chinese social media database with psychological lexicons to enhance its applicability to psychological text analysis. |
| Outcome: | The proposed model performed better on six public datasets and provided relevant predictions given the masked sentences. |
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches to language model alignment assume homogeneous human preferences, but actual human preferences vary widely and are hard to satisfy with a single language model. |
| Approach: | They propose an RL-free extension of Direct Preference Optimization (DPO) that folds language modeling directly into reward modeling and trains language models as collective reward models that combine all objectives with specific weights. |
| Outcome: | The proposed method matches or outperforms existing methods in safety alignment and long-form question answering. |
DORY: Deliberative Prompt Recovery for LLM (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used for their groundbreaking performance across various tasks. |
| Approach: | They propose a method that leverages uncertainty to recover prompts accurately using a single LLM without external resources or models. |
| Outcome: | The proposed approach outperforms baselines across diverse LLMs and prompt benchmarks and establishes a new state-of-the-art record in prompt recovery tasks. |
STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for addressing ambiguities in conversational search systems are one-size-fits-all and struggle to achieve effective domain transferability. |
| Approach: | They propose a method to provide search engines with strategies regarding when to ask clarification questions in a post-hoc manner. |
| Outcome: | The proposed method improves search performance 10% on four unseen domains. |
Evaluating Robustness of Generative Search Engine on Adversarial Factoid Questions (2024.findings-acl)
Copied to clipboard
Xuming Hu, Xiaochuan Li, Junzhe Chen, Yinghui Li, Yangning Li, Xiaoguang Li, Yasheng Wang, Qun Liu, Lijie Wen, Philip Yu, Zhijiang Guo
| Challenge: | Existing large language models (LLMs)-backed generative search engines may not always be accurate. |
| Approach: | They propose to evaluate the robustness of retrieval-augmented generation in a realistic and high-risk setting where adversaries have only black-box system access. |
| Outcome: | The proposed model exhibits higher susceptibility to factual errors compared to LLMs without retrieval. |
Automatic Engineering of Long Prompts (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research has explored automatic prompt engineering for short prompts, typically consisting of one or a few sentences. |
| Approach: | They propose an algorithm that automatically improves long prompts by combining a greedy algorithm with beam-search to enhance the effectiveness of LLM-based mutation. |
| Outcome: | The proposed algorithm achieves 9.2% accuracy gain on eight tasks in Big Bench Hard and consistent improvements on GSM8K with various models. |
AS-ES Learning: Towards efficient CoT learning in small models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to induce Chain-of-Thought (CoT) in LLMs are limited and do not consider the importance of efficiently utilizing existing CoT data. |
| Approach: | They propose a new training paradigm which exploits the inherent information in CoT for iterative generation. |
| Outcome: | The proposed training paradigm surpasses direct seq2seq training on CoT-extensive tasks without data augmentation or altering the model itself. |
II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on assessing the model’s overall accuracy without evaluating it on different reasoning cases. |
| Approach: | They propose a novel idea to identify and improve multi-modal multi-hop reasoning in VQA by using two new language prompts to find a reasoning path to reach its answer. |
| Outcome: | The proposed model improves multi-modal multi-hop reasoning in visual question answering (VQA) it finds that the proposed model is easy to answer, simply demanding “single-hop” reasoning, whereas only a few questions require “multi-hop.” |
TAME-RD: Text Assisted Replication of Image Multi-Adjustments for Reverse Designing (2024.findings-acl)
Copied to clipboard
Pooja Guhan, Uttaran Bhattacharya, Somdeb Sarkhel, Vahid Azizi, Xiang Chen, Saayan Mitra, Aniket Bera, Dinesh Manocha
| Challenge: | a new model to reverse design images can be used to replicate image edits on other images based on human instructions in natural language . a study of a dataset of 100K source and edited images shows improvements in accuracy and concordance correlation coefficient . |
| Approach: | They propose a reverse-designing model that automatically learns from image editing operations and natural language instructions to learn fully specified edit operations. |
| Outcome: | The proposed model improves accuracy and concordance correlation scores on multiple datasets. |
Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a meta-optimization process that affects performance . we develop a batch-based inference algorithm that is order-agnostic to ICL examples . |
| Approach: | They develop an order-agnostic inference algorithm that aggregates ICL examples in batches . they find it outperforms most permutations of ICL, and it even exceeds the best order . |
| Outcome: | The proposed method outperforms standard ICL examples while reducing computational resources. |
IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages (2024.findings-acl)
Copied to clipboard
Tahir Javed, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Deovrat Mehendale, Ishvinder Sethi, Aparna Ananthanarayanan, Hafsah Faquih, Pratiti Palit, Sneha Ravishankar, Saranya Sukumaran, Tripura Panchagnula, Sunjay Murali, Kunal Gandhi, Ambujavalli R, Manickam M, C Vaijayanthi, Krishnan Karunganni, Pratyush Kumar, Mitesh Khapra
| Challenge: | Using INDICVOICES, we build the first ASR model to support all 22 languages listed in the 8th Schedule of the Constitution of India. |
| Approach: | They propose a dataset of natural and spontaneous speech from 16237 speakers covering 145 Indian districts and 22 languages. |
| Outcome: | The proposed dataset contains 7348 hours of read, extempore and conversational audio from 16237 speakers covering 145 Indian districts and 22 languages. |
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual commonsense reasoning (VCR) use pre-trained large language models and pre-training visionlanguage models. |
| Approach: | They propose a collaborative approach where pre-trained LLMs serve as problem classifiers to analyze problem category and either use VLMs to answer directly or actively instruct LLM to gather relevant visual elements to support potential commonsense inferences. |
| Outcome: | The proposed approach outperforms all other methods without in-domain fine-tuning on two VCR benchmark datasets. |
Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm (2024.findings-acl)
Copied to clipboard
Yuanzhen Xie, Xinzhou Jin, Tao Xie, Matrixmxlin Matrixmxlin, Liang Chen, Chenyun Yu, Cheng Lei, Chengxiang Zhuo, Bo Hu, Zang Li
| Challenge: | In-context learning of large-language models has achieved remarkable success in the field of natural language processing . however, the single-step chain-of-thought prompting approach faces challenges such as attention diffusion and inadequate performance in complex tasks like text-to-SQL. |
| Approach: | They propose a workflow paradigm method to enhance the attention and problem-solving scope of large-language models through decomposition. |
| Outcome: | The proposed method outperforms existing methods on three datasets and improves the upper limit of LLM-based approaches. |
Unveiling Opinion Evolution via Prompting and Diffusion for Short Video Fake News Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for short video fake news detection ignore the implicit opinions and evolving nature of opinions across modalities. |
| Approach: | They propose a short video fake news model that mines implicit opinions within short videos and promotes the evolution of both explicit and implicit opinions across all modalities. |
| Outcome: | The proposed model outperforms existing methods on a publicly available dataset for short video fake news detection. |
iSign: A Benchmark for Indian Sign Language Processing (2024.findings-acl)
Copied to clipboard
Abhinav Joshi, Romit Mohanty, Mounika Kanakanti, Andesha Mangla, Sudeep Choudhary, Monali Barbate, Ashutosh Modi
| Challenge: | Indian Sign Language has limited resources for developing machine learning and data-driven approaches for automated language processing. |
| Approach: | They propose to use a sign language dataset to provide a benchmark for Indian Sign Language processing. |
| Outcome: | The proposed benchmarks will help improve sign language translation models and open up various ways for advancing natural language processing. |
Data Contamination Calibration for Black-box LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Despite the rapid advancements of Large Language Models, the unchecked ultra-large-scale training sets introduce a series of potential risks like data contamination. |
| Approach: | They propose a method to detect contaminated training data and diminish the contamination effect by using a to-be-released dataset. |
| Outcome: | The proposed method outperforms existing methods by at least 4.5% on more 4 dataset formats, with more than 10 base LLMs. |
Truth-Aware Context Selection: Mitigating Hallucinations of Large Language Models Being Misled by Untruthful Contexts (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are easily misled by untruthful contexts provided by users or knowledge augmentation tools, leading to hallucinations. |
| Approach: | They propose a lightweight method to adaptively recognize and mask untruthful context from the inputs and a new evaluation metric to further study the LLMs’ ability to accept truthful information and resist untrusted information. |
| Outcome: | The proposed method can detect and mask untruthful context from the inputs and significantly improve the quality of LLMs’ responses when presented with misleading information. |
Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on sentence-level translation have focused on document level machine translation (DOCMT) document level translation is a complex task different from sentence- level translation. |
| Approach: | They propose a Context-Aware Prompting method which generates more accurate, coherent translations via in-context learning. |
| Outcome: | The proposed method is effective in literary translation tasks and zero pronoun translation tasks. |
Improving Grammatical Error Correction via Contextual Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Increasing use of synthetic data due to inconsistent error distribution and noisy labels is limiting the use of these data. |
| Approach: | They propose a method for augmentation of synthetic data with a more consistent error distribution. |
| Outcome: | The proposed method outperforms strong baselines and achieves state-of-the-art with only a few synthetic data. |
RECOST: External Knowledge Guided Data-efficient Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Considering the high computing power overhead, data-efficient instruction tuning is proposed to reduce the training data size. |
| Approach: | They propose a framework to improve instruction tuning by integrating external knowledge into a single pipeline. |
| Outcome: | The proposed method achieves better results with only 1% of the full dataset. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of lifelong prompt tuning are limited due to inherent challenges. |
| Approach: | They propose a similarity heuristic lifelong prompt tuning framework that partitions tasks into two distinct subsets by harnessing a learnable similarity measure. |
| Outcome: | The proposed framework outperforms state-of-the-art techniques in lifelong learning benchmarks and demonstrates robustness against negative transfer in diverse task sequences. |
PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated considerable capabilities across various tasks . however, they often fall short of the performance achieved by domain-specific state-of-the-art models . |
| Approach: | They propose a tuning-free method to augment domain-specific abilities of Large language models . they leverage insights from the response preference of expert models to augment LLMs . |
| Outcome: | The proposed method outperforms the expert model on 4 ScienceWorld tasks. |
Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction (2024.findings-acl)
Copied to clipboard
Albert Sawczyn, Katsiaryna Viarenich, Konrad Wojtasik, Aleksandra Domogała, Marcin Oleksy, Maciej Piasecki, Tomasz Kajdanowicz
| Challenge: | Existing KBQA datasets are outdated and inefficient in human labor, and assisting tools like Large Language Models (LLM) are not utilized to reduce the workload. |
| Approach: | They propose a semi-automated question answering task that uses structured knowledge graphs to answer extensive knowledge-intensive questions. |
| Outcome: | The proposed approach includes KBQA, MRC, and Information Retrieval tasks for low-resource languages. |
Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Generating accurate SQL queries for user questions rely on the capability of large language models (LLMs) however, some knowledge is not explicitly included in the database schema and user question or has been learned by LLMs. |
| Approach: | They propose a Knowledge-to-SQL framework that employs tailored Data Expert LLM (DELLM) to provide helpful knowledge for all text-to SQL models. |
| Outcome: | The proposed framework improves the state-of-the-art approaches for text-to-SQL tasks by leveraging a data expert LLM (DELLM) to provide useful knowledge for all text- to-SqL models. |
Centroid-Based Efficient Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. |
| Approach: | They propose a centroid-based MBR decoding method that clusters the translations in the feature space and calculates the expected score using the centroids of each cluster. |
| Outcome: | The proposed method outperforms vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT’22 EnJa, EnDe, EnZh, and WMT'23 Enja translation tasks. |
Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration (2024.findings-acl)
Copied to clipboard
Han Cheng Yu, Yu An Shih, Kin Man Law, KaiYu Hsieh, Yu Chen Cheng, Hsin Chih Ho, Zih An Lin, Wen-Chuan Hsu, Yao-Chung Fan
| Challenge: | Existing LMs undergo task-agnostic pertaining, but task-specific pretraining has gained prominence. |
| Approach: | They propose retrieval augmented pretraining and task-specific pretraining for DG . they propose to refine language model pretraining to align it more closely with downstream task . |
| Outcome: | The proposed method improves the performance of multiple-choice questions by integrating knowledge graphs and language models. |
Exploiting Positional Bias for Query-Agnostic Generative Content in Search (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that neural ranking models outperform lexical models in text retrieval. |
| Approach: | They propose to exploit transformer attention mechanism to induce exploitable defects in search models through sensitivity to token position within a sequence. |
| Outcome: | The proposed model can generalise beyond a single query or topic without knowledge of topicality. |
ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to curation text-image data are noisy and lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. |
| Approach: | They propose a metric that evaluates caption text without an image reference to measure its concreteness and relevancy. |
| Outcome: | The proposed method detects the concreteness of captions without an image reference and correlates with human evaluation of concreteness in both single-word and caption-level texts. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
When is a Language Process a Language Model? (2024.findings-acl)
Copied to clipboard
| Challenge: | In some pathological situations, such a stochastic process may "leak" probability mass onto the set of infinite strings. |
| Approach: | They propose to view a language model as a discrete stochastic process X t : t = = t + . |
| Outcome: | The proposed conditions of tightness are generalized to language models and the literature. |
Accelerating Multilingual Language Model for Excessively Tokenized Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown a significant degree of multilingual proficiency on a variety of tasks in multiple languages. |
| Approach: | They propose a framework to fine-tune a language model head and fine-track it while preserving its performance. |
| Outcome: | The proposed framework increases the generation speed by 1.7 while maintaining the performance of pre-trained multilingual models on target monolingual tasks. |
Definition Generation for Automatically Induced Semantic Frame (2024.findings-acl)
Copied to clipboard
| Challenge: | Semantic frames are conceptual structures that describe specific types of situations or events. |
| Approach: | They propose to generate frame definitions from a set of frame-evoking words using a large language model. |
| Outcome: | The proposed task incorporates frame element reasoning as chain-of-thought to enhance the inclusion of correct frame elements in the generated definitions. |
Distillation Enhanced Generative Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. |
| Approach: | They propose a framework that leverages generative language models to enhance generative retrieval by distillation. |
| Outcome: | The proposed framework achieves state-of-the-art performance among the generative retrieval methods. |
ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a dataset of 931 videos with 4021 code-mixed Hindi-English utterances, we find that video content with multiple modalities is more accurate and more accurate than textual content. |
| Approach: | They propose to use a dataset to analyze toxic content in video content in non-English languages by leveraging language models. |
| Outcome: | The proposed framework achieves an Accuracy and Weighted F1 score of 94.29% and 94.35% for the first time in its class. |
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2024.findings-acl)
Copied to clipboard
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, Yang Liu
| Challenge: | Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning. |
| Approach: | They propose a virtual API server and stable evaluation system to assess the stability of large-scale real-time APIs. |
| Outcome: | The proposed benchmarks demonstrate the stability of the proposed system and its caching system. |
Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence (2024.findings-acl)
Copied to clipboard
| Challenge: | Emotional Intelligence (EI) is a key concept in the field of human intelligence. |
| Approach: | They propose a method to enhance EI of large language models by naive fine-tuning on EI-related tasks. |
| Outcome: | The proposed method improves EI of two LLM-based assistants without compromising GI. |
KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) must possess an understanding of the nation’s culture and basic knowledge. |
| Approach: | They propose to construct a national alignment benchmark, KorNAT, which measures the alignment between an LLM and a targeted country from two perspectives: social value alignment and common knowledge alignment. |
| Outcome: | The proposed model passes the national alignment score of 7 LLMs, indicating there is room for improvement. |
Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development (2024.findings-acl)
Copied to clipboard
| Challenge: | ADEs are a serious public health concern and cost healthcare systems billions of dollars . despite advancements in healthcare, ADE detection remains a significant challenge . |
| Approach: | They propose a multimodal adverse drug event detection dataset that merges ADE-related textual information with visual aids to enhance patient safety. |
| Outcome: | The proposed dataset integrates ADE-related textual information with visual aids to improve patient safety and healthcare accessibility. |
Space Decomposition for Sentence Embedding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to measure sentence pair similarity are based on a continuous semantic textual similarity scale . however, the score in the range [4,5] indicates an upper-range sample, while the rest are lower-range samples. |
| Approach: | They propose a method to decompose sentences into embedding space space . they use a mixture of specialized projectors to distinguish and rank upper-range and lower-range samples . |
| Outcome: | The proposed method outperforms existing methods on STS and zero-shot benchmarks while reducing overlap between upper-range and lower-range classes. |
Don’t Augment, Rewrite? Assessing Abusive Language Detection with Synthetic Data (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for abusive language detection and content moderation are limited by regulatory bodies and social media platforms. |
| Approach: | They propose to replace existing datasets in English with synthetic data by rewriting original texts with an instruction-based generative model. |
| Outcome: | The proposed model improves performance in cross-dataset training. |
Improving Low-Resource Machine Translation for Formosan Languages Using Bilingual Lexical Resources (2024.findings-acl)
Copied to clipboard
| Challenge: | Using bilingual lexicons for low-resource languages can improve machine translation for low resource languages. |
| Approach: | They propose to use bilingual lexicons to improve machine translation for low-resource languages . they use parallel data and bilingual dictionaries to generate pseudo-parallel sentences . |
| Outcome: | The proposed techniques improve translation between Mandarin and Formosan languages and Spanish and Nahuatl, a language pair consisting of languages from completely different language families. |
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for long-form text outputs are prone to biases and scaling up is expensive. |
| Approach: | They propose to evaluate VLMs with VLM feedback dataset . they use 15K customized score rubrics to train Prometheus-Vision . |
| Outcome: | The proposed model shows highest correlation with human evaluators and GPT-4V among open-source models. |
Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations focus on problem-solving from examiner perspective, overlooking a dual perspective of examiner regarding error identification and correction. |
| Approach: | They propose to use an annotated dataset to evaluate large language models from the examiner perspective and to use diverse prompts to evaluate eleven representative LLMs. |
| Outcome: | The proposed model outperforms all models while LLaMA-2-7B has comparable abilities to closed-source models GPT-3.5 and Gemini Pro. |
Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing pruning algorithms suffer from limitations such as architecture specificity and reliance on demanding calculations. |
| Approach: | They propose a pruning algorithm based on Kernel Density Estimation . it preserves most significant parameters while restoring others to their pre-training state . |
| Outcome: | The proposed pruning algorithm achieves better performance than the original unpruned version. |
When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have difficulty knowing they do not possess certain knowledge and tend to provide specious answers in such cases. |
| Approach: | They propose to use Retrieval Augmentation to enhance LLMs' ability to perceive their knowledge boundaries to reduce overconfidence. |
| Outcome: | The proposed methods reduce overconfidence and improve accuracy in large language models with fewer retrieval calls. |
Hybrid Alignment Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with instructions and preferences are conflicting . et al., 2023b) show that hybrid alignment training can outperform baselines . |
| Approach: | They propose a hybrid alignment training approach based on alternating alignment and modified elastic weight consolidation methods to achieve better collaboration between different alignment tasks. |
| Outcome: | The proposed approach outperforms baseline alignment training methods on summarization and dialogue tasks. |
Graph-Structured Speculative Decoding (2024.findings-acl)
Copied to clipboard
Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan
| Challenge: | Speculative decoding is a promising technique to accelerate the inference of Large Language Models. |
| Approach: | They propose a method that uses a token graph to record multiple sequence hypotheses within a single draft stage. |
| Outcome: | The proposed method significantly accelerates the inference of Large Language Models (LLMs) it allows the LLM to choose from and select the longest sequence that meets its standards. |
Duwak: Dual Watermarks in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing watermark techniques are effective in embedding single human-imperceptible and machine-detectable patterns without significantly affecting generated text quality and semantics. |
| Approach: | They propose to embed dual secret patterns in token probability distribution and sampling schemes to enhance the efficiency of watermarking. |
| Outcome: | The proposed method achieves highest watermark quality at the lowest required token count for detection, up to 70% less than existing techniques, especially under post paraphrasing attacks. |
CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models exhibit remarkable generative capabilities but can be misused for harmful purposes. |
| Approach: | They propose a framework that transforms natural language inputs into code inputs. |
| Outcome: | The proposed framework bypasses the safety guardrails of all models more than 80% of the time. |
Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across diverse tasks, but suffer from the "reversal curse" this limitation poses a challenge to the advancement of artificial general intelligence (AGI) |
| Approach: | They propose to use training data to permute training sentences into entities and feed them into the model. |
| Outcome: | The proposed method improves the performance of large language models (LLMs) on reversed questions and improves existing models. |
wav2vec-S: Adapting Pre-trained Speech Models for Streaming (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained speech models have advanced speech-related tasks, including speech recognition and translation. |
| Approach: | They propose a pre-trained speech model that incorporates modifications to ensure consistent speech representations during training and inference phases for streaming speech inputs. |
| Outcome: | The proposed model outperforms baseline models on speech recognition and translation tasks and achieves a superior balance between quality and latency. |
Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used for question answering . lack of explicit references or attributions hinders ability to verify accuracy of answers . |
| Approach: | They propose a method for attribution in contextual question answering . they use hidden state representations of large language models to identify copied segments . |
| Outcome: | The proposed method performs better than GPT-4 at identifying verbatim copied segments in LLM generations and attributing these segments to their source. |
TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. |
| Approach: | They propose a method that uses adversarial suffixes to get an answer from a target LLM. |
| Outcome: | The proposed method detects the LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models’ Theory-of-Mind (2024.findings-acl)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is the foundation of social interaction and is crucial for social interaction. |
| Approach: | They propose a tool-belief solver that can transform a character’s higher-order beliefs into another character’ s first-order belief under belief communication period. |
| Outcome: | The proposed model improves the ToM capabilities of Large Language Models (LLMs) in multiple scenarios. |
Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis (2024.findings-acl)
Copied to clipboard
Sitiporn Sae Lim, Can Udomcharoenchaikit, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Current NLU models obtain state-of-the-art accuracy on in-distribution benchmarks, but they use annotation bias to make predictions, negatively affecting the models' generalizability. |
| Approach: | They apply causal mediation analysis to gauge how much each component mediates annotation biases and use causal-grounded masking and gradient unlearning to mitigate bias. |
| Outcome: | The proposed methods improve the model's robustness against annotation bias even after employing other training-time debiasing techniques. |
Perturbed examples reveal invariances shared by language models (2024.findings-acl)
Copied to clipboard
| Challenge: | Rapid growth in natural language processing (NLP) research has led to numerous new models outpacing our understanding of how they compare to established ones. |
| Approach: | They propose a framework to compare two NLP models by revealing their shared invariance to interpretable input perturbations targeting a specific linguistic capability. |
| Outcome: | The proposed framework can shed light on the types of invariances retained or emerging in new models. |
Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Stochastic sampling strategies are not widely used in open-domain dialogue systems. |
| Approach: | They propose a dynamic decoding strategy which can adjust the decoding space w.r.t. different contexts. |
| Outcome: | The proposed decoding strategy can improve the performance of pre-trained models when coupled with four well-used stochastic decoding algorithms. |
Discourse Structure-Aware Prefix for Generation-Based End-to-End Argumentation Mining (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in AM models overlook the integration of supplementary discourse structure information, resulting in suboptimal outcomes. |
| Approach: | They propose a framework which generates discourse structure-aware prefixes for each layer of the generation model. |
| Outcome: | The proposed framework achieves state-of-the-art performance on two AM benchmarks. |
Poor-Supervised Evaluation for SuperLLM via Mutual Consistency (2024.findings-acl)
Copied to clipboard
| Challenge: | evaluating superLLMs is especially difficult because of their intelligence-intensive nature. |
| Approach: | They propose an evaluation benchmark with accurate labels for SuperLLMs whose capabilities surpass those of humans . they first prove that consistency between model under evaluation and reference model can equalize the true capabilities of the model to be evaluated . |
| Outcome: | The proposed evaluation benchmarks can assess the true capabilities of the model to be evaluated without accurate labels. |
Addressing Entity Translation Problem via Translation Difficulty and Context Diversity (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation systems often produce inadequate translations for named entities. |
| Approach: | They propose a data augmentation strategy to enhance the accuracy of named entity translation by retraining the target named entity pair. |
| Outcome: | The proposed method improves translation accuracy across test sets and terminology tests. |
ADAM: Dense Retrieval Distillation with Adaptive Dark Examples (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to retrieve data from multiple encoders are too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student. |
| Approach: | They propose a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with adaptive dark examples. |
| Outcome: | The proposed framework can better transfer the dark knowledge held in the teacher with adaptive dark examples. |
Instruction Position Matters in Sequence Generation with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. |
| Approach: | They propose to shift the position of task instructions after the input sentences to enhance the model's instruction-following capability. |
| Outcome: | The proposed method outperforms traditional settings across various model scales (1B / 7B & 13B) and different sequence generation tasks (translation and summarization) without any additional data or annotation costs. |
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | XMoE leverages small experts and a threshold-based router to selectively engage only essential parameters. |
| Approach: | They propose a novel MoE that leverages small experts to selectively engage only essential parameters. |
| Outcome: | The proposed model can reduce computation load at MoE layers by over 50% without sacrificing performance. |
BranchNorm: Robustly Scaling Extremely Deep Transformers (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on DeepNorm scales Transformers into extremely deep (1000 layers) due to the training instability of Transformers, the depths of these SOTA models are still relatively shallow. |
| Approach: | They propose a branch-rescaled model which dynamically rescales the non-residual branch of Transformer in accordance with the training period. |
| Outcome: | The proposed approach significantly outperforms existing shallow models on multiple translation tasks and achieves better training stability and convergent performance. |
MusTQ: A Temporal Knowledge Graph Question Answering Dataset for Multi-Step Temporal Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on fact-centered reasoning with limited attention to temporal reasoning. |
| Approach: | They propose a new TKGQA dataset, MusTQ, which contains 666K multi-step temporal reasoning questions and a TKG. |
| Outcome: | The proposed model achieves state-of-the-art multi-step temporal reasoning ability with entity-time attention mechanism and optimized temporal knowledge graph representation. |
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. |
| Approach: | They propose to calibrate language models to better represent outcome uncertainty . they propose to use two methods to calibrated small open-source models . |
| Outcome: | The proposed fine-tuning strategies can calibrate smaller open-source models to beat pre-trained models 10x their size. |
Knowledge Fusion By Evolving Weights of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimental results on mainstream language models show that Evolver outperforms previous state-of-the-art models by large margins due to the high training costs of large language models. |
| Approach: | They propose a method to integrate multiple models from diverse training scenarios into a unified model. |
| Outcome: | The proposed method outperforms state-of-the-art models on mainstream language models by large margins. |
ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has shown significant practical benefits when using language models . current two stage MTL introduces a substantial number of additional parameters . |
| Approach: | They propose a multi-task learning method that leverages existing knowledge for a target task. |
| Outcome: | The proposed method outperforms baselines on three benchmarks and two encoder LMs with a small number of transfer parameters. |
Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for dialogue-to-image retrieval are constrained by pre-trained vision language models. |
| Approach: | They leverage the reasoning capabilities of large language models to predict potential features in images to be shared based on dialogue context. |
| Outcome: | The proposed method outperforms existing methods significantly in terms of Recall@k. |
MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization (2024.findings-acl)
Copied to clipboard
Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, Maosong Sun
| Challenge: | Scientific data visualization is an essential process in research, but its use of large language models remains unexplored. |
| Approach: | They propose a model-agnostic LLM agent framework to automate scientific data visualization tasks. |
| Outcome: | The proposed framework improves performance of commercial and open-source models. |
Continual Few-shot Relation Extraction via Adaptive Gradient Correction and Knowledge Decomposition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to learn new relations with limited samples neglect the instability of embeddings in the process of different task training, which leads to catastrophic forgetting. |
| Approach: | They propose a method to analyze catastrophic forgetting by limiting embedding instability . they propose to decompose knowledge into general and task-related knowledge . |
| Outcome: | The proposed method outperforms the state-of-the-art model and improves the following degree of embeddings. |
CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models (2024.findings-acl)
Copied to clipboard
Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwang Song, Tingting Cui, Xiaoqing Cheng, Liutao Liutao, Deyi Xiong
| Challenge: | Recent years have witnessed remarkable progress achieved by large language models in both natural language understanding and generation. |
| Approach: | They propose a large benchmark CMoralEval for moral evaluation of Chinese LLMs . they use a Chinese TV program discussing Chinese moral norms and Chinese moral anomies based on various sources . |
| Outcome: | The proposed dataset is characterized by diversity and authenticity. |
Cache & Distil: Optimising API Calls to Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models are expensive to run and expose the entire request stream to external providers. |
| Approach: | They propose to locally train a small private language model on the LLM's predictions to minimise the costs and data exposure associated with calling the API. |
| Outcome: | The proposed model can handle an increasing number of user requests independently and is able to perform better than other policies and baselines across tasks and budgets. |
Investigating the Impact of Model Instability on Explanations and Uncertainty (2024.findings-acl)
Copied to clipboard
| Challenge: | Explainable AI methods are typically evaluated holistically, but small perturbations to inputs can vastly distort explanations. |
| Approach: | They artificially simulate epistemic uncertainty in text input by introducing noise at inference time and measure the effect on the output of pre-trained language models. |
| Outcome: | The proposed model can detect salient tokens when uncertain, but it is not reliable when small perturbations are exposed during training. |
A Two-Stage Adaptation of Large Language Models for Text Ranking (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models (PLMs) have significantly improved ranking performance in text ranking tasks. |
| Approach: | They propose a two-stage progressive paradigm to better adapt LLMs to text ranking by conducting continual pre-training on a large weakly-supervised corpus and performing SFT on high-quality data. |
| Outcome: | The proposed approach outperforms previous methods on in- and out-domain scenarios. |
Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models (2024.findings-acl)
Copied to clipboard
Daniela Occhipinti, Michele Marchi, Irene Mondella, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim, Marco Guerini
| Challenge: | a recent study has focused on the quality of data generated by automatic methods for fine-tuning Language Models in languages less resourced than English. |
| Approach: | They investigate whether human intervention improves the quality of machine-generated dialogues . they use a large-scale dataset to fine-tune three different sizes of an LM . |
| Outcome: | The results show that human intervention can improve the quality of training data . larger models are less sensitive to data quality, while smaller models are more sensitive . |
Analyze, Generate and Refine: Query Expansion with LLMs for Zero-Shot Open-Domain QA (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods like GAR and EAR rely heavily on supervised training and struggle to maintain effectiveness across domains and datasets. |
| Approach: | They propose a QE approach based on a three-step prompting strategy to enhance query expansion by broadening the scope of queries with additional relevant texts. |
| Outcome: | The proposed approach outperforms state-of-the-art methods in out-domain zero-shot scenarios and outperformed existing methods in end-to-end evaluations. |
On the Evaluation of Speech Foundation Models for Spoken Language Understanding (2024.findings-acl)
Copied to clipboard
Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, Shinji Watanabe
| Challenge: | Spoken language understanding evaluation (SLUE) benchmarks are used to benchmark complex spoken language understanding tasks on natural speech. |
| Approach: | They propose a set of benchmark tasks to evaluate spoken language understanding on natural speech . they use pre-trained speech foundation models to evaluate the utility of different SFMs . |
| Outcome: | The proposed framework outperforms pre-trained speech foundation models on natural speech . the proposed framework also outperformed self-supervised SFMs on the sequence generation tasks . |
Towards Multiple References Era – Addressing Data Leakage and Limited Reference Diversity in Machine Translation Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows a weak correlation between n-gram-based metrics and human evaluations in machine translation tasks. |
| Approach: | They propose to use multiple references generated by LLMs to improve alignment between automatic metrics and human evaluations. |
| Outcome: | The proposed approach improves the alignment between automatic metrics and human evaluations on the WMT22 benchmark with 4 languages and achieves a maximum accuracy gain of 9.5%. |
Prompting open-source and commercial language models for grammatical error correction of English learner text (2024.findings-acl)
Copied to clipboard
Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery
| Challenge: | Recent advances in generative AI have enabled us to prompt large language models (LLMs) to produce texts which are fluent and grammatical. |
| Approach: | They evaluate model performance by measuring their performance on established benchmarks. |
| Outcome: | The proposed models outperform supervised English GEC models on fluency correction benchmarks and commercial LLMs on edit benchmarks. |
BATS: BenchmArking Text Simplicity 🦇 (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on text simplification focus on the difference between a source text and its simplified variant. |
| Approach: | They propose to use a dataset to assess the overall simplicity of text. |
| Outcome: | The proposed method compared 15 datasets on text simplification and their impact on the overall simplicity of text. |
AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Scholars call for more personalized mechanisms of content moderation to account for multifaceted differences. |
| Approach: | They propose to annotate a news forum's user comments with a German dialect and identify their spans as vulgar language or offensive statements. |
| Outcome: | The proposed model interpretability improves on fine-tuned Transformer models and large language models in a zero-shot fashion. |
Discovering influential text using convolutional neural networks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for estimating the effects of text on human evaluation are limited to testing a small number of pre-specified text treatments. |
| Approach: | They propose a method for flexibly discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. |
| Outcome: | The proposed method can detect and predict human reactions to texts under certain assumptions. |
LC4EE: LLMs as Good Corrector for Event Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Event extraction (EE) is a critical task in natural language processing, yet deploying a practical EE system remains challenging. |
| Approach: | They propose to leverage the superior extraction capability of LLMs and instruction-following ability of LRMs to construct a robust and highly available EE system. |
| Outcome: | The proposed method can identify and correct errors in SLMs predictions based on automatically generated feedback information and improve performance. |
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Considering the vast size and wide-ranging sources of LLMs’ training data, it could explicitly or implicitly include test data. |
| Approach: | They propose a Contamination Detection via output Distribution (CDD) which detects data contamination only by identifying the peakedness of LLM's output distribution. |
| Outcome: | The proposed method improves performance by 21.8%-30.2% on humanEval and TED: trustworthy evaluation via output distribution. |
Efficient Training of Language Models with Compact and Consistent Next Token Distributions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train language models have focused on maximizing the likelihood of the next token . however, the construction and querying of such n-grams can be costly and impede training speed. |
| Approach: | They propose a method to train language models faster by pre-aggregating corpus with collapsed n-gram distribution. |
| Outcome: | The proposed model improves model quality and convergence rate while reducing variance across mini-batches compared to the standard next-token loss method. |
Ancient Chinese Glyph Identification Powered by Radical Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Currently, about half of ancient Chinese glyphs have not been deciphered yet. |
| Approach: | They propose to use a Chinese glyph knowledge graph to infer the Chinese character label for the unknown ancient Chinese . they propose to combine the visual, textual, and the graph data to create a multimodal Chinese morph identification framework. |
| Outcome: | The proposed method can identify ancient Chinese characters from 1300 BC to 200 BC based on image and radical semantics on a 1000-year-old Chinese glyph dataset. |
PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities (2024.findings-acl)
Copied to clipboard
| Challenge: | Pragmatics understanding is not well studied in LLMs, but their understanding of pragmatics is lacking. |
| Approach: | They propose to use a dataset to measure LLMs' understanding of pragmatics to evaluate their models. |
| Outcome: | The proposed dataset includes 14 tasks in four pragmatics phenomena, namely; Implicature, Presupposition, Reference, and Deixis. |
EmoTransKG: An Innovative Emotion Knowledge Graph to Reveal Emotion Transformation (2024.findings-acl)
Copied to clipboard
| Challenge: | EmoTransKG establishes connections and transformations between emotions across open-textual events. |
| Approach: | They propose an Emotion Knowledge Graph that establishes connections and transformations between emotions across diverse open-textual events. |
| Outcome: | The proposed model integrates with existing conversational emotion recognition models to improve the quality and effectiveness of EmoTransKG. |
How Vocabulary Sharing Facilitates Multilingualism in LLaMA? (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show strong performance on English tasks, but their performance in other languages is limited. |
| Approach: | They conducted an exhaustive analysis of the multilingual capability of LLMs by examining the performance gap before and after embedding fine-tuning across 101 languages. |
| Outcome: | The proposed model improves on the attributes of four quadrants in the model and provides actionable and efficient guidelines for tuning these languages. |
Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language models (LLMs) to specific preferences. |
| Approach: | They propose a training-free alignment method that uses minimal prior tokens to bridge the foundation LLM and the SFT LLM. |
| Outcome: | The proposed method achieves comparable performance without training on machine translation and part-of-speech tagging across seven languages. |
Dual Prompt Tuning based Contrastive Learning for Hierarchical Text Classification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods focus on hierarchy-aware text feature by exploiting explicit parent-child relationships, resulting in label confusion within each layer. |
| Approach: | They propose a dual-prompt tuning method which emphasizes discrimination among peer labels by performing contrastive learning on each hierarchical layer. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets and is available on github. |
Probing the Emergence of Cross-lingual Alignment during LLM Training (2024.findings-acl)
Copied to clipboard
| Challenge: | Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. |
| Approach: | They propose that LLMs can align languages without explicit supervision from parallel sentences without a single linguistic feature. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer even when the vocabularies of two languages have a null intersection, i.e., no tokens are shared. |
STSPL-SSC: Semi-Supervised Few-Shot Short Text Clustering with Semantic text similarity Optimized Pseudo-Labels (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for obtaining task-specific labels require prior knowledge of clustering categories and uncontrollable clustering centers. |
| Approach: | They propose a framework for supervised clustering using a discrete process and a robust Contrastive Learning module. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a real-world dataset with just one label per class . the proposed framework is based on k-means clustering and a robust Contrastive Learning module . |
A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantization studies have focused on instruction-tuned LLMs, leaving their performance on other benchmarks unclear. |
| Approach: | They propose a framework to evaluate quantized large language models using four dimensions . they propose to reduce the bits needed for model weights or activations with minimal performance loss . |
| Outcome: | The proposed framework can retain comparable performance to non-quantized LLMs on most benchmarks. |
Exploiting Target Language Data for Neural Machine Translation Beyond Back Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) encounters challenges when translating in new domains and low-resource languages. |
| Approach: | They propose a variant of k-nearest neighbor machine translation that utilizes target language data by constructing a pseudo datastore. |
| Outcome: | The proposed method exhibits strong domain adaptation capability in both high-resource and low-resourced machine translation. |
Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to quantify uncertainty of pre-trained large language models require specific architectures or retraining strategies. |
| Approach: | They propose a Bayesian Prompts Ensemble approach to accurately quantify LLM uncertainty . they compute output probabilities through a weighted ensemble of different task instruction prompts . |
| Outcome: | The proposed approach achieves significantly superior calibration over baselines over a range of natural language classification tasks. |
X-ACE: Explainable and Multi-factor Audio Captioning Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for automated audio captioning only provide an overall score . current evaluation checklists are inadequate to characterize the nuanced differences . |
| Approach: | They propose an explainable and multi-factor audio captioning evaluation paradigm . they define sound event, source, attribute and relation as four factors tailored for the audio description . |
| Outcome: | The proposed evaluation paradigm improves the quality of audio captions . it can detect mismatches and align with human perception, the authors show . |
Reasons to Reject? Aligning Language Models with Judgments (2024.findings-acl)
Copied to clipboard
| Challenge: | a new framework for aligning large language models with judgments is proposed to help with alignment . a framework that allows for fine-grained inappropriate content detection and correction based on judgments . large language model alignment is critical for making artificial intelligence a reliable ally for humanity . |
| Approach: | They propose a framework that allows for fine-grained inappropriate content detection and correction based on judgments. |
| Outcome: | The proposed framework beats the 175B DaVinci003 and improves on AlpacaEval using judgments. |
Decomposing Argumentative Essay Generation via Dialectical Planning of Complex Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Argumentative Essay Generation (AEG) is a challenging task in computational argumentation, where detailed logical reasoning and effective rhetorical skills are essential. |
| Approach: | They propose an argumentative planning strategy for prompting large language models to generate high-quality essays by sketch planning and dialectical planning. |
| Outcome: | The proposed method generates more dialectical and persuasive essays with higher diversity compared to baselines. |
Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on fallacy recognition is still in its early stages, with limited datasets available. |
| Approach: | They propose to use GPT3.5 to generate synthetic examples and explore prompt settings to improve the representation of the infrequent classes. |
| Outcome: | The proposed model improves on existing models and generates synthetic examples with GPT3.5. |
Concept-aware Data Construction Improves In-context Learning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work curating in-context learners assumes that ICL emerges from vast over-parametrization or the scale of multitask training. |
| Approach: | They propose a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. |
| Outcome: | The proposed framework makes it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations and fares comparably to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data. |
Beyond Text: Leveraging Multi-Task Learning and Cognitive Appraisal Theory for Post-Purchase Intention Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that user-level features can carry more task-related information than the text itself. |
| Approach: | They evaluate multi-task learning frameworks grounded in Cognitive Appraisal Theory to predict user behavior as a function of users’ self-expression and psychological attributes. |
| Outcome: | The proposed models improve on the language and traits of users, while lacking rich annotations of other attributes. |
Non-Autoregressive Machine Translation as Constrained HMM (2024.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive (AR) models have some drawbacks due to slow inference speed and label bias due to local normalization. |
| Approach: | They propose to use a left-to-right Hidden Markov Model (HMM) to control label bias in non-autoregressive translation (NAT) They propose a bi-directional HMM, which can regularize each other's biases via shared parameters. |
| Outcome: | The proposed models can achieve comparable performance to autoregressive Transformers using various decoding methods. |
Multi-modal Stance Detection: New Datasets and Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for stance detection for pure texts have limited results to multi-modal content. |
| Approach: | They propose a multi-modal stance detection framework that leverages target information to learn multi-modal stance features from textual and visual modalities. |
| Outcome: | The proposed framework achieves state-of-the-art in multi-modal stance detection on five datasets based on Twitter . |
Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate long-form and coherent text, yet they often hallucinate facts, which undermines their reliability. |
| Approach: | They propose a Learnable Intervention method for Truthfulness Optimization that automatically identifies the optimal intervention intensity tailored to each query context. |
| Outcome: | Experiments on multiple LLMs and question-answering datasets show that LITO improves truthfulness while preserving task accuracy. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models (2024.findings-acl)
Copied to clipboard
Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Noah Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, Wangchunshu Zhou, Yiming Liang, Lei Zhang, Lei Ma, Jiajun Zhang, Zuowen Li, Wenhao Huang, Chenghua Lin, Jie Fu
| Challenge: | a recent study shows that large language models have limited generalization in low-resource languages like Chinese. |
| Approach: | They propose to evaluate the zero-shot generalizability of large language models to the Chinese language . they release only half of the dataset publicly, with the remainder kept private . |
| Outcome: | The Chinese Instruction-Following Benchmark evaluates the generalizability of LLMs to the Chinese language. |
Countering Reward Over-Optimization in LLM with Demonstration-Guided Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. |
| Approach: | They propose a reinforcement learning approach that leverages human demonstrations and a reward model to recalibrate the reward objective. |
| Outcome: | The proposed approach achieves comparable performance to carefully tuned baselines while mitigating ROO in three RL language tasks. |
Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss (2024.findings-acl)
Copied to clipboard
| Challenge: | Accurately modeling idiomatic or non-compositional language has been a longstanding challenge in natural language processing (NLP). |
| Approach: | They propose an approach to model idiomaticity effectively using a triplet loss that incorporates the asymmetric contribution of components words to an idiomatic meaning by using adaptive contrastive learning and resampling miners. |
| Outcome: | The proposed model outperforms previous models significantly on a SemEval challenge and outperformed previous alternatives in many metrics. |
AdaLomo: Low-memory Optimization with Adaptive Learning Rate (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models require substantial memory for training, thereby setting a high hardware threshold. |
| Approach: | They propose a low-memory optimization technique that reduces memory footprint . they propose an adaptive learning rate for each parameter and a grouped update normalization to stabilize convergence . |
| Outcome: | The proposed low-memory optimization performs better than the prevailing algorithm for large language models, AdamW. |
Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods struggle to effectively propagate updates to interconnected facts, limiting the performance of reasoning tasks based on these updated facts. |
| Approach: | They propose a reasoning-based benchmark, ReCoE, which covers six common reasoning schemes in the real world. |
| Outcome: | The proposed reasoning-based benchmark shows that current models struggle to propagate updated knowledge within reasoning schemes. |
Exciting Mood Changes: A Time-aware Hierarchical Transformer for Change Detection Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on temporally sensitive tasks focuses on predicting mood changes . however, there is little attention given to the importance of longitudinal language modelling . |
| Approach: | They propose a Hawkes process-inspired transformation layer to model the influence of time on users’ posts, capturing both their immediate and historical dynamics. |
| Outcome: | The proposed model outperforms existing models on two existing datasets and shows clear performance gains. |
CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing vision-and-language navigation methods do not incorporate environmental feedback into their decision-making processes. |
| Approach: | They propose a framework that incorporates environmental feedback into decision-making and a 3D simulator that renders realistic scenarios using Unreal Engine 5. |
| Outcome: | The proposed framework outperforms existing vision-and-language navigation methods in a zero-shot multi-task setting by 28.1% on average. |
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval (2024.findings-acl)
Copied to clipboard
Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, Chenghua Lin
| Challenge: | Multi-modal information retrieval (MMIR) is a rapidly evolving field . current benchmarks for image-text pairings overlook the scientific domain . |
| Approach: | They develop a scientific domain-specific MMIR benchmark to evaluate image-text pairings using open-access research paper corpora. |
| Outcome: | The proposed benchmarks are based on 530K image-text pairs extracted from scientific documents with detailed captions. |
Diving Deep into the Motion Representation of Video-Text Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing video-text models that capture motion in videos are lacking in quality. |
| Approach: | They propose a method to improve motion understanding in video-text models by utilizing motion descriptions to capture motion in action videos. |
| Outcome: | The proposed pipeline improves motion understanding on two action datasets and shows that it is effective on fine-grained motion descriptions. |
Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing instruction tuning datasets are limited by the quality of the instruction tuning data. |
| Approach: | They propose a model that converts unannotated text into task-specific training datasets for instruction tuning. |
| Outcome: | The proposed model improves the performance of pretrained and instruction tuned models over the de facto self-supervised baseline. |
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Paraphrasing of offensive content is a better alternative to content removal, but supervised methods often retain a large portion of the offensiveness of the original content. |
| Approach: | They propose to use In-Context Learning (ICL) to generate usable offensive paraphrases by using large language models. |
| Outcome: | The proposed framework is better than supervised methods on human evaluation and lower toxicity by 76%. |
Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on whataboutism have focused on tracking "what about" phrases, but they neglect the unique challenges to its detection. |
| Approach: | They propose to use attention weights to distinguish the ‘what about’ lexical construct from whataboutism by using Twitter/X and YouTube datasets. |
| Outcome: | The proposed method improves by 4% and 10% over previous state-of-the-art methods in Twitter and YouTube datasets. |
Epistemology of Language Models: Do Language Models Have Holistic Knowledge? (2024.findings-acl)
Copied to clipboard
| Challenge: | et al., 2021) explores whether language models exhibit characteristics consistent with epistemological holism . authors examined the epistle of language models from the perspective of abduction, revision, and argument generation tasks. |
| Approach: | They examine whether language models exhibit characteristics consistent with epistemological holism . they created a scientific reasoning dataset and examined the epistology of language models . |
| Outcome: | The language models showed that they did not distinguish between core and peripheral knowledge, compared with other tasks. |
Strong hallucinations from negation and how to fix them (2024.findings-acl)
Copied to clipboard
| Challenge: | Despite great performance on many tasks, language models still struggle with reasoning, sometimes providing responses that cannot possibly be true because they stem from logical incoherence. |
| Approach: | They propose a way to treat negation as an operation over latent representations that constrains how they may evolve. |
| Outcome: | The proposed approach improves model performance in cloze prompting and natural language inference tasks without training on sparse negative data. |
LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation tasks favor text generated by different LMs . human evaluation by experts is the most reliable approach, but it is costly and time-consuming . |
| Approach: | They examine whether language model-driven evaluation metrics exhibit bias toward underlying language models in the context of summarization tasks. |
| Outcome: | The proposed evaluation metrics tend to assign inflated scores to outputs generated by the very model they are based on. |
HelloFresh: LLM Evalutions on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits (2024.findings-acl)
Copied to clipboard
Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, Joao F. Henriques, Jakob Foerster
| Challenge: | a better understanding of LLM capabilities on real world tasks is vital for safe development and deployment. |
| Approach: | They propose a new LLM called HelloFresh that uses real-world data to measure performance . they backtest the model and find it yields a temporally consistent ranking . |
| Outcome: | The proposed benchmarks outperform static evaluation data and test data on Wikipedia pages. |
Chaos with Keywords: Exposing Large Language Models Sycophancy to Misleading Keywords and Evaluating Defense Strategies (2024.findings-acl)
Copied to clipboard
| Challenge: | sycophancy is a type of hallucination in Large Language Models, which can lead to false information being presented. |
| Approach: | They explore the sycophantic tendencies of Large Language Models where models provide accurate answers even if they are not entirely correct. |
| Outcome: | The proposed models generate factually correct statements even when they are not completely correct. |
Empowering Large Language Models for Textual Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | True. True. False |
| Approach: | False slants are proposed to generate a large pool of augmentation instructions and select the most suitable task-informed instructions. |
| Outcome: | False omissions: the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
Argument-Aware Approach To Event Linking (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior research in event linking has mainly borrowed methods from entity linking, overlooking distinct features of events. |
| Approach: | They propose an argument-aware method to improve event linking models by augmenting input text with tagged event argument information. |
| Outcome: | The proposed method improves in-KB and out-of-KB queries and training examples. |
CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate grounded responses are prone to errors due to the irrelevancy of input documents. |
| Approach: | They propose a framework that leverages the insight that a robust grounded response should be consistent with information derived solely from its cited sources. |
| Outcome: | Experiments on three open-domain question-answering datasets show that the proposed framework improves performance by 1.5% to 7% without any model fine-tuning. |
TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2024.findings-acl)
Copied to clipboard
Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji
| Challenge: | Recent studies suggest that event extraction evaluations may not accurately reflect the true performance. |
| Approach: | They propose a standardized, fair, and reproducible benchmark for event extraction . they use standardized scripts and splits for 16 datasets spanning eight domains . |
| Outcome: | The proposed benchmarks show that they struggle to achieve satisfactory performance. |
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers (2024.findings-acl)
Copied to clipboard
| Challenge: | Previous work has examined performance disparities between AAL speakers and White Mainstream English speakers . but, this work has not sought to understand the impacts of these disparities on AAL speaker. |
| Approach: | They examine the experiences of African American Language (AAL) speakers when using language technologies. |
| Outcome: | The authors interview 19 AAL speakers to understand performance disparities . they find that speakers often undertake invisible labor to successfully use language technologies . |
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)
Copied to clipboard
| Challenge: | OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement. |
| Approach: | They propose a family of open-source code systems for generating, executing, and iteratively refining code. |
| Outcome: | The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks. |
Measuring and Addressing Indexical Bias in Information Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Information Retrieval (IR) systems may not optimize rankings for fairness, neutrality, or the balance of ideas. |
| Approach: | They propose to use a framework to automatically audit IR rankings for indexical biases, or biase in the positional order of documents. |
| Outcome: | The proposed bias metric can help predict when and how indexical bias will shift a reader’s opinion. |
CIDAR: Culturally Relevant Instruction Dataset For Arabic (2024.findings-acl)
Copied to clipboard
Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, Maged Al-shaibani
| Challenge: | Instruction tuning datasets predominantly cater to English or are derived from English-dominated LLMs. |
| Approach: | They propose to use an Arabic instruction tuning dataset culturally aligned by native Arabic speakers to address drawbacks of finetuning LLMs on machine-generated or machinetranslated datasets. |
| Outcome: | The proposed datasets show that they achieve better cultural alignment than models fine-tuned on other datasets. |
RadGraph-XL: A Large-Scale Expert-Annotated Dataset for Entity and Relation Extraction from Radiology Reports (2024.findings-acl)
Copied to clipboard
Jean-Benoit Delbrouck, Pierre Chambon, Zhihong Chen, Maya Varma, Andrew Johnston, Louis Blankemeier, Dave Van Veen, Tan Bui, Steven Truong, Curtis Langlotz
| Challenge: | RadGraph-XL is an expert-annotated dataset for structured clinical data extraction. |
| Approach: | They propose a large-scale, expert-annotated dataset for clinical entity and relation extraction using radiology reports. |
| Outcome: | The proposed model outperforms existing methods by up to 52% and outperfies GPT-4 in this domain. |
SMART: Submodular Data Mixture Strategy for Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine tuning language models are manual or rely on intuition. |
| Approach: | They propose a method which uses a submodular function to assign importance scores to tasks and then use them to determine mixture weights. |
| Outcome: | The proposed method outperforms traditional methods such as examples proportional mixing and equal mixing. |
Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)
Copied to clipboard
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu
| Challenge: | ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions. |
| Approach: | They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions. |
| Outcome: | ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions. |
Language Model Priors and Data Augmentation Strategies for Low-resource Machine Translation: A Case Study Using Finnish to Northern Sámi (2024.findings-acl)
Copied to clipboard
| Challenge: | a new study examines the use of monolingual data for improving low-resource machine translation. |
| Approach: | They investigate ways of using monolingual data for improving low-resource machine translation. |
| Outcome: | The proposed model can perform better on the target-side data without augmentation of parallel data. |
Differentially Private Knowledge Distillation via Synthetic Text Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are achieving state-of-the-art performance in many downstream tasks, but data privacy is a major challenge for practitioners. |
| Approach: | They propose a differentially private knowledge distillation algorithm that exploits the knowledge of a teacher LLM and a student's output distribution. |
| Outcome: | The proposed algorithm significantly improves the utility over baselines on the Big Patent dataset, with strong privacy parameters, =2. |
KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as conversational agents. |
| Approach: | They construct a dataset of knowledge-intensive writing instructions to evaluate LLMs' ability to follow user instructions. |
| Outcome: | The proposed model fails to integrate new information into an existing answer and perform precise and unambiguous edits. |
XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags (2024.findings-acl)
Copied to clipboard
| Challenge: | XL-HeadTags is a dataset that includes 20 languages across 6 diverse language families. |
| Approach: | They propose to leverage auxiliary information such as images and captions embedded in news articles to retrieve relevant sentences and utilize instruction tuning with variations to generate both headlines and tags for news articles in a multilingual context. |
| Outcome: | The proposed approach generates headlines and tags in a multilingual context using images and captions embedded in the articles and instruction tuning with variations. |
InFoBench: Evaluating Instruction Following Ability in Large Language Models (2024.findings-acl)
Copied to clipboard
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, Dong Yu
| Challenge: | Existing methods for evaluating Large Language Models (LLMs) ability to follow instructions have not been able to provide a detailed analysis of their compliance with instructions. |
| Approach: | They propose a new metric for evaluating Large Language Models' ability to follow instructions and a benchmark for DRFR. |
| Outcome: | The proposed metric and benchmark compared with traditional scoring methods and explores annotation sources including human experts, crowd-sourced workers, and GPT-4. |
EcoRank: Budget-Constrained Text Re-ranking Using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated dominant performance in text re-ranking. |
| Approach: | They propose a suite of budget-constrained methods to perform text re-ranking using LLMs. |
| Outcome: | The proposed method outperforms other budget-aware methods on four datasets. |
FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | FinTral is a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. |
| Approach: | They introduce FinTral, a suite of state-of-the-art multimodal large language models built upon the Mistral-7b model and tailored for financial analysis. |
| Outcome: | The proposed model outperforms ChatGPT-3.5 and GPT-4 in five out of nine tasks and surpasses GPT-4.5 in five of nine task evaluations. |
Aligning Large Multimodal Models with Factually Augmented RLHF (2024.findings-acl)
Copied to clipboard
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell
| Challenge: | Large Multimodal Models (LMMs) are built across modalities and the misalignment between two modality can result in "hallucination" . developing LMMs faces challenges such as a lack of data and a limited number of data sets. |
| Approach: | They propose a new algorithm that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options. |
| Outcome: | The proposed approach improves on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 and an improvement of 60% on MMHAL-BENCH over other baselines. |
The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work on Large Language Models (LLMs) has identified a number of approaches to protect against their vulnerabilities and safety. |
| Approach: | They propose to use a large collection of safe and unsafe prompts to evaluate various LLM defense strategies over both ‘safety’ and ‘over-defensiveness’. |
| Outcome: | The proposed defense strategies are compared on multiple state-of-the-art LLMs and show that they are effective against both ‘safety’ and ‘over-defensiveness’. |
PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work on Temporal Question Answering (TQA) has focused on questions anchored to specific timestamps or events. |
| Approach: | They introduce a benchmark to address present-anchored temporal QA (PATQA) which includes single and multi-hop temporal questions. |
| Outcome: | The proposed model can be automatically refreshed by re-running SPARQL queries on a knowledge graph. |
360∘REA: Towards A Reusable Experience Accumulation with 360∘ Assessment for Multi-Agent System (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on optimizing the agent team or employing self-reflection to iteratively solve complex tasks. |
| Approach: | They propose a hierarchical multi-agent framework that uses 360 assessment to accumulate experience through fine-grained assessment. |
| Outcome: | The proposed framework is based on corporate organizational practices and employs a dual-level experience pool for agents to accumulate experience through fine-grained assessment. |
Extracting Polymer Nanocomposite Samples from Full-Length Documents (2024.findings-acl)
Copied to clipboard
| Challenge: | Using large language models (LLMs) to extract sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers is challenging due to the complexity of the data. |
| Approach: | They propose a benchmark and evaluation technique for extracting sample lists of polymer nanocomposites from full-length materials science research papers. |
| Outcome: | The proposed method improves the performance of LLMs and incorporates self-consistency to improve the performance. |
Leveraging LLM Reasoning Enhances Personalized Recommender Systems (2024.findings-acl)
Copied to clipboard
Alicia Tsai, Adam Kraft, Long Jin, Chenwei Cai, Anahita Hosseini, Taibai Xu, Zemin Zhang, Lichan Hong, Ed H. Chi, Xinyang Yi
| Challenge: | Recent advances have showcased the potential of Large Language Models (LLMs) in executing reasoning tasks, particularly facilitated by Chain-of-Thought (CoT) prompting. |
| Approach: | They propose to use Large Language Models to perform tasks with subjectivity and personalized preferences as inputs to RecSys. |
| Outcome: | The proposed framework aligns with real human judgment on the coherence and faithfulness of LLM reasoning responses. |
Toucan: Many-to-Many Translation for 150 African Language Pairs (2024.findings-acl)
Copied to clipboard
| Challenge: | We introduce two language models with 1.2 billion and 3.7 billion parameters to improve Machine Translation (MT) for low-resource languages. |
| Approach: | They propose a set of tools to improve Machine Translation (MT) for low-resource languages with a focus on African languages. |
| Outcome: | The proposed model outperforms existing models on MT for African languages and improves translation evaluation metrics for 1K languages including African languages. |
Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning (2024.findings-acl)
Copied to clipboard
Zhouhang Xie, Bodhisattwa Prasad Majumder, Mengjie Zhao, Yoshinori Maeda, Keiichi Yamada, Hiromi Wakaki, Julian McAuley
| Challenge: | Motivational Interviewing (MI) requires a system that can infer how to motivate users to adopt positive lifestyle changes. |
| Approach: | They propose a framework that can learn and apply conversation strategies from expert demonstrations by using natural language inductive rules. |
| Outcome: | The proposed framework outperforms in-context demonstrations that are over 50 times longer and can learn natural language strategies from demonstrations. |
Evaluating Structural Generalization in Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on compositional generalization with semantic parsing, but it remains unclear to what extent models can translate sentences that require structural generalization. |
| Approach: | They construct a machine translation dataset that measures compositional generalization with control of words and sentence structures. |
| Outcome: | The proposed model struggle more in structural generalization than in compositional generalization. |
Figuratively Speaking: Authorship Attribution via Multi-Task Figurative Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models that detect multiple FL features in text are not effective in authorship attribution tasks. |
| Approach: | They propose a multi-task Figurative Language Model that learns to detect multiple FL features in text at once. |
| Outcome: | The proposed model outperforms specialized binary models in AA tasks or outperformed binary models on three datasets. |
CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs’ Mathematical Reasoning Capabilities (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models have shown indications of mathematical reasoning ability on competition-level problems. |
| Approach: | They propose a benchmark dataset to enable such analyses using large language models. |
| Outcome: | The proposed model performs better with concepts and hints than with the best model, but it is difficult to verify. |
Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Commercial machine translation engines are proficient in addressing the majority of translation requirements. |
| Approach: | They propose to combine NMT and MT-oriented LLMs to achieve superior translation quality by combining their strengths. |
| Outcome: | The proposed model can handle complex scenarios beyond the capability of NMT alone. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
Proving membership in LLM pretraining data via data watermarks (2024.findings-acl)
Copied to clipboard
| Challenge: | Detecting whether copyright holders’ works were used in large language model (LLM) training is poised to be an important problem. |
| Approach: | They propose to use data watermarks to enable principled detection with only black-box model access, provided the rightholder contributed multiple training documents and watermarked them before public release. |
| Outcome: | The proposed method can be used to test hypothesis testing on a black-box model . it shows that the watermarks are strong under model and dataset scaling . |
Enhancing Hallucination Detection through Perturbation-Based Synthetic Data Generation in System Responses (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection are expensive and outdated . despite the popularity of LLMs, the issue of hallucinosity poses significant concerns for downstream users. |
| Approach: | They propose an approach that automatically generates both faithful and hallucinated outputs by rewriting system responses. |
| Outcome: | The proposed model outperforms state-of-the-art zero-shot detectors and existing synthetic generation methods in accuracy and latency. |
SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC (2024.findings-acl)
Copied to clipboard
| Challenge: | a growing number of cloud-based inference services are relying on SMPC to protect data privacy. |
| Approach: | They propose a framework for Privacy-Preserving Inference for Transformer models that eliminates exponential and maximum operations in PPI without sacrificing model performance. |
| Outcome: | The proposed framework outperforms MPCFormer in terms of performance and efficiency . it is 3.57 and 3.58 times faster than PUMA for BERTBASE and BERTLARGE . |
Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have markedly shifted the landscape of AI, enabling these models to tackle complex, real-world tasks through natural language instructions. |
| Approach: | They propose a benchmark which evaluates a model's susceptibility to prompt extraction attacks by employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models. |
| Outcome: | The proposed benchmark assesses models under both defenseless and defended scenarios, employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models. |
History-Aware Conversational Dense Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Current approaches for conversational dense retrieval rely on fine-tuning a pre-trained ad-hoc retriever, which can be lengthy and noisy. |
| Approach: | They propose a context-denoised query reformulation and automatic mining of supervision signals based on historical turns. |
| Outcome: | The proposed system improves on two public conversational search datasets. |
Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-Modal Knowledge Graphs (MMKGs) are knowledge graphs that integrate and align information from diverse modalities (e.g., text and images). |
| Approach: | They propose a framework that integrates image-text pairs of long-tailed entities and a concept guidance module that offers explainability and enables human verification. |
| Outcome: | The proposed framework improves the accuracy of recognizing long-tailed image-text pairs compared to baselines and also offers flexibility and explainability. |
ZeroStance: Leveraging ChatGPT for Open-Domain Stance Detection via Dataset Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Until recently, zero-shot stance detection was limited to in-domain tasks. |
| Approach: | They propose a method for stance detection that trains a model that can generalize well to unseen targets across multiple domains. |
| Outcome: | The proposed method generalizes well to unseen targets across multiple domains over baselines on most benchmarks. |
Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate task-specific data via zero-shot prompting and promote cross-lingual transfer for low-resource target languages. |
| Approach: | They propose to use a teacher model to label LLM generations and use their label probabilities to identify a representative subset of diverse generations that boost zero-shot accuracies while being efficient. |
| Outcome: | The proposed models generate task-specific data via zero-shot prompting and promote cross-lingual transfer for low-resource target languages. |
Reinforcement Tuning for Detecting Stances and Debunking Rumors Jointly with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media has become a fertile ground for nurturing rumors and misinformation due to its lack of systematic moderation. |
| Approach: | They propose a framework to enhance the joint predictive capabilities of LLMs for stance detection and rumor verification tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art methods and generalizes to non-LLMs accommodated as task models. |
Exploring the Potential of Dense Information in Multimodal Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance captions have limitations such as insufficient detail and excessive hallucinations, resulting in compromised alignment and masking the true potential of dense information. |
| Approach: | They propose a pipeline that generates highly detailed captions for images that facilitates in-depth analysis of the potential for dense information in multimodal alignment. |
| Outcome: | The proposed pipeline significantly improves multimodal alignment and compositional reasoning abilities, surpassing hard negative samples. |
Referral Augmentation for Zero-Shot Information Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Referral-augmented retrieval improves zero-shot document retrieval in a variety of tasks . prior work shows sparse models struggle to reconcile with dense models . |
| Approach: | They propose a technique that concatenates document indices with referrals from other documents that cite or link to the given document. |
| Outcome: | The proposed technique outperforms generative text expansion techniques on structured tasks and improves on ACL paper retrieval. |
InstructEval: Instruction-Tuned Text Evaluator from Human Preference (2024.findings-acl)
Copied to clipboard
| Challenge: | InstructEval is a general text evaluator based on open-source Large Language Models (LLMs). |
| Approach: | They propose to build a general multi-aspect text evaluator based on open-source Large Language Models (LLMs) they use extensive open Human Preference Modeling datasets and a small set of multi-spect annotated data to overcome the shortage of annotation resources for multi-task evaluations. |
| Outcome: | The proposed model performs comparable or superior to commercial LLMs like ChatGPT or GPT-4 in terms of both overall and multi-aspect evaluation tasks. |
A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that fine-tuned textual transformer models are vulnerable to adversarial text perturbations. |
| Approach: | They extract 13 different features representing a wide range of input fine-tuning corpora properties and use them to predict adversarial robustness of the fine- tuned models. |
| Outcome: | The proposed framework can be used as an additional tool for robustness evaluation since it saves 30x-193x runtime compared to the traditional technique and can be easily used under adversarial training. |
InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) can solve graph reasoning and generation tasks with parameter updates without sacrificing performance. |
| Approach: | They propose a structured format verbalizer to unify all graph data into a universal code-like format, which can simply represent the graph without any external graph-specific encoders. |
| Outcome: | The proposed framework outperforms GPT-4 and LLaMA2 in graph reasoning and generation tasks by more than 13% and 38%, respectively. |
RaDA: Retrieval-augmented Web Agent Planning with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Agents powered by large language models inherit important limitations such as the restricted context length, dependency on human-engineered exemplars, and insufficient generalization. |
| Approach: | They propose a novel planning method for Web agents that disentangles planning into two stages: for a new given task, it decomposes tasks into high-level subtasks; and then iteratively synthesizes actions based on dynamically retrieved exemplars. |
| Outcome: | The proposed method decomposes tasks into high-level subtasks and iteratively synthesizes actions based on dynamically retrieved exemplars. |
Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)
Copied to clipboard
Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen
| Challenge: | Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about their capabilities and the potential data contamination problem. |
| Approach: | They propose to evaluate the reasoning capabilities of large language models in solving recent competition-level programming problems in Codeforces. |
| Outcome: | The proposed model has experienced a cliff-like decline in problems after September 2021, which shows the potential data contamination and the challenges for any existing LLM to solve unseen complex reasoning problems. |
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research on hypothetical induction is limited by the observation annotations in the dataset and the ground truth hypotheses are mostly commonsense knowledge. |
| Approach: | They propose a first dataset for social science academic hypotheses discovery using raw web corpus as observations and propose valid, useful scientific hypothese . they propose 'a multi-module framework' that includes feedback mechanisms to boost performance. |
| Outcome: | The proposed dataset generates valid, novel, and helpful scientific hypotheses, even new to humanity, using open-domain data and a web corpus as observations. |
GRADUAL: Granularity-aware Dual Prototype Learning for Better Few-Shot Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for few-shot relation extraction use text labels and context sentences to learn prototype representations. |
| Approach: | They propose a "dual prototype learning method" that integrates text labels and context sentences into prototype representations. |
| Outcome: | The proposed method achieves state-of-the-art performance in few-shot relation extraction. |
Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spelling Correction models are prone to over-correct and poor generalization for error patterns outside the standard distribution. |
| Approach: | They propose a teacher network guided by prior knowledge for distillation learning of CSC models. |
| Outcome: | The proposed method significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. |
The Revolution of Multimodal Large Language Models: A Survey (2024.findings-acl)
Copied to clipboard
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
| Challenge: | Recent advances in large language models have led to the development of multimodal large language model. |
| Approach: | They present a review of recent visual-based Large Language Models and analyze their architectures and alignment strategies. |
| Outcome: | The proposed models can integrate visual and textual modalities while providing a dialogue-based interface and instruction-following capabilities. |
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | None Large language models (LLMs) are emerging as a key tool for automated programming. |
| Approach: | They compare performance of None Large language models with language understanding models on functional programming and object-oriented programming benchmarks. |
| Outcome: | The models perform relatively well on functional programming (FP) and object-oriented programming (OOP) benchmarks, while exhibiting poor performance on OOP benchmarks. |
Code Needs Comments: Enhancing Code LLMs with Comment Augmentation (2024.findings-acl)
Copied to clipboard
Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, Dahua Lin
| Challenge: | Large Language Models (LLMs) require a deep understanding of programming languages and their correlation with natural languages (NLs). |
| Approach: | They propose a data augmentation method that generates comments for existing code and a filtering strategy that filters out code data poorly correlated with natural language. |
| Outcome: | The proposed method outperforms the model trained on the augmented data and the model further trained on data without augmentation on two widely-used programming skill benchmarks. |
Efficient Domain Adaptation for Non-Autoregressive Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing non-parametric approaches like nearest neighbor machine translation have made small Autoregressive translation models less efficient . despite their impressive generalization and task performance, LLMs suffer from prohibitive inference cost when confronted with specific domains. |
| Approach: | They propose a domain adaptation approach that tailors a k-nearest-neighbor algorithm for NAT models that incorporates the parallel nature of NAT. |
| Outcome: | The proposed approach achieves significant improvements over the Base-NAT model and exhibits enhanced efficiency. |
Exploring Reversal Mathematical Reasoning Ability for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been a success in the wide range of natural language understanding and reasoning tasks. |
| Approach: | They propose a training method to improve general and reversal reasoning abilities by using a reversed dataset. |
| Outcome: | The proposed method improves general and reversal reasoning abilities and alleviates the reverse curse. |
A Unified Joint Approach with Topological Context Learning and Rule Augmentation for Knowledge Graph Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge graph completion methods perform simple linear update on relation representation, and only local neighborhood information is aggregated, making it difficult to capture logic semantic between relations and global topological context information. |
| Approach: | They propose a joint approach with Topological Context learning and Rule Augmentation (TCRA) it uses a topological context learning mechanism and a relation rule context learning system . |
| Outcome: | The proposed approach performs better on three benchmark datasets and is widely used in knowledgeintensive applications. |
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024.findings-acl)
Copied to clipboard
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong
| Challenge: | Modern large language models often "hallucinate" plausible but factually incorrect information, which reduces their trustworthiness especially in settings where accurate and up-to-date information is critical. |
| Approach: | They develop a human evaluation procedure to measure correctness and hallucination and use it to benchmark both closed and open-source LLMs. |
| Outcome: | The proposed method outperforms both competing search engine-augmented prompting methods and commercial systems on search-augmented QA. |
ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning LLMs output with expected safety require substantial training efforts and expensive computational resources. |
| Approach: | They propose a method to directly boost the safety of existing instruction-tuned large language models without additional training. |
| Outcome: | The proposed method improves safety of instruction-tuned large language models without training and requires expensive computational resources. |
CR-LLM: A Dataset and Optimization for Concept Reasoning of Large Language Models (2024.findings-acl)
Copied to clipboard
Nianqi Li, Jingping Liu, Sihang Jiang, Haiyun Jiang, Yanghua Xiao, Jiaqing Liang, Zujie Liang, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han
| Challenge: | Existing concept reasoning related datasets suffer from modeledge leakage and context leakage. |
| Approach: | They propose a concept reasoning for large language models with modeledge leakage prevention and context leakage preventive methods to improve the models' conceptual reasoning abilities. |
| Outcome: | The proposed method significantly improves the existing models and reasoning methods, achieving a 7% increase in accuracy compared to CoT and showing better granularity. |
DATA-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve sentence representation learning (SRL) ignore the potential interference problems across tasks and instances. |
| Approach: | They propose a multi-task instruction tuning method that arranges the order of multi- task data for training to minimize interference risks. |
| Outcome: | The proposed method can boost the performance of state-of-the-art methods. |
Combating Label Sparsity in Short Text Topic Modeling via Nearest Neighbor Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing topic models suffer from poor performance when applied to short text contents due to the limited length of a single topic. |
| Approach: | They propose a neural short text topic model that augments reconstruction labels with k-nearest documents to complement relevant but unobserved words. |
| Outcome: | The proposed model outperforms the state-of-the-art models on multiple public short-text datasets and can derive high-quality topics and document representations. |
RefuteBench: Evaluating Refuting Instruction-Following for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | The application scope of large language models (LLMs) is expanding . however, evaluating whether models can respond to user feedback has not been thoroughly analyzed. |
| Approach: | They propose a benchmark to assess whether large language models can respond to refuting feedback and adhere to user demands throughout the conversation. |
| Outcome: | The proposed benchmark covers tasks such as question answering, machine translation, and email writing. |
Complex Logical Query Answering by Calibrating Knowledge Graph Completion Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to solve complex logical queries are not well-calibrated . CKGC is lightweight and effective, allowing the model to quickly converge . |
| Approach: | They propose a method for calibrating KGC models to adapt to complex logical queries . they map the values of predictions of KGC to the range [0, 1] . |
| Outcome: | The proposed method can significantly boost model performance in complex logical query answering task. |
Argument-Based Sentiment Analysis on Forward-Looking Statements (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for argument mining are limited in interpreting future-oriented arguments. |
| Approach: | They propose a categorization of argument units into claims, premises, and scenarios coupled with a unique sentiment analysis framework. |
| Outcome: | The proposed framework outperforms existing models in most tasks and is more efficient than existing methods. |
Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack explicit alignment between source and target contexts, leading to unfaithful translations. |
| Approach: | They propose three learning strategies to encourage LLMs to pay more attention to source context . they use a dataset to test the effectiveness of their model across multiple language pairs . |
| Outcome: | The proposed model reduces hallucinatory translation and improves fidelity across multiple languages. |
Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion (2024.findings-acl)
Copied to clipboard
| Challenge: | Document Artificial Intelligence (Document AI) is gaining momentum across industries for streamlining document processes, enhancing efficiency, and extracting insights from unstructured data. |
| Approach: | They propose a fusion framework that summarizes block diagrams by integrating local and global information, catering to both English and Korean languages. |
| Outcome: | The proposed framework surpasses all previous methods and models for block diagram summarization on a dataset of BD-EnKo in English and Korean. |
MultiSQL: A Schema-Integrated Context-Dependent Text2SQL Dataset with Diverse SQL Operations (2024.findings-acl)
Copied to clipboard
| Challenge: | Text2SQL is a task that translates natural language into SQL statements. |
| Approach: | They propose a task that translates natural language into SQL statements. |
| Outcome: | The proposed task enables users to convert natural language into SQL statements. |
Towards Demonstration-Aware Large Language Models for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models for machine translation often face difficulties in leveraging demonstrations to further improve their performance. |
| Approach: | They propose a novel approach that integrates demonstration-aware training and inference strategies within the framework of tuning-based LTMs. |
| Outcome: | The proposed model integrates demonstration-aware training and inference strategies within tuning-based LTMs. |
DADA: Distribution-Aware Domain Adaptation of PLMs for Information Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models struggle with out-of-domain data due to distribution shifts . generative domain adaptation (DA) methods have been proposed to address these issues . |
| Approach: | They propose a distribution-aware domain adaptation method to address distribution shifts in domains . they use observation-level feedback and observation- level feedback to adapt to the target domain . |
| Outcome: | The proposed method adapts to the domain distribution knowledge at the level of a single document and the corpus and expands document representation to unseen gold query terms using domain and observation feedback. |
LLMs cannot find reasoning errors, but can correct them given the error location (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in poor performance overall. |
| Approach: | They propose to use a backtracking setup to test the correction abilities of LLMs on their mistake-finding ability to find logical mistakes. |
| Outcome: | The proposed model improves on 5 reasoning tasks, showing that it can correct logical mistakes without ground truth labels or training data. |
Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL translation (2024.findings-acl)
Copied to clipboard
Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto
| Challenge: | Large Language Models (LLMs) understand textual description to generate code in zero-shot scenarios, but there is a possibility that this ability may be influenced by having seen target textual descriptions and the related code. |
| Approach: | They propose a method to detect Data Contamination in Large Language Models (LLMs) and analyze their results on Termite and Spider Datasets to investigate their method. |
| Outcome: | The proposed method detects data contamination in GPTs and analyzes its performance on unfamiliar datasets. |
ChartCheck: Explainable Fact-Checking over Real-World Chart Images (2024.findings-acl)
Copied to clipboard
| Challenge: | Data visualizations are often used to summarize and communicate key information, but they can also be misused to spread misinformation and promote agendas. |
| Approach: | They propose a dataset for explainable fact-checking against real-world charts that uses vision-language and chart-to-table models to evaluate the validity of the dataset. |
| Outcome: | The proposed model is based on vision-language and chart-to-table models and proposes a baseline to the community. |
Real World Conversational Entity Linking Requires More Than Zero-Shots (2024.findings-acl)
Copied to clipboard
| Challenge: | Entity linking (EL) in conversations is a key component of many downstream tasks such as semantic search. |
| Approach: | They propose to use Fandom and Wikipedia to evaluate EL models' ability to generalize to a new unfamiliar KB without prior training. |
| Outcome: | The proposed evaluation framework and dataset are tailored to facilitate the study. |
CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling (2024.findings-acl)
Copied to clipboard
Chenhao Zhang, Renhao Li, Minghuan Tan, Min Yang, Jingwei Zhu, Di Yang, Jiahao Zhao, Guancheng Ye, Chengming Li, Xiping Hu
| Challenge: | Existing datasets lack consulting knowledge, resulting in LLMs lacking professional consulting competence. |
| Approach: | They propose a report-based multi-turn dialogue reconstruction framework for Chinese psychological counseling that uses large language models to assist counseling. |
| Outcome: | The proposed framework is open-source and can be used in future research. |
Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models to generate implicit hate explanations are lacking in many fields. |
| Approach: | They propose to use language models to generate explicit hate posts to make it clear . they find that simpler models incorporating external toxicity signals outperform KG-infused models . |
| Outcome: | The proposed setup produces more precise explanations than zero-shot GPT-3.5, highlighting the intricate nature of the task. |
TextGenSHAP: Scalable Post-Hoc Explanations in Text Generation with Long Documents (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are difficult to explain and understand due to long input contexts and autoregressive output generation. |
| Approach: | They propose a post-hoc explanation method which incorporates LLM-specific techniques. |
| Outcome: | The proposed method improves retrieval recall and prediction accuracy significantly on open-domain question answering benchmarks. |
Balanced Data Sampling for Language Model Training with Clustering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a fundamental part of the training process. |
| Approach: | They propose to use clustering to balance the text distribution of training data for better model training. |
| Outcome: | Extensive experiments validate the effectiveness of ClusterClip Sampling under various training datasets and large language models. |
Length Generalization of Causal Transformers without Position Encoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Besides Transformers without position encodings, the success of NoPE provides a new way to overcome the challenge of generalizing to longer sentences. |
| Approach: | They propose a parameter-efficient tuning for searching attention heads’ best temperature hyper-parameters, which substantially expands NoPE’s context size. |
| Outcome: | The proposed tuning significantly expands NoPE's context size, allowing it to generalize to longer sentences with state-of-the-art generalization algorithms. |
Unsupervised Sign Language Translation and Generation (2024.findings-acl)
Copied to clipboard
Zhengsheng Guo, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Kehai Chen, Zhaopeng Tu, Yong Xu, Min Zhang
| Challenge: | Experimental results on the BBC-Oxford Sign Language dataset reveal that USLNet achieves competitive results compared to supervised baseline models. |
| Approach: | They propose an unsupervised sign language translation and generation network that learns from abundant single-modality data without parallel sign language data. |
| Outcome: | The proposed model achieves competitive results compared to baseline models on the BBC-Oxford Sign Language dataset and Open-Domain American Sign Language data. |
Mitigating Data Scarcity in Semantic Parsing across Languages with the Multilingual Semantic Layer and its Dataset (2024.findings-acl)
Copied to clipboard
Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Karim Ghonim, Lu Xu, Hee-Soo Choi, Alberte Fernández-Castro, Roberto Navigli
| Challenge: | Large Language Models (LLMs) have advanced significantly in understanding human text, but semantic representations remain crucial for various applications. |
| Approach: | They introduce a multilingual semantic layer which decouples from disambiguation and external inventories and simplifies the task. |
| Outcome: | The proposed model reduces performance gap between languages and annotators by enabling them to understand semantic relations between concepts in any language. |
Efficient Sparse Attention needs Adaptive Token Release (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks, however, their ‘large’ scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. |
| Approach: | They propose to release resources from caches and rebuild key-value states by a lightweight controller module to approximate an ideal top-K sparse attention. |
| Outcome: | The proposed method achieves a significant throughput improvement of 221.8% over full attention and a model with 7 billion tokens. |
Learning Fine-Grained Grounded Citations for Attributed Large Language Models (2024.findings-acl)
Copied to clipboard
Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, Bing Qin
| Challenge: | despite impressive performance, large language models still struggle with hallucinations . current approaches suffer from suboptimal citation quality due to reliance on in-context learning . |
| Approach: | They propose a framework that teaches large language models to generate fine-grained citations. |
| Outcome: | The proposed framework outperforms all baselines on the ALCE benchmark and achieves an average improvement of 14.21% in citation quality. |
ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (2024.findings-acl)
Copied to clipboard
| Challenge: | Entity Linking and Relation Extraction (EL) are fundamental tasks in Natural Language Processing. |
| Approach: | They propose a Retriever-Reader architecture for Entity Linking and Relation Extraction . they propose an input representation that incorporates the candidate entities alongside the text . |
| Outcome: | The proposed architecture achieves state-of-the-art in in- and out-of domain benchmarks while using academic budget training and with 40x inference speed compared to competitors. |
Synergizing Large Language Models and Pre-Trained Smaller Models for Conversational Intent Discovery (2024.findings-acl)
Copied to clipboard
| Challenge: | Current attempts at CID rely on pretrained Small Language Models (SLMs) this lacks the ability to label new intents and is a challenge for small language models. |
| Approach: | They propose to combine Large Language Models (LLMs) with pre-trained SLMs for CID to enhance the semantic comprehension of LLMs. |
| Outcome: | The proposed approach improves the semantic comprehension of LLMs and the operational agility of SLMs by realigning existing descriptors within the SLM’s feature space to correct cluster distortion and promote robust learning of representations. |
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text summarization have shown remarkable performance, but a significant number of summaries exhibit factual inconsistencies, such as hallucinations. |
| Approach: | They propose a factuality-oriented metric that evaluates text summarization for accuracy . they use a human annotation process to examine the accuracy of automatically generated summaries . |
| Outcome: | The proposed metric sets a new state-of-the-art on AGGREFACT, the de-facto benchmark for factuality evaluation. |
Self-Para-Consistency: Improving Reasoning Tasks at Low Cost for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that self-consistency decoding can improve performance for complex reasoning tasks with large language models. |
| Approach: | They propose a self-consistency decoding strategy that generates multiple paraphrases for each test question and then generates reasoning paths for the original and all the paraphrased questions based on greedy decoding. |
| Outcome: | The proposed strategy reduces the sampling number and improves performance on complex reasoning tasks. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |
mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans (2024.findings-acl)
Copied to clipboard
| Challenge: | Currently, multilingual datasets are created through translation, which cannot evaluate such language-specific aspects. |
| Approach: | They propose to curate a dataset for language-specific knowledge and commonsense . they propose to use multilingual commonsensiaq to leverage language models for a more efficient construction . |
| Outcome: | The proposed method reduces the creation cost by using multilingual LMs to create QAs . the proposed approach is based on the construction process of CSQA but with language models . |
Dual-Stage Multi-Task Syntax-Oriented Pre-Training for Syntactically Controlled Paraphrase Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactically controlled paraphrase generation (SCPG) aims to generate sentences with syntactic structures resembling given exemplars. |
| Approach: | They propose a dual-stage multi-task pre-training scheme that uses a series of structure-oriented and syntax-oriented tasks to generate sentences with syntactic structures resembling given exemplars. |
| Outcome: | The proposed method outperforms existing methods on all possible variants of SCPG tasks and significantly outperformed the popular T5 model. |
Demonstration Augmentation for Zero-shot In-context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated an impressive capability known as In-context Learning (ICL), which enables them to acquire knowledge from textual demonstrations without the need for parameter updates. |
| Approach: | They propose to use model’s previously predicted historical samples as demonstrations for subsequent ones to improve model’ s performance. |
| Outcome: | The proposed method significantly outperforms the previous method and its predecessors in terms of inference cost and time. |
Pushing the Limits of Zero-shot End-to-End Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end Speech Translation (ST) systems require limited data, which can cause data scarcity and performance degradation. |
| Approach: | They propose a method for zero-shot ST that bridges the modality gap without any paired ST data. |
| Outcome: | The proposed method bridges the modality gap without any paired ST data on a speech encoder and on MT models. |
NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs are not able to handle numerals and units of measurement, but they can be improved by introducing perturbations. |
| Approach: | They propose to analyze existing LLMs on processing numerals and units of measurement by perturbing their datasets. |
| Outcome: | The proposed model improves on ancient Chinese arithmetic problems and can handle numeral and measurement conversions. |
On The Persona-based Summarization of Domain-Specific Documents (2024.findings-acl)
Copied to clipboard
Ankan Mullick, Sombit Bose, Rounak Saha, Ayan Bhowmick, Pawan Goyal, Niloy Ganguly, Prasenjit Dey, Ravi Kokku
| Challenge: | In an ever-expanding world of domain-specific knowledge, summarization of information is a complex task . persona-based summarizing of domain specific information by humans is deemed not preferred . |
| Approach: | They propose a framework for efficient training of a small foundation LLM on a healthcare corpus. |
| Outcome: | The proposed framework fine-tunes a domain-specific small foundation LLM using a healthcare corpus and evaluates its quality using AI-based critiquing. |
Evaluating Large Language Models for Health-related Queries with Presuppositions (2024.findings-acl)
Copied to clipboard
| Challenge: | a large number of health-related queries require factually accurate answers . however, the lack of accurate answers may cause real-world harm . |
| Approach: | They evaluate the factual accuracy and consistency of large language models using a dataset consisting of health-related queries with varying degrees of presuppositions. |
| Outcome: | The proposed model responses agree with 23-32% of existing false claims and 49-55% with novel fabricated claims. |
Word Sense Linking: Disambiguating Outside the Sandbox (2024.findings-acl)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) systems have performed well on several evaluation benchmarks, but it still struggles to find downstream applications. |
| Approach: | They propose a task where systems have to identify which spans to disambiguate and link them to their most suitable meaning. |
| Outcome: | The proposed task performs above the estimated inter-annotator agreement on a set of words . the proposed system is based on 'transformer-based' architectures and iteratively relaxes the assumptions . |
Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Memorisation in neural models is concerned due to overfitting and privacy concerns . a dominant hypothesis based on image classification is that lower layers learn generalisable features and deeper layers specialise and memorise. |
| Approach: | They apply 4 techniques to localise and edit models' memories. |
| Outcome: | The proposed method shows that memorisation is a gradual process rather than a localised one. |
Towards Multi-Relational Multi-Hop Reasoning over Dense Temporal Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Temporal knowledge graph reasoning is a crucial task for answering time-dependent questions within a knowledge graph (KG). |
| Approach: | They propose a temporal KG reasoning benchmark with over 200k entities and 960k questions that facilitate complex, multi-relational and multi-hop reasoning. |
| Outcome: | The proposed model is able to conduct pattern-aware and time-sensitive reasoning across temporal KGs and is scalable to a wide range of data conditions. |
Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on hallucination detection for LLMs focus on how to identify possible factrelated errors in outputs. |
| Approach: | They propose an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods in hallucination detection. |
Progressive Tuning: Towards Generic Sentiment Abilities for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models of sentiment understanding do not consider interrelated sentiment knowledge . et al., 2023; Zhao e.t., 20, 21; Shu e t. 2021) focus on individual sentiment subtasks . |
| Approach: | They propose an open-source large language model specific to the sentiment domain that explores hierarchical relationships between subtasks. |
| Outcome: | The proposed model performs well across all datasets in the progressive sentiment reasoning benchmark. |
Fooling the Textual Fooler via Randomizing Latent Representations (2024.findings-acl)
Copied to clipboard
| Challenge: | Several adversarial attacks can compromise the model without accessing the model architecture or model parameters (i.e., a blackbox setting) Several studies have revealed that deep NLP models are vulnerable to adversarials that slightly perturb the input to cause the models to misbehave. |
| Approach: | They propose a lightweight and attack-agnostic defense that perplexes the process of generating an adversarial example in query-based black-box attacks. |
| Outcome: | The proposed defense is lightweight and attack-agnostic and does not necessitate additional computational overhead during training nor does it rely on assumptions about the potential adversarial perturbation set while having a negligible impact on the model’s accuracy. |
Part-of-speech Tagging for Extremely Low-resource Indian Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | Modern natural language processing systems thrive when given access to large datasets, but a large fraction of the world’s languages are not privy to such benefits due to sparse documentation and inadequate digital representation. |
| Approach: | They propose a parallel part-of-speech evaluation dataset for Angika, Magahi, Bhojpuri and Hindi. |
| Outcome: | The proposed approach improves F1 scores by up to 8% on Angika, Magahi, Bhojpuri and Hindi while ignoring the tokenization challenge. |
FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate original text using pre-trained language models are problematic as they are trained on corpora constructed by human authors. |
| Approach: | They propose a unique “self-plagiarism” contrastive decoding strategy that modifies prompts in LLMs to develop an amateur model and a professional model. |
| Outcome: | The proposed method enables the development of an amateur model and a professional model while maintaining its standard language model status. |
Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Emotion recognition is a multimodal learning method that can be used for data scarcity. |
| Approach: | They propose to use Adaptively modality-balanced domain adaptation to balance the alignment of different modalities for multimodal emotion recognition. |
| Outcome: | The proposed model outperforms competing models on common datasets on multimodal emotion recognition. |
MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can encode knowledge during pre-training on large text corpora, enabling downstream tasks like question answering (QA). |
| Approach: | They construct a dataset derived from systematic reviews to examine their ability to encode medical knowledge and their recall. |
| Outcome: | The proposed model performs well on the biomedical QA dataset. |
Deepfake Defense: Constructing and Evaluating a Specialized Urdu Deepfake Audio Dataset (2024.findings-acl)
Copied to clipboard
Sheza Munir, Wassay Sajjad, Mukeet Raza, Emaan Abbas, Abdul Hameed Azeemi, Ihsan Ayyub Qazi, Agha Ali Raza
| Challenge: | Automatic speaker verification systems are facing escalating challenges due to deepfake attacks. |
| Approach: | They propose a Urdu deepfake audio dataset for deepfak detection focusing on two spoofing attacks – Tacotron and VITS TTS. |
| Outcome: | The proposed dataset evaluates two spoofing attacks in Urdu with a human evaluation to gauge whether people are able to distinguish deepfake audios from real (bonafide) audios. |
Leveraging Entailment Judgements in Cross-Lingual Summarisation (2024.findings-acl)
Copied to clipboard
| Challenge: | Synthetically created cross-lingual summarisation datasets are prone to include document-summary pairs where the reference summary is unfaithful to the corresponding document. |
| Approach: | They propose to use off-the-shelf cross-lingual Natural Language Inference to evaluate faithfulness of reference and model generated summaries and use unlikelihood loss to teach a model about unfaithful summary sequences. |
| Outcome: | The proposed approach evaluates faithfulness of reference and model generated summaries and uses unlikelihood loss to teach a model about unfaithful summary sequences. |
Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on IE tasks have focused on recognizing and analyzing cross-modal information . a multimodal large language model (MLLM) is developed to analyze IE across modalities . |
| Approach: | They propose a multimodal large language model (MLLM) capable of grounding information from all modalities. |
| Outcome: | The proposed framework provides a framework to analyze IE tasks over various modalities and their fine-grained groundings. |
Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data (2024.findings-acl)
Copied to clipboard
Yanda Li, Chi Zhang, Gang Yu, Wanqi Yang, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei
| Challenge: | OpenAI's GPT-4 has demonstrated remarkable multimodal capabilities, but specific mechanics of GPT4 remain unknown. |
| Approach: | They propose a data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. |
| Outcome: | The proposed method improves on ten commonly assessed models and provides greater flexibility compared to existing methods. |
Modeling Overregularization in Children with Small Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. |
| Approach: | They hypothesize that language models that imitate errors children make during language acquisition have a learning process more similar to humans. |
| Outcome: | The proposed model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children. |
Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research suggests that contextual representations of large language models exhibit subpar performance in downstream tasks, struggling to fully capture the semantic nuances of words. |
| Approach: | They investigate the bottom-up evolution of lexical semantics for a popular LLM . they probing its hidden states at the end of each layer using a contextualized word identification task . |
| Outcome: | The proposed model is able to encode lexical semantics in lower layers while achieving weaker induction in higher layers. |
Harnessing Large Language Models as Post-hoc Correctors (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their effectiveness in a wide range of tasks, including machine translation and commonsense reasoning. |
| Approach: | They propose a training-free framework that can work as a post-hoc corrector to propose corrections for ML models. |
| Outcome: | The proposed framework improves the performance of a number of models by up to 39% on text analysis and the challenging molecular predictions. |
Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on short dialogues, but mainly on short debates. |
| Approach: | They propose to use Large Language Models to construct an automated debate judge to evaluate multi-turn debates. |
| Outcome: | The proposed system improves on the PanelBench benchmark, which compares its performance to actual debate outcomes. |
CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models that generate harmful responses are constrained by their inherent capability. |
| Approach: | They propose to align large language models with human preferences from AI feedback. |
| Outcome: | The proposed framework improves the alignment of large language models with human preferences from AI feedback. |
Towards a new research agenda for multimodal enterprise document understanding: What are we missing? (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we discuss the limitations of multimodal document understanding models in enterprise settings. |
| Approach: | They propose a research agenda that is aimed at driving the field towards higher impact in enterprise applications. |
| Outcome: | The proposed research agenda is aimed at driving the field towards higher impact in enterprise applications. |
CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have evaluated user satisfaction estimation in TOD systems . current benchmarks for user satisfaction estimates are highly skewed towards dialogues for which the user is satisfied. |
| Approach: | They leverage large language models to generate satisfaction-aware counterfactual dialogues to augment original dialogues of a test collection. |
| Outcome: | The proposed models show higher robustness to increase in dissatisfaction labels than fine-tuned models. |
Measuring Retrieval Complexity in Question Answering Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a new metric, retrieval complexity (RC), measures the difficulty of answering questions. |
| Approach: | They propose a retrieval complexity metric conditioned on the completeness of retrieved documents . they propose an unsupervised pipeline to measure RC given an arbitrary retrieval system . |
| Outcome: | The proposed pipeline measures RC more accurately than alternative estimators on six challenging QA benchmarks. |
Combining Hierachical VAEs with LLMs for clinically meaningful timeline summarisation in social media (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that social media users' posts can help identify depression, bipolar disorder or self-harm. |
| Approach: | They propose a hybrid abstractive summarisation approach combining hierarchical VAEs with LLMs to produce clinically meaningful summaries from social media timelines. |
| Outcome: | The proposed approach produces clinically meaningful summaries from social media user timelines, suitable for mental health monitoring. |
PIXAR: Auto-Regressive Language Modeling in Pixel Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows the possibility of building open-vocabulary large language models that operate on pixel representations. |
| Approach: | They propose a pixel-based autoregressive LLM that performs generative tasks . they propose an adversarial pretraining stage that improves the readability and accuracy of PIXAR by 8.1 on LAMBADA and 8.5 on bAbI. |
| Outcome: | The proposed model performs free-form generative tasks while keeping the number of parameters on par with previous encoder-decoder models. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
Preemptive Answer “Attacks” on Chain-of-Thought Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a scenario where the LLM obtains an answer before engaging in reasoning, and propose two measures to bolster the robustness of this approach. |
| Outcome: | The proposed model significantly impairs its reasoning capability across various CoT methods and a broad spectrum of datasets. |
Do Language Models Exhibit Human-like Structural Priming Effects? (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent exposure to a structure facilitates processing of the same structure, a study finds . structural priming is well attested in humans, for both language production and comprehension . |
| Approach: | They use the structural priming paradigm to investigate where priming effects manifest . they find that rarer elements within a prime increase priming effect . |
| Outcome: | The findings provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models. |
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models (2024.findings-acl)
Copied to clipboard
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, Junran Peng
| Challenge: | Large Language Models (LLMs) have paved the way for complex tasks such as role-playing. |
| Approach: | They propose a framework to benchmark, elicit, and enhance role-playing abilities in Large Language Models. |
| Outcome: | The proposed framework improves role-playing abilities with 168,093 samples. |
LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. |
| Approach: | They propose a flexible and simulation-free testbed that simulates 6 representative embodied tasks in textual embodies. |
| Outcome: | The proposed testbed offers adaptability to diverse environments without multiple simulation engines and allows easy customization of communication and action strategies. |
Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground (2024.findings-acl)
Copied to clipboard
Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow
| Challenge: | Existing benchmarks for theory of mind (ToM) use synthetic data, which can misalign with human behavior. |
| Approach: | They propose a question-answer benchmark based on naturally occurring spoken dialogs to evaluate theory of mind capabilities of language models. |
| Outcome: | The proposed dataset shows that LMs struggle to demonstrate theory of mind (ToM) . |
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
MoE-SLU: Towards ASR-Robust Spoken Language Understanding via Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
| Challenge: | Spoken language understanding (SLU) is a crucial task in task-oriented dialogue systems. |
| Approach: | They propose an ASR-Robust SLU framework based on the mixture-of-experts technique to generate additional transcripts from clean transcripts and use it to weigh the representations of the generated transcripts, ASR transcripts . |
| Outcome: | The proposed framework achieves state-of-the-art on three benchmark SLU datasets. |
Multi-Task Transfer Matters During Instruction-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction-tuning improves a model’s ability to learn in-context, but the mechanisms that drive in-constext learning are poorly understood. |
| Approach: | They propose to train a model on hundreds of tasks to improve its ability to learn in-context. |
| Outcome: | The proposed methods improve model transfer and in-context generalization, suggesting catastrophic forgetting may impact in-constext learning. |
What Makes a Good Order of Examples in In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate impressive few-shot learning capabilities via in-context learning (ICL). |
| Approach: | They propose to use unlabeled data to evaluate order performance . they propose to filter out subsets of orders with label fairness and select the most influential order for each test instance. |
| Outcome: | The proposed method is superior over strong baselines and validates generalizability across settings. |
BloomVQA: Assessing Hierarchical Multi-modal Comprehension (2024.findings-acl)
Copied to clipboard
Yunye Gong, Robik Shrestha, Jared Claypoole, Michael Cogswell, Arijit Ray, Christopher Kanan, Ajay Divakaran
| Challenge: | Recent advances of machine intelligence solutions have demonstrated tremendous success in a wide range of language and multi-modal tasks over diverse domains. |
| Approach: | They propose a VQA dataset to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. |
| Outcome: | The proposed dataset shows improved accuracy over all comprehension levels and a tendency to bypass visual inputs especially for higher-level tasks. |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |
Diffusion Guided Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing guidance methods for text generation are prone to decoding errors and degrade performance. |
| Approach: | They propose a model that steers an auto-regressive language model to generate text with desired properties. |
| Outcome: | The proposed model outperforms existing guidance methods on a wide range of benchmark data sets. |
InstructEd: Soft-Instruction Tuning for Model Editing with Hops (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for model editing are limited due to excessive memorization and knowledge conflict issues. |
| Approach: | They propose to insert soft instructions into the attention module to facilitate interactions between instructions and questions and to understand and utilize new facts. |
| Outcome: | The proposed method achieves 10% improvement in one-hop (multi-hop) model editing on three datasets with LLaMAs and GPT2 . |
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback (2024.findings-acl)
Copied to clipboard
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, Chang Yoo
| Challenge: | Existing approaches to provide token-level rewards fail to account for varying degrees of preference inherent to each token. |
| Approach: | They propose a reward model that uses a discriminator to assign token-based continuous rewards to each token considering the context. |
| Outcome: | Extensive experiments show that the proposed reward model improves on open-ended language generation benchmarks. |
Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization (2024.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
| Challenge: | Large language models struggle to capture relevant information located in the middle of their input. |
| Approach: | They propose a calibration mechanism that allows the model to attend to contexts faithfully according to their relevance even when they are in the middle. |
| Outcome: | The proposed calibration mechanism mitigates this positional bias and improves retrieval-augmented generation performance. |
S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMs (2024.findings-acl)
Copied to clipboard
Sarkar Snigdha Sarathi Das, Chirag Shah, Mengting Wan, Jennifer Neville, Longqi Yang, Reid Andersen, Georg Buscher, Tara Safavi
| Challenge: | Dialogue state tracking (DST) was based on narrow task-oriented conversations . however, large language models have ushered in more flexible open-domain chat systems . |
| Approach: | They propose a method that combines dialogue segmentation and state tracking within open-domain dialogues to improve long context tracking. |
| Outcome: | The proposed method outperforms the state-of-the-art on open-domain dialogue datasets and publicly available datasets. |
Set the Clock: Temporal Alignment of Pretrained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. |
| Approach: | They construct a time-sensitive question dataset and use it to examine temporal alignment methods to align their internal knowledge to a target time. |
| Outcome: | The proposed methods improve LLaMa2's performance by 62% if they are fine tuned to the year 2022 . |
From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | toxicity mitigation in language models has been focused on single-language settings . however, widespread adoption of LLMs has introduced a range of unknown -harms . |
| Approach: | They employ translated data to evaluate and enhance mitigation techniques in the absence of sufficient annotated datasets across languages. |
| Outcome: | The proposed approach compares translation quality and retrieval-augmented mitigation techniques under static and continual toxicity mitigation scenarios. |
Here’s a Free Lunch: Sanitizing Backdoored Models with Model Merge (2024.findings-acl)
Copied to clipboard
| Challenge: | democratization of pre-trained language models brings significant security risks, including backdoor attacks. |
| Approach: | They propose to merge a backdoored model with other homogeneous models to remediate backdoor vulnerabilities. |
| Outcome: | The proposed model merging approach outperforms other models on classification tasks without additional resources or specific knowledge. |
Enhancing Sentence Simplification in Portuguese: Leveraging Paraphrases, Context, and Linguistic Features (2024.findings-acl)
Copied to clipboard
| Challenge: | Automated text simplification requires (paired) datasets that are scarce in languages other than English. |
| Approach: | They propose a method that leverages paraphrases, context, and linguistic attributes to overcome the absence of paired texts in Portuguese. |
| Outcome: | The proposed model surpasses the current state-of-the-art while competing with a Large Language Model. |
How Far can 100 Samples Go? Unlocking Zero-Shot Translation with Tiny Multi-Parallel Data (2024.findings-acl)
Copied to clipboard
| Challenge: | a common solution to zero-shot translation is to add as many related translation directions as possible to the training corpus. |
| Approach: | They show that a small amount of multi-parallel data can achieve significant zero-shot improvements . they say that the resulting non-English performance is close to the complete translation upper bound . |
| Outcome: | The proposed model achieves +21.7 ChrF++ non-English translation improvements on EC30 dataset . the resulting non- English performance exceeds M2M100 by an average of 5.9 ChrF+ . |
Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset (2024.findings-acl)
Copied to clipboard
Satanu Ghosh, Neal Brodnik, Carolina Frey, Collin Holgate, Tresa Pollock, Samantha Daly, Samuel Carton
| Challenge: | Existing methods for ad-hoc schema-based information extraction are brittle and non-transferable, limiting their practicality for this type of one-off extraction task. |
| Approach: | They propose to use GPT-4 to perform ad-hoc schema-based information extraction from scientific literature. |
| Outcome: | The proposed model can replicate two existing material science datasets, one pertaining to multi-principal element alloys and one to silicate diffusion, and draw on their insights to suggest future research directions. |
Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction (2024.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised grammar induction models lack analysis of traditional challenges, especially regarding the training loss. |
| Approach: | They propose a method to reduce the parse pool per sentence for loss evaluation using structural bias from pre-trained parsers. |
| Outcome: | The proposed method significantly improves performance while reducing variance and bias toward overly simplistic parses. |
LMDX: Language Model-based Document Information Extraction and Localization (2024.findings-acl)
Copied to clipboard
Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Zifeng Wang, Jiaqi Mu, Hao Zhang, Chen-Yu Lee, Nan Hua
| Challenge: | Large Language Models have revolutionized Natural Language Processing but their application in extracting information from visually rich documents has not been successful. |
| Approach: | They propose a language model-based document information extraction and localization methodology to reframe the document information extract task for a LLM. |
| Outcome: | The proposed method enables extraction of singular, repeated, and hierarchical entities with and without training data. |
DBQR-QA: A Question Answering Dataset on a Hybrid of Database Querying and Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Question answering (QA) is a fundamental task in the field of Natural Language Processing (NLP). |
| Approach: | They propose a database querying and reasoning dataset for question answering that is designed to accommodate sequential questions and multi-hop queries. |
| Outcome: | The proposed dataset better mirrors the dynamics of real-world information retrieval and analysis with a particular focus on the financial reports of US companies. |
NoteChat: A Dataset of Synthetic Patient-Physician Conversations Conditioned on Clinical Notes (2024.findings-acl)
Copied to clipboard
| Challenge: | NoteChat is a cooperative multi-agent framework for generating patient-physician dialogues . evaluator finds it outperforms state-of-the-art models for generating clinical notes . clinical documentation is largely done by physicians at both steps . |
| Approach: | They propose a cooperative multi-agent framework leveraging Large Language Models to generate patient-physician dialogues. |
| Outcome: | The proposed framework outperforms state-of-the-art models for generating clinical notes . it can engage patients directly and help clinical documentation, a leading cause of physician burnout . |
Model Editing at Scale leads to Gradual and Catastrophic Forgetting (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing model editing methods are evaluated using metrics for reliability, specificity and generalization over one or few edits. |
| Approach: | They evaluate model editing methods for three crucial properties - editing proficiency, fact forgetting and downstream performance. |
| Outcome: | The proposed methods are based on two state-of-the-art models - ROME and MEMIT. |
3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for visually rich document understanding do not account for the diverse carriers of document versions and their associated noises. |
| Approach: | They propose a multimodal, multi-task, multiteacher joint-grained knowledge distillation model for visually-rich form document understanding. |
| Outcome: | The proposed model outperforms baselines on a comprehensive evaluation of public datasets showing it can handle complex structures and content of visually-rich forms. |
Faithful Persona-based Conversational Dataset Generation with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for training conversational AI models do not sufficiently model their users. |
| Approach: | They propose a generator-critic architecture framework to expand the initial dataset while improving the quality of its conversations. |
| Outcome: | The proposed framework expands the initial dataset while improving the quality of its conversations. |
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024.findings-acl)
Copied to clipboard
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang
| Challenge: | Recent vision-language models (VLMs) have shown impressive capabilities as general visual assistants, but there are two challenges to their performance: (1) lacking task diversity in pretraining and visual instruction tuning; (2) annotation error and bias in GPT-4 synthesized instruction tuning data. |
| Approach: | They propose a two-stage instruction tuning framework that fine tunes VLMs firstly and further tuned on GPT-4 synthesized data. |
| Outcome: | The proposed framework outperforms the traditional single-stage visual instruction tuning framework and achieves state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. |
TAXI: Evaluating Categorical Knowledge Editing for Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge editing aims to inject new facts into language models to improve factuality, but current benchmarks fail to evaluate consistency, which is critical to ensure efficient, accurate, and generalizable edits. |
| Approach: | They manually create a new benchmark dataset specifically created to evaluate consistency in categorical knowledge edits. |
| Outcome: | The results show that the editors achieve marginal, yet non-random consistency, and their consistency far underperforms human baselines. |
Automatic Bug Detection in LLM-Powered Text-Based Games Using LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Advancements in large language models (LLMs) are revolutionizing interactive game design, but they may exhibit flaws such as hallucinations, forgetfulness, or misinterpretation of prompts. |
| Approach: | They propose a method for automatically identifying LLM bugs from player game logs . their method surpasses unstructured bug-catching methods and fills the gap . |
| Outcome: | The proposed method surpasses unstructured bug-catching methods and fills the gap in detection of logical and design flaws. |
Embodied Language Learning: Opportunities, Challenges, and Future Directions (2024.findings-acl)
Copied to clipboard
| Challenge: | embodied language learning is a form of language understanding where the language learner is situated in the world, perceives it, and interacts with it. |
| Approach: | They propose to use a concept of World Scopes to measure progress in language understanding research. |
| Outcome: | The proposed framework identifies gaps and suggests future directions for language understanding research. |
Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that evaluations of CR models on multiple datasets conflate different factors concerning what is being measured. |
| Approach: | They propose to view evaluations through the lens of measurement modeling . they show that evaluations risk conflating different factors concerning what is being measured . |
| Outcome: | The evaluations on seven datasets show that models that reflect coreference generalization are often correlated with differences in how coreference is defined and operationalized. |
SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that understanding goal-driven actions is challenging because of the small variations in the narrative that lead to vastly different goals, achievement outcomes and future actions. |
| Approach: | They propose to use a participant achievement lens to interpret and assess goal driven actions . they collect 6.3K high quality goal and action annotations reflecting their proposed lens . |
| Outcome: | The proposed lens can be used to interpret and assess goal driven actions . it can be fine-tuned on the dataset to achieve performance surpassing larger models . |
SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate open domain dialogues have a one-to-many problem . existing approaches lack commonsense reasoning biases and perform poorly in domain-specific scenarios. |
| Approach: | They propose a framework that leverages both a small, specialised model and LLMs for the evaluation of open-domain dialogues. |
| Outcome: | The proposed framework achieves state-of-the-art performance in both classification and evaluation tasks and exhibits better correlation with human judgements. |
Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on instruction tuning to show cross-lingual generalization . a novel non-English meta-dataset is used to study instruction tuning . |
| Approach: | They perform instruction tuning individually for two distinct language meta-datasets and assess the performance on unseen tasks in a non-English language. |
| Outcome: | The proposed model outperforms baseline training in English and Korean by 20.7% and 13.6%. |
What Makes Language Models Good-enough? (2024.findings-acl)
Copied to clipboard
| Challenge: | Psycholinguistic research suggests that humans may build a representation of linguistic input that is ‘good-enough’ for the task at hand. |
| Approach: | They create a good-enough language processing evaluation dataset to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. |
| Outcome: | The proposed model evaluation against the annotated GELP reveals that models with shallower depth and fewer heads exhibit good-enough performance. |
Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spelling Correction (CSC) lacks large-scale high-quality corpora due to labor-intensive labeling of spelling errors in real-life writing or typing scenarios. |
| Approach: | They propose to use OCR/ASR-based generation to refine Chinese Spelling Correction models on random replacement-based corpora and filter them based on prediction confidence. |
| Outcome: | The proposed model outperforms existing models on three widely-used benchmarks while significantly alleviating over-correction. |
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples (2024.findings-acl)
Copied to clipboard
| Challenge: | Using high-performing text and image generation models, we can improve compositional reasoning capabilities for large multimodal models. |
| Approach: | They propose a framework to improve visio-linguistic compositional reasoning capability for multimodal models by leveraging physical grounding and using image-based models for semantic counterfactual fine-tuning. |
| Outcome: | The proposed framework outperforms GPT-4V and DALLE-3 on a new dataset and checkpoints. |
Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models (2024.findings-acl)
Copied to clipboard
Ran Xu, Hejie Cui, Yue Yu, Xuan Kan, Wenqi Shi, Yuchen Zhuang, May Dongmei Wang, Wei Jin, Joyce Ho, Carl Yang
| Challenge: | Clinical natural language processing (NLP) is a subfield that requires the extraction, analysis, and interpretation of unstructured clinical text. |
| Approach: | They propose a model which infuses knowledge into clinical text generation with LLMs for clinical NLP tasks. |
| Outcome: | The proposed model improves performance across 8 clinical NLP tasks and 18 datasets by 7.7%-8.7% on average. |
Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent expressive speech-to-speech translation systems have achieved impressive expressivity preservation performances by cascading unit-to speech (U2S) generator to the speech- to-unit translation model. |
| Approach: | They propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST) They aim to address this limitation by incorporating a distillation with no label (DINO) self-controlled training strategy into the model’s pretraining process. |
| Outcome: | The proposed model significantly improved the expressive speech-to-speech translation system in noisy environments while maintaining competitive performance in clean environments. |
Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed a substantial increase in the demand for legal services, especially for individuals with modest means. |
| Approach: | They propose a diagnostic legal large language model which uses adaptive lawyer-like diagnostic questions to collect additional case information and then provides high-quality feedback. |
| Outcome: | The proposed model surpasses classical LLMs by providing outstanding performance and a remarkable user experience in the legal domain. |
TELLER: A Trustworthy Framework for Explainable, Generalizable and Controllable Fake News Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for detecting fake news are limited due to non-transparent reasoning processes and inherent risks of integration with large language models. |
| Approach: | They propose a framework for trustworthy fake news detection that prioritizes explainability, generalizability and controllability of models. |
| Outcome: | The proposed framework prioritizes explainability, generalizability and controllability of models. |
Verifiable Generation with Subsentence-Level Fine-Grained Citations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work mainly targets the generation of sentence-level citations, lacking specificity about which parts of a sentence are backed by the cited sources. |
| Approach: | They propose to use subsentence-level fine-grained citations to generate more precise location of generated content supported by the cited sources. |
| Outcome: | The proposed model improves the accuracy and trustworthiness of large language models by allowing users to trace the information back to its source and verify its correctness. |
Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a set of over 200 WikiHow procedures, we test several simple multi-LLM-agent architectures for customization. |
| Approach: | They propose to use a set of WikiHow procedures to test how-to procedures can be customized by multiple LLMs. |
| Outcome: | The proposed architecture outperforms an end-to-end LLM in the evaluation set of over 200 WikiHow procedures. |
A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Mechanisms of the Transformer architecture for causal language modeling are not well understood. |
| Approach: | They propose a meta-learning view of the Transformer architecture when trained for a causal language modeling task by explicating an inner optimization process that may happen within the Transformer. |
| Outcome: | The proposed model is based on a self-attention mechanism and has been widely used in natural language processing, computer vision, and scientific discovery. |
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024.findings-acl)
Copied to clipboard
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang
| Challenge: | Knowledge distillation (KD) is a technique for transferring expertise from large teacher models to compact student models with reduced memory footprints and inference costs. |
| Approach: | They propose to transfer knowledge from large teacher models to compact student models by exploiting teacher-student capacity discrepancies to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
| Outcome: | The proposed framework exploits teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
Small Language Models Need Strong Verifiers to Self-Correct Reasoning (2024.findings-acl)
Copied to clipboard
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
| Challenge: | Existing studies show that large language models can self-correct their outputs by generating a critique and revising it based on the critique. |
| Approach: | They propose a pipeline that prompts small language models to collect self-correction data that supports the training of self-refinement abilities. |
| Outcome: | The proposed pipeline improves the self-correction abilities of two models on five datasets spanning math and commonsense reasoning. |
Hire a Linguist!: Learning Endangered Languages in LLMs with In-Context Linguistic Descriptions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs rarely perform well in unseen, endangered languages . Existing models such as Llama and GPT-4 lack a rich corpus of training data . |
| Approach: | They propose a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. |
| Outcome: | The proposed approach elevates translation capability from GPT-4’s 0 to 10.5 BLEU for 10 language directions. |
From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often output text at a native level of speech, making them difficult to use for contexts where end-users are not fully proficient. |
| Approach: | They propose a framework to control the difficulty level of text generated by Large Language Models for contexts where end-users are not fully proficient. |
| Outcome: | The proposed framework surpasses GPT-4 and other models at fraction of the cost. |
From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards (2024.findings-acl)
Copied to clipboard
Khaoula Chehbouni, Megha Roshan, Emmanuel Ma, Futian Wei, Afaf Taik, Jackie Cheung, Golnoosh Farnadi
| Challenge: | Recent advances in large language models have also introduced additional safety risks and raised concerns regarding their detrimental impact on already marginalized populations. |
| Approach: | They propose to use LLMs to evaluate their safety responses on already mitigated biases by evaluating models on already encoded assumptions. |
| Outcome: | The proposed model can encode harmful assumptions, but it can also be harmful for certain demographic groups. |
CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions (2024.findings-acl)
Copied to clipboard
| Challenge: | a benchmark is designed to evaluate the capabilities of large language models (LLMs) as agents in decision making and operational tasks. |
| Approach: | They propose a benchmark to evaluate LLMs in the context of Chinese societal applications . they propose he benchmark will evaluate tool invocation ability of LLM and task completion ability . |
| Outcome: | The proposed benchmark features 398 APIs across 27 widely-used Apps across 14 domains. |
Token Alignment via Character Matching for Subword Completion (2024.findings-acl)
Copied to clipboard
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Robert Kwiatkowski, Ramesh Nallapati, Parminder Bhatia, Bing Xiang
| Challenge: | Generative models struggle with prompts corresponding to partial tokens due to tokenization, where partial token is out-of-distribution during inference. |
| Approach: | They propose a method to alleviate tokenization artifact on text completion by backtracking to the last complete tokens and aligning subsequent generations to match with the prompt. |
| Outcome: | The proposed method shows that it improves on partial token scenarios with only a minor time increase. |
Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | a limited number of human annotations are required to evaluate multilingual summarization evaluation metrics. |
| Approach: | They propose a multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual metaevaluations dataset into multilingual versions. |
| Outcome: | The proposed framework outperforms classical text-matching-based metrics in non-English languages. |
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for speech emotion recognition are not suitable for emotional tasks. |
| Approach: | They propose a universal speech emotion representation model that is pre-trained on open-source emotion data. |
| Outcome: | euphoria2vec outperforms state-of-the-art models and emotion specialist models . it shows consistent improvements among 10 different languages of speech emotion recognition datasets . |
Language-Informed Beam Search Decoding for Multilingual Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, but decoding multilingual NMT models produces off-target translations . |
| Approach: | They propose a general decoding algorithm incorporating an off-the-shelf Language Identification (LiD) model into beam search decoding to reduce off-target translations. |
| Outcome: | The proposed language-informed beam search improves +1.1 BLEU and +0.9 BLUE on WMT and OPUS datasets and reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% respectively. |
RA-LoRA: Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) with their extensive parameters and high memory demands are challenging to fine-tune for specific applications with limited resources. |
| Approach: | They propose a method that dynamically adjusts the adapter’s rank using rank-subspace analysis, optimizing performance with fewer parameters. |
| Outcome: | The proposed method improves model accuracy with minimal parameter changes and demonstrates the importance of rank dynamics in optimizing quantized LLMs. |
The PGNSC Benchmark: How Do We Predict Where Information Spreads? (2024.findings-acl)
Copied to clipboard
| Challenge: | Social networks have become ideal vehicles for news dissemination because posted content is easily able to reach users beyond a news outlet’s direct audience. |
| Approach: | They propose a benchmark that builds information pathways based on the audiences of influential news sources and uses their content to characterize the communities. |
| Outcome: | The proposed benchmark builds information pathways based on the audiences of influential news sources and uses their content to characterize the communities. |
STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing environments for text-based RL are domain-specific or time-consuming to generate and do not train the agents to master a specific set of skills. |
| Approach: | They propose an interactive environment for self-supervised RL that bootstraps the text-based RL agents with automatically generated games to boost their generalization capabilities. |
| Outcome: | The proposed environment bootstraps the agents with automatically generated games to boost their generalization capabilities to reach a goal of the target environment. |
Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) demonstrate exceptional capabilities on tasks, but are vulnerable to extraction attacks. |
| Approach: | They propose Privacy Protection via Optimal Parameters (POP) which induces the model to forget about some of its training data. |
| Outcome: | The proposed method outperforms the state-of-the-art in retaining LM performance on 9 classification and 4 dialogue benchmarks. |
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research in large vision-language models has shown promising results, but the issue of hallucination remains. |
| Approach: | They propose an instruction-based method to reduce hallucinations in large vision-language models . they use disturbance instructions to exacerbate hallucinosity in multimodal fusion modules . |
| Outcome: | The proposed method reduces hallucinations in multimodal fusion modules by reducing alignment uncertainty and subtracting hallucines from the original distribution. |
Fine-tuning Language Models for Joint Rewriting and Completion of Code with Potential Bugs (2024.findings-acl)
Copied to clipboard
| Challenge: | Previous work has demonstrated shortcomings of large language models of code (CodeLLMs) in completing drafty partial code with potential bugs. |
| Approach: | They propose to use large language models of code to fine-tune their models to rewrite and complete drafty partial code into functional full programs. |
| Outcome: | The proposed approach achieves superior pass rates over baselines and preserves the integrity of the original partial implementations. |
A Critical Study of What Code-LLMs (Do Not) Learn (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models trained on code corpora have limitations such as suggesting codes with syntactic errors, variable misuse etc. |
| Approach: | They conduct a fine-grained analysis of attention maps and hidden representations of large-scale Large Language Models (cLLMs) trained on a large corpus of code and natural language -programming language pairs. |
| Outcome: | The proposed models encode relations among syntactic tokens and identifiers, but fail to encode relations between syntaktic token and identifier. |
Visual In-Context Learning for Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of Large Visual Language Models (LVLMs) are limited by cross-modal interactions and representation disparities. |
| Approach: | They propose a Visual In-Context Learning method that retrieves images via a 'Retrieval & Rerank' paradigm and summarises images with task intent and task-specific visual parsing to compose language-based demonstrations that reduce token count. |
| Outcome: | The proposed method reduces token count and alleviates cross-modal interaction problem on visual reasoning datasets. |
SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Approach: | They propose a collaborative framework that connects a Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Outcome: | The proposed framework outperforms both LLMs and supervised models in high-resource or challenging low-resourced settings. |
No perspective, no perception!! Perspective-aware Healthcare Answer Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Healthcare Community Question Answering forums are prone to off-topic discussions and diverse answers can be challenging for readers to sift through. |
| Approach: | They propose a task of perspective-specific answer summarization to identify different perspectives within healthcare-related responses and frame a perspective-driven abstractive summary covering all responses. |
| Outcome: | The proposed model outperforms existing models against five baselines and shows that it is more accurate than existing models. |
Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever (2024.findings-acl)
Copied to clipboard
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, Daxin Jiang
| Challenge: | Large-scale retrieval is indispensable in information-seeking tasks such as open-domain question answering and knowledgegrounded dialogue. |
| Approach: | They propose to use a large language model (LLM) to augment a query with its potential answers by prompting LLMs with a composition of the query and the query’s in-domain candidates. |
| Outcome: | The proposed method breaks brute-force combinations of retrievers with LLMs and lifts the performance of zero-shot retrieval to be very competitive on benchmark datasets. |
A Survey on Predicting the Factuality and the Bias of News Media (2024.findings-acl)
Copied to clipboard
| Challenge: | a growing number of scholars are profiling entire news outlets to profile fake content . political bias detection is also an important topic, but the two problems have been addressed separately . |
| Approach: | They argue that media profiling should be based on factuality and bias together . they argue that it is difficult to fact-check every single suspicious claim or article manually . |
| Outcome: | The present level of proliferation of fake, biased, and propagandistic content online has made it impossible to fact-check every single suspicious claim or article, either manually or automatically. |
Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform (2024.findings-acl)
Copied to clipboard
| Challenge: | Discrete Wavelet Transforms (DWT) can be applied to NLP capturing a variety of linguistic and semantic properties. |
| Approach: | They propose to use Discrete Wavelet Transforms to analyze word and sentence embeddings . they show that DWT can reduce dimensionality of embeddables by 50-93% . |
| Outcome: | The proposed paradigm reduces embeddings' dimensionality by 50-93% while maintaining their quality. |
Improving Multi-hop Logical Reasoning in Knowledge Graphs with Context-Aware Query Representation Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods rely on linear sequential operations to solve First-Order Logic queries. |
| Approach: | They propose a model-agnostic approach that fully integrates the context of the query graph. |
| Outcome: | The proposed method improves performance on two datasets by 19.5%. |
ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable adaptability across domains, but they are often not suitable for structured knowledge extraction tasks such as named entity recognition (NER). |
| Approach: | They propose a method that instructs LLMs to self-reflect on the specific domain and generates domain-relevant attributes for creating attribute-rich training data. |
| Outcome: | The proposed method produces NER datasets in domains with domain-relevant attributes and generates entity terms and NER context data around these entities. |
Defending LLMs against Jailbreaking Attacks via Backtranslation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advancement in large language models (LLMs) has shown their extensive applications and transformative potential to reshape people's lives. |
| Approach: | They propose a method which uses backtranslation to infer an input prompt from an input input prompt and then run it again on the backtranslated prompt. |
| Outcome: | The proposed method outperforms baselines and has little impact on the generation quality for benign input prompts. |
A Large Collection of Model-generated Contradictory Responses for Consistency-aware Dialogue Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large-scale neural response generation models (RGMs) have made significant progress but still struggle to generate semantically appropriate responses. |
| Approach: | They build a large dataset of model-generated contradictions for the first time and analyze the results to gain valuable insights into their characteristics. |
| Outcome: | The proposed dataset significantly improves the performance of data-driven contradiction suppression methods. |
Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models of large language reasoning exhibit reasoning biases similar to humans, a study shows . syllogistic reasoning is one of the basic forms of deductive reasoning . |
| Approach: | They propose to use a syllogism dataset to evaluate models' reasoning abilities . they propose to ask LLMs to translate slogismatic slurs into abstract logical expressions . |
| Outcome: | The proposed method shows that models exhibit reasoning biases similar to humans, and that there is room for improvement in reasoning problems where premises and hypotheses are neither entailment nor contradiction. |
Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation (2024.findings-acl)
Copied to clipboard
| Challenge: | Data contamination is a problem in Large language models due to the reliance on extensive internet-derived training corpora. |
| Approach: | They present a survey on the topic of data contamination in large language models. |
| Outcome: | The results of the first survey on data contamination in large language models provide a comprehensive guide for NLP researchers seeking a systematic understanding of the issue. |
DIMSIM: Distilled Multilingual Critics for Indic Text Simplification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve the quality of responses generated by large language models (LLMs) however, these critique-refine steps require multiple expensive LLM calls. |
| Approach: | They propose to use critique distillation to train critic models that are trained on input-critique pairs generated by an LLM. |
| Outcome: | The proposed model trains two separate critics that focus on lexical and structure complexity, and is more effective than using an LLM directly as a critic in both 0-shot and few-shot settings. |
MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing retrieval-augmented models typically retrieve information from a single type of knowledge source. |
| Approach: | They propose an efficient memory-augmented transformer to retrieve relevant knowledge from multiple knowledge sources. |
| Outcome: | The proposed model outperforms existing retrieval-augmented models on popular QA benchmarks in terms of accuracy and speed. |
Ask LLMs Directly, “What shapes your bias?”: Measuring Social Bias in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate social bias in large language models have limitations . et al., 1995: stereotypes shape social perceptions without objective basis . |
| Approach: | They propose a method to intuitively quantify social perceptions and suggest metrics to evaluate biases within LLMs. |
| Outcome: | The proposed metrics capture the multi-dimensional aspects of social bias, the paper shows . they show that the proposed metrics can be used to evaluate bias in large language models . |
Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing graph-based models excel at capturing structural information within TKGs but lack semantic comprehension abilities. |
| Approach: | They propose a plug-and-play module to enhance the performance of graph-based TKG models by exploring high-order histories step-by-step. |
| Outcome: | Experiments on three datasets and backbones show that CoH is effective in capturing high-order historical information for LLMs. |
Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack sufficient controllability to generate statements supporting diverse or even controversial perspectives. |
| Approach: | They develop a pipeline that fine tunes LLMs to generate statements generated via debate. |
| Outcome: | The proposed pipeline improves the controllability of LLMs in generating statements supporting an argument the user defined in the prompt. |
Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for implicit hate speech detection do not have significant advantage over cross-entropy loss-based learning. |
| Approach: | They propose a label-aware hard negative sampling strategy that encourages the model to learn detailed features from hard negative samples instead of random batch. |
| Outcome: | The proposed models outperform existing models for implicit hate speech detection both in- and cross-datasets. |
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction tuning is critical to large language models but its success heavily relies on the training data quality. |
| Approach: | They propose a paradigm that synergizes a teacher LLM’s reflection and introspection with the data selection capability of the student LLM to automatically refine existing instruction-tuning data. |
| Outcome: | The proposed method achieves much stronger and top-tier 7B and 13B LLMs without collecting brand-new data. |
Selective Prompting Tuning for Personalized Conversations with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Personalization in conversational AI requires persona profiles and contextual understanding to create meaningful conversations. |
| Approach: | They propose a method that softly prompts LLMs for personalized conversations in a selective way. |
| Outcome: | The proposed approach improves response diversity by up to 90% on the CONVAI2 dataset. |
Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) face challenges in maintaining accuracy due to the dynamic nature of world knowledge. |
| Approach: | They propose to use a benchmark dataset to investigate the effects of model edits on model safety metrics and guardrails. |
| Outcome: | The proposed dataset sheds light on how the edits, impact the model’s safety metrics and guardrails. |
ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions (2024.findings-acl)
Copied to clipboard
Honglin Lin, Siyu Li, Guoshun Nan, Chaoyue Tang, Xueting Wang, Jingxin Xu, Rong Yankai, Zhouzhili Zhouzhili, Yutong Gao, Qimei Cui, Xiaofeng Tao
| Challenge: | Existing approaches to image retrieval from contextual descriptions (IRCD) lag behind human performance in IRCD. |
| Approach: | They propose a method that relies on a doubly contextual alignment scheme for challenging IRCD. |
| Outcome: | The proposed method can yield comparable results with GPT-4V, despite fewer parameters. |
PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns (2024.findings-acl)
Copied to clipboard
| Challenge: | recognizing patterns and abstracting concepts are key to general intelligence, we show . state-of-the-art large multimodal models struggle to generalize well to simple abstract patterns . |
| Approach: | They evaluate large multimodal models with abstract patterns based on colors, numbers, sizes, and shapes. |
| Outcome: | The proposed model fails to generalize well to simple abstract patterns, the study shows . the model fails on single-concept puzzles, despite its sophistication . |
How Do Moral Emotions Shape Political Participation? A Cross-Cultural Analysis of Online Petitions Using Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Using cross-cultural online petition data, we examine how moral emotions influence user participation and political participation. |
| Approach: | They construct and share a moral emotion dataset comprising 50,000 petition sentences in Korean and English each, along with emotion labels annotated by a fine-tuned LLM. |
| Outcome: | The results show that moral emotions like other-suffering increase both forms of participation and help petitions go viral, while self-conscious have the opposite effect. |
VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraft (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-agent collaboration using LLMs is a challenging research topic that aims to enable multiple autonomous agents to coordinate their actions and achieve a common goal. |
| Approach: | They propose a benchmark for multi-agent collaboration in the Minecraft environment and introduce a Directed Acyclic Graph Multi-Agent Framework to resolve complex inter-ag dependencies. |
| Outcome: | The proposed framework outperforms existing ModelVerse, reducing hallucinations and improving task decomposition efficacy. |
CF-TCIR: A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to text-conditioned image retrieval use attention-driven compositors . instead, we reformulate the retrieval process as a cross-modal interaction between a synthesized image feature and its corresponding text descriptor. |
| Approach: | They propose a compositor-free framework for text-conditioned image retrieval . they propose combining a reference image and modification text to form a query tuple . |
| Outcome: | The proposed framework offers advantages in terms of computational efficiency, scalability, and performance. |
DMIN: A Discourse-specific Multi-granularity Integration Network for Conversational Aspect-based Sentiment Quadruple Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on enhancing token-level interactions, but lack sufficient modeling of discourse structure information. |
| Approach: | They propose to use a discourse structure called "thread" to enhance token interaction among different utterances. |
| Outcome: | The proposed model achieves state-of-the-art on two datasets. |
Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models are highly effective tools for solving different kinds of problems in natural language processing. |
| Approach: | They propose to use large language models to solve a myriad of problems. |
| Outcome: | The proposed model performs worse on word meaning comprehension than an encoder-only model with vastly fewer parameters. |
FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to process contexts with unlimited length are limited to finite expansion length or prone to performance degradation when dealing with very long contexts. |
| Approach: | They propose to exploit fragment-level relations in external memory to hierarchically process the long text. |
| Outcome: | The proposed model improves story understanding, repository-level code generation, and long-term chatting. |
On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing DocRE models which perform well may make more mistakes when merely changing the entity names in the document, hindering the generalization to novel entity names. |
| Approach: | They propose a pipeline to generate entity-renamed documents by replacing the original entity names with names from Wikidata. |
| Outcome: | The proposed pipeline generates entity-renamed documents by replacing the original entity names with names from Wikidata. |
RESEMO: A Benchmark Chinese Dataset for Studying Responsive Emotion from Social Media Content (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on social media text processing do not focus on responsive emotion analysis. |
| Approach: | They propose a Chinese dataset named ResEmo for responsive emotion analysis, including 3813 posts with 68,781 comments collected from Weibo, the largest social media platform in China. |
| Outcome: | The proposed dataset includes 3813 posts with 68,781 comments collected from weibo, the largest social media platform in China. |
EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records (2024.findings-acl)
Copied to clipboard
| Challenge: | EHR-SeqSQL is the first text-to-SQl dataset to include sequential and contextual questions. |
| Approach: | They propose a sequential text-to-SQL dataset for electronic health records databases that addresses critical yet underexplored aspects in text- to-SqL parsing. |
| Outcome: | The proposed dataset improves compositional generalization efficiency and improves interactivity and compositionality. |
KEEP CHATTING! An Attractive Dataset for Continuous Conversation Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing works about persona dialogue such as PersonaChat have greatly facilitated the chatbot with configurable and persistent personalities. |
| Approach: | They propose to collect a dataset called ContinuousChat and rewrite it in style-specific ways to increase users' willingness to continue chatting. |
| Outcome: | The proposed model increases users' willingness to continue talking to the chatbot by increasing their personas to detailed-personas through experiences, daily life, future plans, or interesting stories. |
RePair: Automated Program Repair with Process-based Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Commercial-scale language models (LMs) have taken APR to unprecedented levels, but they are limited by parameters and humans interact with them through explicit prompts. |
| Approach: | They propose a method that utilizes process supervision to improve program repair by allowing users to input feedback from compilers and test cases. |
| Outcome: | The proposed method outperforms large outcome-based generation methods and is inspired by strategies used in programming competitions. |
Concise and Precise Context Compression for Tool-Using Language Models (2024.findings-acl)
Copied to clipboard
Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, Min Zhang, Wanxiang Che
| Challenge: | Existing methods suffer from key information loss and difficulty in adjusting the length of compressed sequences based on documentation lengths. |
| Approach: | They propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. |
| Outcome: | The proposed approach achieves comparable performance to the upper-bound baseline under 16x compression ratio. |
MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical decisions directly impact individuals’ health and well-being. |
| Approach: | They propose to use a dataset to jointly extract and classify medical decisions within clinical notes. |
| Outcome: | The proposed dataset contains clinical notes of eleven different phenotypes (diseases) annotated by ten types of medical decisions. |