Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

MAGE: Machine-generated Text Detection in the Wild (2024.acl-long)

Challenge:	Existing methods for semantics discovery focus on text, video, and audio, failing to leverage the rich multimodal information in the real world.
Approach:	They propose a method to construct augmentation views for multimodal data and use them to perform pre-training to establish well-initialized representations for subsequent clustering.
Outcome:	The proposed method improves on benchmark multimodal intent and dialogue act datasets by 2-6% over state-of-the-art methods.

Copied to clipboard

Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, Yue Zhang

PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models (2024.acl-long)

Challenge:	Existing research has focused on evaluating detection methods for specific domains or language models.
Approach:	They build a testbed to detect texts from diverse human writings and LLMs using different detection methods.
Outcome:	Empirical results show that the top performing detector can identify 84.12% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.

Copied to clipboard

Haoran Li, Dadi Guo, Donghao Li, Wei Fan, Qi Hu, Xin Liu, Chunkit Chan, Duanyi Yao, Yuan Yao, Yangqiu Song

GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators (2024.acl-long)

Challenge:	generative large language models (LLMs) exhibit surprising capability and integrate previous tasks into a unified text generation formulation.
Approach:	They propose a privacy evaluation benchmark to quantify the privacy leakage of language models.
Outcome:	The proposed benchmark compares PPLMs with different privacy implementations to find out how privacy leakage is handled.

Copied to clipboard

Yuchen Hu, Chen Chen, Chao-Han Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, EngSiong Chng

Exploring Chain-of-Thought for Multi-modal Metaphor Detection (2024.acl-long)

Challenge:	Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge.
Approach:	They propose a generative paradigm for translation tasks that integrates the diverse translation versions in N-best list.
Outcome:	The proposed model outperforms the state-of-the-art model on speech and machine translation benchmarks on various languages.

Copied to clipboard

Yanzhi Xu, Yueying Hua, Shichen Li, Zhongqing Wang

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation (2024.acl-long)

Challenge:	Metaphors are commonly found in advertising and internet memes, but lack of high-quality textual data is a challenge for language models . a new framework for multi-modal metaphor detection is being developed to address these challenges .
Approach:	They propose a framework that extracts and integrates knowledge from Large Language Models into smaller ones to improve model performance.
Outcome:	The proposed framework outperforms existing models on the MET-MEME dataset.

Copied to clipboard

DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, Ningyi Xu

A Unified Temporal Knowledge Graph Reasoning Model Towards Interpolation and Extrapolation (2024.acl-long)

Challenge:	Weight quantization has emerged as a popular solution to reduce memory and computational demands.
Approach:	They propose a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at sub-4-bit.
Outcome:	The proposed framework outperforms existing QAT methods on language understanding and complex reasoning benchmarks on sub-4-bit models.

Copied to clipboard

Kai Chen, Ye Wang, Yitong Li, Aiping Li, Han Yu, Xin Song

Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation (2024.acl-long)

Challenge:	Existing methods for temporal knowledge graphs de-emphasize temporal correlations between facts sequences and ignore inferring clues from missing facts.
Approach:	They propose a Temporal PAth-based reasoning model that is robust to ambiguous temporal data.
Outcome:	The proposed model outperforms SOTA methods on the link prediction task.

Copied to clipboard

Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, Jie Zhou

CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers (2024.acl-long)

Challenge:	Existing studies show that LLMs face challenges in effectively using retrieved information . authors propose a method that considers LLM as "Information Refiner"
Approach:	They propose a method that considers LLMs as "Information Refiners" they propose INFO-RAG, which is low-cost and general across various tasks .
Outcome:	The proposed method improves performance of LLaMA2 by 9.39% relative points . it is low-cost and general across various tasks, and is robust and in-context learning is possible .

Copied to clipboard

Yong Hu, Fandong Meng, Jie Zhou

Challenge:	Existing datasets aimed at Chinese learners and native speakers are limited in size and quality.
Approach:	They propose a method that simulates the input process through an input method and generates large-scale pseudo data that closely resembles the actual error distribution.
Outcome:	The proposed method outperforms existing methods and outperformed existing models.

Evaluating Dynamic Topic Models (2024.acl-long)

Copied to clipboard

Charu Karakkaparambil James, Mayank Nagda, Nooshin Haji Ghassemi, Marius Kloft, Sophie Fellenz

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024.acl-long)

Challenge:	Existing evaluation measures to evaluate the progression of topics in dynamic topic models (DTMs) are difficult due to their unsupervised nature, but are crucial for detecting trends in time-indexed documents.
Approach:	They propose to combine topic quality and temporal consistency to evaluate the progression of topics over time in dynamic topic models.
Outcome:	The proposed measure correlates well with human judgment and can be used to identify changing topics and evaluate different models and LLMs.

Copied to clipboard

Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou

Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification (2024.acl-long)

Challenge:	supervised fine-tuning (SFT) is a technique used to enhance multiple abilities in large language models.
Approach:	They propose to study the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during supervised fine-tuning.
Outcome:	The proposed model improves math reasoning and code generation with increasing data amount . the proposed model size and SFT strategies can be used to learn multiple skills with different scaling patterns.

Copied to clipboard

Shanshan Xu, Santosh T.y.s.s, Oana Ichim, Barbara Plank, Matthias Grabmair

Inference to the Best Explanation in Large Language Models (2024.acl-long)

Challenge:	Existing methods for NLP calibration ignore inherent human label variation (HLV) split votes are a problem in high-stakes domains such as legal and medical decisions .
Approach:	They present a case outcome classification dataset with judges' vote distributions and build a taxonomy of disagreement with SV-specific subcategories.
Outcome:	The proposed model is compared against a judge vote distribution and assesses the alignment of perceived difficulty between models and humans.

Copied to clipboard

Dhairya Dalal, Marco Valentino, Andre Freitas, Paul Buitelaar

A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus (2024.acl-long)

Challenge:	Large Language Models (LLMs) have found success in real-world applications, but their underlying explanatory process is still poorly understood.
Approach:	They propose to use a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLMs’ explanations.
Outcome:	The proposed framework can identify the best explanation with up to 77% accuracy (27% above random) while being intrinsically more efficient and interpretable.

Copied to clipboard

Eduard Poesina, Cornelia Caragea, Radu Ionescu

MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering (2024.acl-long)

Challenge:	Natural language inference (NLI) is an actively studied topic serving as a proxy for natural language understanding.
Approach:	They propose to use a Romanian NLI corpus to analyze sentence pairs . they use multiple machine learning methods to establish competitive baselines .
Outcome:	The proposed model improves on the best model by employing a new curriculum learning strategy based on data cartography.

Copied to clipboard

Xiusi Chen, Jyun-Yu Jiang, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Wei Wang

SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs (2024.acl-long)

Challenge:	Recent advances in few-shot question answering rely on pre-trained large language models and fine-tuning in specific settings.
Approach:	They propose to select the most informative data for fine-tuning to improve efficiency . they use an approximate graph algorithm and unsupervised question generation to generate QA pairs .
Outcome:	The proposed framework improves the performance of the few-shot question answering task on the open-domain QA task.

Copied to clipboard

Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu

Numerical Reasoning

Challenge:	Large language models can handle text and data, but blending text and numerical data presents significant challenges.
Approach:	They propose four tasks to evaluate the numerical reasoning and information fusion capabilities of large language models in sports data analytics.
Outcome:	The proposed tasks evaluate the numerical reasoning and information fusion capabilities of large language models in sports data analytics.

SciMON: Scientific Inspiration Machines Optimized for Novelty (2024.acl-long)

Copied to clipboard

Qingyun Wang, Doug Downey, Heng Ji, Tom Hope

Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction (2024.acl-long)

Challenge:	Existing literature-based hypothesis generation models focus on binary link prediction, limiting expressivity of hypotheses.
Approach:	They propose a framework that uses literature-based hypothesis generation as input . they use literature-derived literature as background and output natural language ideas .
Outcome:	The proposed model improves the ability of language models to generate new scientific directions grounded in literature.

Copied to clipboard

Yiren Jian, Tingkai Liu, Yunzhe Tao, Chunhui Zhang, Soroush Vosoughi, Hongxia Yang

Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024.acl-long)

Challenge:	EVLGen is a framework for visual-language pre-training with high computational demands.
Approach:	They propose a streamlined framework for the pre-training of visually conditioned language generation models with high computational demands.
Outcome:	The proposed framework accelerates training of vision-language models by a factor of 5 without compromising performance.

Copied to clipboard

Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, Ali Emami

Retrieval-Augmented Multilingual Knowledge Editing (2024.acl-long)

Challenge:	Large Language Models (LLMs) are increasingly used in high-stakes areas such as healthcare, law, and education.
Approach:	They propose a concept of Confidence-Probability Alignment that connects an LLM’s internal confidence to the confidence conveyed in the model’s response when explicitly asked about its certainty.
Outcome:	The proposed model shows the strongest confidence-probability alignment across a wide range of tasks.

Copied to clipboard

Weixuan Wang, Barry Haddow, Alexandra Birch

Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge (2024.acl-long)

Challenge:	Knowledge editing (KE) is an effective and economical alternative to inject new knowledge or to fix factual errors in Large Language Models (LLMs).
Approach:	They propose a multilingual knowledge editing method that can be used to update knowledge in LLMs by concatenating new knowledge retrieved from a knowledge base with users’ prompts before querying an LLM.
Outcome:	The proposed method outperforms baseline knowledge editing methods by a significant margin and is scalable to real-word application scenarios.

Copied to clipboard

Brendan Park, Madeline Janecek, Naser Ezzati-Jivan, Yifeng Li, Ali Emami

Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models (2024.acl-long)

Challenge:	Large Language Models have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning.
Approach:	They propose a framework to isolate models' ability in pronoun disambiguation from other visual processing challenges.
Outcome:	The proposed framework isolates the models’ ability in pronoun disambiguation from other visual processing challenges.

Copied to clipboard

Abhishek Kumar, Sarfaroz Yunusov, Ali Emami

Framing in the Presence of Supporting Data: A Case Study in U.S. Economic News (2024.acl-long)

Challenge:	Representative bias is a tendency of Large Language Models to generate outputs that mirror the experiences of certain identity groups, and affinity bias is an evaluative preference for specific narratives.
Approach:	They propose two new metrics to measure representative bias and affinity bias within large language models and present a new set of tasks designed with customized rubrics to detect these biases.
Outcome:	The proposed model identifies representative biases in prominent LLMs, with a preference for identities associated with being white, straight, and men.

Copied to clipboard

Alexandria Leto, Elliot Pickens, Coen Needell, David Rothschild, Maria Pacheco

Challenge:	a lack of objective measures to evaluate editorial choices makes research difficult . a computational framework is proposed to analyze newsworthy topics with objective measures .
Approach:	They propose a computational framework to analyze editorial choices in newsworthy topics . they focus on the economy because economic indicators provide a relatively easy way to determine selection and framing .
Outcome:	The proposed framework analyzes editorial choices on newsworthy topics . it finds that there are topics where objective measures exist in the form of supporting data .

Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences (2024.acl-long)

Copied to clipboard

Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Fuxiao Liu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, Furong Huang

TTM-RE: Memory-Augmented Document-Level Relation Extraction (2024.acl-long)

Challenge:	Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks, but their ability to extrapolate from image sequences has been less investigated.
Approach:	They propose a new benchmark to assess MLLMs’ sequential image reasoning abilities.
Outcome:	The proposed benchmark features 4,761 diverse image sequences with varying lengths.

Copied to clipboard

Chufan Gao, Xuan Wang, Jimeng Sun

Answer is All You Need: Instruction-following Text Embedding via Answering the Question (2024.acl-long)

Challenge:	Existing methods for document-level relation extraction are ineffective in exploiting the full potential of large amounts of training data with varied noise levels.
Approach:	They propose a novel approach that integrates a trainable memory module with a noisy-robust loss function that accounts for the positive-unlabeled setting to unlock the full potential of large-scale noisy training data.
Outcome:	The proposed model outperforms existing methods on a ReDocRED benchmark dataset with an absolute F1 score improvement of over 3%.

Copied to clipboard

Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, Jingbo Shang

Explore Spurious Correlations at the Concept Level in Language Models for Text Classification (2024.acl-long)

Challenge:	Existing methods for encoding instruction information fail to be sensitive to clearer criteria like “evaluate similarity based on emotion” . instead, we propose a different approach, which treats the instruction as a “question” about the input text and encodes the expected answers to obtain the representation accordingly.
Approach:	They propose a text embedder that captures characteristics of texts specified by user instructions clarifying the similarity criterion.
Outcome:	The proposed model improves instruction-following capabilities when applied to large language models and encoder-based LMs.

Copied to clipboard

Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, Furong Huang

Every Answer Matters: Evaluating Commonsense with Probabilistic Measures (2024.acl-long)

Challenge:	Language models have demonstrated remarkable performance in numerous NLP tasks, employing both fine-tuning and in-context learning (ICL) methods.
Approach:	They propose a method to assess concept bias in models during fine-tuning and in-context learning using ChatGPT.
Outcome:	The proposed method outperforms token removal approaches and is validated through extensive testing.

Copied to clipboard

Qi Cheng, Michael Boratko, Pranay Kumar Yelugam, Tim O’Gorman, Nalini Singh, Andrew McCallum, Xiang Li

GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis (2024.acl-long)

Challenge:	Existing commonsense evaluations are often posed as multiple-choice questions, allowing models to exploit systematic biases.
Approach:	They propose a generative task that evaluates common sense via multiple open-ended generations and a method that strongly correlates with human judgments.
Outcome:	The proposed method outperforms strong language model baselines on a dataset of human and machine common sense.

Copied to clipboard

Yueqi Xie, Minghong Fang, Renjie Pi, Neil Gong

Pouring Your Heart Out: Investigating the Role of Figurative Language in Online Expressions of Empathy (2024.acl-long)

Challenge:	Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs.
Approach:	They propose a method which scrutinizes the gradients of safety-critical parameters in large LLMs to detect jailbreak prompts.
Outcome:	The proposed method outperforms Llama Guard in detecting jailbreak prompts despite extensive finetuning with a large dataset.

Copied to clipboard

Gyeongeun Lee, Christina Wong, Meghan Guo, Natalie Parde

An Information-Theoretic Approach to Analyze NLP Classification Tasks (2024.acl-long)

Challenge:	Empathy is a social mechanism used to support and strengthen emotional connection with others, including in online communities.
Approach:	They propose to use a subcategory of linguistic phenomena, figurative language, to detect online expressions of empathy.
Outcome:	The proposed model improves on the use of metaphor, idiom, and hyperbole in online expressions of empathy, with 0.942 and 0.809 F1 scores.

Copied to clipboard

Luran Wang, Mark Gales, Vatsal Raina

Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024.acl-long)

Challenge:	Natural language processing (NLP) tasks take either a single or multiple text elements to predict an output variable.
Approach:	They propose an information-theoretic framework to analyse the influence of inputs on the output of text classification tasks.
Outcome:	The proposed framework is available at: https://github.com/WangLuran/nlp-element-influence.

Copied to clipboard

Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, Saab Mansour

Wav2Gloss: Generating Interlinear Glossed Text from Speech (2024.acl-long)

Challenge:	Conversational systems often rely on embedding models for intent classification and intent clustering tasks.
Approach:	They propose a toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task.
Outcome:	The proposed model improves on the linguistic dimensions while affecting performance on downstream task metrics.

Copied to clipboard

Taiqi He, Kwanghee Choi, Lindia Tjuatja, Nathaniel Robinson, Jiatong Shi, Shinji Watanabe, Graham Neubig, David Mortensen, Lori Levin

Speech & Audio in NLP

Challenge:	Interlinear Glossed Text (IGT) is a form of linguistic annotation that can support documentation and resource creation for endangered languages.
Approach:	They propose a task in which these four annotation components are extracted automatically from speech and introduce a dataset to lay the groundwork for future research on IGT generation from speech.
Outcome:	The proposed dataset provides the first dataset to lay the groundwork for future research on IGT generation from speech, including end-to-end versus cascaded, monolingual versus multilingual, and single-task versus multiple-task approaches.

Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification (2024.acl-long)

Copied to clipboard

Yibo Hu, Erick Skorupa Parolin, Latifur Khan, Patrick Brandt, Javier Osorio, Vito D’Orazio

SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation (2024.acl-long)

Challenge:	Existing annotation codebook is labor-intensive for coding events from large datasets.
Approach:	They propose to use existing annotation codebook to classify political relations without extensive annotations.
Outcome:	The proposed methods outperform dictionary-based methods and the existing ontology annotation codebook and improve interpretability and efficiency.

Copied to clipboard

Ziyao Xu, Houfeng Wang

OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following (2024.acl-long)

Challenge:	Existing studies on compositional generalization in data-to-text generation focus on one manifestation, Systematicity, Productivity, Order invariance, and Rule learnability.
Approach:	They propose a method for evaluation of compositional generalization in data-to-text generation that includes four aspects of manifestations and allows high-quality evaluation without additional manual annotations.
Outcome:	The proposed method is based on two datasets and evaluates existing language models including LLMs.

Copied to clipboard

Haochen Shi, Zhiyuan Sun, Xingdi Yuan, Marc-Alexandre Côté, Bang Liu

Multimodal Instruction Tuning with Conditional Mixture of LoRA (2024.acl-long)

Challenge:	Embodied Instruction Following (EIF) is a crucial task in embodied learning . however, there is n'a unified understanding regarding the impact of various components on task performance .
Approach:	They propose a framework that delineates the core components essential for embodied learning tasks . they integrate a multi-agent design into the Planner component of their LLM-centric architecture .
Outcome:	OPEx delineates the core components essential for solving embodied learning tasks . integrating a multi-agent design into the Planner component of the LLM-centric architecture further elevates performance.

Copied to clipboard

Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang

DocLens: Multi-aspect Fine-grained Medical Text Evaluation (2024.acl-long)

Challenge:	Multimodal Large Language Models (MLLMs) have demonstrated proficiency in diverse tasks across different domains.
Approach:	They propose a method that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA.
Outcome:	Experimental results show that MixLoRA outperforms LoRA with the same or higher ranks . MLLMs have demonstrated remarkable proficiency in diverse tasks across domains .

Copied to clipboard

Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, Carolyn Rose

FOFO: A Benchmark to Evaluate LLMs’ Format-Following Capability (2024.acl-long)

Challenge:	Medical text generation systems are widely used to assist with administrative work and highlight salient information to support decision-making.
Approach:	They propose a set of metrics to evaluate completeness, conciseness, and attribution of medical text at a fine-grained level.
Outcome:	The proposed framework exhibits substantially higher agreement with medical experts than existing metrics.

Copied to clipboard

Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, Caiming Xiong

Challenge:	Existing benchmarks fail to assess large language models’ format-following proficiency adequately.
Approach:	They propose a benchmark to evaluate large language models' ability to follow complex, domain-specific formats.
Outcome:	The proposed framework evaluates large language models' ability to follow complex, domain-specific formats across open-source and closed-source models.

Hyper-CL: Conditioning Sentence Representations with Hypernetworks (2024.acl-long)

Copied to clipboard

Young Yoo, Jii Cha, Changhyeon Kim, Taeuk Kim

Analysis of Multi-Source Language Training in Cross-Lingual Transfer (2024.acl-long)

Challenge:	Existing approaches to sentence embeddings do not capture fine-grained semantics of sentences.
Approach:	They propose a method that integrates hypernetworks with contrastive learning to generate conditioned sentence representations.
Outcome:	The proposed method narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoding approach.

Copied to clipboard

Seonghoon Lim, Taejun Yun, Jinhyeon Kim, Jihun Choi, Taeuk Kim

ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions (2024.acl-long)

Challenge:	Existing studies on cross-lingual transfer (XLT) methods address data scarcity problem . cross-linguistic transfer (xLT) techniques are effective at fine-tuning multilingual LMs .
Approach:	They propose to use multiple source languages to improve XLT by fine-tuning multilingual models . they propose to employ arbitrary combinations of source languages for XL to improve performance .
Outcome:	The proposed technique improves performance on language-agnostic or task-specific features by using multiple source languages.

Copied to clipboard

Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Evuru, Ramaneswaran S, S Sakshi, Dinesh Manocha

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2024.acl-long)

Challenge:	ABEX is a novel and effective generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks.
Approach:	They propose a novel generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks based on a paradigm for generating diverse forms of an input document .
Outcome:	The proposed method outperforms all baselines qualitatively with improvements of 0.04% - 38.8%.

Copied to clipboard

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, Madian Khabsa

Learn from Failure: Fine-tuning LLMs with Trial-and-Error Data for Intuitionistic Propositional Logic Proving (2024.acl-long)

Challenge:	Existing benchmarks for text comprehension only cover 30 languages, but lack of labeled data is a major obstacle to building functional systems in most languages.
Approach:	They present a multiple-choice machine reading comprehension dataset spanning 122 languages . they use it to evaluate the capabilities of multilingual masked language models and large language models .
Outcome:	The proposed dataset enables the evaluation of text models in high-, medium- and low-resource languages.

Copied to clipboard

Chenyang An, Zhibo Chen, Qihao Ye, Emily First, Letian Peng, Jiayun Zhang, Zihan Wang, Sorin Lerner, Jingbo Shang

Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach (2024.acl-long)

Challenge:	Recent advances in Automated Theorem Proving have shown the effectiveness of leveraging a (large) language model that generates tactics (i.e. proof steps) to search through proof states.
Approach:	They propose to use a large language model that generates tactics to search through proof states.
Outcome:	The proposed model solves more unseen theorems with lower trial searches than the current model, which only learns from failed attempts.

Copied to clipboard

Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, Sungroh Yoon

IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction (2024.acl-long)

Challenge:	primarily addressed in text-to-image retrieval task using dialogue-form context query . conventionally, text-based retrieval methods rely on initial text descriptions .
Approach:	They propose a plug-based retrieval method that uses large language models as questioners to generate non-redundant questions about the attributes of the target image.
Outcome:	The proposed method performs better than zero-shot and fine-tuned baselines in benchmarks.

Copied to clipboard

Inna Lin, Ashish Sharma, Christopher Rytting, Adam Miner, Jina Suh, Tim Althoff

Token-wise Influential Training Data Retrieval for Large Language Models (2024.acl-long)

Challenge:	Various communication frameworks assist individuals in conducting difficult conversations by providing a set of skills to apply.
Approach:	They propose a language-based simulation system that provides just-in-time feedback to support the practice and learning of interpersonal effectiveness skills.
Outcome:	The proposed training system improves self-efficacy and reduces negative emotions by 27% compared to the GPT-4 training system.

Copied to clipboard

Huawei Lin, Jikai Long, Zhaozhuo Xu, Weijie Zhao

Tree-of-Counterfactual Prompting for Zero-Shot Stance Detection (2024.acl-long)

Challenge:	Large language models (LLMs) have been widely used in various industries due to their unprecedented scale and impressive capabilities derived from the massive training dataset.
Approach:	They propose a framework that can estimate the influence of training data by caching and retrieval.
Outcome:	The proposed framework can estimate the influence of training data within minutes, achieving over a speedup of 6,326x.

Copied to clipboard

Maxwell Weinzierl, Sanda Harabagiu

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024.acl-long)

Challenge:	Stance detection is used to infer attitudes from human communications . stance decisions involve complex judgments generated by LLMs .
Approach:	They propose a method for stance detection which relies on a new prompting framework . it allows for more than one stance object type and no examples of stance attribution .
Outcome:	The proposed method outperforms fine-tuned stance detection systems.

Copied to clipboard

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried

FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)

Challenge:	Existing benchmarks focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve.
Approach:	They propose a benchmark to assess the performance of multimodal web agents . they use visual and textual inputs to process and interpret natural language instructions .
Outcome:	a new benchmark assesses the performance of multimodal agents on visually grounded tasks . the benchmark identifies limitations of text-only agents and offers insights towards building stronger agents for the web .

Copied to clipboard

Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, Saab Mansour

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback (2024.acl-long)

Challenge:	Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis.
Approach:	They propose a fine-grained evaluator specifically tailored for the summarization task using large language models.
Outcome:	The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods.

Copied to clipboard

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

Prompt Refinement with Image Pivot for Text-to-Image Generation (2024.acl-long)

Challenge:	Recent advances in large language models have influenced the development of video large multimodal models (VLMMs).
Approach:	They propose a method that integrates video descriptions as context into a multimodal AI system to enrich the understanding of video content.
Outcome:	Empirical evaluations show that the proposed approach outperforms existing approaches for video large multimodal models (VLMMs)

Copied to clipboard

Jingtao Zhan, Qingyao Ai, Yiqun Liu, Yingwei Pan, Ting Yao, Jiaxin Mao, Shaoping Ma, Tao Mei

Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation (2024.acl-long)

Challenge:	Recent advances in text-to-image generation have markedly expanded the boundaries of digital artistry, enabling the creation of visually compelling images with unprecedented ease.
Approach:	They propose to decompose the prompt refinement process into two tasks: inferring user-preferred images from user languages and translating them into system languages.
Outcome:	Experiments show that PRIP outperforms baselines and transfers to unseen systems in a zero-shot manner.

Copied to clipboard

Masato Mita, Soichiro Murakami, Akihiko Kato, Peinan Zhang

AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation (2024.acl-long)

Challenge:	Existing benchmarks and problem sets for automatic ad text generation are lacking . however, the growing volume of search queries has fueled research on the automatic generation of ads.
Approach:	They propose to standardize the task of automatic ad text generation (ATG) using a benchmark dataset, CAMERA, to enable the utilization of multi-modal information and facilitate industry-wise evaluations.
Outcome:	The proposed dataset standardizes the task of automatic ad text generation (ATG) it shows that existing metrics align with human evaluations and that the proposed methods can be used to improve the quality of the results.

Copied to clipboard

Zhaowei Wang, Wei Fan, Qing Zong, Hongming Zhang, Sehyun Choi, Tianqing Fang, Xin Liu, Yangqiu Song, Ginny Wong, Simon See

Reflect-RL: Two-Player Online RL Fine-Tuning for LMs (2024.acl-long)

Challenge:	Existing work shows that LLMs are deficient in abstract ability, and how to improve it remains unexplored.
Approach:	They propose a framework AbsInstruct to enhance LLMs’ abstract ability through instruction tuning.
Outcome:	The proposed framework can enhance LLMs’ abstraction ability with strong generalization performance while maintaining their general instruction-following abilities.

Copied to clipboard

Runlong Zhou, Simon Du, Beibin Li

Can ChatGPT’s Performance be Improved on Verb Metaphor Detection Tasks? Bootstrapping and Combining Tacit Knowledge (2024.acl-long)

Challenge:	supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance.
Approach:	They propose a two-player system to fine-tune an LM using SFT and online RL . they use negative example generation to enhance error-correction ability of the reflection model .
Outcome:	The proposed system outperforms SFT and online RL without reflection on a GPT-2 XL 1.56B model.

Copied to clipboard

Cheng Yang, Puli Chen, Qingbao Huang

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning (2024.acl-long)

Challenge:	Existing methods for supervised metaphor detection are limited by their performance.
Approach:	They propose to use ChatGPT to detect most prevalent verb metaphors among metaphors . they use literal collocations of target verbs and subject-object pairs of verbs to detect them .
Outcome:	The proposed method achieves the best performance on the unsupervised verb metaphors detection task compared to existing unsupervised methods or direct prediction using ChatGPT.

Copied to clipboard

Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu

An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation (2024.acl-long)

Challenge:	Experimental results show that fine-tuning of large language models for specific tasks can be challenging . distribution shift during fine-timing can lead to performance degradation in general task capabilities .
Approach:	They propose a new approach that bridges the distribution gap between task datasets and LLMs by guiding fine-tuning with a distilled dataset generated by the model itself.
Outcome:	The proposed approach achieves comparable or superior performance on downstream tasks compared to the vanilla approach.

Copied to clipboard

Kun Zhu, Xiaocheng Feng, Xiyuan Du, Yuxuan Gu, Weijiang Yu, Haotian Wang, Qianglong Chen, Zheng Chu, Jingchang Chen, Bing Qin

RORA: Robust Free-Text Rationale Evaluation (2024.acl-long)

Challenge:	Experimental results show that retrieval-augmented generation improves accuracy and relevance of large language models.
Approach:	They propose to introduce the information bottleneck theory into retrieval-augmented generation by maximizing mutual information between compression and ground output while minimizing mutual information .
Outcome:	The proposed approach improves accuracy and correctness of answer generation and conciseness with 2.5% compression rate.

Copied to clipboard

Zhengping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van Durme, Anqi Liu

Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents (2024.acl-long)

Challenge:	Existing metrics rely on degree to which rationale supports a label, but they fail to evaluate rationales that inadvertently leak the label.
Approach:	They propose a RObust free-text RAtionale evaluation against label leakage that quantifies the new information supplied by a rationale to justify the label.
Outcome:	The proposed evaluation outperforms existing methods in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage.

Copied to clipboard

Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

InstructProtein: Aligning Human and Protein Language via Knowledge Instruction (2024.acl-long)

Challenge:	Current language model-driven agents lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions.
Approach:	They propose a benchmark to inspect users’ implicit intentions through explicit queries and a model expert as the upstream in agent design to enhance user-agent interaction.
Outcome:	The proposed approach excels at identifying vague user tasks, recovering and summarizing critical missing information, setting precise and necessary agent execution goals, and minimizing redundant tool usage, thus boosting overall efficiency.

Copied to clipboard

Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, Huajun Chen

ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) are a promising new approach to understanding biological sequences such as proteins.
Approach:	They propose an LLM that can generate protein sequences in human and protein languages by pre-training an Lm on protein and natural language corpora and supervised instruction tuning to facilitate alignment.
Outcome:	The proposed model outperforms state-of-the-art LLMs on protein-text generation tasks by a large margin.

Copied to clipboard

Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, Dan Roth

Linguistically Conditioned Semantic Textual Similarity (2024.acl-long)

Challenge:	In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon the insights from disciplines such as user experience research and human behavioral psychology to ensure that the results are reliable.
Approach:	They propose a framework for human evaluation of generative large language models that takes into account usability, aesthetics and cognitive biases.
Outcome:	The proposed framework is based on the framework proposed by Deutsch and alnajjar . it is aimed at ensuring that human evaluation is accurate in the age of generative AI .

Copied to clipboard

Jingxuan Tu, Keer Xu, Liulu Yue, Bingyang Ye, Kyeongmin Rim, James Pustejovsky

Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future (2024.acl-long)

Challenge:	Semantic textual similarity (STS) is a fundamental NLP task that measures the semantic similarity between two sentences.
Approach:	They propose to use a conditional STS dataset to measure sentences’ similarity conditioned on a certain aspect to reduce the inherent ambiguity posed by the sentences.
Outcome:	The proposed method improves the performance over baselines on the C-STS dataset with over 80% F1 score.

Copied to clipboard

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, Ting Liu

TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models (2024.acl-long)

Challenge:	Recent studies have revealed that chain-of-thought prompting significantly enhances LLM’s reasoning capabilities, which attracts widespread attention from both academics and industry.
Approach:	They propose to summarize advanced methods through a taxonomy that offers novel perspectives.
Outcome:	The proposed method delineates the challenges and future directions, thereby shedding light on future research.

Copied to clipboard

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, Bing Qin

BeamAggR: Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering (2024.acl-long)

Challenge:	Grasping the concept of time is a fundamental facet of human cognition.
Approach:	They propose a hierarchical temporal reasoning benchmark that covers a broad spectrum of temporal phenomena.
Outcome:	The proposed benchmark shows that state-of-the-art LLMs are still far behind humans in temporal reasoning .

Copied to clipboard

Zheng Chu, Jingchang Chen, Qianglong Chen, Haotian Wang, Kun Zhu, Xiyuan Du, Weijiang Yu, Ming Liu, Bing Qin

ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated strong reasoning capabilities, but they still suffer from factual errors when tackling knowledge-intensive tasks.
Approach:	They propose a reasoning framework for knowledge-intensive multi-hop QA that prioritizes promising answers at each hop of question.
Outcome:	The proposed framework outperforms SOTA methods on four open-domain multi-hop reasoning datasets by 8.5%.

Copied to clipboard

Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, Deqing Yang

TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation (2024.acl-long)

Challenge:	ANALOGYKB is a million-scale analogy knowledge base based on existing knowledge graphs (KGs) based upon relational knowledge triples, we can discover new analogies using the corresponding relations between concepts.
Approach:	They propose a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) ANALOGYKB identifies analogies of the same relations and analogies from analogous relations .
Outcome:	The proposed model enables both smaller LMs and LLMs to gain better analogical reasoning capabilities.

Copied to clipboard

Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, Xiao-Ming Wu

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024.acl-long)

Challenge:	Current methods for Continual Dialogue State Tracking (DST) struggle with catastrophic forgetting and knowledge transfer between tasks.
Approach:	They propose a framework for task skill localization and consolidation that enables effective knowledge transfer without relying on memory replay.
Outcome:	The proposed framework shows a 7.6% increase in Avg. JGA and 11% rise in BWT metrics over existing state-of-the-art methods.

Copied to clipboard

Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang

Grounding Language Model with Chunking-Free In-Context Retrieval (2024.acl-long)

Challenge:	Mixture-of-Experts (MoE) architectures face challenges in ensuring expert specialization . despite the promising performance, scaling language models to an extremely large scale is associated with exceedingly high computational costs.
Approach:	They propose an architecture that allows for ultimate expert specialization by segmenting experts into mN ones and activating mK from them.
Outcome:	The proposed architecture achieves comparable performance with GShard with 2B parameters and computation.

Copied to clipboard

Hongjin Qian, Zheng Liu, Kelong Mao, Yujia Zhou, Zhicheng Dou

Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generation (2024.acl-long)

Challenge:	CFIC retrieval approach eliminates the need for document chunking and provides a more efficient and efficient method for RAG systems.
Approach:	They propose a Chunking-Free In-Context retrieval approach specifically tailored for RAG systems . they employ auto-aggressive decoding to accurately identify specific evidence text .
Outcome:	The proposed method is better than traditional methods on open question answering datasets.

Copied to clipboard

Jiaxin Bai, Yicheng Wang, Tianshi Zheng, Yue Guo, Xin Liu, Yangqiu Song

Active Prompting with Chain-of-Thought for Large Language Models (2024.acl-long)

Challenge:	Abductive reasoning is the process of making educated guesses to provide explanations for observations.
Approach:	They propose a task of complex logical hypothesis generation to generate a complex logique hypothesis that can explain a set of observations.
Outcome:	The proposed model generates logical hypotheses closer to the reference hypothesis, but not better on unseen observations.

Copied to clipboard

Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, Tong Zhang

EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs (2024.acl-long)

Challenge:	Existing methods to annotate large language models rely on a fixed set of human-annotated exemplars, which are not always the most effective for different tasks.
Approach:	They propose a method to adapt large language models to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning) they introduce several metrics to characterize uncertainty so as to select the most uncertain questions for annotation.
Outcome:	The proposed method significantly improves performance on eight complex reasoning tasks.

Copied to clipboard

Xiangyu Zhao, Bo Liu, Qijiong Liu, Guangyuan Shi, Xiao-Ming Wu

Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search (2024.acl-long)

Challenge:	Existing multimodal models that depend on encoders like CLIP or ImageBind need ample amounts of training data to bridge modalities.
Approach:	They propose an efficient model that leverages bidirectional conditional diffusion model to foster more efficient modality interactions.
Outcome:	The proposed model is able to train a projection layer linking an LLM and an adapter to align the LLM’s text space with the bidirectional diffusion model.

Copied to clipboard

Haochen Li, Xin Zhou, Zhiqi Shen

A Multidimensional Framework for Evaluating Lexical Semantic Change with Social Science Applications (2024.acl-long)

Challenge:	Experimental results show that ReCo significantly boosts retrieval accuracy across sparse, zero-shot dense and fine-tuned dense search settings.
Approach:	They propose a generation-augmented retrieval framework that additionally Rewrites the Code (ReCo) within the codebase for style normalization.
Outcome:	The proposed method significantly boosts retrieval accuracy across sparse, zero-shot dense, and fine-tuned dense retrieval settings in diverse search scenarios.

Copied to clipboard

Naomi Baes, Nick Haslam, Ekaterina Vylomova

Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal (2024.acl-long)

Challenge:	Historical linguists have identified multiple forms of lexical semantic change.
Approach:	They propose a framework for integrating and evaluating lexical semantic changes in historical linguists and a unified computational methodology for evaluating them concurrently.
Outcome:	The proposed framework enables lexical semantic change to be mapped economically and systematically and has applications in computational social science.

Copied to clipboard

Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	Existing methods to train LLMs on previous training data are not feasible in real-world applications because of catastrophic forgetting.
Approach:	They propose a framework that uses the LLM to generate synthetic instances for rehearsal and refine the instance outputs based on the synthetic inputs.
Outcome:	The proposed framework achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

Citation-Enhanced Generation for LLM-based Chatbots (2024.acl-long)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Weitao Li, Junkai Li, Weizhi Ma, Yang Liu

Transitive Consistency Constrained Learning for Entity-to-Entity Stance Detection (2024.acl-long)

Challenge:	Existing efforts to alleviate hallucination in chatbots require additional training and data annotation.
Approach:	They propose a Citation-Enhanced Generation approach that uses retrieval argumentation to generate citations and a natural language inference-based citation generation module to generate content.
Outcome:	The proposed method outperforms state-of-the-art methods on three benchmarks.

Copied to clipboard

Haoyang Wen, Eduard Hovy, Alexander Hauptmann

Feature-Adaptive and Data-Scalable In-Context Learning (2024.acl-long)

Challenge:	Entity-to-entity stance detection is a streamlined task without the complex dependency structure for structural sentiment analysis.
Approach:	They propose a method that models transitive consistency constraints during training to help train entity-to-entity stance detection models.
Outcome:	The proposed method improves both classification-based and generation-based models while large language models struggle with predicting link direction and neutral labels.

Copied to clipboard

Jiahao Li, Quan Wang, Licheng Zhang, Guoqing Jin, Zhendong Mao

Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games (2024.acl-long)

Challenge:	In-context learning (ICL) is a popular way to stimulate LLM capabilities for downstream tasks due to context length constraints.
Approach:	They propose a feature-adaptive and data-scalable in-context learning framework which leverages task-adaptives to promote inference on the downstream task.
Outcome:	The proposed framework outperforms state-of-the-art methods on 10 datasets under different data settings and LLM scale.

Copied to clipboard

Yizhe Zhang, Jiarui Lu, Navdeep Jaitly

WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models (2024.acl-long)

Challenge:	Large language models (LLMs) are effective at answering clear questions but when faced with ambiguous queries they act unpredictably and produce incorrect outputs.
Approach:	They propose to use a surrogate problem to assess an LLMs’s ability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries.
Outcome:	The proposed model outperforms human players on the entity-deducing task by a large margin.

Copied to clipboard

Shangqing Tu, Yuliang Sun, Yushi Bai, Jifan Yu, Lei Hou, Juanzi Li

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models (2024.acl-long)

Challenge:	Recent studies have developed watermarking algorithms which restrict the generation process to leave an invisible trace for watermark detection.
Approach:	They propose a benchmarking procedure that compares different methods to ensure consistent watermarking strength and jointly evaluates their generation and detection performance.
Outcome:	The proposed benchmark compares 4 open-source watermarks on 2 LLMs under 2 watermarking strengths and observes the common struggles for current methods on maintaining the generation quality.

Copied to clipboard

Yida Zhao, Chao Lou, Kewei Tu

A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation (2024.acl-long)

Challenge:	Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences.
Approach:	They propose a class of Transformer language models with explicit dependency-based inductive bias.
Outcome:	Experiments show that the proposed models outperform constituency-based models on sentences annotated with dependency trees and achieve better generalization.

Copied to clipboard

Zhengrui Ma, Qingkai Fang, Shaolei Zhang, Shoutao Guo, Yang Feng, Min Zhang

Probing Language Models for Pre-training Data Detection (2024.acl-long)

Challenge:	Existing translation pipelines require additional cascade components to achieve speech-to-speech translation.
Approach:	They propose a non-autoregressive generation framework for simultaneous speech translation . it integrates both text-to-text and speech-tospeech tasks into a unified framework .
Outcome:	The proposed framework outperforms state-of-the-art models in speech-to-text and speech- to-speech tasks.

Copied to clipboard

Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Bing Liu, Haonan Lu, Wenliang Chen

Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding (2024.acl-long)

Challenge:	Large Language Models (LLMs) have shown impressive capabilities, while raising concerns about the data contamination due to privacy issues and leakage of benchmark datasets in the pre-training phase.
Approach:	They propose to utilize the probing technique to examine the model’s internal activations to detect pre-training data contamination by examining the model's internal activates.
Outcome:	The proposed method outperforms baselines and achieves state-of-the-art performance on both WikiMIA and ArxivMIA, with additional experiments confirming its efficacy.

Copied to clipboard

Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, Tat-Seng Chua

IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generation (2024.acl-long)

Challenge:	Existing research in complex event analysis has made significant strides but is constrained by inadequate natural language processing techniques.
Approach:	They propose a novel approach using Large Language Models to extract and analyze the event chain within TCE, characterized by their key points and timestamps.
Outcome:	The proposed model performs comparable to models with long context window and retrieval-augmented generation method in three distinct tasks .

Copied to clipboard

Senyu Han, Lu Chen, Li-Min Lin, Zhengshan Xu, Kai Yu

Language Model Adaption for Reinforcement Learning with Natural Language Action Space (2024.acl-long)

Challenge:	Language models have demonstrated their capabilities in storyline creation and human-like character role-playing.
Approach:	They propose a director-actor coordinate agent framework that generates drama scripts . framework allows actors to role-play their characters while maintaining plot development .
Outcome:	The proposed framework generates drama scripts from a drama plot outline and human actors can play their characters.

Copied to clipboard

Jiangxing Wang, Jiachen Li, Xiao Han, Deheng Ye, Zongqing Lu

Evaluating Intention Detection Capability of Large Language Models in Persuasive Dialogues (2024.acl-long)

Challenge:	Previous research has focused on reducing the size of the natural language action space due to the combinatorial nature of the language.
Approach:	They propose mutual-information regularized policy optimization to reduce the action space by dynamically adjusting the prior provided by the pretrained model.
Outcome:	The proposed method improves monotonically on the mutual-information regularized RL objective.

Copied to clipboard

Hiromasa Sakurai, Yusuke Miyao

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024.acl-long)

Challenge:	Existing studies measure the intention detection capability of machine learning models without considering the conversational history.
Approach:	They modified existing persuasive conversation datasets and created a dataset using a multiple-choice paradigm to evaluate LLMs' intention detection capability.
Outcome:	The proposed model can detect speakers' intentions well in persuasive multi-turn dialogs using the largest available Large Language Models (LLMs).

Copied to clipboard

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Persuading across Diverse Domains: a Dataset and Persuasion Large Language Model (2024.acl-long)

Challenge:	Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance.
Approach:	They propose a prompt compression tool that improves LLMs' perception of key information in input prompts by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo.
Outcome:	The proposed solution improves performance and reduces costs and latency by up to 21.4% with around 4x fewer tokens in the NaturalQuestions benchmark.

Copied to clipboard

Chuhao Jin, Kening Ren, Lingzhen Kong, Xiting Wang, Ruihua Song, Huan Chen

HealMe: Harnessing Cognitive Reframing in Large Language Models for Psychotherapy (2024.acl-long)

Challenge:	Persuasive dialogue requires multi-turn following and planning abilities to achieve the goal of persuating users.
Approach:	They propose a general method to learn a persuasive model based on LLMs through intent-to-strategy reasoning, which summarizes the intent of user’s utterance and reasons next strategy to respond.
Outcome:	The proposed method outperforms baselines on automatic evaluation metric Win-Rate and human evaluation on two datasets.

Copied to clipboard

Mengxi Xiao, Qianqian Xie, Ziyan Kuang, Zhicheng Liu, Kailai Yang, Min Peng, Weiguang Han, Jimin Huang

Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)

Challenge:	Large Language Models (LLMs) can be used in psychotherapy to overcome challenges such as shame, distrust, and resource scarcity.
Approach:	They propose a cognitive reframing therapy method that uses empathetic dialogue to address deep-rooted negative thoughts and fosters rational, balanced perspectives.
Outcome:	The proposed model outperforms other models in terms of empathy, guidance, and logical coherence, demonstrating its effectiveness and potential positive impact on psychotherapy.

Copied to clipboard

Zirun Guo, Tao Jin, Zhou Zhao

An Effective Pronunciation Assessment Approach Leveraging Hierarchical Transformers and Pre-training Strategies (2024.acl-long)

Challenge:	Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like.
Approach:	They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities.
Outcome:	The proposed framework outperforms existing methods significantly across evaluation metrics.

Copied to clipboard

Bi-Cheng Yan, Jiun-Ting Li, Yi-Cheng Wang, Hsin Wei Wang, Tien-Hong Lo, Yung-Chang Hsu, Wei-Cheng Chao, Berlin Chen

Detection-Correction Structure via General Language Model for Grammatical Error Correction (2024.acl-long)

Challenge:	Existing attempts to quantify a second language learner’s pronunciation proficiency in a target language often sideline the hierarchy of linguistic units and relatedness among the pronunciation aspects.
Approach:	They propose a hierarchical automatic pronunciation assessment method that models the intrinsic structures of an utterance while considering the relatedness among the pronunciation aspects.
Outcome:	The proposed method can be used to quantify a second language learner’s pronunciation proficiency in a target language by providing fine-grained feedback with multiple pronunciation aspect scores at various linguistic levels.

Copied to clipboard

Wei Li, Houfeng Wang

Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024.acl-long)

Challenge:	Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits.
Approach:	They propose a detection-correction structure based on the general language model which integrates detection and correction into a single model.
Outcome:	The proposed model outperforms the state-of-the-art models on English and Chinese datasets.

Copied to clipboard

Yongxin Zhu, Dan Su, Liqiang He, Linli Xu, Dong Yu

Speech & Audio in NLP

Challenge:	Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity.
Approach:	They propose a hierarchical transformer that quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a transformer architecture.
Outcome:	The proposed model outperforms existing speech language models in word error rate, speech quality, and speaker similarity.

Selene: Pioneering Automated Proof in Software Verification (2024.acl-long)

Copied to clipboard

Lichen Zhang, Shuai Lu, Nan Duan

Dissecting Human and LLM Preferences (2024.acl-long)

Challenge:	Currently, software verification is resource-intensive and manpower-consuming.
Approach:	They propose a project-level automated proof benchmark based on the seL4 operating system . they propose augmentations to enhance the flexibility of the framework and lightweight verification environment .
Outcome:	The proposed framework provides a comprehensive framework for end-to-end proof generation and a lightweight verification environment.

Copied to clipboard

Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, Pengfei Liu

UniCoder: Scaling Code Large Language Model via Universal Code (2024.acl-long)

Challenge:	a recent study shows that human and Large Language Model preferences are important for model fine-tuning and evaluation.
Approach:	They dissect the preferences of human and 32 different Large Language Models to understand their quantitative composition.
Outcome:	The proposed model is compared with 32 different large language models using real-world user-model conversations.

Copied to clipboard

Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, Zhoujun Li

AoE: Angle-optimized Embeddings for Semantic Textual Similarity (2024.acl-long)

Challenge:	Experimental results show that UniCoder with the universal code significantly outperforms the previous prompting methods by a large margin.
Approach:	They introduce the universal code (UniCode) as the intermediate representation of algorithm steps using conventions of programming languages.
Outcome:	The proposed model outperforms previous prompting methods by a large margin . the proposed model is based on a dataset of natural-language questions and code solutions .

Copied to clipboard

Xianming Li, Jing Li

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews (2024.acl-long)

Challenge:	Existing text embedding models neglect cosine saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeds.
Approach:	They propose a novel angle-optimized Embedding model that optimizes angle differences in complex space to explore similarity in saturation zones better.
Outcome:	The proposed model outperforms existing models neglecting cosine saturation zones on STS and MTEB benchmarks and can produce high-quality embeddings and benefit downstream tasks.

Copied to clipboard

Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua Xiao

Challenge:	Existing methods focus on knowledge and linguistic patterns of characters.
Approach:	They propose to evaluate character fidelity of role-playing agents with psychological scales . they propose to use psychological scale to measure personality traits of RPAs based on personality traits.
Outcome:	The proposed model reproduces character fidelity with psychological scales and shows that it is effective in measuring personality traits.

Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better (2024.acl-long)

Copied to clipboard

Shengchao Liu, Xiaoming Liu, Yichen Wang, Zehua Cheng, Chengzhengxu Li, Zhaohan Zhang, Yu Lan, Chao Shen

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators (2024.acl-long)

Challenge:	Existing methods to detect MGT from human-written texts are inadequate . existing methods are fine-tuned and zero-shot metric-based, but they can be more accurate.
Approach:	They propose a novel fine-tuned detector that can detect MGT from human-written texts by contrastive learning on selective perturbation.
Outcome:	The proposed method outperforms the state-of-the-art by 1.20% on four public datasets.

Copied to clipboard

Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, Markus Leippold

Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering (2024.acl-long)

Challenge:	generative AI is a counter-measure to misinformation, but factual claim detection suffers from inconsistency in definitions and high cost of manual annotation.
Approach:	They propose a framework that assists in the annotation of factual claims with the help of large language models.
Outcome:	The proposed framework can be used to annotate factual claims with the help of large language models and can work with or without expert supervision.

Copied to clipboard

Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, Markus Leippold

Question Answering

Challenge:	Evidence-Based QA has proved insufficiently faithful with Large Language Models . a typical application of LLMs is in Evidence-based Question Answering (QA).
Approach:	They propose a data generation pipeline with automated data quality filters to fine-tune LLMs for better source quality and answer attributability.
Outcome:	The proposed model can synthesize high-quality training and testing data at scale.

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin (2024.acl-long)

Copied to clipboard

Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation (2024.acl-long)

Challenge:	Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM.
Approach:	They propose a framework that introduces several low-rank adapters and integrates them by using a router network to freeze the backbone model and force a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks.
Outcome:	The proposed framework freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge forgetting.

Copied to clipboard

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, Helen Meng

M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions (2024.acl-long)

Challenge:	Existing approaches to addressing factual inaccuracies require high-quality human factuality annotations to mitigate these hallucinations.
Approach:	They propose to leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality.
Outcome:	The proposed approach significantly improves factual accuracy over LLMs across three key knowledge-intensive tasks on TruthfulQA and BioGEN.

Copied to clipboard

Zheng Wang, Shu Teo, Jieer Ouyang, Yongjun Xu, Wei Shi

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (2024.acl-long)

Challenge:	Existing methods for retrieving relevant memories from an external database are coarse-grained and can cause noise and focus on crucial memories.
Approach:	They propose a multiple partition paradigm for RAG where each database partition serves as a basic unit for execution.
Outcome:	The proposed framework outperforms baseline methods on three language generation tasks on seven datasets.

Copied to clipboard

Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, Jingren Zhou

Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024.acl-long)

Challenge:	Existing benchmarks for audio-centric interaction have impeded advancements in this field . AIR-Bench evaluates LALMs' ability to understand audio signals and interact with humans .
Approach:	They propose a benchmark to evaluate the ability of large audio-language models to understand audio signals . they use 19 tasks with approximately 19k single-choice questions to examine single-task ability .
Outcome:	The proposed framework evaluates the ability of large audio-language models to understand audio signals and interact with humans in the textual format.

Copied to clipboard

Tom Kocmi, Vilém Zouhar, Christian Federmann, Matt Post

ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models (2024.acl-long)

Challenge:	a decade ago a single metric, BLEU, governed progress in machine translation research.
Approach:	They investigate the "dynamic range" of a number of modern machine translation metrics to provide a collective understanding of differences in scores . they use a large dataset to discover deltas at which metrics achieve system-level differences that are meaningful to humans .
Outcome:	The proposed method is more stable than statistical p-values in regards to testset size.

Copied to clipboard

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, Guojie Song

DM-BLI: Dynamic Multiple Subspaces Alignment for Unsupervised Bilingual Lexicon Induction (2024.acl-long)

Challenge:	Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies.
Approach:	They propose a psychometric evaluation pipeline grounded in realistic human-AI interactions to probe value orientations and novel tasks for evaluating value understanding in an open-ended value space.
Outcome:	The proposed evaluation pipeline is grounded in realistic human-AI interactions and performs tasks that approximate expert conclusions in value-related extraction and generation tasks.

Copied to clipboard

Ling Hu, Yuemei Xu

SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations (2024.acl-long)

Challenge:	Existing approaches to unsupervised bilingual lexicon induction (BLI) fail on distant or low-resource language pairs, achieving less than half the performance observed in rich-resourced languages.
Approach:	They propose a framework for unsupervised bilingual lexicon induction that uses multiple subspace alignments instead of a single mapping.
Outcome:	Experiments on 12 language pairs show that the proposed framework improves language alignment by utilizing multiple subspace alignments instead of a single mapping.

Copied to clipboard

Jesus Solano, Mardhiyah Sanni, Oana-Maria Camburu, Pasquale Minervini

Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024.acl-long)

Challenge:	Models that generate natural language explanations (NLEs) for their predictions often require large datasets of human-written NLEs at training time, which can be expensive and time-consuming to collect.
Approach:	They propose a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs.
Outcome:	The proposed approach compares sparse few-shot fine-tuning with existing parametric fine- tuning techniques on three sizes of the T5 language model and four datasets and produces competitive results for both task performance and NLE quality.

Copied to clipboard

Wen Wu, Bo Li, Chao Zhang, Chung-Cheng Chiu, Qiujia Li, Junwen Bai, Tara Sainath, Phil Woodland

REANO: Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation (2024.acl-long)

Challenge:	Experimental results show that incorporating utterances without majority-agreed labels into an additional class reduces the classification performance of the other emotion classes.
Approach:	They propose to combine utterances without majority-agreed labels into an additional class . they propose to quantify uncertainty in emotion classification using evidential deep learning .
Outcome:	The proposed method retains classification accuracy while effectively detects ambiguous emotion expressions.

Copied to clipboard

Jinyuan Fang, Zaiqiao Meng, Craig MacDonald

Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks (2024.acl-long)

Challenge:	Existing knowledge graphs suffer from incompleteness and lack information critical for answering given questions.
Approach:	They propose to enhance the open domain question answering model with a knowledge graph generation module that generates KGs from the passages and an answer predictor.
Outcome:	The proposed model improves the exact match score by 2.7% on the EntityQuestion dataset, with an average improvement of 1.8% across all the datasets.

Copied to clipboard

Yingji Zhang, Danilo Carvalho, Andre Freitas

MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generation (2024.acl-long)

Challenge:	Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and controllable data generation.
Approach:	They propose a flow-based invertible neural network mechanism integrated with a transformer-based language Autoencoder to deliver latent spaces with better semantic separability and geometrical properties.
Outcome:	The proposed model can deliver latent spaces with better separability properties compared to the current state-of-the-art models.

Copied to clipboard

Yan Ma, Yu Qiao, Pengfei Liu

Open-Set Semi-Supervised Text Classification via Adversarial Disagreement Maximization (2024.acl-long)

Challenge:	Existing sources of story premises are limited by a lack of diversity, uneven quality, and high costs that make them difficult to scale.
Approach:	They propose a method which breaks down story premises into modules like background and persona for automated design and generation.
Outcome:	The proposed framework excels in diversity, fascination, completeness, and originality compared to those induced from large language models and captured from public datasets.

Copied to clipboard

Junfan Chen, Richong Zhang, Junchi Chen, Chunming Hu

ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages (2024.acl-long)

Challenge:	Open-Set Semi-Supervised Text Classification (OSTC) aims to train a classification model on a limited set of labeled texts along with plenty of unlabeled examples.
Approach:	They propose to train a classification model on a limited set of labeled texts alongside plenty of unlabeled examples that include both in-distribution and out-of-difference examples.
Outcome:	The proposed model improves on outlier detection and abnormal example detection and calibration.

Copied to clipboard

Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, Xuanjing Huang

A synthetic data approach for domain generalization of NLI models (2024.acl-long)

Challenge:	Existing research focuses on enhancing LLMs capabilities through tool utilization.
Approach:	They propose a framework to investigate safety issues in large language models in tool learning . they propose malicious queries and jailbreak attacks in the input stage .
Outcome:	The proposed framework investigates six safety scenarios for LLMs in tool learning . the data will be released upon acceptance of the proposed framework .

Copied to clipboard

Mohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, Annie Louis

Challenge:	Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset .
Approach:	They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings .
Outcome:	The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy.

Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild (2024.acl-long)

Copied to clipboard

Ting Wu, Jingyi Liu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

LRQuant: Learnable and Robust Post-Training Quantization for Large Language Models (2024.acl-long)

Challenge:	Existing methods for continual relation extraction (CRE) excel in preserving old knowledge but falter when confronted with contaminated data streams.
Approach:	They propose a noise-resistant contrastive framework for continual relation extraction (CRE) that preserves old knowledge while learning incremental corrupted relations.
Outcome:	The proposed framework outperforms state-of-the-art methods on various benchmarks with increasing noise rates.

Copied to clipboard

Jiaqi Zhao, Miao Zhang, Chao Zeng, Ming Wang, Xuebo Liu, Liqiang Nie

VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)

Challenge:	Existing methods for post-training quantization (PTQ) are limited by the complexity of the quantization parameter and performance degradations when tested on unseen datasets.
Approach:	They propose a learnable smooth-based PTQ framework that allows for rapid adaptation during testing.
Outcome:	The proposed framework improves performance on unseen datasets and reduces memory constraints.

Copied to clipboard

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, Barbara Plank

Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation (2024.acl-long)

Challenge:	Existing work on label variation and annotation errors has focused on them in isolation.
Approach:	They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations.
Outcome:	The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items.

Copied to clipboard

Xunjian Yin, Xu Zhang, Jie Ruan, Xiaojun Wan

ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval (2024.acl-long)

Challenge:	Recent advances in large language models have improved performance across tasks . however, the sensitivity of LLMs to prompt leads to unreliability of evaluation results .
Approach:	They propose a new concept to evaluate language models with a fixed question or limited paraphrases as the query.
Outcome:	The proposed method outperforms existing benchmarks on multiple language models . it avoids prompt sensitivity, rendering models more reliable and robust .

Copied to clipboard

Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, Seung-won Hwang

Exploring the Potential of Large Language Models in Computational Argumentation (2024.acl-long)

Challenge:	Existing listwise reranking models rely on pointwise sizing of each passage . Until now, listwise models lack the ability to compare between passages at inference time .
Approach:	They propose a listwise reranking approach based on Fusion-in-Decoder that handles multiple candidate passages at train and inference time.
Outcome:	The proposed model outperforms the state-of-the-art RankT5 model on the BEIR benchmark for zero-shot retrieval task with a notable +1.3 gain in the average NDCG@10 score.

Copied to clipboard

Guizhen Chen, Liying Cheng, Anh Tuan Luu, Lidong Bing

TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks (2024.acl-long)

Challenge:	Argumentation is an essential tool in various domains, including law, public policy, and artificial intelligence.
Approach:	They propose to evaluate LLMs on various computational argumentation tasks . they organize existing tasks into six main categories and standardize the format of 14 datasets .
Outcome:	The proposed model performs well on argument mining and argument generation tasks.

Copied to clipboard

Viktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, Irina Nikishina

CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning (2024.acl-long)

Challenge:	Recent studies in Natural Language Processing widely utilize Large Language Models (LLMs) for their capability to store extensive knowledge.
Approach:	They propose an LLM-based model that captures lexical-semantic knowledge from WordNet and test it on multiple lexicals.
Outcome:	The proposed model achieves 11 SOTA results and 4 top-2 results out of 16 taxonomy-related tasks.

Copied to clipboard

Weiqi Wang, Tianqing Fang, Chunyang Li, Haochen Shi, Wenxuan Ding, Baixuan Xu, Zhaowei Wang, Jiaxin Bai, Xin Liu, Cheng Jiayang, Chunkit Chan, Yangqiu Song

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter (2024.acl-long)

Challenge:	Existing approaches to generalize commonsense reasoning lack instantiated knowledge and require pre-built concept taxonomies and annotations.
Approach:	They propose a framework that iteratively performs contextualized conceptualization and instantiation over commonsense knowledge bases by instructing large language models to generate both types of knowledge with critic filtering.
Outcome:	Empirical results show that distilling CANDLE on student models provides benefits across three downstream tasks.

Copied to clipboard

Jitai Hao, Weiwei Sun, Xin Xin, Qi Meng, Zhumin Chen, Pengjie Ren, Zhaochun Ren

Surgical Feature-Space Decomposition of LLMs: Why, When and How? (2024.acl-long)

Challenge:	Parameter-Efficient Fine-tuning (PEFT) methods are limited on knowledge-intensive tasks due to the limited number of trainable parameters.
Approach:	They propose a mechanism that fine-tunes Large Language Models with larger adapters . they store and update the parameters of larger adapter adapters on the CPU .
Outcome:	The proposed method achieves comparable results to those obtained with larger memory capacities over the limited bandwidth of PCI Express (PCIe).

Copied to clipboard

Arnav Chavan, Nahush Lele, Deepak Gupta

Reasoning in Flux: Enhancing Large Language Models Reasoning through Uncertainty-aware Adaptive Guidance (2024.acl-long)

Challenge:	Low-rank approximations of the weight and feature space can enhance the performance of large language models.
Approach:	They propose to use weight and feature space decomposition to improve LLM performance . they also extend their investigation to the implications of low-rank approximations on model bias .
Outcome:	The proposed low-rank approximations can improve performance of large language models . the authors show that the approximate can improve generalization and inference performance .

Copied to clipboard

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuanjing Huang, Xipeng Qiu

Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering (2024.acl-long)

Challenge:	Extensive experiments across various reasoning tasks demonstrate that UAG not only enhances the reasoning abilities of LLMs but consistently outperforms several strong baselines with minimal computational overhead.
Approach:	They propose an approach to guide LLMs onto an accurate and reliable trajectory by identifying and adjusting uncertainty signals within each step of the reasoning chain.
Outcome:	The proposed approach outperforms strong baselines and outperformed strong models with minimal computational overhead.

Copied to clipboard

Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, Xiao Huang

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression (2024.acl-long)

Challenge:	Existing methods to integrate multimodal knowledge in a modality-agnostic manner can be sub-optimal.
Approach:	They propose a modality-aware integration with large language models (LLMs) that leverages multimodal knowledge for both image understanding and knowledge reasoning.
Outcome:	The proposed model is able to bridge a tight inter-modal exchange while preserving insightful intra-modal learning.

Copied to clipboard

Peiyu Liu, Ze-Feng Gao, Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen

VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models (2024.acl-long)

Challenge:	Existing methods to compress KV cache compromise precision or require extra data for calibration, limiting their practicality in LLM deployment.
Approach:	They propose a low-bit quantization technique based on tensor decomposition to effectively compress KV cache.
Outcome:	The proposed method reduces memory footprint and performance by 75% . it is compared with existing methods that compromise precision or require extra data for calibration .

Copied to clipboard

Seoyeon Kim, Kwangwook Seo, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee

Making Long-Context Language Models Better Multi-Hop Reasoners (2024.acl-long)

Challenge:	Recent approaches in domain-specific named entity recognition (NER) have shown remarkable advances, but they still lack faithfulness, producing erroneous predictions.
Approach:	They propose a framework that revises errors from existing NER methods using knowledge to produce more faithful predictions.
Outcome:	The proposed framework can validate errors from existing models as a model-agnostic approach.

Copied to clipboard

Yanyang Li, Shuo Liang, Michael Lyu, Liwei Wang

TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024.acl-long)

Challenge:	Recent advances in long-context modeling have enhanced language models for complex tasks, but they struggle with multi-hop reasoning and noisy contexts.
Approach:	They propose an approach that prompts LMs to supply attributions for each assertion during reasoning.
Outcome:	The proposed model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.

Copied to clipboard

Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schuetze

Extreme Miscalibration and the Illusion of Adversarial Robustness (2024.acl-long)

Challenge:	The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap.
Approach:	They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script.
Outcome:	The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts.

Copied to clipboard

Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis

HyCoRec: Hypergraph-Enhanced Multi-Preference Learning for Alleviating Matthew Effect in Conversational Recommendation (2024.acl-long)

Challenge:	emergence of the Adversarial Training paradigm has shown some success in training models to be more robust to these small adversarial perturbations.
Approach:	They propose to use adversarial examples to detect adversarials by miscalibrating models to mask gradients in a way that interferes with adversarial attack search methods.
Outcome:	The proposed model gains are an illusion of robustness (IOR) and urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine.

Copied to clipboard

Yongsen Zheng, Ruilin Xu, Ziliang Chen, Guohua Wang, Mingjie Qian, Jinghui Qin, Liang Lin

Co-training for Low Resource Scientific Natural Language Inference (2024.acl-long)

Challenge:	Existing methods to study the Matthew effect in Recommender Systems (RSs) however, it is amplified when the user interacts with the system over time.
Approach:	They propose a paradigm to alleviate the Matthew effect in conversational recommendation by learning multi-aspect preferences.
Outcome:	The proposed paradigm achieves state-of-the-art performance and superior of alleviating Matthew effect in conversational recommendation tasks.

Copied to clipboard

Mobashir Sadat, Cornelia Caragea

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models (2024.acl-long)

Challenge:	Scientific Natural Language Inference (NLI) is a task to predict the semantic relation between sentences extracted from research articles.
Approach:	They propose a co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels.
Outcome:	The proposed method achieves an improvement of 1.5% in Macro F1 over the distant supervision baseline and substantial improvements over several other strong SSL baselines.

Copied to clipboard

Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao

Time is Encoded in the Weights of Finetuned Language Models (2024.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) have significantly enhanced the capabilities in natural language processing.
Approach:	They propose a method to poison large language models by using annotators to rank a set of collected responses to generate longer tokens.
Outcome:	The proposed method can generate longer tokens without harming the original safety alignment performance.

Copied to clipboard

Kai Nylund, Suchin Gururangan, Noah Smith

Long-Context Language Modeling with Parallel Context Encoding (2024.acl-long)

Challenge:	Time vectors are used to customize language models to new time periods.
Approach:	They propose a tool to customize language models to new time periods by using time vectors . they show that time is encoded in the weight space of finetuned models .
Outcome:	The proposed tool improves performance on text from a time period without training.

Copied to clipboard

Howard Yen, Tianyu Gao, Danqi Chen

SirLLM: Streaming Infinite Retentive LLM (2024.acl-long)

Challenge:	Existing long-context models degenerate with retrieved contexts.
Approach:	They propose a framework that can be applied to existing decoder-only LLMs for context expansion.
Outcome:	The proposed framework can be applied to any existing decoder-only LLMs for context expansion.

Copied to clipboard

Yao Yao, Zuchao Li, Hai Zhao

IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) are becoming increasingly prevalent in various domains, requiring a one-off input of overly long texts to maintain a degree of memory.
Approach:	They propose a Streaming Infinite Retentive LLM which allows LLMs to maintain longer memory during infinite-length dialogues without fine-tuning.
Outcome:	The proposed model can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness.

Copied to clipboard

Tao Feng, Lizhen Qu, Zhuang Li, Haolan Zhan, Yuncheng Hua, Reza Haf

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (2024.acl-long)

Challenge:	IMO is a machine learning model that learns invariant features from unseen domains.
Approach:	They propose IMO: Invariant features Masks for Out-of-Distribution text classification to achieve OOD generalization by learning invariant feature masks.
Outcome:	The proposed model outperforms baseline models in various evaluation metrics and settings.

Copied to clipboard

Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, Kewei Tu

MELA: Multilingual Evaluation of Linguistic Acceptability (2024.acl-long)

Challenge:	Existing syntactic language models require a gold tree and sequential training to generate sentences.
Approach:	They propose an unsupervised syntactic language model that incrementally generates a sentence with its syntaktic tree in a left-to-right manner.
Outcome:	The proposed model outperforms existing models on grammar induction and comprehension tasks while holding a substantial acceleration on training.

Copied to clipboard

Ziyin Zhang, Yikang Liu, Weifang Huang, Junyu Mao, Rui Wang, Hai Hu

Syntactic Text Processing

Challenge:	Existing benchmarks on linguistic acceptability have been used to evaluate language models' ability to distinguish between acceptable and unacceptable sentences.
Approach:	They present the largest benchmark to date on linguistic acceptability: MELA . they establish LLM baselines on this benchmark and investigate cross-lingual transfer in acceptability judgements with XLM-R.
Outcome:	The proposed model outperforms open-source models on cross-lingual transfer in acceptability judgements.

CopyNE: Better Contextual ASR by Copying Named Entities (2024.acl-long)

Copied to clipboard

Shilin Zhou, Zhenghua Li, Yu Hong, Min Zhang, Zhefeng Wang, Baoxing Huai

Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval (2024.acl-long)

Challenge:	Existing approaches to transcribe contextual named entities (NEs) treat entities as tokens and generate them token-by-token, which may result in incomplete transcriptions of entities.
Approach:	They propose a mechanism that can copy entities from the NE dictionary and reduce errors during entity transcription.
Outcome:	The proposed mechanism can copy entities from the NE dictionary, reducing errors during entity transcription, ensuring the completeness of the entity.

Copied to clipboard

Peter Baile Chen, Yi Zhang, Dan Roth

Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation (2024.acl-long)

Challenge:	Existing methods for retrieving relevant tables are not sufficient as many questions require retrieving multiple tables and joining them through a join plan that cannot be discerned from the user query itself.
Approach:	They propose a method that uncovers useful join relations during table retrieval.
Outcome:	The proposed method outperforms the state-of-the-art methods for table retrieval by up to 9.3% in F1 score and for end-to-end QA by up 5.4% in accuracy.

Copied to clipboard

Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, Ziliang Zhao

ItD: Large Language Models Can Teach Themselves Induction through Deduction (2024.acl-long)

Challenge:	Existing conversational dense retrieval models view a conversation as a fixed sequence of questions and responses, and these alternate conversations are unrecorded.
Approach:	They propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug) they first generate multi-level augmented conversations to capture the diverse nature of conversational contexts.
Outcome:	The proposed framework generalizes Conversational dense retrieval via LLM-cognition data Augmentation on four public datasets.

Copied to clipboard

Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, Kang Liu

MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs (2024.acl-long)

Challenge:	Recent studies have shown that Large Language Models (LLMs) have limited ability to conduct induction.
Approach:	They propose a framework to enable LLMs to teach themselves induction through deduction.
Outcome:	The proposed framework improves performance on two induction benchmarks and shows that it can be used to teach induction through deduction.

Copied to clipboard

Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

Rethinking Task-Oriented Dialogue Systems: From Complex Modularity to Zero-Shot Autonomous Agent (2024.acl-long)

Challenge:	Existing models have demonstrated outstanding capabilities in mathematical reasoning, but there is a performance gap between open-source models and closed-source ones.
Approach:	They propose a method for generating diverse and reliable math problems by leveraging the ground-truth solutions of the seed data.
Outcome:	The proposed model outperforms open-source models across five representative mathematical reasoning datasets.

Copied to clipboard

Heng-Da Xu, Xian-Ling Mao, Puhai Yang, Fanshu Sun, Heyan Huang

On Context Utilization in Summarization with Large Language Models (2024.acl-long)

Challenge:	Task-oriented dialogue systems are designed to be composed of several functional modules, but lacks a general-purpose instruction-following language model.
Approach:	They propose a fully zero-shot autonomous TOD agent that leverages a general-purpose instruction-following language model to decide what to do at each dialogue turn.
Outcome:	The proposed agent can perform tasks in real-life scenarios with a general-purpose instruction-following language model.

Copied to clipboard

Mathieu Ravaut, Aixin Sun, Nancy Chen, Shafiq Joty

INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024.acl-long)

Challenge:	Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries.
Approach:	They conduct the first comprehensive study on context utilization and position bias in summarization.
Outcome:	The proposed benchmark compares two methods to alleviate position bias in summarization tasks.

Copied to clipboard

Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, Zhicheng Dou

Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks, but their application to information retrieval tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language.
Approach:	They propose to use instruction tuning to enhance LLMs' proficiency in IR tasks by combining a dataset with manually written templates to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions.
Outcome:	The proposed model can be used to perform query understanding, document understanding, and query-document relationship understanding tasks.

Copied to clipboard

Xiaoling Zhou, Wei Ye, Yidong Wang, Chaoya Jiang, Zhemg Lee, Rui Xie, Shikun Zhang

PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA (2024.acl-long)

Challenge:	In-context learning (ICL) is a new paradigm for pre-trained language models that can make predictions for unseen inputs without updating parameters.
Approach:	They propose a method that enables a model to augmented copies of a demonstration by leveraging their deep feature distribution and a logit calibration mechanism.
Outcome:	The proposed method significantly improves the average and worst-case accuracy across diverse PLMs and tasks.

Copied to clipboard

Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, Chuan Wu

Improving Event Definition Following For Zero-Shot Event Detection (2024.acl-long)

Challenge:	Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA) is an intra-layer sharing mechanism that circumvents the drawbacks of peer parameter-sharing methods.
Approach:	They propose a partially rotation-enhanced low-rank adaptation (PRoLoRA) that shares four components to reduce the cost of LoRA and improves model capacity.
Outcome:	Empirical results show that PRoLoRA outperforms LoRA on multiple instruction tuning datasets.

Copied to clipboard

Zefan Cai, Po-Nien Kung, Ashima Suvarna, Mingyu Ma, Hritik Bansal, Baobao Chang, P. Jeffrey Brantingham, Wei Wang, Nanyun Peng

Through the MUD: A Multi-Defendant Charge Prediction Benchmark with Linked Crime Elements (2024.acl-long)

Challenge:	Existing approaches on zero-shot event detection train models on datasets annotated with known event types and prompt them with unseen event definitions.
Approach:	They propose to train models to better follow event definitions by using an automatic generated Diverse Event Definition dataset.
Outcome:	The proposed model outperforms existing models on three open benchmarks on zero-shot event detection.

Copied to clipboard

Xiao Wei, Qi Xu, Hang Yu, Qian Liu, Erik Cambria

Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding (2024.acl-long)

Challenge:	Existing charge prediction datasets focus on single-defendant cases, but real-world cases involve multiple defendants.
Approach:	They propose a benchmark that encompasses legal cases involving multiple defendants . they develop an interpretable model called EJudge that incorporates crime elements and legal rules to infer charges.
Outcome:	The proposed model outperforms state-of-the-art models in predicting crime charges while providing corresponding rationales.

Copied to clipboard

Yiruo Cheng, Kelong Mao, Zhicheng Dou

Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks (2024.acl-long)

Challenge:	Conversational dense retrieval models lack interpretability, hindering intuitive understanding of model behaviors . a major limitation of conversational dense search is their lack of interpretability .
Approach:	They propose to transform opaque session embeddings into explicit interpretable text . they propose to incorporate external interpretable query rewrites into the transformation process .
Outcome:	The proposed approach yields more interpretable text and preserves original retrieval performance over baselines.

Copied to clipboard

Yichen Wang, Shangbin Feng, Abe Hou, Xiao Pu, Chao Shen, Xiaoming Liu, Yulia Tsvetkov, Tianxing He

Training Language Models to Generate Text with Citations via Fine-grained Rewards (2024.acl-long)

Challenge:	Existing studies on this topic focus on the robustness of specific detectors or particular attack methods.
Approach:	They stress test the detectors’ robustness to malicious attacks under realistic scenarios using LLMs and metric-based detectors.
Outcome:	The proposed methods are based on a set of LLM-based models and their performance is compared under different budget levels.

Copied to clipboard

Chengyu Huang, Zeqiu Wu, Yushi Hu, Wenya Wang

Hypergraph based Understanding for Document Semantic Entity Recognition (2024.acl-long)

Challenge:	Recent Large Language Models (LLMs) are prone to hallucination and their outputs often contain incorrect or unverifiable claims.
Approach:	They propose a training framework using fine-grained rewards to teach LLMs to generate highly supportive and relevant citations while ensuring the correctness of their responses.
Outcome:	The proposed training framework outperforms existing methods on QA datasets and surpasses GPT-3.5-turbo on LLaMA-2-7B.

Copied to clipboard

Qiwei Li, Zuchao Li, Ping Wang, Haojun Ai, Hai Zhao

GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers (2024.acl-long)

Challenge:	Existing document understanding models focus on entity categories while ignoring the extraction of entity boundaries.
Approach:	They propose a hypergraph attention document semantic entity recognition framework which uses hypergraph focus to focus on entity boundaries and entity categories at the same time.
Outcome:	The proposed framework can improve the performance of existing models on FUNSD, CORD, XFUND and SROIE.

Copied to clipboard

Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, Wei Bi

Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated impressive performance across various mathematical reasoning benchmarks.
Approach:	They introduce an adversarial grade school math dataset and explore whether LLMs can be more robust when questions are slightly changed.
Outcome:	The proposed method generates and verifies each intermediate thought based on its reasoning goal and calculation result.

Copied to clipboard

Qingkai Min, Qipeng Guo, Xiangkun Hu, Songfang Huang, Zheng Zhang, Yue Zhang

AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning (2024.acl-long)

Challenge:	Existing approaches to cross-document event coreference resolution are prone to learning simple co-occurrences due to the complexity of contexts.
Approach:	They propose a collaborative approach to cross-document event coreference resolution that leverages both a universally capable LLM and a task-specific SLM.
Outcome:	The proposed approach surpasses the performance of both large and small language models individually, underscoring its effectiveness in diverse scenarios.

Copied to clipboard

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, Huajun Chen

Question Answering

Challenge:	Existing language agent systems struggle with costly data reliance and need multiple models for multiple functions.
Approach:	They propose an automatic agent learning framework for QA that synthesizes planning trajectories without human intervention.
Outcome:	The proposed framework outperforms existing models on question-answering tasks with a division-of-labor strategy.

ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks (2024.acl-long)

Copied to clipboard

Santosh T.y.s.s, Tuan-Quang Vuong, Matthias Grabmair

Virtual Compiler Is All You Need For Assembly Code Search (2024.acl-long)

Challenge:	Existing models overlook the temporal dimension in their training process, leading to suboptimal performance over time.
Approach:	They propose a training paradigm that trains models on chronological splits, preserving the temporal order of the data.
Outcome:	The proposed model fails to fit to recent data, despite continual learning and temporal invariant methods.

Copied to clipboard

Zeyu Gao, Hao Wang, Yuanda Wang, Chao Zhang

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Using a large dataset, we find that assembly code search is a significant task for reverse engineers.
Approach:	They propose to train a Large Language Model (LLM) to emulate a general compiler.
Outcome:	The proposed model surpasses the baseline by 26%.

Copied to clipboard

Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei

Can LLMs Learn from Previous Mistakes? Investigating LLMs’ Errors to Boost for Reasoning (2024.acl-long)

Challenge:	Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible.
Approach:	They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times .
Outcome:	The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.

Copied to clipboard

Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, Jingbo Shang

An Iterative Associative Memory Model for Empathetic Response Generation (2024.acl-long)

Challenge:	Recent studies have shown the benefits to LLMs from fine-tuning golden-standard Chain-of-Thought rationales or using them as correct examples in few-shot prompting.
Approach:	They propose a new benchmark to test the effectiveness of large language models by leveraging errors to enhance reasoning capabilities.
Outcome:	The proposed methods can be used to fine-tune models in correct and incorrect domains, rather than tuning models to learn ground truth in traditional methods.

Copied to clipboard

Zhou Yang, Zhaochun Ren, Wang Yufeng, Haizhou Sun, Chao Chen, Xiaofei Zhu, Xiangwen Liao

Detoxifying Large Language Models via Knowledge Editing (2024.acl-long)

Challenge:	Existing methods for empathetic response generation ignore the associated words between dialogue utterances.
Approach:	They propose an iterative associative memory model to capture associated words between dialogue utterances and situations, dialogue history, and a memory module for storing associated words.
Outcome:	The proposed model captures key words between dialogue utterances and situations, dialogue history, and a memory module, thereby accurately and nuancedly comprehending the utterables.

Copied to clipboard

Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, Huajun Chen

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)

Challenge:	Existing methods to detoxify Large Language Models (LLMs) are limiting, but knowledge editing can be effective.
Approach:	They propose a baseline method to detoxify Large Language Models (LLMs) they propose supervised fine-tuning and reinforcement learning from human feedback (RLHF)
Outcome:	The proposed method reduces toxicity of large language models with one instance of tuning . it reduces the toxicity, while minimizing the toxins, the authors show .

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models (2024.acl-long)

Challenge:	Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases.
Approach:	They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities.
Outcome:	The proposed model outperforms open-source models but struggles on longer contexts.

Copied to clipboard

Yuyan Chen, Songzhou Yan, Panjun Liu, Yanghua Xiao

UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages (2024.acl-long)

Challenge:	Recognizing LLMs’ capability to generate educational content can lead to advances in automated and personalized learning.
Approach:	They propose to evaluate the questioning capability in education as a teacher of large language models by evaluating their generated educational questions.
Outcome:	The proposed model can generate educational content that aligns with human perspectives and is more apt as an interdisciplinary teacher.

Copied to clipboard

Trinh Pham, Khoi Le, Anh Tuan Luu

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024.acl-long)

Challenge:	Existing pre-trained language models are weak in addressing cross-lingual transfer tasks.
Approach:	They propose a method for initializing embeddings and choosing the right vocabulary size for cross-lingual systems.
Outcome:	The proposed method improves the F1-Score in several languages .

Copied to clipboard

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)

Challenge:	Existing methods for multimodal retrieval are mostly text-oriented, which lack the capability to process visual information.
Approach:	They propose a multi-modal multi-text embedding model VISTA which extends a powerful text encoder with the image understanding capability by introducing visual token embedds.
Outcome:	The proposed model achieves superior performance across a variety of multi-modal retrieval tasks in zero-shot and supervised settings.

Copied to clipboard

Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang

Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark (2024.acl-long)

Challenge:	Large language models are often not well aligned with human intents, which requires additional training.
Approach:	They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents.
Outcome:	The proposed model outperforms existing models and is model-agnostic.

Copied to clipboard

Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, Hwalsuk Lee

Unified Hallucination Detection for Multimodal Large Language Models (2024.acl-long)

Challenge:	Existing benchmarks for evaluating Large Language Models are limited to the English language.
Approach:	They introduce the Open Ko-LLM Leaderboard and Ko-H5 Benchmark as tools for evaluating Large Language Models in Korean using private test sets.
Outcome:	The proposed evaluation framework is well integrated in the Korean LLM community.

Copied to clipboard

Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen

Empowering Character-level Text Infilling by Eliminating Sub-Tokens (2024.acl-long)

Challenge:	despite significant strides in multimodal tasks, MLLMs are plagued by the critical issue of hallucination.
Approach:	They propose a meta-evaluation benchmark to facilitate evaluation of advancements in hallucination detection methods.
Outcome:	The proposed framework validates hallucinations robustly and provides strategic insights . MHaluBench is a meta-evaluation benchmark designed to facilitate evaluation .

Copied to clipboard

Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Hongsheng Li

Challenge:	Existing methods for character-level infilling relied on predicting sub-tokens, but this strategy was ineffective.
Approach:	They propose a method to fill-in-the-mid with Starting and Ending character constraints that avoids predicting sub-tokens in inference.
Outcome:	The proposed method surpasses existing methods and offers significant performance gains.

Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models (2024.acl-long)

Copied to clipboard

Kun Luo, Zheng Liu, Shitao Xiao, Tong Zhou, Yubo Chen, Jun Zhao, Kang Liu

GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge? (2024.acl-long)

Challenge:	Existing methods for retrieval augmentation work with chunked contexts, which leads to poor quality of semantic representation and incomplete retrieval of useful information.
Approach:	They propose a method for retrieval augmentation of long-context language modeling using landmark embedding.
Outcome:	The proposed method outperforms existing retrieval methods with a notable advantage.

Copied to clipboard

Dayoon Ko, Jinyoung Kim, Hahyeon Choi, Gunhee Kim

Attribute First, then Generate: Locally-attributable Grounded Text Generation (2024.acl-long)

Challenge:	Existing knowledge-based datasets are outdated due to the rapid evolution of knowledge.
Approach:	They propose a retrieval-interactive language model framework that evaluates and reflects on its answers for further re-retrieval.
Outcome:	The proposed framework performs comparably to or surpasses continuously trained language models.

Copied to clipboard

Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, Ido Dagan

T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text (2024.acl-long)

Challenge:	Recent efforts to address hallucinations in Large Language Models have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections.
Approach:	They propose a locally-attributable text generation approach prioritizing concise attributions by identifying relevant source segments and conditioning the generation process on them.
Outcome:	The proposed method yields more concise citations than baselines and significantly reduces time required for fact verification by human assessors.

Copied to clipboard

Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, Yueting Zhuang

OceanGPT: A Large Language Model for Ocean Science Tasks (2024.acl-long)

Challenge:	Existing vector quantization methods are fixed-length encodings, overlooking the uneven information density in sign language.
Approach:	They propose a two-stage sign language production paradigm that encodes sign language sequences into discrete codes and autoregressively generates sign languages from text.
Outcome:	The proposed model can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoded enccoding.

Copied to clipboard

Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, Huajun Chen

Beyond Memorization: The Challenge of Random Memory Access in Language Models (2024.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) have transformed the paradigm in ocean science.
Approach:	They propose a framework to automatically obtain large volume of ocean domain instruction data, which generates instructions based on multi-agent collaboration.
Outcome:	The proposed framework shows a higher level of knowledge expertise for ocean science tasks and gains preliminary embodied intelligence capabilities in ocean technology.

Copied to clipboard

Tongyao Zhu, Qian Liu, Liang Pang, Zhengbao Jiang, Min-Yen Kan, Min Lin

BIPED: Pedagogically Informed Tutoring System for ESL Education (2024.acl-long)

Challenge:	Recent advances in Language Models (LMs) have shown their effectiveness in knowledge-intensive tasks.
Approach:	They investigate whether a generative language model is able to access its memory sequentially or randomly.
Outcome:	The proposed LMs are able to access memory sequentially or randomly.

Copied to clipboard

Soonwoo Kwon, Sojung Kim, Minju Park, Seunghyun Lee, Kyuseok Kim

Challenge:	Existing Large Language Models (LLMs) are limited in scope and lack pedagogical depth.
Approach:	They construct a BIlingual PEDagogically-informed Tutoring Dataset of one-on-one, human-to-human tutoring interactions using a post-hoc analysis.
Outcome:	The proposed models replicate the style of human teachers and employ diverse and contextually appropriate pedagogical strategies.

Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction (2024.acl-long)

Copied to clipboard

Jianhao Chen, Haoyuan Ouyang, Junyang Ren, Wentao Ding, Wei Hu, Yuzhong Qu

Collaboration or Corporate Capture? Quantifying NLP’s Reliance on Industry Artifacts and Contributions (2024.acl-long)

Challenge:	Recent research on temporal fact extraction fails to establish time-to-fact correspondences in complex sentences.
Approach:	They propose a timeline-based sentence decomposition strategy using large language models with in-context learning to extract temporal facts from natural language text.
Outcome:	The proposed method achieves state-of-the-art on a complex temporal fact extraction dataset.

Copied to clipboard

Will Aitken, Mohamed Abdalla, Karen Rudie, Catherine Stinson

Prompt Expansion for Adaptive Text-to-Image Generation (2024.acl-long)

Challenge:	EMNLP 2022 citations are three times greater than expected for pre-trained models . industry participation in the Association of Computational Linguistics (ACL) anthology has increased 180% from 2017 to 2022.
Approach:	They surveyed 100 papers published at EMNLP 2022 to determine the ratio of their citations to industry models.
Outcome:	a new study shows that industry citations are three times greater than expected . the study aims to better understand whether industry collaboration is still collaboration . industry participation in the 2023 AI index report is the top takeaway .

Copied to clipboard

Siddhartha Datta, Alexander Ku, Deepak Ramachandran, Peter Anderson

Progressively Modality Freezing for Multi-Modal Entity Alignment (2024.acl-long)

Challenge:	Text-to-image generation models are powerful but difficult to use. Users craft specific prompts to get better images, though the prompts can be repetitive.
Approach:	They propose a framework that takes a text query as input and outputs a set of expanded text prompts that are optimized to generate a wider variety of appealing images.
Outcome:	The proposed framework generates high-quality images from text prompts with less effort and is more aesthetically pleasing than baseline models.

Copied to clipboard

Yani Huang, Xuefeng Zhang, Richong Zhang, Junfan Chen, Jaein Kim

Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024.acl-long)

Challenge:	Multi-Modal Entity Alignment aims to discover identical entities across heterogeneous knowledge graphs.
Approach:	They propose a strategy of progressive modality freezing that focuses on alignment-relevant features and enhances multi-modal feature fusion.
Outcome:	The proposed approach demonstrates state-of-the-art performance and the rationale for freezing modalities.

Copied to clipboard

Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, Defu Lian

Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts (2024.acl-long)

Challenge:	Dense retrieval requires discriminative embeddings to represent the semantic relationship between query and document.
Approach:	They propose an unsupervised approach that performs unsupervised adaptation of large language models for dense retrieval.
Outcome:	The proposed model improves on a variety of dense retrieval benchmarks and is available on github.

Copied to clipboard

Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, Lidong Bing

Metaphor Understanding Challenge Dataset for LLMs (2024.acl-long)

Challenge:	Large language models (LLMs) are known to perform tasks by simply observing few exemplars, but performance among under-represented languages falls behind due to pre-training data imbalance.
Approach:	They propose to assemble synthetic exemplars from high-resource languages to prompt LLMs to translate from any language into English and use them to create intra-lingual exemplar models to perform tasks in target languages.
Outcome:	The proposed method outperforms supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.

Copied to clipboard

Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova

A Multi-Task Embedder For Retrieval Augmented LLMs (2024.acl-long)

Challenge:	Metaphor understanding is an essential task for large language models (LLMs).
Approach:	They propose to evaluate the metaphor understanding capabilities of large language models (LLMs) the metaphor understanding challenge dataset provides over 10k paraphrases and 1.5k instances of inapt paraphrase.
Outcome:	The metaphor understanding challenge dataset evaluates the performance of large language models on a range of NLU tasks.

Copied to clipboard

Peitian Zhang, Zheng Liu, Shitao Xiao, Zhicheng Dou, Jian-Yun Nie

Language Models Don’t Learn the Physical Manifestation of Language (2024.acl-long)

Challenge:	Existing retrieval methods face limitations in terms of knowledge, memory, and action.
Approach:	They propose a retrieval enhancement mechanism that brings in useful information from external sources to augment the LLM.
Outcome:	The proposed method significantly improves the LLM’s performance in various downstream tasks while introducing superior retrieval augmentation’s effect over both general and task-specifc retrievers.

Copied to clipboard

Bruce Lee, Jaehyuk Lim

What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection (2024.acl-long)

Challenge:	a new study examines the differences between language-only models and humans . we show that language-based models do not learn the physical manifestation of language .
Approach:	They propose to use a series of tasks to investigate visual-auditory properties of language to test their hypothesis.
Outcome:	The proposed model does not learn the physical manifestation of language . the results highlight the limitations of linguistic knowledge acquired without sensory experience .

Copied to clipboard

Shangbin Feng, Herun Wan, Ningnan Wang, Zhaoxuan Tan, Minnan Luo, Yulia Tsvetkov

Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives (2024.acl-long)

Challenge:	Social media bot detection has always been an arms race between advancements in machine learning and adversarial bot strategies to evade detection.
Approach:	They propose a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities and propose LLM-guided manipulation of user textual and structured information to evade detection.
Outcome:	The proposed framework outperforms state-of-the-art baselines on 1,000 annotated examples while bringing down existing detectors by 29.6% and harming calibration and reliability of bot detection systems.

Copied to clipboard

Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, Weiming Lu

Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty (2024.acl-long)

Challenge:	Recent research indicates without external feedback, LLM’s intrinsic reflection is unstable.
Approach:	They propose a method that combines self-evaluated and external feedback to improve LLM's reflection.
Outcome:	The proposed method improves the quality of self-evaluated feedback and can catalyze more accurate and stable reflection.

Copied to clipboard

Kaitlyn Zhou, Jena Hwang, Xiang Ren, Maarten Sap

Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources (2024.acl-long)

Challenge:	a pivotal aspect of fostering reliable human-AI interactions lies in the apt communication of model confidences.
Approach:	They examine how LMs incorporate confidence in responses via natural language . they also examine how downstream users behave in response to LM-articulated uncertainties .
Outcome:	The proposed model overconfidences are high in LMs, and humans are biased against uncertainty-rich texts.

Copied to clipboard

Xiaochen Wang, Junyu Luo, Jiaqi Wang, Yuan Zhong, Xiaokun Zhang, Yaqing Wang, Parminder Bhatia, Cao Xiao, Fenglong Ma

When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP (2024.acl-long)

Challenge:	Current pre-training techniques rely on a limited scope of medical data, limiting the range of downstream tasks.
Approach:	They propose a pre-training strategy that unifies patient data within individual sources and captures explicit and implicit correlations between patients across different sources.
Outcome:	The proposed strategy bridges the gap between multimodal medical sources by aggregating patient data within individual sources and capturing explicit and implicit correlations between patients across sources.

Copied to clipboard

Sara Papi, Marco Gaido, Andrea Pilzer, Matteo Negri

SBAAM! Eliminating Transcript Dependency in Automatic Subtitling (2024.acl-long)

Challenge:	despite its crucial role in research experiments, code correctness is often presumed on the perceived quality of results.
Approach:	They propose to promote code-quality checklists to promote coding best practices . they propose to fix bugs in conformer implementations to mitigate this risk .
Outcome:	The proposed checklists aim to promote coding best practices and improve software quality within the NLP community.

Copied to clipboard

Marco Gaido, Sara Papi, Matteo Negri, Mauro Cettolo, Luisa Bentivogli

StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection (2024.acl-long)

Challenge:	Subtitling is a crucial task for enhancing the accessibility of audiovisual content and relying on automatic transcripts for the three subtasks is uncharted territory.
Approach:	They propose a model capable of producing automatic subtitles, completely eliminating any dependence on intermediate transcripts also for timestamp prediction.
Outcome:	Experimental results show that the proposed model eliminates the need for intermediate transcripts for timestamp prediction across multiple language pairs and diverse conditions.

Copied to clipboard

Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli

ARL2: Aligning Retrievers with Black-box Large Language Models via Self-guided Adaptive Relevance Labeling (2024.acl-long)

Challenge:	Existing studies on streaming translation focus on SimulST only focusing on StreamST . StreamAtt is the first Stream ST policy and proposes StreamLAAL .
Approach:	They propose StreamAtt, the first StreamST policy, and StreamLAAL, the second Stream ST latency metric.
Outcome:	Experiments in 8 languages show that StreamAtt is more efficient than SimulST . StreamLAAL is the first StreamST latency metric comparable with existing metrics for Simul ST.

Copied to clipboard

LingXi Zhang, Yue Yu, Kuan Wang, Chao Zhang

Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference (2024.acl-long)

Challenge:	Existing retrievers are misaligned with large language models due to separate training processes and inherent black-box nature of LLMs.
Approach:	They propose a retriever learning technique that harnesses LLMs as labelers to annotate and score adaptive relevance evidence.
Outcome:	Extensive experiments show that ARL2 improves accuracy and reduces the cost of API calls.

Copied to clipboard

Jihwan Bang, Juntae Lee, Kyuhong Shim, Seunghan Yang, Simyung Chang

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model (2024.acl-long)

Challenge:	Large language models are highly advanced to user-requested tasks.
Approach:	They propose a device-server hybrid inference strategy for on-device LLM customization . they construct a pool of diverse base adapters and then blend them into a customized adapter .
Outcome:	The proposed method can be used on a large scale without extra training . it can be applied to large-scale LLMs without sacrificing the benefits of on-device customization.

Copied to clipboard

Yebin Lee, Imseong Park, Myungjoo Kang

MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations (2024.acl-long)

Challenge:	Existing image captioning evaluation metrics do not provide an explanation for the assigned numerical score.
Approach:	They propose an explainable reference-free metric to provide an explanation for captions . they introduce score smoothing to align as closely as possible with human judgment .
Outcome:	The proposed metric achieves high correlations with human judgment across image captioning evaluation benchmarks and is publicly available at https://github.com/Yebin46/FLEUR.

Copied to clipboard

Yuxin Wang, Ivory Yang, Saeed Hassanpour, Soroush Vosoughi

MPCoder: Multi-user Personalized Code Generator with Explicit and Implicit Style Representation Learning (2024.acl-long)

Challenge:	Existing studies on mental manipulation focus on context-free content and face challenges in identifying implicit toxicity.
Approach:	They propose a dataset that analyzes mental manipulation and its components . they propose to use 4,000 fictional dialogues to identify the techniques utilized for manipulation .
Outcome:	The proposed dataset enables a comprehensive analysis of mental manipulation . it shows that leading-edge models inadequately identify and categorize manipulative content .

Copied to clipboard

Zhenlong Dai, Chang Yao, WenKang Han, Yuanying Yuanying, Zhipeng Gao, Jingyuan Chen

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows (2024.acl-long)

Challenge:	Existing Large Language Models are usually generalized with large programming corpus, therefore the generated code is difficult to adapt to personalized and/or customized requests.
Approach:	They propose a method to use Large Language Models to generate personalized code for multiple users.
Outcome:	The proposed model can generate personalized code for multiple users . it can be used to improve code generation and reduce maintenance costs.

Copied to clipboard

Ajay Patel, Colin Raffel, Chris Callison-Burch

Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective (2024.acl-long)

Challenge:	Large language models (LLMs) have become a dominant tool for NLP researchers in a wide range of tasks.
Approach:	They propose an open source Python library that allows researchers to write simple code to implement powerful LLM workflows.
Outcome:	The proposed library is open source and can be used to implement powerful LLM workflows.

Copied to clipboard

Chenze Shao, Fandong Meng, Jiali Zeng, Jie Zhou

Identifying while Learning for Document Event Causality Identification (2024.acl-long)

Challenge:	Neural Machine Translation (NMT) has made remarkable progress over the past years, but under-translation and over-translatation remain challenging obstacles faced by NMT systems.
Approach:	They propose to employ the confidence of predicting the end of sentence (EOS) as a detector for under-translation and strengthen the confidence-based penalty to penalize candidates with a high risk of under-translated.
Outcome:	The proposed method can detect and rectify under-translated outputs, with minor impact on other correct translations.

Copied to clipboard

Cheng Liu, Wei Xiang, Bang Wang

Challenge:	Existing studies focus on causality existence, but ignore causal direction.
Approach:	They propose a new identifying while learning mode for the ECI task that takes care of the causal direction and updates events’ representations for boosting next round of causality identification.
Outcome:	The proposed method outperforms the state-of-the-art methods on two public datasets.

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024.acl-long)

Copied to clipboard

Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, Maosong Sun

Insert or Attach: Taxonomy Completion via Box Embedding (2024.acl-long)

Challenge:	Large Language Models (LLMs) and Large Multimodal Models have exceeded general human capabilities in various tasks.
Approach:	They present an Olympiad-level bilingual multimodal scientific benchmark featuring 8,476 problems from Olympiad level mathematics and physics competitions.
Outcome:	The best performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning.

Copied to clipboard

Wei Xue, Yongliang Shen, Wenqi Ren, Jietian Guo, Shiliang Pu, Weiming Lu

Semiparametric Token-Sequence Co-Supervision (2024.acl-long)

Challenge:	Existing taxonomy expansion methods embed concepts as vectors in Euclidean space, causing incorrectly model asymmetric relations.
Approach:	They propose to use box containment and center closeness to create geometric scorers that capture intrinsic relationships between concepts.
Outcome:	The proposed framework outperforms existing methods on four real-world datasets.

Copied to clipboard

Hyunji Lee, Doyoung Kim, Jihoon Jun, Se June Joo, Joel Jang, Kyoung-Woon On, Minjoon Seo

Instruction Fusion: Advancing Prompt Evolution through Hybridization (2024.acl-long)

Challenge:	Using semiparametric token-sequence co-supervision, language models are trained using a finite parametric vocabulary space.
Approach:	They propose a semiparametric token-sequence co-supervision training method that leverages supervision from two different supervisions.
Outcome:	The proposed method outperforms models trained via each supervision independently and shows that it encourages a broader generalization capability across the model.

Copied to clipboard

Weidong Guo, Jiuding Yang, Kaitong Yang, Xiangyang Li, Zhuwei Rao, Yu Xu, Di Niu

TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation (2024.acl-long)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks, namely HumanEval, HumanEva+, MBPP, mbap+ and MultiPL-E.

Copied to clipboard

Yikai Zhang, Siyu Yuan, Caiyu Hu, Kyle Richardson, Yanghua Xiao, Jiangjie Chen

Exploring Memorization in Fine-tuned Language Models (2024.acl-long)

Challenge:	e.g., GPT-4 still lag behind humans in effective multitasking, a study finds . current textual simulations do not adequately address the notion of time .
Approach:	They propose a textual simulated environment that incorporates complex temporal dynamics and constraints that better reflect real-life planning scenarios.
Outcome:	The proposed model incorporates complex temporal dynamics and constraints that better reflect real-life planning scenarios.

Copied to clipboard

Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin

Towards Real-world Scenario: Imbalanced New Intent Discovery (2024.acl-long)

Challenge:	Existing studies have shown that pre-trained langauge models tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately.
Approach:	They conduct the first comprehensive analysis to explore language models’ memorization during fine-tuning across tasks.
Outcome:	The proposed analysis shows that memorization presents a strong disparity among different fine-tuning tasks.

Copied to clipboard

Shun Zhang, Yan Chaoran, Jian Yang, Jiaheng Liu, Ying Mo, Jiaqi Bai, Tongliang Li, Zhoujun Li

M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection (2024.acl-long)

Challenge:	Existing studies focus on detecting known and previously undefined categories of user intent . skewed and long-tailed distributions often encountered in open-world scenarios .
Approach:	They propose to use imbalanced new intent discovery task to identify familiar and novel intent categories within long-tailed distributions.
Outcome:	The proposed model outperforms the existing benchmark on three datasets to simulate the real-world long-tail distributions.

Copied to clipboard

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov

Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialogue (2024.acl-long)

Challenge:	Large Language Models (LLMs) have brought an unprecedented surge in machine-generated text (MGT) societal implications are posed by their potential misuse and lack of training data.
Approach:	They propose a benchmark to detect machine-generated text in multiple languages . they use multi-domain and multi-generator corpus to identify which model generated the text .
Outcome:	The proposed benchmark compares a multilingual, multi-domain and multi-generator corpus of MGTs with human-generated content.

Copied to clipboard

Jian Wang, Chak Tou Leong, Jiashuo Wang, Dongding Lin, Wenjie Li, Xiaoyong Wei

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024.acl-long)

Challenge:	Tuning language models for dialogue generation has been a prevalent paradigm for building capable dialogue agents.
Approach:	They propose a multi-round interactive dialogue tuning framework that models the speaker roles of agent and user separately.
Outcome:	The proposed framework performs superior to fine-tuning and improves dialogue consistency.

Copied to clipboard

Nan He, Weichen Xiong, Hanwen Liu, Yi Liao, Lei Ding, Kai Zhang, Guohua Tang, Xiao Han, Yang Wei

Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? (2024.acl-long)

Challenge:	Current methods focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication.
Approach:	They propose a method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness.
Outcome:	The proposed method significantly improves training efficiency on deduplicated datasets and improves downstream accuracy by 1.77%.

Copied to clipboard

Ning Bian, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, Le Sun

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Experimental results show that stories outperform rules as the expression for retrieving commonsense from LLMs, exhibiting higher generation confidence and commonsensense accuracy.
Approach:	They investigate the commonsense ability of large language models expressed through stories and rules to retrieve commonsensing knowledge from LLMs.
Outcome:	The stories outperform rules as commonsense expressions on 28 commonsensense QA datasets, exhibiting higher generation confidence and commonsence accuracy.

Copied to clipboard

Zeqi Tan, Yongliang Shen, Xiaoxia Cheng, Chang Zong, Wenqi Zhang, Jian Shao, Weiming Lu, Yueting Zhuang

CaMML: Context-Aware Multimodal Learner for Large Models (2024.acl-long)

Challenge:	Large language models (LLMs) have shown remarkable performance, but their training costs are exorbitant.
Approach:	They propose a parameter-efficient method for exploring optimal solutions within latent space by using latent units to extract input representations from LLMs.
Outcome:	The proposed method improves performance on a range of natural language processing tasks.

Copied to clipboard

Yixin Chen, Shuai Zhang, Boran Han, Tong He, Bo Li

MAVEN-ARG: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation (2024.acl-long)

Challenge:	a lightweight module for tuning large multimodal models is introduced . CaMML integrates contextual samples into large models, enabling them to make inferences .
Approach:	They introduce a lightweight module for tuning large multimodal models . they have developed two models that have shown exceptional performance .
Outcome:	The proposed model outperforms LLaVA-1.5 on ten widely recognized datasets with a noticeable margin.

Copied to clipboard

Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, Juanzi Li

NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes (2024.acl-long)

Challenge:	Existing datasets for event understanding have limited coverage due to complexity of tasks.
Approach:	They propose a dataset that augments MAVEN datasets with event argument annotations . they propose 98,591 events and 290,613 arguments obtained with laborious human annotation .
Outcome:	The proposed dataset is the first all-in-one dataset supporting event detection, event argument extraction, and event relation extraction.

Copied to clipboard

Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, Yongfeng Zhang

Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models (2024.acl-long)

Challenge:	Complex reasoning ability is one of the most important features of Large Language Models.
Approach:	They propose a new benchmark that measures the reasoning ability of Large Language Models . it contains 900 algorithmic questions belonging to the NP-Hard complexity class .
Outcome:	The proposed benchmark contains 900 questions belonging to the NP-Hard complexity class and is updated on a monthly basis.

Copied to clipboard

Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, Rui Wang

Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors (2024.acl-long)

Challenge:	Existing text watermarking technologies lack consistency when texts are translated into different languages.
Approach:	They propose a cross-lingual watermark removal attack to bypass watermarking by first obtaining a response from an LLM in a pivot language and then translating it into the target language.
Outcome:	The proposed method can remove watermarks without performance loss by obtaining a response from an LLM in a pivot language and then translating it into the target language.

Copied to clipboard

Alicja Chaszczewicz, Raj Shah, Ryan Louie, Bruce Arnow, Robert Kraut, Diyi Yang

In-context Mixing (ICM): Code-mixed Prompts for Multilingual LLMs (2024.acl-long)

Challenge:	Existing mechanisms of providing feedback rely on human supervision . existing mechanisms of delivering feedback largely rely only on human oversight .
Approach:	They propose to leverage large language models to provide contextualized feedback to peer counselors . they construct a publicly available dataset with detailed feedback annotations of 400 conversations .
Outcome:	The proposed method minimizes the risk of potentially harmful and low-quality feedback generation.

Copied to clipboard

Bhavani Shankar, Preethi Jyothi, Pushpak Bhattacharyya

Respond in my Language: Mitigating Language Inconsistency in Response Generation based on Large Language Models (2024.acl-long)

Challenge:	In-context mixing is a prompting technique for effective in-contact learning with multilingual large language models.
Approach:	They propose a prompting technique called in-context mixing for effective in-constext learning with multilingual large language models.
Outcome:	The proposed prompts perform better with multilingual large language models.

Copied to clipboard

Liang Zhang, Qin Jin, Haoyang Huang, Dongdong Zhang, Furu Wei

Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries (2024.acl-long)

Challenge:	Large Language Models (LLMs) show strong instruction understanding ability across multiple languages, but are easily biased towards English in instruction tuning.
Approach:	They propose to use a model with Pseudo-Inconsistent Penalization to prevent the model from generating English responses when given non-English language prompts during training and prior Enhanced decoding to improve the language consistency of the model.
Outcome:	The proposed methods significantly improve the language consistency of the model without multilingual data.

Copied to clipboard

Yu-Hsiang Huang, Yuche Tsai, Hsiang Hsiao, Hong-Yi Lin, Shou-De Lin

Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding (2024.acl-long)

Challenge:	Recent advances in text embedding models have significantly streamlined the process of generating embeddables.
Approach:	They develop a transfer attack method that uses a surrogate model to mimic the victim model's behavior and infers sensitive information from embeddings without direct access.
Outcome:	The proposed method outperforms existing methods and reveals potential privacy vulnerabilities in embedding technologies.

Copied to clipboard

Kuo Liao, Shuang Li, Meng Zhao, Liqun Liu, Mengge Xue, Zhenyu Hu, Honglin Han, Chengguo Yin

Challenge:	Recent advances in large language models (LLMs) have yielded remarkable performance, but objective mismatch issues hinder RLHF learning.
Approach:	They propose a Reinforcement Learning framework enhanced with Label-sensitive reward to enhance LLMs' alignment and generation capabilities.
Outcome:	The proposed framework improves performance on five diverse models across eight tasks.

Intuitive or Dependent? Investigating LLMs’ Behavior Style to Conflicting Prompts (2024.acl-long)

Copied to clipboard

Jiahao Ying, Yixin Cao, Kai Xiong, Long Cui, Yidong He, Yongbin Liu

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending (2024.acl-long)

Challenge:	Extensive experiments with seven Large Language Models reveal their varying behaviors.
Approach:	They investigate the behaviors of Large Language Models when faced with conflicting prompts versus their internal memory.
Outcome:	Extensive experiments with seven LLMs reveal their varying behaviors.

Copied to clipboard

Shiyi Zhu, Jing Ye, Wei Jiang, Siqiao Xue, Qi Zhang, Yifan Wu, Jianguo Li

InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification (2024.acl-long)

Challenge:	Existing models that use self-attention and position embedding have anomalous behavior that hinder long context window extrapolation.
Approach:	They propose a collinear constraint between Q and K to integrate RoPE and self-attention.
Outcome:	The proposed model integrates self-attention and position embedding into LLMs without fine-tuning.

Copied to clipboard

Jan Trienes, Sebastian Joseph, Jörg Schlötterer, Christin Seifert, Kyle Lo, Wei Xu, Byron Wallace, Junyi Jessy Li

Paraphrasing

Challenge:	Text simplification aims to make technical texts more accessible to laypeople but often results in deletion of information and vagueness.
Approach:	They propose a framework to characterize and recover simplification-induced information loss in form of question-and-answer (QA) pairs.
Outcome:	The proposed framework characterizes and recovers simplification-induced information loss in form of question-and-answer (QA) pairs.

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following (2024.acl-long)

Copied to clipboard

Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, Bowen Zhou

DAPR: A Benchmark on Document-Aware Passage Retrieval (2024.acl-long)

Challenge:	Large-scale language models (LLMs) are increasingly exposed to private data and are becoming more and more prevalent.
Approach:	They propose a collaborative generation framework that integrates large and small language models to address privacy concerns logically.
Outcome:	The proposed framework combines large and small models to address privacy concerns logically.

Copied to clipboard

Kexin Wang, Nils Reimers, Iryna Gurevych

Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors (2024.acl-long)

Challenge:	Recent approaches to retrieve short passages are limited due to expensive operations such as selfattention.
Approach:	They propose to use a task to find passages from a large corpus of documents . they extend the SoTA passage retrievers with document context to achieve this task .
Outcome:	The proposed task is called Document-Aware Passage Retrieval (DAPR) it performs best on easy and hard queries, but fails on hard queries requiring document-context understanding .

Copied to clipboard

Mengge Xue, Zhenyu Hu, Liqun Liu, Kuo Liao, Shuang Li, Honglin Han, Meng Zhao, Chengguo Yin

SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graph (2024.acl-long)

Challenge:	Multiple-Choice Questions (MCQs) are a critical area of research in the study of Large Language models (LLMs).
Approach:	They propose an efficient SFT algorithm for MCQs, termed Point-wise Intelligent Feedback, which constructs negative instances by randomly combing the incorrect option contents with all candidate symbols.
Outcome:	The proposed algorithm significantly reduces the model’s selection bias by improving its MCSB capability.

Copied to clipboard

Hanzhu Chen, Xu Shen, Qitan Lv, Jie Wang, Xiaoqi Ni, Jieping Ye

Uncertainty-Guided Modal Rebalance for Hateful Memes Detection (2024.acl-long)

Challenge:	Existing KG construction methods rely on human intervention to attain qualified KGs, which severely hinders the practical application of domain KG.
Approach:	They propose a general KG construction framework that uses large language models as "Skilled" Automatic Constructors for domain knowledge (Graph)
Outcome:	The proposed framework generates specialized multi-level knowledge graphs at the scale of over one million nodes and achieves 89.32% precision rate compared to state-of-the-art methods.

Copied to clipboard

Chuanpeng Yang, Yaxin Liu, Fuqing Zhu, Jizhong Han, Songlin Hu

Missci: Reconstructing Fallacies in Misrepresented Science (2024.acl-long)

Challenge:	Existing methods for integrating hate information from different modalities ignore the modality uncertainty caused by the contribution degree of each modality to hate sentiment.
Approach:	They propose an Uncertainty-guided Modal Rebalance framework for hateful memes detection . they propose to combine cross-modal fusion features with unimodal features .
Outcome:	The proposed framework produces state-of-the-art performance on four widely-used datasets.

Copied to clipboard

Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych

Uncovering the Full Potential of Visual Grounding Methods in VQA (2024.acl-long)

Challenge:	False or misleading narratives spread rapidly on social networks, posing challenges for non-experts in discerning credible information.
Approach:	They propose a model for fallacious reasoning that focuses on implicit fallacies between relevant content and the inaccurate claim and requires models to verbalize the fallacious thinking in addition to classifying it.
Outcome:	The proposed model focuses on implicit fallacies between relevant content and the inaccurate claim and requires models to verbalize the fallacious reasoning in addition to classifying it.

Copied to clipboard

Daniel Reich, Tanja Schultz

Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs (2024.acl-long)

Challenge:	Visual Grounding (VG) methods in VQA aim to strengthen a model's reliance on question-relevant visual information.
Approach:	They propose to strengthen a model's reliance on question-relevant visual information by using a visual grounding method that is based on a question-related visual input.
Outcome:	The proposed methods can be much more effective when evaluation conditions are corrected.

Copied to clipboard

Jiejun Tan, Zhicheng Dou, Yutao Zhu, Peidong Guo, Kun Fang, Ji-Rong Wen

Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation (2024.acl-long)

Challenge:	Existing methods to determine the knowledge an LLM already possesses and the knowledge that requires the help of a search engine are expensive and require excessive computational costs.
Approach:	They propose a slim proxy model that detects missing knowledge in LLMs with a proxy model and use it to perform retrieval for the missing knowledge.
Outcome:	The proposed approach detects missing knowledge in LLMs with a slim proxy model and takes its answers as heuristic answers.

Copied to clipboard

Pius Von Däniken, Jan Deriu, Don Tuggener, Mark Cieliebak

LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback (2024.acl-long)

Challenge:	Generative AI systems are becoming ubiquitous for all kinds of modalities . evaluation of generated outputs is increasingly difficult due to cost and complexity of human evaluations.
Approach:	They propose to evaluate preference ratings on sign accuracy and favoritism . they propose to use automated metrics to assess generated outputs .
Outcome:	The proposed evaluations of preference ratings rely on correlation to human judgments or sign accuracy scores, but this does not tell the whole story.

Copied to clipboard

Timon Ziegenbein, Gabriella Skitalinskaya, Alireza Bayat Makou, Henning Wachsmuth

Graph Language Models (2024.acl-long)

Challenge:	Creating trusted and safe online spaces for people with different backgrounds and opinions is a challenge for social media platforms.
Approach:	They propose a reinforcement learning-based rewriting approach that balances content preservation and appropriateness based on existing classifiers.
Outcome:	The proposed approach significantly outperforms baselines including few-shot learning, prompting, and humans.

Copied to clipboard

Moritz Plenz, Anette Frank

Analyzing Semantic Change through Lexical Replacements (2024.acl-long)

Challenge:	Language Models (LMs) are the workhorses of NLP, but their interplay with structured knowledge graphs (KGs) is still actively researched.
Approach:	They propose a Graph Language Model (GLM) that integrates the strengths of both approaches and mitigates their weaknesses.
Outcome:	Empirical evaluations show that the proposed model surpasses both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.

Copied to clipboard

Francesco Periti, Pierluigi Cassotti, Haim Dubossarsky, Nina Tahmasebi

Challenge:	Modern language models can contextualize words based on their surrounding contexts, but semantic change can compromise this capability.
Approach:	They propose a replacement schema where a target word is replaced with lexical replacements of varying relatedness . they leverage the replacement schema as a basis for a novel interpretable model for semantic change .
Outcome:	The proposed model is the first to evaluate LLaMa for semantic change detection . it shows that lexical replacements can detect unexpected contexts .

Exploiting Intrinsic Multilateral Logical Rules for Weakly Supervised Natural Language Video Localization (2024.acl-long)

Copied to clipboard

Zhe Xu, Kun Wei, Xu Yang, Cheng Deng

Interpretability of Language Models via Task Spaces (2024.acl-long)

Challenge:	Existing methods for WS-NLVL rarely consider complex temporal relations enclosing the language query, yielding illogical predictions.
Approach:	They propose a plug-and-play method to exploit temporal relations and logical rules for WS-NLVL.
Outcome:	The proposed method is able to retrieve the moment corresponding to a language query in a video with only video-language pairs utilized during training.

Copied to clipboard

Lucas Weber, Jaap Jumelet, Elia Bruni, Dieuwke Hupkes

Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types (2024.acl-long)

Challenge:	linguistic interpretability is a method used to assess language models' ability to interpret outputs.
Approach:	They propose a method to assess LMs' language conceptualisations by 'similarity probing' and a technique to fine tune them via gradient differentials to disentangle the learning signals of linguistic phenomena.
Outcome:	The proposed method generalises larger models to overarching general concepts for linguistic tasks, and the generalisation patterns are stable throughout training and not marked by incisive stages.

Copied to clipboard

Pierluigi Cassotti, Stefano De Pascale, Nina Tahmasebi

Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators (2024.acl-long)

Challenge:	Existing models for detecting semantic change in corpora have been disregarded due to lack of knowledge of the nature of semantic change and the way it takes place.
Approach:	They propose a model that leverages synchronic lexical relations and definitions of word meanings to detect these types of change.
Outcome:	The proposed model can detect changes in a digitized version of Blank's dataset and improve human judgments of semantic relatedness and binary Lexical Semantic Change Detection.

Copied to clipboard

Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluis Marquez

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback (2024.acl-long)

Challenge:	Large Language Models (LLMs) tend to be unreliable on fact-based answers.
Approach:	They propose a framework for comparing LLMs' confidence over fact-based answers with hidden-state probes that are more reliable than hidden-status probes.
Outcome:	The proposed methods show that hidden-state probes provide the most reliable confidence estimates despite requiring access to weights and supervision data.

Copied to clipboard

Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang

One-Shot Learning as Instruction Data Prospector for Large Language Models (2024.acl-long)

Challenge:	Existing work integrates reinforcement learning with compiler feedback to enhance code generation quality but the long code generated by LLMs makes RL exploration ineffective.
Approach:	They propose a framework that integrates reinforcement learning and compiler feedback to enhance code generation quality.
Outcome:	The proposed framework outperforms state-of-the-art approaches in corresponding benchmarks and integrates reinforcement learning with compiler feedback to improve code generation quality.

Copied to clipboard

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, Yongbin Li

Navigating the OverKill in Large Language Models (2024.acl-long)

Challenge:	Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality.
Approach:	They propose a method that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets.
Outcome:	Nuggets outperforms existing methods on MT-Bench and Alpaca-Eval benchmarks.

Copied to clipboard

Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, Dahua Lin

A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)

Challenge:	Recent studies have highlighted a tendency among large language models to refuse to answer benign queries.
Approach:	They propose a model-agnostic approach to reduce excessive attention to harmful words like ‘kill’ and a method to decode the next-token predictions by contrastive decoding.
Outcome:	The proposed approach reduces the refusal rate by 20% while having little impact on safety.

Copied to clipboard

Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva

Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision (2024.acl-long)

Challenge:	Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods.
Approach:	They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step.
Outcome:	The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions.

Copied to clipboard

Qian Ruan, Ilia Kuznetsov, Iryna Gurevych

Challenge:	a framework for collaborative document revision is lacking for empirical analysis and NLP.
Approach:	They propose a framework for joint analysis of collaborative document revision that instantiates a corpus of aligned scientific paper revisions manually labeled according to their action and intent.
Outcome:	The proposed framework provides first empirical insights into collaborative document revision in the academic domain and assesses its capabilities.

NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024.acl-long)

Copied to clipboard

Tamara Czinczoll, Christoph Hönes, Maximilian Schall, Gerard De Melo

FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models (2024.acl-long)

Challenge:	(large) language models struggle to process long sequences due to the quadratic scaling of the underlying attention mechanism.
Approach:	They propose a Masked Language Model operating on higher-level semantic representations in the form of text embeddings to solve this problem.
Outcome:	The proposed model outperforms larger embedding models on three types of tasks.

Copied to clipboard

Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, Wei Wang

Learning to Edit: Aligning LLMs with Knowledge Editing (2024.acl-long)

Challenge:	Existing benchmarks focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction.
Approach:	They propose a Multi-level Fine-grained Constraints Following Benchmark for Large Language Models that adds a single constraint to the initial instruction at each increased level.
Outcome:	The proposed model can follow instructions with more constraints, and is deemed to have better instruction-following ability.

Copied to clipboard

Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, Wei Wang

Challenge:	Existing knowledge editing techniques rely on memorizing updated knowledge, impeding LLMs from effectively combining the new knowledge with their inherent knowledge when answering questions.
Approach:	They propose a Learning to Edit framework that equips LLMs with the ability to apply updated knowledge to input questions through a two-phase process .
Outcome:	The proposed framework outperforms existing methods in knowledge editing tasks and compares it with four benchmarks and two LLM architectures.

DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024.acl-long)

Copied to clipboard

Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Weiran Xu, Jingang Wang, Mengdi Zhang, Xunliang Cai

When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024.acl-long)

Challenge:	Numerous code large language models (LLMs) have been proposed to enhance code generation performance.
Approach:	They propose a diverse instruction model DolphCoder with self-evaluating for code generation that learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Outcome:	The proposed model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work.

Copied to clipboard

Brielen Madureira, Patrick Kahardipraja, David Schlangen

SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models (2024.acl-long)

Challenge:	In incremental models, one interpretation is possible, but models that can revise can do so if the ambiguity is resolved.
Approach:	They propose an interpretable way to analyse incremental states in a bidirectional way . they propose to use a model that can update internal states to reflect the garden path effect .
Outcome:	The proposed model shows that it can perform revisions and recover if the label is incorrect.

Copied to clipboard

Md Imbesat Rizvi, Xiaodan Zhu, Iryna Gurevych

Planning Like Human: A Dual-process Framework for Dialogue Planning (2024.acl-long)

Challenge:	Existing large language models (LLMs) do not perform well on the datasets.
Approach:	They propose to use a Spatial Reasoning Characterization framework and a spatial reasoning path framework to study spatial reasoning.
Outcome:	The proposed framework and datasets outperform state-of-the-art models in spatial reasoning.

Copied to clipboard

Tao He, Lizi Liao, Yixin Cao, Yuanxing Liu, Ming Liu, Zerui Chen, Bing Qin

Spectral Filters, Dark Signals, and Attention Sinks (2024.acl-long)

Challenge:	Large Language Models (LLMs) operate in a reactive mode, often resulting in efficiency issues or suboptimal performance.
Approach:	They propose a dual-process dialogue planning framework that leverages the dual-process theory of human cognition and a deliberative Monte Carlo Tree Search mechanism to emulate human-like conversational dynamics.
Outcome:	The proposed framework outperforms existing methods in achieving high-quality dialogues and operational efficiency.

Copied to clipboard

Nicola Cancedda

DiffuCOMET: Contextual Commonsense Knowledge Diffusion (2024.acl-long)

Challenge:	Recent work assigns a central role to the model's residual stream as the shared communication channel between model components.
Approach:	They propose a quantitative extension of the logit lens approach by partitioning the embedding and unembedding matrices into bands and spectral filters on intermediate representations.
Outcome:	The proposed model can suppress the tail end of the embedding spectrum, but it is not able to suppress large parts of the spectrum.

Copied to clipboard

Silin Gao, Mete Ismayilzada, Mengjie Zhao, Hiromi Wakaki, Yuki Mitsufuji, Antoine Bosselut

Systematic Task Exploration with LLMs: A Study in Citation Text Generation (2024.acl-long)

Challenge:	Recent methods for identifying contextually relevant commonsense inferences are weak . knowledge models are trained to verbalize tuples from general commonsens knowledge graphs .
Approach:	They develop a series of knowledge models that leverage diffusion to reconstruct semantic connections between narrative contexts and relevant commonsense knowledge.
Outcome:	The proposed model improves on two benchmarks, ComFact and WebNLG+, to measure commonsense diversity and contextual relevance.

Copied to clipboard

Furkan Şahinuç, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych

Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition (2024.acl-long)

Challenge:	Large language models (LLMs) provide unprecedented flexibility in defining and executing complex, creative natural language generation tasks.
Approach:	They propose a framework that consists of input manipulation, reference data, and output measurement to explore citation text generation.
Outcome:	The proposed framework explores citation text generation, a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm.

Copied to clipboard

Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, Andreas Bulling

Temporal Knowledge Question Answering via Abstract Reasoning Induction (2024.acl-long)

Challenge:	Recent work on dialogue-based collaborative plan acquisition (CPA) suggests Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge.
Approach:	They propose to use task-specific constraints to represent plans as graphs and exploit task-related constraints to improve missing knowledge prediction in CPA.
Outcome:	The proposed model improves missing knowledge prediction in contexts with asymmetric skill-sets and knowledge, but the improvements diminish . the proposed model is compared with baseline models and found to be more effective than existing models.

Copied to clipboard

Ziyang Chen, Dongfang Li, Xiang Zhao, Baotian Hu, Min Zhang

Who Wrote this Code? Watermarking for Code Generation (2024.acl-long)

Challenge:	a new method to enhance temporal knowledge reasoning in large language models addresses this challenge . Abstract Reasoning Induction (ARI) framework provides factual knowledge support to LLMs .
Approach:	They propose an abstract reasoning induction framework which divides temporal reasoning into two phases: Knowledge agnostic and Knowledge-based.
Outcome:	The proposed method achieves significant gains on two temporal QA datasets.

Copied to clipboard

Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, Gunhee Kim

MapCoder: Multi-Agent Code Generation for Competitive Problem Solving (2024.acl-long)

Challenge:	Existing methods to detect machine-generated text by embedding watermarks fail to function appropriately in code generation tasks due to the task’s nature of having low entropy.
Approach:	They propose a logit-modifying watermark method which enhances detection ability and mitigates code quality degeneration by removing low-entropy segments at generating and detecting watermarks.
Outcome:	The proposed method outperforms baseline methods in detecting machine-generated code text while preserving code quality.

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

RelayAttention for Efficient Large Language Model Serving with Long System Prompts (2024.acl-long)

Challenge:	Large language models (LLMs) have impressive proficiency in natural language processing, but performance in code generation tasks remains limited.
Approach:	They propose a framework that emulates the full cycle of program synthesis as observed in humans.
Outcome:	The proposed framework replicates the full cycle of program synthesis as observed in human developers.

Copied to clipboard

Lei Zhu, Xinjiang Wang, Wayne Zhang, Rynson Lau

Boosting Language Models Reasoning with Chain-of-Knowledge Prompting (2024.acl-long)

Challenge:	a long system prompt causes throughput/latency bottlenecks as the cost of generating the next token increases w.r.t the sequence length.
Approach:	They propose an attention algorithm that reads hidden state from DRAM exactly once for a batch of input tokens.
Outcome:	The proposed algorithm reduces the need for redundant memory accesses in existing algorithms.

Copied to clipboard

Jianing Wang, Qiushi Sun, Xiang Li, Ming Gao

Open Grounded Planning: Challenges and Benchmark Construction (2024.acl-long)

Challenge:	Recent studies have shown that Chain-of-Thought (CoT) prompting can be effective on complex reasoning tasks but generates unfaithful and unfactual reasoning chains.
Approach:	They propose a chain-of-knowledge prompting that elicits Large Language Models to generate explicit pieces of knowledge evidence in the form of structure triple.
Outcome:	The proposed method improves commonsense, factual, symbolic, and arithmetic reasoning tasks by estimating the reliability of the reasoning chains in terms of factuality and faithfulness.

Copied to clipboard

Shiguang Guo, Ziliang Deng, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun

Challenge:	Existing work on LLM-based planning uses language generation to produce free-style plans . however, these plans are not grounded in an executable set of actions .
Approach:	They propose a new task for open grounded planning that asks the model to generate an executable plan based on a variable action set.
Outcome:	The proposed task is open grounded planning, which is based on a set of variables.

LLM Knows Body Language, Too: Translating Speech Voices into Human Gestures (2024.acl-long)

Copied to clipboard

Chenghao Xu, Guangtao Lyu, Jiexi Yan, Muli Yang, Cheng Deng

QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback based Self-Correction (2024.acl-long)

Challenge:	despite advances in the generation of realistic human gestures, the process often includes unintended, meaningless, or non-realistic gestures.
Approach:	They propose a framework that leverages large language models to generate human gestures . the primary stage employs a transformer-based auto-encoder network to encode human gesture into discrete symbols .
Outcome:	The proposed framework has demonstrated state-of-the-art performance on public TED and TED-Expressive datasets.

Copied to clipboard

Xiang Huang, Sitao Cheng, Shanshan Huang, Jiayu Shen, Yong Xu, Chaoyun Zhang, Yuzhong Qu

PITA: Prompting Task Interaction for Argumentation Mining (2024.acl-long)

Challenge:	Existing methods for semantic parsing fail when hallucinations are encountered . QueryAgent solves a question step-by-step and performs stepwise self-correction .
Approach:	They propose a framework that solves a query step-by-step and performs stepwise self-correction.
Outcome:	The proposed framework outperforms existing methods on GrailQA and GraphQ by 5.7 and 15.0 points.

Copied to clipboard

Yang Sun, Muyi Wang, Jianzhu Bao, Bin Liang, Xiaoyan Zhao, Caihua Yang, Min Yang, Ruifeng Xu

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)

Challenge:	Argumentation mining (AM) aims to detect arguments and their inherent relations from textual compositions.
Approach:	They propose a method to model the inter-relationships among three subtasks within a generative framework.
Outcome:	The proposed method achieves state-of-the-art performance on two AM benchmarks.

Copied to clipboard

Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu

Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations (2024.acl-long)

Challenge:	Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable.
Approach:	They propose to shift attention to more relevant components at token- and sentence-levels for better UQ.
Outcome:	The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters.

Copied to clipboard

Gregor Geigle, Radu Timofte, Goran Glavaš

Estimating Agreement by Chance for Sequence Annotation (2024.acl-long)

Challenge:	Vision-and-language models with separate encoders for each modality are limited in availability.
Approach:	They propose a multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation.
Outcome:	The proposed model outperforms models on English and low-resource languages.

Copied to clipboard

Diya Li, Carolyn Rose, Ao Yuan, Chunxiao Zhou

Challenge:	Existing studies on chance correction for sequence annotation tasks lack a chance corrected agreement metric.
Approach:	They propose a model for generating random annotations which serves as the foundation for estimating chance agreement in sequence annotation tasks.
Outcome:	The proposed model is validated in simulation and corpus-based evaluation.

Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)

Copied to clipboard

Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, Iryna Gurevych

WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning (2024.acl-long)

Challenge:	Large language models have been claimed to acquire certain capabilities without having been specifically trained on them.
Approach:	They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments.
Outcome:	The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge.

Copied to clipboard

Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, Qiufeng Yin

Eliciting Better Multilingual Structured Reasoning from LLMs through Code (2024.acl-long)

Challenge:	Recent work shows that Code Large Language Models can address a wide range of code-related tasks.
Approach:	They propose a method to generate widespread and versatile instruction data from open source code datasets and use it to train code-related models.
Outcome:	The proposed model outperforms open-source models in generalization ability across code-related tasks.

Copied to clipboard

Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour

OLIVE: Object Level In-Context Visual Embeddings (2024.acl-long)

Challenge:	xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks.
Approach:	They propose a multilingual structured reasoning and explanation dataset that covers four tasks across six languages and extends the English STREET benchmark to 5 additional diverse languages.
Outcome:	The proposed models show improved multilingual performance on scientific commonsense reasoning subtasks and no regression on non-reasoning tasks.

Copied to clipboard

Timothy Ossowski, Junjie Hu

Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness (2024.acl-long)

Challenge:	Existing vision-language models lack fine-grained object-level understanding and grounding . existing models implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and introduces noisy spurious background features.
Approach:	They propose a method to prompt large language models with in-context visual object vectors . this method allows for controllable object-level reasoning .
Outcome:	The proposed method achieves competitive referring object classification and captioning performance while offering zero-shot generalization and robustness to visually challenging contexts.

Copied to clipboard

Jiuhai Chen, Jonas Mueller

Marathon: A Race Through the Realm of Long Context with Large Language Models (2024.acl-long)

Challenge:	BSDetector is a method for detecting bad and speculative answers from a pretrained Large Language Model.
Approach:	They propose a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a confidence score for any output it generated.
Outcome:	Experiments on open-form Question-Answer benchmarks show that BSDetector more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures.

Copied to clipboard

Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, Min Yang

Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph (2024.acl-long)

Challenge:	Existing long-context benchmarks do not accurately evaluate large language models’ comprehension and reasoning abilities in extended texts.
Approach:	They propose a new evaluation benchmark that adopts a multiple-choice question format and uses a multi-choke question format to assess the comprehension and reasoning skills of large language models.
Outcome:	The proposed benchmark provides a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models.

Copied to clipboard

Xiaochen Gao, Feng Yao, Kewen Zhao, Beilei He, Animesh Kumar, Vish Krishnan, Jingbo Shang

PCAD: Towards ASR-Robust Spoken Language Understanding via Prototype Calibration and Asymmetric Decoupling (2024.acl-long)

Challenge:	Scaling up language models has demonstrated predictable improvement and unprecedented abilities in many language tasks.
Approach:	They propose a fine-grained cLAim depeNdency graph that captures the dependencies within the patent data and extends the embedding-based state-of-the-art (SOTA) they then explore prompt-based methods to harness proprietary LLMs' potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up.
Outcome:	The proposed graph methods outperform the standard model scaling methods in the patent approval prediction task and show that they are cost-effective.

Copied to clipboard

Xianwei Zhuang, Xuxin Cheng, Liming Liang, Yuxin Xie, Zhichang Wang, Zhiqi Huang, Yuexian Zou

Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment (2024.acl-long)

Challenge:	Spoken language understanding (SLU) suffers from error propagation from automatic speech recognition (ASR) in actual scenarios.
Approach:	They propose a framework which calibrates bias and errors and achieves adaptive-balanced decoupling training by a prototype-based loss model.
Outcome:	The proposed framework outperforms existing approaches and achieves state-of-the-art performance on three datasets.

Copied to clipboard

Tao Jin, Wang Lin, Ye Wang, Linjun Li, Xize Cheng, Zhou Zhao

UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation (2024.acl-long)

Challenge:	Existing studies have focused on the alignment of multimodal sequential learning using transformers.
Approach:	They propose a constrained scheme to align the multiple attentional results from both local and global perspectives.
Outcome:	The proposed scheme could align the multiple attentional results from both local and global perspectives, making the information capture more efficient.

Copied to clipboard

Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Yezhaohui Wang, Dawei He, Cheng Peng, Zhonghao Wang, Haiying Deng

PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)

Challenge:	Large language models (LLMs) produce hallucinated text, compromising their practical utility in professional contexts.
Approach:	They have developed an unconstrained hallucination generation evaluation benchmark that contains hallucines generated by large language models with minimal restrictions.
Outcome:	The proposed benchmarks are based on a Chinese-language dataset that is lacking in the field.

Copied to clipboard

Weizhe Lin, Jingbiao Mei, Jinghong Chen, Bill Byrne

Triple-Encoders: Representations That Fire Together, Wire Together (2024.acl-long)

Challenge:	Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA).
Approach:	They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks.
Outcome:	The proposed framework is used to train and evaluate multi-modal retrievers.

Copied to clipboard

Justus-Jonas Erker, Florian Mai, Nils Reimers, Gerasimos Spanakis, Iryna Gurevych

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning (2024.acl-long)

Challenge:	Existing search-based dialog models re-encode the dialog history at every turn, causing high cost.
Approach:	They propose a method that encodes relative distances between utterances via a bi-encoder . they use a hebbian-inspired co-occurrence learning objective to compute distributed utterrance mixtures .
Outcome:	The proposed method improves on bi-encoded utterances and even better generalization . it can encode 2020 and Erker et al. (2023) with the same results.

Copied to clipboard

Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, Marcus Tomalin

Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization (2024.acl-long)

Challenge:	Existing systems for detecting hateful memes lack sensitivity to subtle differences in memes that are vital for correct hatefulness classification.
Approach:	They propose to construct a hatefulness-aware embedding space through retrieval-guided contrastive training to identify hatefulness based on data unseen in training.
Outcome:	The proposed system outperforms existing models on the HatefulMemes dataset with an AUROC of 87.0 and improves contextual understanding across domains.

Copied to clipboard

Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, Weiming Lu

Your Transformer is Secretly Linear (2024.acl-long)

Challenge:	Large Language Models (LLMs) are designed as specific task solvers with sophisticated prompt engineering, but are inherently incapacitating to address complex dynamic scenarios.
Approach:	They propose an LLM-based agent with policy-level reflection and optimization that can learn from interactive experiences and progressively elevate its behavioral policy.
Outcome:	The proposed agent outperforms vanilla LLM and specialized models in blackjack and Texas hold’em.

Copied to clipboard

Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov

Noise Correction on Subjective Datasets (2024.acl-long)

Challenge:	a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity.
Approach:	They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders.
Outcome:	The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models.

Copied to clipboard

Uthman Jinadu, Yi Ding

Challenge:	Annotator fatigue and changing opinions over time can distort dataset annotations.
Approach:	They propose to use multitask learning to learn a more accurate representation of diverse opinions by using loss-based label correction.
Outcome:	The proposed method can cleanly separate agreeing and disagreeing annotations and provides a controllable way to encourage or discourage disagreement.

Generative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizers (2024.acl-long)

Copied to clipboard

Lütfi Kerem Senel, Besnik Fetahu, Davis Yoshida, Zhiyu Chen, Giuseppe Castellucci, Nikhita Vedula, Jason Ingyu Choi, Shervin Malmasi

Challenge:	Large Language Models (LLMs) have given rise to generative recommenders . however, improving the generated content through user feedback is prohibitively expensive .
Approach:	They propose a generative explore-exploit method that exploits items with high engagement and actively explores hidden population preferences to improve recommendation quality.
Outcome:	The proposed approach exploits items with high engagement and actively explores hidden population preferences to improve recommendation quality.

Instruction-tuned Language Models are Better Knowledge Learners (2024.acl-long)

Copied to clipboard

Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, Srini Iyer

What Do Language Models Hear? Probing for Auditory Representations in Language Models (2024.acl-long)

Challenge:	Large language models store factual knowledge in parameters, but it can become outdated as the work evolves . pre-instruction-tuning improves ability of LLMs to absorb knowledge from new documents .
Approach:	They propose a method that instruction-tunes on questions prior to training on documents . they propose to use QA pairs to update factual knowledge of large language models .
Outcome:	The proposed method outperforms instruction-tuning on documents by 17.8%.

Copied to clipboard

Jerry Ngo, Yoon Kim

Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs (2024.acl-long)

Challenge:	a linear probe is used to retrieve the correct text representation of an object given a snippet of audio related to that object.
Approach:	They develop a linear probe that retrieves the correct text representation of an object . they then test the probe's generalization to objects that were not seen during training .
Outcome:	The proposed model generalizes to objects that were not seen during training, the study finds . the model can learn representations of perceptual concepts that plausibly mirror the grounded representations .

Copied to clipboard

Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, Dongyeop Kang

Challenge:	Empirical findings show that although both LLMs and humans generate distinct discourse patterns influenced by specific domains, human-written texts exhibit more structural variability, reflecting the nuanced nature of human writing in different domains.
Approach:	They propose a method to leverage hierarchical parse trees and recursive hypergraphs to uncover distinctive discourse patterns in texts written by humans and LLMs.
Outcome:	The proposed method combines hierarchical parse trees and recursive hypergraphs to uncover distinctive discourse patterns in texts produced by both LLMs and humans.

Jailbreak Open-Sourced Large Language Models via Enforced Decoding (2024.acl-long)

Copied to clipboard

Hangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, Dinghao Wu

NICE: To Optimize In-Context Examples or Not? (2024.acl-long)

Challenge:	Existing studies show that Large Language Models can be misused to generate undesired content.
Approach:	They propose to use large language models to manipulate the generation process to generate undesired content without heavy computations or prompt designs.
Outcome:	The proposed method shows that open-sourced large language models could be misused to generate undesired content without heavy computations or prompt designs.

Copied to clipboard

Pragya Srivastava, Satvik Golechha, Amit Deshpande, Amit Sharma

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)

Challenge:	Recent work shows that in-context learning and optimization of in-const examples (ICE) can improve the accuracy of large language models on a wide range of tasks.
Approach:	They propose a task-specific metric called Normalized Invariability to Choice of Examples (NICE) metric measures the learnability of tasks from a given instruction and provides a heuristic to decide whether to optimize ICE for a new task.
Outcome:	The proposed metric predicts the utility of optimizing ICE for a given task compared to random ICE.

Copied to clipboard

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng

Digital Socrates: Evaluating LLMs through Explanation Critiques (2024.acl-long)

Challenge:	Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks .
Approach:	They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency .
Outcome:	The proposed benchmark covers 43 programming languages and eight coding tasks.

Copied to clipboard

Yuling Gu, Oyvind Tafjord, Peter Clark

SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)

Challenge:	Large language models (LLMs) can provide reasoned explanations, but the nature and quality of those explanations are still poorly understood.
Approach:	They propose to define a task of explanation critiquing and train an open-source automatic critique model using this data.
Outcome:	The proposed model can provide high-quality, nuanced evaluations without expensive API calls or human annotations.

Copied to clipboard

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran

Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? (2024.acl-long)

Challenge:	Despite advances in large language models, they face substantial challenges in terms of safety.
Approach:	They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks.
Outcome:	The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs.

Copied to clipboard

Guijin Son, SangWon Baek, Sangdae Nam, Ilgyun Jeong, Seungone Kim

Experiential Co-Learning of Software-Developing Agents (2024.acl-long)

Challenge:	Large language models (LLMs) are typically trained to follow a single instruction per inference call.
Approach:	They introduce a benchmark to evaluate Large language models' ability to follow one instruction per inference call.
Outcome:	The proposed model reduces the total inference time by 1.46 times in average since it does not require multiple inference calls.

Copied to clipboard

Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, YiFei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, Zhiyuan Liu, Maosong Sun

Learning Geometry-Aware Representations for New Intent Discovery (2024.acl-long)

Challenge:	Recent advances in large language models (LLMs) have brought significant changes to various domains, especially through autonomous agents.
Approach:	They propose a framework that lets agents learn shortcuts from their past tasks and use them for future task execution.
Outcome:	The proposed framework enables agents to tackle unseen software-developing tasks more effectively.

Copied to clipboard

Kai Tang, Junbo Zhao, Xiao Ding, Runze Wu, Lei Feng, Gang Chen, Haobo Wang

Speaker Verification in Agent-generated Conversations (2024.acl-long)

Challenge:	Existing methods for intent classification fail to distinguish new intents due to intertwined centers . a novel framework that learns geometry-aware representations to maximally separate all intents is proposed .
Approach:	They propose a new intent discovery framework that learns geometry-aware representations to maximally separate all intents.
Outcome:	The proposed framework achieves a new state-of-the-art performance on three benchmarking datasets.

Copied to clipboard

Yizhe Yang, Palakorn Achananuparp, Heyan Huang, Jing Jiang, Ee-Peng Lim

Benchmarking Data Science Agents (2024.acl-long)

Challenge:	Recent advances in large language models have increased the capabilities of conversational AI to solve challenging dialogue problems.
Approach:	They propose a task to verify whether two sets of utterances originate from the same speaker.
Outcome:	The proposed task aims to verify whether two sets of utterances originate from the same speaker.

Copied to clipboard

Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuqing Yang, Kan Ren

Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) have emerged as promising data science aids, assisting humans in data analysis and processing.
Approach:	They propose an evaluation paradigm and benchmarks that assess the performance of data science agents throughout the entire data science lifecycle.
Outcome:	The proposed evaluation paradigm streamlines dataset preparation, improves coverage, and expands benchmarking comprehensiveness.

Copied to clipboard

Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models (2024.acl-long)

Challenge:	Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent.
Approach:	They propose a novel detection method to pinpoint language-specific neurons within LLMs by selectively activating or deactivating these neurons.
Outcome:	The proposed method can “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons.

Copied to clipboard

Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang

A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques (2024.acl-long)

Challenge:	Existing methods of model editing and knowledge updating add additional network parameters, knowledge bases, knowledge base, and model parameters.
Approach:	They propose a new paradigm for fine-tuning called F-Learning that employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge.
Outcome:	The proposed model outperforms existing models on two datasets and is comparable to full fine-tuning and LoRA fine-uning.

Copied to clipboard

Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, Sarath Chandar

Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation (2024.acl-long)

Challenge:	Large language models are pre-trained on trillions of tokens and instruction-tuned or aligned to specific preferences.
Approach:	They propose guidelines to help researchers perform more effective parameter-efficient LLM alignment.
Outcome:	The proposed methods outperform preference optimization and outperformed pre-trained models on three key axes.

Copied to clipboard

Xiang Luo, Zhiwen Tang, Jin Wang, Xuejie Zhang

PRP-Graph: Pairwise Ranking Prompting to LLMs with Graph Aggregation for Effective Text Re-ranking (2024.acl-long)

Challenge:	Existing approaches to zero-shot dialogue state tracking (DST) involve embedding prompts into language models, but these methods have inherent limitations.
Approach:	They propose a plug-and-play architecture designed for zero-shot dialogue state tracking (DST) dual low-rank adaptation targets dialogue context processing and prompt optimization without incurring additional inference latency.
Outcome:	The proposed architecture outperforms baseline methods on multi-domain datasets and the MultiWOZ dataset.

Copied to clipboard

Jian Luo, Xuanang Chen, Ben He, Le Sun

RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)

Challenge:	Existing methods for pairing ranking prompting only output the same label for comparison results of different confidence intervals without considering the uncertainty of pairwise comparison.
Approach:	They propose a pairwise ranking prompting approach that exploits the output probabilities of target labels to capture the degree of certainty of comparison results.
Outcome:	The proposed method shows strong robustness and acceptable efficiency on the BEIR benchmark.

Copied to clipboard

Zhichao Huang, Chutong Meng, Tom Ko

GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick (2024.acl-long)

Challenge:	Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance.
Approach:	They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec.
Outcome:	The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation.

Copied to clipboard

Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, Yanghua Xiao

Event-Radar: Event-driven Multi-View Learning for Multimodal Fake News Detection (2024.acl-long)

Challenge:	Large language models generate human-like content, but they also pose a problem with generation diversity, negatively impacting generation diversity and user experience.
Approach:	They propose a Logits-Addition watermark and three variants that aim to enhance diversity to overcome generation diversity challenges.
Outcome:	The Logits-Addition watermark outperforms the Logits+Trick-based watermark in diversity tests and outperformed other decoding-based methods by 0.1 to 0.3.

Copied to clipboard

Zihan Ma, Minnan Luo, Hao Guo, Zhi Zeng, Yiran Hao, Xiang Zhao

Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions (2024.acl-long)

Challenge:	Existing methods for detecting multimedia fake news have demonstrated excellent results . however, addressing event-level inconsistency and learning from poor-quality news remains a challenge .
Approach:	They propose an Event-diven fake news detection framework that integrates visual manipulation, textual emotion and multimodal inconsistency at event-level for fake news identification.
Outcome:	The proposed framework performs well on three large-scale fake news detection benchmarks.

Copied to clipboard

Liyan Xu, Jiangnan Li, Mo Yu, Jie Zhou

Stealthy Attack on Large Language Model based Recommendation (2024.acl-long)

Challenge:	a novel graph for narrative comprehension captures coherence between passages in narratives . end-to-end paradigms are effective for comprehension tasks, but may not be sufficient for all comprehension scenarios.
Approach:	They propose a graph dubbed NarCo which explicitly depicts task-agnostic coherence dependencies that are ready to be consumed by downstream tasks.
Outcome:	The proposed graph is practically instantiated by LLMs without human annotations.

Copied to clipboard

Jinghao Zhang, Yuting Liu, Qiang Liu, Shu Wu, Guibing Guo, Liang Wang

Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning (2024.acl-long)

Challenge:	Recent advances in recommender systems have been overlooked due to their emphasis on textual content.
Approach:	They propose to introduce large language models into recommendation models to exploit the semantic understanding and strong transferability of LLMs.
Outcome:	The proposed approach significantly boosts an item’s exposure by altering its textual content during the testing phase, without requiring direct interference with the model’s training process.

Copied to clipboard

Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Lee, Jungseul Ok

Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models (2024.acl-long)

Challenge:	Existing summarization methods target a specific dimension, resulting in poor quality summaries.
Approach:	They propose multi-objective reinforcement learning tailored to generate balanced summaries across all dimensions.
Outcome:	The proposed model achieves significant performance gains compared to baseline models on representative summarization datasets on four dimensions.

Copied to clipboard

Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, Yongbin Li

SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning (2024.acl-long)

Challenge:	Despite the advances in large language models, they still face difficulties with multi-step reasoning tasks.
Approach:	They propose a method that randomly masks certain tokens within the chain of thought to improve model accuracy by 5% over standard supervised fine-tuning.
Outcome:	The proposed method improves accuracy and accuracy by 5% over standard fine-tuning with a few codes modified.

Copied to clipboard

Guoxin Chen, Kexin Tang, Chao Yang, Fuying Ye, Yu Qiao, Yiming Qian

Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning (2024.acl-long)

Challenge:	Existing methods focus on single-step reasoning, ignoring logical dependencies between steps.
Approach:	They propose a method that maximizes a structure-based return to facilitate structured reasoning and explanation.
Outcome:	The proposed method outperforms state-of-the-art methods on EntailmentBank and STREET benchmarks.

Copied to clipboard

Yeachan Kim, Junho Kim, SangKeun Lee

SparseFlow: Accelerating Transformers by Sparsifying Information Flows (2024.acl-long)

Challenge:	Parameter-efficient fine-tuning (PEFT) has enabled efficient optimization of cumbersome language models in real-world environments.
Approach:	They propose a routing-based PEFT approach that adaptively activates PEFT modules.
Outcome:	The proposed method is more sensitive to noise interference than other methods.

Copied to clipboard

Yeachan Kim, SangKeun Lee

ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024.acl-long)

Challenge:	SparseFlow is an efficient method to sparsify the dense information flows within transformers.
Approach:	They propose a method to sparsify the dense pathways of token representations across all transformer blocks by parameterizing them to be sparse.
Outcome:	The proposed method reduces computational costs by half on average without compromising task accuracy.

Copied to clipboard

Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua

KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models (2024.acl-long)

Challenge:	Language Models excel in understanding textual descriptions of proteins, but struggle to process texts.
Approach:	They propose a framework for Protein-to-Text Generation for Text-based Protein Understanding that integrates a PLM as its protein understanding module.
Outcome:	The proposed framework surpasses existing baselines and is highly efficient in protein-to-text generation.

Copied to clipboard

Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, Shikun Zhang

EmoBench: Evaluating the Emotional Intelligence of Large Language Models (2024.acl-long)

Challenge:	Existing methods to detect contaminated texts focus on quantifying contamination status instead of accurately gauging model performance.
Approach:	They propose a Knowledge-grounded Interactive Evaluation framework which incorporates an LLM-powered “interactor” role for the first time to accomplish a dynamic contamination-resilient evaluation.
Outcome:	The proposed framework is based on a question in a standard LLM benchmark and can be used to evaluate models in real-world conversations.

Copied to clipboard

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, Minlie Huang

Are AI-Generated Text Detectors Robust to Adversarial Perturbations? (2024.acl-long)

Challenge:	Existing benchmarks for Emotional Intelligence (EI) focus on emotion recognition, neglecting essential EI capabilities.
Approach:	They propose a benchmark that proposes a comprehensive definition for machine EI . they propose 400 hand-crafted questions in English and Chinese to evaluate EI.
Outcome:	The proposed benchmarks focus on emotion recognition, neglecting EI capabilities . they are constructed from existing datasets, which include frequent patterns and errors . the proposed benchmark includes questions in English and Chinese that require thorough reasoning and understanding .

Copied to clipboard

Guanhua Huang, Yuchen Zhang, Zhe Li, Yongjian You, Mingze Wang, Zhouwang Yang

FinTextQA: A Dataset for Long-form Financial Question Answering (2024.acl-long)

Challenge:	Existing detectors for AI-generated text lack robustness against adversarial perturbations, with even minor changes in characters or words causing a reversal in distinguishing between human-created and AI-generated text.
Approach:	They propose a siamese calibration technique to train the model to make equally confident predictions under different noise, which improves the model’s robustness against adversarial perturbations.
Outcome:	The proposed detector outperforms baseline methods on four datasets and is more generalizable in cross-domain, cross-genre, and mixed-source scenarios.

Copied to clipboard

Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, Junwei Liang

On Measuring Faithfulness or Self-consistency of Natural Language Explanations (2024.acl-long)

Challenge:	Existing financial question answering datasets lack scope diversity and question complexity.
Approach:	They propose to use a dataset for long-form question answering in finance to evaluate QA systems.
Outcome:	The proposed dataset includes 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.

Copied to clipboard

Letitia Parcalabescu, Anette Frank

Learning or Self-aligning? Rethinking Instruction Fine-tuning (2024.acl-long)

Challenge:	Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought explanations, but they can also make unfaithful explanations that hide their sensitivity to biasing inputs.
Approach:	They propose to use a model-based consistency test to judge the faithfulness of post-hoc or Chain-of-Thought explanations rather than model-specific faithfulness tests.
Outcome:	The proposed tests do not measure faithfulness to model explainability but rather their self-consistency at output level.

Copied to clipboard

Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Wan Guanglu, Xunliang Cai, Le Sun

Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? (2024.acl-long)

Challenge:	Instruction fine-tuning (IFT) is a crucial phase in building large language models (LLMs).
Approach:	They propose a knowledge intervention framework to decouple the potential underlying factors of IFT and enable individual analysis of different factors.
Outcome:	The proposed framework decouples the potential underlying factors of IFT, enabling individual analysis of different factors.

Copied to clipboard

Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, Yangqiu Song

Soft Knowledge Prompt: Help External Knowledge Become a Better Teacher to Instruct LLM in Knowledge-based VQA (2024.acl-long)

Challenge:	Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLM.
Approach:	They propose a group discussion framework to enrich the set of discussion mechanisms.
Outcome:	The proposed framework performs better on a wide range of reasoning tasks and backbone LLMs.

Copied to clipboard

Qunbo Wang, Ruyi Ji, Tianhao Peng, Wenjun Wu, Zechao Li, Jing Liu

TasTe: Teaching Large Language Models to Translate through Self-Reflection (2024.acl-long)

Challenge:	Recent research focuses on improving prediction performance and reliability of LLM.
Approach:	They propose a method to actively extract valuable information from the knowledge to produce a latent vector as a soft prompt, which is fused with the image embedding to form a knowledge-enhanced context to instruct LLM.
Outcome:	The proposed method improves performance on knowledge-based VQA benchmarks.

Copied to clipboard

Yutong Wang, Jiali Zeng, Xuebo Liu, Fandong Meng, Jie Zhou, Min Zhang

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)

Challenge:	Existing approaches to enhance LLMs' performance in machine translation are unable to fully exploit their instruction-following capabilities.
Approach:	They propose a framework for translating through self-reflection that involves two stages of inference . they propose to use the framework to refine LLMs' preliminary translations .
Outcome:	The proposed framework can produce translation outputs that match the quality of NMT systems.

Copied to clipboard

Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion (2024.acl-long)

Challenge:	Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes.
Approach:	They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques.
Outcome:	The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks.

Copied to clipboard

Wei Li, Xue Xu, Jiachen Liu, Xinyan Xiao

The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities (2024.acl-long)

Challenge:	Existing text-to-image diffusion models generate images from text prompts due to inherent brevity of textual descriptions . however, the ability to accurately synthesize images with intricate details, such as specific entities or scenes, is limited due to the inherent bribery of text descriptions.
Approach:	They propose a multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs.
Outcome:	The proposed framework excels in both text-to-image generation and zero-shot subject-driven synthesis.

Copied to clipboard

David Stap, Eva Hasler, Bill Byrne, Christof Monz, Ke Tran

Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? (2024.acl-long)

Challenge:	Recent studies have shown that fine-tuning large language models improves their translations, but it is unclear what is the impact on desirable LLM behaviors that are not present in neural machine translation models.
Approach:	They perform an extensive translation evaluation on LLaMA and Falcon models with model size ranging from 7 billion up to 65 billion parameters.
Outcome:	The proposed model produces less literal translations after fine-tuning on parallel data.

Copied to clipboard

Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, Xueqi Cheng

Unveiling Linguistic Regions in Large Language Models (2024.acl-long)

Challenge:	Recent advances in augmenting Large Language Models (LLMs) with auxiliary information have significantly revolutionized their efficacy in knowledge-intensive tasks.
Approach:	They propose a systematic framework to identify whether LLMs’ responses are attributed to either generated or retrieved contexts.
Outcome:	The proposed framework identifies whether LLMs’ responses are attributed to either generated or retrieved contexts.

Copied to clipboard

Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, Xuanjing Huang

Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment (2024.acl-long)

Challenge:	Existing studies on how LLMs achieve cross-lingual alignment and generalization have not explored the intrinsic mechanisms of how they achieve crosslingual alignment.
Approach:	They propose to remove a core region that corresponds to linguistic competence and set parameters to zero to reduce performance across 30 different languages.
Outcome:	The proposed model can be used to perform tasks requiring abstract knowledge and reasoning in complex languages.

Copied to clipboard

Zhiqing Hong, Rongjie Huang, Xize Cheng, Yongqi Wang, Ruiqi Li, Fuming You, Zhou Zhao, Zhimeng Zhang

FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection (2024.acl-long)

Challenge:	Existing studies focus on singing voice synthesis and music generation independently.
Approach:	They propose a novel task called Text-to-Song synthesis which incorporates both vocal and accompaniment generation.
Outcome:	The proposed method can synthesize songs with comparable quality and style consistency.

Copied to clipboard

Yufei Huang, Xu Han, Maosong Sun

Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models’ Understanding of Discourse Relations (2024.acl-long)

Challenge:	Open Domain Question Answering (ODQA) is a longstanding task in Natural Language Processing that involves generating an answer solely based on a given question.
Approach:	They propose a novel approach that executes sentence selection on the encoded passages to enhance the inference speed while reducing the context length required for generating answers.
Outcome:	The proposed approach can increase inference speed by 2.3X-5.7X while maintaining the model’s performance.

Copied to clipboard

Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, Min-Yen Kan

An Open Multilingual System for Scoring Readability of Wikipedia (2024.acl-long)

Challenge:	Discursive Socratic Questioning (DISQ) assesses a model's understanding of discourse relations by requiring systematic accuracy over multiple questions.
Approach:	They propose a method that evaluates faithfulness of understanding discourse based on question answering.
Outcome:	The proposed method evaluates the faithfulness of understanding discourse based on question answering.

Copied to clipboard

Mykola Trokhymovych, Indira Sen, Martin Gerlach

Unlearning Traces the Influential Training Data of Language Models (2024.acl-long)

Challenge:	Existing studies on the readability of Wikipedia have focused on English only and there are currently no systems supporting automatic readability assessment of the 300+ languages in Wikipedia.
Approach:	They propose a multilingual model to assess Wikipedia's readability using a dataset spanning 14 languages.
Outcome:	The proposed model outperforms existing models in a zero-shot scenario and is more accurate than previous benchmarks.

Copied to clipboard

Masaru Isonuma, Ivan Titov

Exploring Alignment in Shared Cross-lingual Spaces (2024.acl-long)

Challenge:	Large language models (LLMs) exhibit remarkable abilities without being explicitly trained for such tasks.
Approach:	They propose a method that unlearns a test dataset and evaluates the unlearned model on training datasets.
Outcome:	The proposed method resembles UnTrac, while being efficient for massive training datasets.

Copied to clipboard

Basel Mousi, Nadir Durrani, Fahim Dalvi, Majd Hawasly, Ahmed Abdelali

Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models (2024.acl-long)

Challenge:	a new study examines the degree of alignment between languages in multilingual embeddings . cross-lingual embeds are designed to encode linguistic concepts that bridge equivalent semantic meaning . a comprehensive approach is needed to address these questions.
Approach:	They employ clustering to uncover latent concepts within multilingual models . they introduce two metrics to quantify alignment and overlap of these concepts .
Outcome:	The proposed model can capture linguistic nuances across languages, but is not language-agnostic? the proposed model is able to capture nuances in multiple languages, the authors say.

Copied to clipboard

Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, Michael Lyu

Self-Evolving GPT: A Lifelong Autonomous Experiential Learner (2024.acl-long)

Challenge:	e.g., ChatGPT often provides inappropriate English-culture-related answers when users ask in non-English languages.
Approach:	They build a benchmark of concrete and abstract cultural objects to evaluate the cultural dominance issue in large language models.
Outcome:	The proposed model can significantly mitigate cultural dominance issue in large language models . the model can provide accurate answers in English, while the model is ethically sound .

Copied to clipboard

Jinglong Gao, Xiao Ding, Yiming Cui, Jianbai Zhao, Hepeng Wang, Ting Liu, Bing Qin

WRP: Weight Recover Prune for Structured Sparsity (2024.acl-long)

Challenge:	Existing approaches to provide LLMs with textual task-solving experience rely on manual efforts to acquire and apply such experience for each task.
Approach:	They propose a lifelong autonomous experiential learning framework based on LLMs that learns and accumulates experience through experience transfer and induction.
Outcome:	The proposed framework performs reliably in each intermediate step and improves GPT-3.5 and GPT-4 on widely used NLP datasets.

Copied to clipboard

Zhendong Tan, Xingjun Zhang, Zheng Wei

Error-preserving Automatic Speech Recognition of Young English Learners’ Language (2024.acl-long)

Challenge:	Existing pruning and quantization algorithms could compress LLMs to 4 bits without retraining.
Approach:	They propose a method to prune models using a minimal set of critical weights . they compare the method to a 2:4 structured sparsity method .
Outcome:	The proposed method outperforms other methods on LLAMA2 and OPT models while maintaining the efficiency of the compression.

Copied to clipboard

Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk, Mark Cieliebak

DiFiNet: Boundary-Aware Semantic Differentiation and Filtration Network for Nested Named Entity Recognition (2024.acl-long)

Challenge:	State-of-the-art speech recognition models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners’ speech.
Approach:	They propose to use an automated speech recognition module to train language learners' speaking skills on spontaneous speech by young language learners.
Outcome:	The proposed model improves on 85 hours of English audio spoken by Swiss learners and preserves their mistakes.

Copied to clipboard

Yuxiang Cai, Qiao Liu, Yanglei Gan, Run Lin, Changlin Li, Xueyi Liu, Da Luo, JiayeYang JiayeYang

Legal Case Retrieval: A Survey of the State of the Art (2024.acl-long)

Challenge:	Existing approaches to Named Entity Recognition focus on identifying non-nested entities, but there is no explicit guidance for boundary detection.
Approach:	They propose a Boundary-aware Semantic Differentiation and Filtration Network for nested NER that leverages a biaffine attention mechanism to generate a span representation matrix.
Outcome:	Extensive experiments on three benchmark datasets demonstrate the proposed model yields a new state-of-the-art performance.

Copied to clipboard

Yi Feng, Chuanyi Li, Vincent Ng

Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation (2024.acl-long)

Challenge:	Recent years have seen increasing attention on Legal Case Retrieval (LCR) this task involves retrieving cases from a legal database of historical cases that are similar to a given query case.
Approach:	They present a survey of the major milestones made in legal case retrieval research . they seek to understand the datasets and recent neural models and their performances .
Outcome:	The proposed task is based on a dataset of historical cases similar to a given query case.

Copied to clipboard

Tianqi Zhong, Zhaoyi Li, Quan Wang, Linqi Song, Ying Wei, Defu Lian, Zhendong Mao

LLaMA Pro: Progressive LLaMA with Block Expansion (2024.acl-long)

Challenge:	Existing MCTG methods face a noticeable performance drop in compositional testing.
Approach:	They propose a benchmark to evaluate compositional generalization of MCTG methods by combining multi-aspect labeled datasets and a crafted three-dimensional evaluation protocol.
Outcome:	The proposed framework improves compositional generalization performance by 3.64% and 94.4% in compositional testing.

Copied to clipboard

Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo

Generating Contrastive Narratives Using the Brownian Bridge Process for Narrative Coherence Learning (2024.acl-long)

Challenge:	Existing studies have demonstrated that pre-trained LLMs are limited in certain domains, such as programming, mathematics, biomedical, or finance.
Approach:	They propose a new post-pretraining method with an expansion of Transformer blocks to tune the expanded blocks using only new corpus, efficiently and effectively improving the model’s knowledge while mitigating forgetting.
Outcome:	The proposed model outperforms existing models in programming and math and its instruction-following counterpart LLaMA Pro-8.3B in general tasks, programming, and mathematics.

Copied to clipboard

Feiteng Mu, Wenjie Li

Challenge:	Existing methods for learning narrative coherence are coarse-grained and superficial . existing methods are inadequate for learning negative samples, which are irrelevant or repetitive .
Approach:	They propose two strategies for mining hard negatives using the Brownian Bridge process . they evaluate the method on several tasks and show it is applicable to many applications .
Outcome:	The proposed method proves that it is applicable to many applications.

A Causal Approach for Counterfactual Reasoning in Narratives (2024.acl-long)

Copied to clipboard

Feiteng Mu, Wenjie Li

SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation (2024.acl-long)

Challenge:	Existing methods for counterfactual reasoning in narratives are based on dataset-specific heuristics, but they are abusing unique patterns, i.e., the feature of minimum editing, in the dataset, which limits the generality of their methods.
Approach:	They propose a basic VAE module for counterfactual reasoning in narratives and introduce a pre-trained classifier and external event commonsense to mitigate the posterior collapse problem.
Outcome:	The proposed method improves the causality between the counterfactual condition and the generated counterf actual outcome on two public benchmarks.

Copied to clipboard

Matthias Lindemann, Alexander Koller, Ivan Titov

The Hidden Space of Transformer Language Adapters (2024.acl-long)

Challenge:	Popular neural architectures lack strong structural inductive biases for seq2seq NLP tasks . previous work shows that these models struggle with systematic generalization .
Approach:	They propose to inject a structural inductive bias into a seq2seq model by pre-training it to simulate structural transformations on synthetic data.
Outcome:	The proposed method improves few-shot learning and generalization of FST-like models.

Copied to clipboard

Jesujoba Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, Mor Geva

A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Texts (2024.acl-long)

Challenge:	Adapters are small modules trained on top of a frozen language model to adapt predictions to new target languages.
Approach:	They propose to train transformer language adapters on top of a frozen model to adapt predictions to new target languages.
Outcome:	The transformer language adapters are trained on top of a frozen model to adapt predictions to new target languages.

Copied to clipboard

Nafis Irtiza Tripto, Saranya Venkatraman, Dominik Macko, Robert Moro, Ivan Srba, Adaku Uchendu, Thai Le, Dongwon Lee

Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations (2024.acl-long)

Challenge:	Using a computational approach, we discover that diminishing performance in text classification models is closely associated with the extent of deviation from the original author’s style.
Approach:	They propose to use large language models to determine whether a text retains original authorship when it undergoes numerous paraphrasing iterations.
Outcome:	The results suggest that authorship should be task-dependent .

Copied to clipboard

Guan-Ting Lin, Cheng-Han Chiang, Hung-yi Lee

RetinaQA: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questions (2024.acl-long)

Challenge:	In spoken dialogue, even if two current turns are the same sentence, their responses might differ when they are spoken in different styles.
Approach:	They propose a language-to-speech dataset that can model linguistic content and speaking styles.
Outcome:	The proposed framework outperforms text-only baselines and prior speech LLMs methods.

Copied to clipboard

Prayushi Faldu, Indrajit Bhattacharya, Mausam .

GroundingGPT: Language Enhanced Multi-modal Grounding Model (2024.acl-long)

Challenge:	Existing knowledge base question answering models assume all questions to be answerable.
Approach:	They propose a new KBQA model that unifies two key ideas in a single architecture . they propose logical form discrimination and sketch-filling-based construction for unanswerable questions .
Outcome:	The proposed model outperforms existing models in handling answerable and unanswerable questions.

Copied to clipboard

Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, YiQing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Vu Tu, Zhida Huang, Tao Wang

Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches (2024.acl-long)

Challenge:	Existing multi-modal large language models focus on capturing global information while neglecting the fine-grained local information in multimodal inputs.
Approach:	They propose an end-to-end language enhanced multi-modal grounding model that performs fine-grained grounding tasks for image, video and audio.
Outcome:	The proposed model achieves impressive fine-grained understanding of multi-modal inputs while maintaining or improving its global comprehension capabilities.

Copied to clipboard

Islam Eldifrawi, Shengrui Wang, Amine Trabelsi

Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages (2024.acl-long)

Challenge:	Current research focuses on predicting claim veracity through metadata analysis and language scrutiny, with an emphasis on justifying verdicts.
Approach:	They propose a comprehensive taxonomy for categorizing works based on various criteria and propose scalable methodologies for improving fact-checking explainability.
Outcome:	The proposed taxonomy identifies challenges while proposing future directions in fact-checking explainability.

Copied to clipboard

Carlos Mullov, Quan Pham, Alexander Waibel

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget (2024.acl-long)

Challenge:	Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space.
Approach:	They propose a setup where we decouple learning of vocabulary and syntax and train to translate while keeping those word representations frozen.
Outcome:	The proposed setup achieves near parity with a supervised setting on the TED domain with varying number of languages seen by the encoder.

Copied to clipboard

Rui Kong, Yuanchun Li, Qingtian Feng, Weijun Wang, Xiaozhou Ye, Ye Ouyang, Linghe Kong, Yunxin Liu

PixT3: Pixel-based Table-To-Text Generation (2024.acl-long)

Challenge:	Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) but memory-constrained devices are a major concern in edge AI training and serving.
Approach:	They propose a framework for efficient serving of MoE-based large language models with tunable memory budgets.
Outcome:	Experiments show that SwapMoE can reduce memory consumption while maintaining reasonable accuracy.

Copied to clipboard

Iñigo Alonso, Eneko Agirre, Mirella Lapata

Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers (2024.acl-long)

Challenge:	Table-to-text generation is a visual recognition task that uses textual descriptions from structured inputs.
Approach:	They propose to rethink data-to-text generation as a visual recognition task by removing the need for rendering the input in a string format.
Outcome:	The proposed model overcomes the challenges of linearization and input size limitations and is applicable to open-ended and controlled generation settings.

Copied to clipboard

Gal Yona, Roee Aharoni, Mor Geva

TAMS: Translation-Assisted Morphological Segmentation (2024.acl-long)

Challenge:	Standard question answering evaluation protocols do not take factual errors into account and compare a predicted answer against answers of a single granularity level.
Approach:	They propose a new evaluation setting where a predicted answer is evaluated in terms of accuracy and informativeness against a set of multi-granularity answers.
Outcome:	The proposed evaluation setting compares a predicted answer against a set of multi-granularity answers.

Copied to clipboard

Enora Rice, Ali Marashian, Luke Gessler, Alexis Palmer, Katharina Wense

XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)

Challenge:	Canonical morphological segmentation is a key task in endangered language documentation . training data for canonical segmentation can be difficult, making it difficult to train high quality models.
Approach:	They propose a model that leverages translation data to speed up canonical segmentation . they propose to use translation data as an additional signal to leverage the data .
Outcome:	The proposed model outperforms baseline models in a super-low resource setting but yields mixed results on training splits with more data.

Copied to clipboard

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, Shafiq Joty

ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024.acl-long)

Challenge:	Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments.
Approach:	They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages.
Outcome:	The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens)

Copied to clipboard

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song

A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia (2024.acl-long)

Challenge:	Existing evaluation methods for large language models are labor-intensive and lack efficiency.
Approach:	They propose a framework dedicated to assessing long-text generation that includes in-depth human-curated meta-questions spanning various domains . they use a set of proxy-quests with pre-annotated answers to assess the content's quality by incorporating the generated texts as contextual background.
Outcome:	The proposed framework assesses the quality of long-text content by matching it with references through human evaluation or automated metrics.

Copied to clipboard

Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, Robert West

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA (2024.acl-long)

Challenge:	Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context, yet the mechanisms underlying contextual grounding remain unknown.
Approach:	They propose a method to study grounding abilities using a counterfactual dataset constructed to clash with a model's parametric knowledge using Fakepedia.
Outcome:	The proposed method evaluates grounding abilities when the internal parametric knowledge clashes with the contextual information.

Copied to clipboard

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Yang Zhao, Xinze Guan, Xin Wang

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)

Challenge:	Multipanel images are a common form of visual representations, and humans can achieve approximately 99% accuracy on these questions.
Approach:	They propose a benchmark that tests multipanel visual reasoning models with 6,600 triplets of questions, answers, and multipanel images.
Outcome:	The proposed benchmark features 6,600 triplets of questions, answers, and multipanel images that challenge state-of-the-art Multimodal Large Language Models (MLLMs) human users can attain approximately 99% accuracy on these questions, compared with previous benchmarks.

Copied to clipboard

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu

Translation-based Lexicalization Generation and Lexical Gap Detection: Application to Kinship Terms (2024.acl-long)

Challenge:	Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios.
Approach:	They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites.
Outcome:	The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups.

Copied to clipboard

Senyu Li, Bradley Hauer, Ning Shi, Grzegorz Kondrak

Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations (2024.acl-long)

Challenge:	Existing methods for identifying lexical gaps have been limited . kinship terms are well-suited for investigations into lexicons and lexicals .
Approach:	They propose an algorithm to automatically generate concept lexicalizations based on machine translation and hypernymy relations between concepts.
Outcome:	Empirical evaluations show that the proposed method is more accurate than BabelNet and ChatGPT.

Copied to clipboard

Ritam Dutt, Zhen Wu, Jiaxin Shi, Divyanshu Sheth, Prakhar Gupta, Carolyn Rose

Robust Frame-Semantic Models with Lexical Unit Trees and Negative Samples (2024.acl-long)

Challenge:	Existing models for language from a social perspective are gaining popularity . we present a generalizable classification approach that leverages Large Language Models .
Approach:	They propose a generalizable classification approach that leverages Large Language Models to detect social meaning in conversations.
Outcome:	The proposed approach improves on two social meaning detection tasks over 2,340 settings.

Copied to clipboard

Jacob Devasier, Yogesh Gurjar, Chengkai Li

Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation (2024.acl-long)

Challenge:	Using a RoBERTa-based filter, we achieve an F1 score of 0.775, surpassing the previous state-of-the-art solution by +0.012.
Approach:	They propose a new prefix tree modification to enable robust support for multi-word lexical units and a RoBERTa-based filter to achieve an F1 score of 0.775.
Outcome:	The proposed model achieves an F1 score of 0.775, surpassing the state-of-the-art model by +0.012.

Copied to clipboard

Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, Faramarz Fekri

Lightweight reranking for language model generations (2024.acl-long)

Challenge:	Logic-based approaches to reasoning have lost popularity due to limited scalability and coverage.
Approach:	They present a dataset of 28K sentence-level NL-FOL pairs from GPT4 and a LogicLLaMA2-7B/13B fine-tuned on MALLS for NL translation.
Outcome:	The proposed model can be used standalone or to correct previously generated rules by GPT3.5.

Copied to clipboard

Siddhartha Jain, Xiaofei Ma, Anoop Deoras, Bing Xiang

ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews (2024.acl-long)

Challenge:	Large Language Models (LLMs) can exhibit considerable variation in quality of sampled outputs.
Approach:	They propose a method for reranking LLM generations using pairwise statistics . they show strong improvements for selecting the best k generations for code generation tasks .
Outcome:	The proposed approach improves selection and generation quality for code generation tasks and autoformalization, summarization, and translation tasks.

Copied to clipboard

Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, Doug Downey

Challenge:	Existing systems that can interpret complex writing feedback and edit documents in response are limited on the most demanding writing tasks.
Approach:	They propose to use peer feedback to revise scientific papers based on peer feedback . they provide labels linking each reviewer comment to the specific paper edits made by the author .
Outcome:	The proposed model fails to identify which edits correspond to a comment and the original paper.

The Unreasonable Effectiveness of Easy Training Data for Hard Tasks (2024.acl-long)

Copied to clipboard

Peter Hase, Mohit Bansal, Peter Clark, Sarah Wiegreffe

PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning (2024.acl-long)

Challenge:	Existing pretrained language models perform well on hard data, but hard data is noisier and costlier to collect.
Approach:	They propose to use in-context learning, linear classifier heads, and QLoRA to show that pretrained language models generalize relatively well from easy to hard data.
Outcome:	The proposed model generalizes well from easy to hard data even better than oracle models finetuned on hard data.

Copied to clipboard

Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, Francesco Barbieri

Cross-Lingual Transfer

Challenge:	Instruction tuning has advanced large language models (LLMs) but its application in lower-resource languages faces challenges due to the imbalanced foundational abilities of LLMs across different languages.
Approach:	They propose a pivot language guided generation approach that utilizes a high-resource language as the pivot to enhance instruction tuning in lower-resourced languages.
Outcome:	The proposed approach improves instruction-following abilities of LLMs by 29% on average compared to directly responding in the target language alone.

MIDGARD: Self-Consistency Using Minimum Description Length for Structured Commonsense Reasoning (2024.acl-long)

Copied to clipboard

Inderjeet Nair, Lu Wang

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2024.acl-long)

Challenge:	Existing approaches to generate a reasoning graph from natural language input suffer from error propagation due to autoregressive nature and single-pass-based decoding.
Approach:	They propose a method that uses minimum description length to identify consistent properties among different graph samples generated by large language models.
Outcome:	The proposed method outperforms previous approaches for generating reasoning graphs from natural language input using large language models.

Copied to clipboard

Justin Chen, Swarnadeep Saha, Mohit Bansal

Mirror: Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning (2024.acl-long)

Challenge:	Large Language Models (LLMs) still struggle with natural language reasoning tasks.
Approach:	They propose a multi-model multi-agent framework that enhances collaborative reasoning between LLM agents via multiple rounds of discussion.
Outcome:	Experiments on seven benchmarks show that ReConcile significantly improves LLM reasoning on individual and team levels.

Copied to clipboard

Hanqi Yan, Qinglin Zhu, Xinyu Wang, Lin Gui, Yulan He

Where Do People Tell Stories Online? Story Detection Across Online Communities (2024.acl-long)

Challenge:	Large language models (LLMs) struggle with knowledge-rich problems without external resources.
Approach:	They propose a Multiple-perspective self-reflection method that allows LLMs to reflect from multiple-perceptive clues, achieved through a heuristic interaction between a Navigator and a Reasoner.
Outcome:	The proposed method is superior to other self-reflection methods on five reasoning datasets.

Copied to clipboard

Maria Antoniak, Joel Mire, Maarten Sap, Elliott Ash, Andrew Piper

Challenge:	Story detection in online communities is a challenging task as stories are scattered across communities and interwoven with non-storytelling spans within a single text.
Approach:	They propose a toolkit to detect stories in online communities using an annotated reddit dataset and a codebook adapted to social media context.
Outcome:	The proposed toolkit includes an annotation-rich dataset of 502 Reddit posts and comments . it also includes a codebook adapted to the social media context and models to predict storytelling at document and span levels.

Large Language Models Are No Longer Shallow Parsers (2024.acl-long)

Copied to clipboard

Yuanhe Tian, Fei Xia, Yan Song

Dialogue Summarization with Mixture of Experts based on Large Language Models (2024.acl-long)

Challenge:	Recent advances in large language models (LLMs) have reshaped the field of natural language processing (NLP) however, fundamental NLP tasks that involve linguistic analysis still play essential roles in the field.
Approach:	They propose to use constituency parsing to improve performance of LLMs on deep syntactic parse trees to prompt LLM chunking, filter out low-quality chunks and add remaining chunks to prompts to instruct LLM for parser.
Outcome:	The proposed approach improves LLMs' performance on constituency parsing on English and Chinese benchmark datasets.

Copied to clipboard

Yuanhe Tian, Fei Xia, Yan Song

ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences (2024.acl-long)

Challenge:	Existing studies for dialogue summarization use one model at a time or treat it as a black box.
Approach:	They propose an LLM-based approach with role-oriented routing and fusion generation to utilize mixture of experts for dialogue summarization.
Outcome:	The proposed approach produces informative and accurate dialogue summarization on widely used datasets.

Copied to clipboard

Yuanhe Tian, Ruyi Gan, Yan Song, Jiaxing Zhang, Yongdong Zhang

An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs (2024.acl-long)

Challenge:	Current large language models (LLMs) are ineffective in learning domain knowledge and aligning with human preference.
Approach:	They propose a benchmark LLM for Chinese medical domain that uses pre-training, supervised fine-tuning and RLHF to train LLMs.
Outcome:	The proposed LLM performs better than existing LLMs in the Chinese medical domain.

Copied to clipboard

Daking Rai, Ziyu Yao

Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling (2024.acl-long)

Challenge:	Prior work has focused on ablating components in the CoT prompt, but the reason why these components are important to LLM reasoning is not explored.
Approach:	They investigate "neuron activation" as a lens to provide a unified explanation to previous work . they propose an approach to automatically identify neurons that imply arithmetic reasoning .
Outcome:	The proposed approach can explain the importance of components in a CoT prompt . it also automatically identifies neurons that imply arithmetic reasoning .

Copied to clipboard

Hang Jiang, Xiajie Zhang, Robert Mahari, Daniel Kessler, Eric Ma, Tal August, Irene Li, Alex Pentland, Yoon Kim, Deb Roy, Jad Kabbara

Intrinsic Task-based Evaluation for Referring Expression Generation (2024.acl-long)

Challenge:	a novel application of large language models (LLMs) to legal education helps non-experts learn complex legal concepts . authors find storytelling helps nonexperts understand complex legal terms and concepts compared to definitions .
Approach:	They propose a novel application of large language models to legal education . they use LLMs to generate legal stories explaining complex legal concepts .
Outcome:	The proposed method improves comprehension and interest among non-native speakers compared to definitions . the novel method also shows that non-experts retain more stories .

Copied to clipboard

Guanyi Chen, Fahime Same, Kees Van Deemter

From Moments to Milestones: Incremental Timeline Summarization Leveraging Large Language Models (2024.acl-long)

Challenge:	Referring Expression Generation (REG) models generate referring expressions that refer to referents at different points in a discourse.
Approach:	They propose to use a purely ratings-based human evaluation to evaluate REG models by completing two meta-level tasks.
Outcome:	The proposed evaluation makes the models more reliable and discriminable, and improves the quality of the REs.

Copied to clipboard

Qisheng Hu, Geonsik Moon, Hwee Tou Ng

End-to-end Learning of Logical Rules for Enhancing Document-level Relation Extraction (2024.acl-long)

Challenge:	Prior work on timeline summarization has neglected the potential synergy between the two forms of timelines.
Approach:	They propose a timeline summarization approach that leverages large language models to generate both event and topic timelines.
Outcome:	The proposed approach outperforms the best existing approaches in four TLS benchmarks.

Copied to clipboard

Kunxun Qi, Jianfeng Du, Hai Wan

Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)

Challenge:	Document-level relation extraction (DocRE) aims to extract relations between entities in a document.
Approach:	They propose a rule-based framework that jointly learns DocRE and logical rules . they parameterize a Rule reasoning module to simulate the inference of logical rule .
Outcome:	The proposed framework improves DocRE models by a significant margin on four benchmark datasets.

Copied to clipboard

Qingkai Fang, Shaolei Zhang, Zhengrui Ma, Min Zhang, Yang Feng

Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder (2024.acl-long)

Challenge:	Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect.
Approach:	They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model.
Outcome:	The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data.

Copied to clipboard

Jiaqi Wang, Zhenxi Song, Zhengyu Ma, Xipeng Qiu, Min Zhang, Zhiguo Zhang

CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers (2024.acl-long)

Challenge:	EEG-based language decoding is still in its nascent stages, despite promising applications in brain-computer interfaces.
Approach:	They propose a novel EEG-text Masked Autoencoder that orchestrates compound self-supervised learning across and within EEG and text through a dedicated multi-stream encoder.
Outcome:	The proposed model outperforms baseline framework in ROUGE-1 F1 and BLEU-4 scores and an LLM (specifically BART) to improve downstream tasks involving EEG and text.

Copied to clipboard

Longwei Zou, Qingyang Wang, Han Zhao, Jiangangkong Jiangangkong, Yi Yang, Yangdong Deng

Prompt Optimization via Adversarial In-Context Learning (2024.acl-long)

Challenge:	Existing methods to improve inference efficiency target to reduce per-layer latency, but ignore cumulative latency due to number of layers.
Approach:	They propose to identify quasi-independent layers that can be concurrently computed to significantly decrease inference latency.
Outcome:	Empirical results show that the proposed method reduces latency by 48.3% on LLaMA-33B while maintaining close level of performance.

Copied to clipboard

Do Long, Yiran Zhao, Hannah Brown, Yuxi Xie, James Zhao, Nancy Chen, Kenji Kawaguchi, Michael Shieh, Junxian He

StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024.acl-long)

Challenge:	Existing methods to optimize prompts for in-context learning are based on adversarial learning and are computationally efficient and extensible to other LLMs and tasks.
Approach:	They propose a method to optimize prompts for in-context learning by a generator and a discriminator.
Outcome:	The proposed method improves state-of-the-art prompt optimization techniques on 13 generation and classification tasks including summarization, arithmetic reasoning, machine translation, data-to-text generation, and the MMLU and big-bench hard benchmarks.

Copied to clipboard

Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Lei Xie, Yuping Wang

Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering (2024.acl-long)

Challenge:	Existing LM-based VC models require offline conversion from source semantics to acoustic features, limiting their deployment to real-time applications.
Approach:	They propose a streaming LM-based model for zero-shot voice conversion that uses a fully causal context-aware LM with a temporal-independent acoustic predictor to facilitate real-time conversion given arbitrary speaker prompts and source speech.
Outcome:	The proposed model achieves comparable performance to non-streaming VC systems while maintaining a fully causal context-aware LM with a temporal-independent acoustic predictor.

Copied to clipboard

Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, Zhaochun Ren

Multimodal Contextualized Semantic Parsing from Speech (2024.acl-long)

Challenge:	Existing approaches to solve multi-hop question are constrained by the retriever and the noise in the retrieved documents.
Approach:	They propose a framework that integrates parametric knowledge of large language models with external documents to solve a multi-hop question.
Outcome:	The proposed framework is based on the parametric knowledge of LLMs and external documents to solve a multi-hop question.

Copied to clipboard

Jordan Voas, David Harwath, Ray Mooney

LaMP: When Large Language Models Meet Personalization (2024.acl-long)

Challenge:	Towards this goal, we introduce Semantic Parsing in Contextual Environments (SPICE) task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts.
Approach:	They introduce a task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts.
Outcome:	The proposed task is based on the VG-SPICE dataset and the Audio-Vision Dialogue Scene Parser (AViD-SP) it allows agents to maintain their contextual state within a structured, dense information framework that is scalable and interpretable .

Copied to clipboard

Alireza Salemi, Sheshera Mysore, Michael Bendersky, Hamed Zamani

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters (2024.acl-long)

Challenge:	Existing benchmarks for personalization in large language models are understudied .
Approach:	They propose a benchmark for training and evaluating language models for producing personalized outputs using a set of seven personalized tasks . they propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs.
Outcome:	The proposed approach is effective for a set of zero-shot and fine-tuned language models and highlights the impact of personalization in various natural language tasks.

Copied to clipboard

Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, Jesse Dodge

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024.acl-long)

Challenge:	Large language models' (LLMs) abilities are drawn from their pretraining data. however, decisions around what data is retained or removed during this initial stage are under-scrutinized.
Approach:	They ground web text, a popular pretraining data source, to its social and geographic contexts.
Outcome:	The results show that some quality classifiers act like topical domain filters, and langID overlook English content from some regions of the world.

Copied to clipboard

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang

EFSA: Towards Event-Level Financial Sentiment Analysis (2024.acl-long)

Challenge:	Large Language Models (LLMs) have greatly enhanced dialogue systems, but evaluation of their capabilities remains a challenge.
Approach:	They propose a model to evaluate the fine-grained abilities of Large Language Models in multi-turn dialogues.
Outcome:	The proposed model evaluates 21 popular chatbots based on MT-Bench-101 . it includes 3 overarching abilities and 13 distinct tasks within multi-turn dialogue scenarios.

Copied to clipboard

Tianyu Chen, Yiming Zhang, Guoxin Yu, Dapeng Zhang, Li Zeng, Qing He, Xiang Ao

What Evidence Do Language Models Find Convincing? (2024.acl-long)

Challenge:	a large-scale Chinese dataset contains 12,160 news articles and 13,725 quintuples . a four-hop Chain-of-Thought LLM-based approach is devised for this task .
Approach:	They propose to extend financial sentiment analysis to event-level since events usually serve as the subject of the sentiment in financial text.
Outcome:	The proposed method can reach the current state-of-the-art on a large-scale Chinese dataset.

Copied to clipboard

Alexander Wan, Eric Wallace, Dan Klein

Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024.acl-long)

Challenge:	Current retrieval-augmented language models are tasked with subjective, contentious, and conflicting queries.
Approach:	They construct a dataset that pairs controversial queries with real-world evidence documents . they find current models rely heavily on relevance of a website to the query .
Outcome:	The proposed dataset pairs controversial queries with real-world evidence documents that contain different facts, arguments, and answers.

Copied to clipboard

Qihang Ai, Jiafan Li, Jincheng Dai, Jianwu Zhou, Lemao Liu, Haiyun Jiang, Shuming Shi

LangBridge: Multilingual Reasoning Without Multilingual Supervision (2024.acl-long)

Challenge:	Graph data organizes complex relationships and interactions between objects . Graph neural networks (GNNs) are becoming more popular in graph learning .
Approach:	They propose a new paradigm for interactive and instructional graph data understanding and reasoning . they first evaluate the capabilities of public VLMs in graph learning from multiple aspects .
Outcome:	The proposed model achieves an accuracy increase of 5%-15% compared to baseline models . the best-performing model achieve scores comparable to Gemini in GPT-asissted Evaluation .

Copied to clipboard

Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, Minjoon Seo

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs (2024.acl-long)

Challenge:	Existing approaches to adapt language models for multilingual reasoning tasks require multilingual supervision.
Approach:	They propose a zero-shot approach to adapt language models for multilingual reasoning tasks without multilingual supervision by bridging two models by introducing minimal trainable parameters between them.
Outcome:	The proposed approach significantly improves multilingual reasoning capabilities on low-resource languages.

Copied to clipboard

Siyuan Wang, Zhongyu Wei, Yejin Choi, Xiang Ren

SEGO: Sequential Subgoal Optimization for Mathematical Problem-Solving (2024.acl-long)

Challenge:	Large language models (LLMs) have impressive human-like performance across various reasoning tasks, but their mastery of underlying inferential rules falls short of human capabilities.
Approach:	They propose a logic scaffolding inferential rule generation framework to construct an infer- ential rule base, ULogic, comprising both primitive and compositional rules across five domains.
Outcome:	The proposed model improves the ability to generate accurate, complex and abstract conclusions and premises and improves various commonsense reasoning tasks.

Copied to clipboard

Xueliang Zhao, Xinting Huang, Wei Bi, Lingpeng Kong

Numerical Reasoning

Challenge:	Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of tasks, including mathematical problem-solving.
Approach:	They propose a framework that connects the subgoal breakdown process and the probability of solving problems by identifying better subgoals with theoretical guarantees.
Outcome:	The proposed framework outperforms existing methods on two benchmarks, GSM8K and MATH, highlighting the potential of SEGO in AI-driven mathematical problem-solving.

Unlocking the Power of Large Language Models for Entity Alignment (2024.acl-long)

Copied to clipboard

Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen, Yuanzhuo Wang

Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (2024.acl-long)

Challenge:	Entity Alignment (EA) is a crucial step in unifying data from heterogeneous sources and plays a critical role in data-driven AI applications.
Approach:	They propose a framework that incorporates large language models to improve EA.
Outcome:	The proposed framework incorporates large language models (LLMs) to improve EA accuracy while preserving efficiency.

Copied to clipboard

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, Bill Yuchen Lin

Challenge:	Large Language Models (LLMs) have become integral components in various autonomous agent systems.
Approach:	They propose an exploration-based trajectory optimization approach that allows agents to learn from their exploration failures.
Outcome:	The proposed method outperforms baseline methods on three complex tasks by a large margin.

ReFT: Reasoning with Reinforced Fine-Tuning (2024.acl-long)

Copied to clipboard

Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, Hang Li

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment (2024.acl-long)

Challenge:	Existing approaches to improve the generalization of large language models are using Supervised Fine-Tuning (SFT) this approach does not show sufficient generalization ability because it only relies on the given CoT data.
Approach:	They propose to use Chain-of-Thought annotations to train Large Language Models using supervised fine-tuning to improve generalization.
Outcome:	The proposed approach outperforms SFT on GSM8K, MathQA, and SVAMP datasets and shows a superior generalization ability.

Copied to clipboard

Yunxin Li, Xinyu Chen, Baotian Hu, Haoyuan Shi, Min Zhang

FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation (2024.acl-long)

Challenge:	Recent Large Multimodal Models (LMMs) focus on visual knowledge-dimension alignment, but ignore visual knowledge.
Approach:	They propose a cognitive visual-language mapper that integrates visual-linguistic knowledge alignment with a fine-grained knowledge Adapter.
Outcome:	The proposed model significantly improves LMMs on knowledge-based visual question answering (VQA) it also improves the performance of other models, including GPT-4V and Gemini-Pro.

Copied to clipboard

Zijian Feng, Hanzhang Zhou, Kezhi Mao, Zixiao Zhu

HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition (2024.acl-long)

Challenge:	Controllable text generation (CTG) focuses on crafting texts adhering to specific attributes . studies show learning-based methods require extensive computational and data resources .
Approach:	They propose a learning-free approach that dynamically adjusts the weights of selected feedforward neural network vectors to steer the outputs of large language models.
Outcome:	The proposed approach outperforms learning-based and learning-free methods on multi-attribute control.

Copied to clipboard

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Conundrums in Cross-Prompt Automated Essay Scoring: Making Sense of the State of the Art (2024.acl-long)

Challenge:	Large language models (LLMs) are a promising alternative to expensive human evaluations.
Approach:	They propose a framework that iteratively aligns LLM-based evaluators with human preference . they decompose a given evaluation task into finer-grained criteria .
Outcome:	The proposed framework iteratively aligns LLM-based evaluators with human preference . it decomposes a given evaluation task into finer-grained criteria . the framework is efficient to train and more explainable than relying solely on prompts .

Copied to clipboard

Shengjie Li, Vincent Ng

Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution (2024.acl-long)

Challenge:	Automated essay scoring (AES) is a task of assigning a single score to an essay . authors abandon sophisticated neural architectures and develop a simple feature-based approach .
Approach:	a team of researchers develop a feature-based approach to cross-prompt automated essay scoring that adopts a simple neural architecture.
Outcome:	a new approach to cross-prompt automated essay scoring can achieve state-of-the-art results.

Copied to clipboard

Flor Plaza-del-Arco, Amanda Curry, Alba Cercas Curry, Gavin Abercrombie, Dirk Hovy

Label Augmentation for Zero-Shot Hierarchical Text Classification (2024.acl-long)

Challenge:	Large language models reflect societal norms and biases, especially about gender.
Approach:	They propose to use large language models to examine gendered emotion attribution in five state-of-the-art LLMs to investigate whether emotions are genderes and whether they are influenced by societal stereotypes.
Outcome:	The proposed models exhibit gendered emotions, influenced by gender stereotypes, and the results are consistent with established research in psychology and gender studies.

Copied to clipboard

Lorenzo Paletto, Valerio Basile, Roberto Esposito

STICKERCONV: Generating Multimodal Empathetic Responses from Scratch (2024.acl-long)

Challenge:	Hierarchical Text Classification is a difficult problem due to the lack of labeled data and the cost of manually annotating data samples.
Approach:	They propose a method that uses a Large Language Model to augment the deepest layer of the labels hierarchy to enhance its specificity.
Outcome:	The proposed method achieves state-of-the-art on four public datasets and a strong correlation between the metric values and the classification performance.

Copied to clipboard

Yiqun Zhang, Fanheng Kong, Peidong Wang, Shuang Sun, SWangLing SWangLing, Shi Feng, Daling Wang, Yifei Zhang, Kaisong Song

EIT: Enhanced Interactive Transformer (2024.acl-long)

Challenge:	Prior studies on stickers focused on sentiment analysis and recommendation systems, overlooking their vast potential in empathetic response generation.
Approach:	They propose a multimodal empathetic dialogue dataset, STICKERCONV, which simulates human behavior with stickers, and propose evaluative metrics based on LLM.
Outcome:	The proposed framework generates contextually relevant and emotionally resonant multimodal empathetic responses, contributing to the advancement of more nuanced and engaging e-dialog systems.

Copied to clipboard

Tong Zheng, Bei Li, Huiwen Bao, Tong Xiao, JingBo Zhu

MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs (2024.acl-long)

Challenge:	Existing multi-view learning models prioritize complementarity while ignoring consensus . EMHA allows for efficient modeling of global dependencies among tokens in parallel .
Approach:	They propose an enhanced multi-head self-attention (EMHA) that prioritizes complementarity while ignoring consensus.
Outcome:	The proposed method favors consensus among heads by introducing two models . it is superior on a wide range of language tasks with a modest increase in model size .

Copied to clipboard

Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, Salman Avestimehr

Challenge:	Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. however, their tendency to produce inaccurate or misleading outputs poses a potential risk.
Approach:	They propose a new scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question.
Outcome:	The proposed scoring function improves UE performance on a medical QA dataset.

EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models (2024.acl-long)

Copied to clipboard

Rocktim Das, Simeon Hristov, Haonan Li, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov

Order-Agnostic Data Augmentation for Few-Shot Named Entity Recognition (2024.acl-long)

Challenge:	Existing benchmarks for vision language models are outdated and unable to accurately assess their performance.
Approach:	They propose a multi-discipline multimodal multilingual exam benchmark for vision language models . they collect multiple-choice questions across 20 disciplines across 11 languages from 7 language families .
Outcome:	The EXAMS-V exam includes 20,932 multiple-choice questions across 20 disciplines . the questions come in 11 languages from 7 language families and require advanced reasoning skills .

Copied to clipboard

Huiming Wang, Liying Cheng, Wenxuan Zhang, De Wen Soh, Lidong Bing

Text Embedding Inversion Security for Multilingual Language Models (2024.acl-long)

Challenge:	Existing DA methods for named entity recognition (NER) are costly and labor-intensive to acquire, necessitating innovative approaches to data scarcity.
Approach:	They propose an order-agnostic data augmentation solution that exploits the order-based property in the training phase of sequence-to-sequence NER methods for data augmented.
Outcome:	The proposed method significantly enhances the few-shot capabilities of pre-trained language models in low-resource settings.

Copied to clipboard

Yiyi Chen, Heather Lent, Johannes Bjerva

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment (2024.acl-long)

Challenge:	storing sensitive information as embeddings is susceptible to security breaches, as text can be reconstructed from embeddables . study explores multilingual inversion attacks using a masking defense .
Approach:	They propose a simple masking defense that can be used to decode embedded text . they define the problem of black-box multilingual and crosslingual inversion attacks .
Outcome:	The proposed defense is effective for both monolingual and multilingual models.

Copied to clipboard

Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator (2024.acl-long)

Challenge:	Existing work cheaply emulates LLMs, allowing users to create profiles for their preferred characters.
Approach:	They propose a self-alignment method that encourages an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension.
Outcome:	The proposed model outperforms open-source role-play benchmarks and the roleplay subset of MT-Bench in multiple parameters.

Copied to clipboard

Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, Benyou Wang

Synthesizing Text-to-SQL Data from Weak and Strong LLMs (2024.acl-long)

Challenge:	Recent efforts to democratize ChatGPT have focused on leveraging real user and ChatGPP dialogues, but the most direct human needs are often ignored.
Approach:	They propose a method to simulate human behavior better by using real human-like questions extracted from real human conversations as a learning goal and a user simulator called ‘Socratic’.
Outcome:	The proposed model achieves SoTA performance among LLaMA-based 7B models in MT-Bench.

Copied to clipboard

Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, Chang Zhou

STRUCTSUM Generation for Faster Text Comprehension (2024.acl-long)

Challenge:	a capability gap exists between open-source and closed-source large language models (LLMs) . the adoption of closed-sourced LLMs introduces concerns pertaining to openness, privacy, and substantial costs.
Approach:	They propose a synthetic data approach that combines strong and weak models for error information . they demonstrate the effectiveness of SENSE, a specialized text-to-SQL model .
Outcome:	The proposed method enhances the domain generalization of text-to-SQL models and explores the potential of error data supervision through preference learning.

Copied to clipboard

Parag Jain, Andreea Marzoca, Francesco Piccinno

Analysing The Impact of Sequence Composition on Language Model Pre-Training (2024.acl-long)

Challenge:	Current large language models (LLMs) fail to adequately structure and organize information in a way that facilitates comprehension.
Approach:	They propose a taxonomy of problems around factuality, global and local structure common to both modalities and propose 'auto-QA' to improve the accuracy of generated structured representations.
Outcome:	The proposed models improve accuracy and speed without loss of accuracy.

Copied to clipboard

Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miłoś, Yuxiang Wu, Pasquale Minervini

NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time (2024.acl-long)

Challenge:	Existing studies show that pretraining sequence composition strategy can lead to distracting information from previous documents.
Approach:	They propose to use a sequence construction method to concatenate documents into fixed-length sequences to compute the likelihood of each token given its context.
Outcome:	The proposed method can improve in-context learning, knowledge memorisation and context utilisation without sacrificing efficiency.

Copied to clipboard

Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu

SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network (2024.acl-long)

Challenge:	Large Language Models (LLMs) with extended context windows are expensive and infeasible on fixed memory hardware due to the surprisingly large memory consumption of KV Cache.
Approach:	They propose a general framework for long-context KV cache eviction that achieves more optimal and efficient evict in a single operation during the encoding phase.
Outcome:	The proposed framework improves performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 5 with over 95% performance maintenance.

Copied to clipboard

Kexin Wang, Jiahong Zhang, Yong Ren, Man Yao, Di Shang, Bo Xu, Guoqi Li

Context-aware Difference Distilling for Multi-change Captioning (2024.acl-long)

Challenge:	SpikeVoice performs high-quality Text-To-Speech (TTS) via SNN . major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies.
Approach:	They propose a brain-inspired Spiking Neural Network (SNN) which performs high-quality Text-To-Speech (TTS) via SNN and explores the potential of SNN to "speak".
Outcome:	The proposed model achieves comparable results to Artificial Neural Networks (ANN) with only 10.5% energy consumption of ANN.

Copied to clipboard

Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, Qingming Huang

Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion (2024.acl-long)

Challenge:	Existing methods for multi-change captioning are difficult because it requires a higher level of cognition to reason an arbitrary number of changes.
Approach:	They propose a context-aware difference distilling network to capture all genuine changes for yielding sentences.
Outcome:	The proposed network captures all genuine changes for yielding sentences on three public datasets.

Copied to clipboard

Wei Cheng, Yuhan Wu, Wei Hu

Chain-of-Exemplar: Enhancing Distractor Generation for Multimodal Educational Question Generation (2024.acl-long)

Challenge:	Existing methods to generate correct code completions in private repositories are insufficiently relevant.
Approach:	They propose a dataflow-guided retrieval augmentation approach for repository-level code completion . they parses a private repository into code entities and establishes their relations through an extended dataflow analysis .
Outcome:	The proposed method improves code exact match and identifier F1-score by 3.43% compared to the state-of-the-art approach.

Copied to clipboard

Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, Tat-Seng Chua

LLMEmbed: Rethinking Lightweight LLM’s Genuine Function in Text Classification (2024.acl-long)

Challenge:	Existing methods for multiple choice questions focus on text inputs and lack visual information.
Approach:	They propose a framework to generate subject-specific educational questions with plausible distractors based on multimodal content.
Outcome:	The proposed framework improves question generation and distractor generation over existing methods across subjects and educational levels.

Copied to clipboard

ChunLiu ChunLiu, Hongguang Zhang, Kainan Zhao, Xinghai Ju, Lin Yang

LEMON: Reviving Stronger and Smaller LMs from Larger LMs with Linear Parameter Fusion (2024.acl-long)

Challenge:	Recent attempts to improve text classification performance are based on heuristic Chain-of-Thought (CoT) LLMEmbed is a simple and effective transfer learning strategy that can be used to improve the performance of large language models.
Approach:	They propose a simple transfer learning strategy to improve text classification using heuristic Chain-of-Thought.
Outcome:	The proposed method achieves strong performance on publicly available datasets while using low training overhead.

Copied to clipboard

Yilong Chen, Junyuan Shang, Zhenyu Zhang, Shiyao Cui, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu

Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation (2024.acl-long)

Challenge:	Existing methods to train a stronger and smaller model with the help of large models are limited by the model size and performance.
Approach:	They propose to learn competent initial points for smaller models by fusing parameters from larger models and introduce controllable receptive fields to model prior parameter characteristics.
Outcome:	The proposed method outperforms baselines in terms of effectiveness and efficiency.

Copied to clipboard

Tengfei Yu, Xuebo Liu, Liang Ding, Kehai Chen, Dacheng Tao, Min Zhang

To be Continuous, or to be Discrete, Those are Bits of Questions (2024.acl-long)

Challenge:	End-to-end speech translation (ST) models require simultaneous crossmodal and crosslingual transformations to be effective.
Approach:	They propose a homophone-aware contrastive learning approach that integrates a speech-text masking strategy to reduce ambiguity.
Outcome:	The proposed approach achieves SOTA results on BLEU scores on different MuST-C and CoVoST ST tasks, underlining its effectiveness in reducing speech sense ambiguity.

Copied to clipboard

Yiran Wang, Masao Utiyama

Moûsai: Efficient Text-to-Music Diffusion Models (2024.acl-long)

Challenge:	Existing studies have shown that binary representations preserve information on the output side, but they are not available for continuous representations.
Approach:	They propose to extend the contrastive hashing method to output binary labels instead of continuous vectors.
Outcome:	The proposed representation preserves structural information along with label information . it achieves competitive performance on various structured prediction tasks .

Copied to clipboard

Flavio Schneider, Ojasv Kamal, Zhijing Jin, Bernhard Schölkopf

PokeMQA: Programmable knowledge editing for Multi-hop Question Answering (2024.acl-long)

Challenge:	Recent years have seen the rapid development of large generative models for text; however, little research has explored the connection between text and another “language” of communication – music.
Approach:	They develop a text-to-music generation model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions.
Outcome:	The proposed model can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions.

Copied to clipboard

Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, Xin Wang

MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Intervention (2024.acl-long)

Challenge:	Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine’s comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance.
Approach:	They propose a framework to edit multi-hop question models to update model with up-to-date facts while avoiding expensive re-training or fine-tuning.
Outcome:	The proposed framework outperforms all competitors in multi-hop question answering tasks and consistently produces reliable reasoning process.

Copied to clipboard

Prince Jha, Raghav Jain, Konika Mandal, Aman Chadha, Sriparna Saha, Pushpak Bhattacharyya

Efficient OCR for Building a Diverse Digital History (2024.acl-long)

Challenge:	Existing studies on content moderation of toxic memes focus on text-based content . current research neglects the widespread influence of multimodal content like memes .
Approach:	They propose a framework leveraging Large Language Models and Visual Language Model (VLMs) for meme intervention.
Outcome:	The proposed framework enables users to generate relevant and effective responses to toxic memes.

Copied to clipboard

Jacob Carlson, Tom Bryan, Melissa Dell

Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space (2024.acl-long)

Challenge:	Current optical character recognition (OCR) systems are poorly extensible to low-resource document collections, as learning a language-vision model requires extensive labeled sequences and compute.
Approach:	They propose to model optical character recognition as a character level image retrieval problem using a contrastively trained vision encoder.
Outcome:	The proposed model is more sample efficient and extensible than existing architectures, enabling accurate OCR in settings where existing solutions fail.

Copied to clipboard

Zongru Wu, Zhuosheng Zhang, Pengzhou Cheng, Gongshen Liu

ANAH: Analytical Annotation of Hallucinations in Large Language Models (2024.acl-long)

Challenge:	Prior work attempts to mitigate backdoor learning during training LMs on poisoned datasets . backdoor attack poisons a small portion of training data by implanting specific text patterns .
Approach:	They propose a multi-scale low-rank adaptive model that prioritizes learning of clean mapping . they propose radial scalings to reduce the success rate of diverse backdoor attacks .
Outcome:	The proposed model outperforms baselines significantly in the frequency space . it reduces the success rate of diverse backdoor attacks to below 15% across datasets .

Copied to clipboard

Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, Kai Chen

Aligning Large Language Models for Controllable Recommendations (2024.acl-long)

Challenge:	a comprehensive and fine-grained measurement of the hallucination is crucial for LLMs' wide applications.
Approach:	They propose a dataset that offers ANalytical Annotation of Hallucinations in Large Language Models.
Outcome:	The proposed dataset can be used to train and evaluate hallucination annotators.

Copied to clipboard

Wensheng Lu, Jianxun Lian, Wei Zhang, Guanghua Li, Mingyang Zhou, Hao Liao, Xing Xie

Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods (2024.acl-long)

Challenge:	Existing literature focuses on integrating domain-specific knowledge into LLMs to enhance accuracy using a fixed task template.
Approach:	They propose a collection of supervised learning tasks augmented with labels derived from a conventional recommender model to improve LLMs’ proficiency in adhering to recommendation-specific instructions.
Outcome:	The proposed approach significantly improves the capability of LLMs to respond to instructions within recommender systems, reducing formatting errors while maintaining a high level of accuracy.

Copied to clipboard

Haeun Yu, Pepa Atanasova, Isabelle Augenstein

Full Parameter Fine-tuning for Large Language Models with Limited Resources (2024.acl-long)

Challenge:	Language Models acquire parametric knowledge from their training process, embedding it within their weights.
Approach:	They propose a new evaluation framework to quantify and compare the knowledge revealed by Instance Attribution and Neuron Attributions.
Outcome:	The proposed evaluation framework compares the knowledge revealed by IA and NA with that of neuron attribution methods.

Copied to clipboard

Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo, Xipeng Qiu

M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought (2024.acl-long)

Challenge:	Large Language Models (LLMs) require massive GPU resources for training.
Approach:	They propose a parameter-efficient optimization that fuses the gradient computation and parameter update in one step to reduce memory usage.
Outcome:	The proposed method reduces memory usage to 10.8% compared to the standard approach.

Copied to clipboard

Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, Wanxiang Che

Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models (2024.acl-long)

Challenge:	MCoT requires models to leverage knowledge from both textual and visual modalities for step-by-step reasoning.
Approach:	They propose a benchmark to address the challenges of MCoT, and evaluate it using vision large language models.
Outcome:	The proposed benchmark addresses the above challenges and shows that current models still struggle to reason in M3CoT.

Copied to clipboard

Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, Min Yang

Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation (2024.acl-long)

Challenge:	Long-context modeling capabilities are important for large language models (LLMs) however, training LLMs with long context windows is insufficient since some samples do not exhibit strong semantic dependencies across long contexts.
Approach:	They propose a data mining framework ProLong that assigns each training sample with a long dependency score and ranks and filters them according to their results.
Outcome:	The proposed framework can rank and filter training samples that exhibit more powerful long-context modeling abilities.

Copied to clipboard

Keqi Deng, Phil Woodland

Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)

Challenge:	LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire (AIF) mechanism.
Approach:	They propose a label-synchronous neural transducer for simultaneous speech translation which dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire mechanism.
Outcome:	The proposed model can predict translation tokens on monolingual text-only data or can be used in both decoding and training.

Copied to clipboard

Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim

A Modular Approach for Multimodal Summarization of TV Shows (2024.acl-long)

Challenge:	Prompt tuning is an important technique for directing model behaviors and eliciting desired responses.
Approach:	They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning.
Outcome:	The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable.

Copied to clipboard

Louis Mahon, Mirella Lapata

Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities (2024.acl-long)

Challenge:	In this paper, we address the task of summarizing television shows, which touches key areas in AI research.
Approach:	They propose a modular approach where separate components perform specialized sub-tasks . they propose atomic facts to measure precision and recall of generated summaries .
Outcome:	The proposed method produces higher quality summaries than comparison models on a recently released dataset.

Copied to clipboard

Alex Wilf, Sihyun Lee, Paul Pu Liang, Louis-Philippe Morency

BizBench: A Quantitative Reasoning Benchmark for Business and Finance (2024.acl-long)

Challenge:	Recent advances to LLMs’ reasoning capabilities from simple yet effective prompting techniques such as Chain-of-Thought have seen limited applicability to ToM.
Approach:	They propose a two-stage prompting framework inspired by Simulation Theory's notion of perspective-taking to elicit Theory-of-Mind capabilities in Large Language Models.
Outcome:	The proposed framework shows that it is much more effective than existing prompts.

Copied to clipboard

Michael Krumdick, Rik Koncel-Kedziorski, Viet Lai, Varshini Reddy, Charles Lovering, Chris Tanner

Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization (2024.acl-long)

Challenge:	Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge.
Approach:	They propose a benchmark for evaluating models’ ability to reason about realistic financial problems by focusing on question-answering over financial data via program synthesis.
Outcome:	The proposed benchmark evaluates models' financial background knowledge, ability to parse financial documents, and capacity to solve complex problems with code.

Copied to clipboard

Takumi Takada, Yuma Suzuki, Hiroki Takushima, Hayato Tanoue, Haruki Sato, Aiswariya Kumar, Hiroki Nishihara, Takayuki Hori, Kazuya Ueki

Deciphering Hate: Identifying Hateful Memes and Their Targets (2024.acl-long)

Challenge:	Recent large-scale vision language models (VLMs) lack continuity between learning objective and performance metrics.
Approach:	They propose a lightweight final-metric-optimizing training method that replaces the expensive exploration process in RL with an offline, diverse text data augmentation method.
Outcome:	The proposed method achieves comparable performance to state-of-the-art RL method while saving hundreds of times more model forwarding iterations and greater amounts of computation time.

Copied to clipboard

Eftekhar Hossain, Omar Sharif, Mohammed Moshiul Hoque, Sarah Masud Preum

Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings (2024.acl-long)

Challenge:	a growing body of research has focused on the negative aspects of memes in high-resource languages like Bengali . a new dataset for Bengali hateful memes is designed to detect their targeted entities .
Approach:	They propose a multimodal dataset that analyzes the modality of memes and compares them with other datasets.
Outcome:	The proposed dataset outperforms state-of-the-art datasets on Bengali hateful memes . the proposed dataset is generalizable on other low-resource hateful memes datasets compared with baselines based on the proposed model .

Copied to clipboard

Yichen Jiang, Xiang Zhou, Mohit Bansal

Label-Efficient Model Selection for Text Generation (2024.acl-long)

Challenge:	Empirically, we show SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on low-complexity datasets.
Approach:	They propose a Transformer that explicitly encourages systematicity in the embeddings and attention layers even with low-complexity data.
Outcome:	Empirically, the proposed model achieves stronger compositional generalization than the vanilla Transformer on low-complexity datasets.

Copied to clipboard

Shir Ashury Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, Eyal Shnarch

Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)

Challenge:	Model selection for a given task can entail extensive annotation of the quality of outputs of different models.
Approach:	They propose a model-agnostic method to make an informed decision between candidate text generation models based on preference annotations.
Outcome:	The proposed method reduces the required number of annotations by up to 75% while maintaining high evaluation reliability.

Copied to clipboard

Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, Xiang Yue

Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals (2024.acl-long)

Challenge:	Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining.
Approach:	They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness.
Outcome:	The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process.

Copied to clipboard

Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, Bernhard Schölkopf

FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence (2024.acl-long)

Challenge:	Existing interpretability research focused on analyzing a single mechanism . et al., 2023) focused on finding how models copy or recall factual knowledge .
Approach:	They propose a competition of mechanisms that focuses on the interplay of multiple mechanisms instead of individual mechanisms . they uncover how and where the competition of mechanism happens within LLMs using logit inspection and attention modification methods.
Outcome:	The proposed model is based on two interpretability methods, logit inspection and attention modification.

Copied to clipboard

Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, Junyi Jessy Li

BvSP: Broad-view Soft Prompting for Few-Shot Aspect Sentiment Quad Prediction (2024.acl-long)

Challenge:	FactPICO is a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials . existing metrics for factual summarizing medical evidence are poorly correlated with expert judgments on the instance level.
Approach:	They propose a factuality benchmark for plain language summarization of medical texts . they assess factuality of critical elements of RCTs in those summaries .
Outcome:	The proposed benchmark assesses the factuality of medical summaries using LLMs . the summary summators are based on 345 plain language summaires with fine-grained evaluation .

Copied to clipboard

Yinhao Bai, Yalan Xie, Xiaoyi Liu, Yuhua Zhao, Zhixin Han, Mengting Hu, Hang Gao, Renhong Cheng

Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack (2024.acl-long)

Challenge:	Aspect sentiment quad prediction aims to predict aspects due to distinct data distribution.
Approach:	They propose a method that aggregates multiple templates with a broader view . they first construct a few-shot ASQP dataset that contains richer categories .
Outcome:	The proposed method outperforms the state-of-the-art methods under four few-shot settings and other public datasets.

Copied to clipboard

Yu Fu, Yufei Li, Wen Xiao, Cong Liu, Yue Dong

Speech language models lack important brain-relevant semantics (2024.acl-long)

Challenge:	Recent developments in balancing usefulness and safety of large language models raise a critical question . current attacks, especially adversarial ones that manipulate malicious prompts, often aim to manipulate the input .
Approach:	They show that LLMs can effectively summarize malicious long documents but often refuse to translate them.
Outcome:	The findings highlight a vulnerability in LLMs that can't translate or summarize documents . the study focuses on LLM models, Gemini and GPT-4, which can' be exploited .

Copied to clipboard

Subba Reddy Oota, Emin Çelik, Fatma Deniz, Mariya Toneva

DocLLM: A Layout-Aware Generative Language Model for Multimodal Document Understanding (2024.acl-long)

Challenge:	Recent work shows that text-based language models predict both text- and speech-evoked brain activity.
Approach:	They remove low-level stimulus features from language models to assess their impact on alignment with fMRI brain recordings during reading and listening.
Outcome:	The proposed model removes low-level features from fMRI brain recordings to assess their impact on alignment with fmr recordings.

Copied to clipboard

Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, Xiaomo Liu

Bypassing LLM Watermarks with Color-Aware Substitutions (2024.acl-long)

Challenge:	Documents with rich layouts are a significant portion of enterprise corpora and document AI is still a challenge.
Approach:	They propose a lightweight extension to traditional large language models for reasoning over visual documents that takes into account both textual semantics and spatial layout.
Outcome:	The proposed model outperforms existing large language models on 14 out of 16 datasets and generalizes well to 4 out of 5 previously unseen datasets.

Copied to clipboard

Qilong Wu, Varun Chandrasekaran

Parallel Structures in Pre-training Data Yield In-Context Learning (2024.acl-long)

Challenge:	Existing methods to detect large language models (LLMs) generated text are susceptible to a large number of false positives.
Approach:	They propose a watermarking approach that uses color information to determine token colors and substitute green tokens with non-green ones.
Outcome:	The proposed method evades detection with fewer edits and removes the watermark for arbitrarily long watermarked text.

Copied to clipboard

Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, He He

OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models (2024.acl-long)

Challenge:	Pre-trained language models (LMs) are capable of in-context learning (ICL) however, it is unclear where this ability comes from as there is a stark distribution shift between pre-training text and ICL prompts.
Approach:	They find that pre-trained language models are capable of in-context learning (ICL) they detect parallel structures in the pre-training data and conduct ablation experiments to study their effect on ICL.
Outcome:	The proposed model can adapt to a task with a few examples given in the prompt without any parameter update.

Copied to clipboard

Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, Yulan He

Towards Privacy-Aware Sign Language Translation at Scale (2024.acl-long)

Challenge:	Existing N-ToM benchmarks lack ambiguous and artificial narratives, lack of personality traits and preferences, and limited diversity in the questions posed.
Approach:	They propose a benchmark to assess Neural Theory-of-Mind (N-ToM) with longer and clearer narrative stories, characters with explicit personality traits, actions triggered by character intentions, and questions designed to challenge LLMs’ abilities of modeling characters’ mental states.
Outcome:	The proposed test aims to assess the performance of LLMs in the physical and psychological worlds.

Copied to clipboard

Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, Jean Maillard

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards (2024.acl-long)

Challenge:	Existing sign language training systems require detailed and time aligned annotations to be effective.
Approach:	They propose a two-stage framework for privacy-aware SLT at scale that leverages self-supervised video pretraining on anonymized and unannotated videos followed by supervised SLT finetuning on a curated parallel dataset.
Outcome:	The proposed framework outperforms baselines on the How2Sign dataset and achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance.

Copied to clipboard

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang

Challenge:	Reinforcement Learning from Human Feedback (RLHF) relies on scalar rewards to capture user preferences.
Approach:	They propose a framework that integrates multi-objective reward modeling to represent diverse preference profiles.
Outcome:	The proposed method improves performance across reward objectives and targets.

Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characters (2024.acl-long)

Copied to clipboard

Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, Ying Shen

RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (2024.acl-long)

Challenge:	Existing studies focus on misspelled characters, ignoring faked characters which are more common and difficult to correct.
Approach:	They propose to use Chinese character checking to identify and correct wrong characters in texts by human annotation.
Outcome:	The proposed dataset is the first real-world visual and the largest human-crafted dataset for the Chinese character checking scenario.

Copied to clipboard

Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, Atticus Geiger

Large Language Models as Zero-shot Dialogue State Tracker through Function Calling (2024.acl-long)

Challenge:	Existing methods to disentangle individual neurons from multiple high-level concepts are not yet benchmarked.
Approach:	They propose a method of Multi-task Distributed Alignment Search that allows to find distributed representations satisfying multiple causal criteria.
Outcome:	The proposed method achieves state-of-the-art on the target language model with Llama2-7B .

Copied to clipboard

Zekun Li, Zhiyu Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Dong, Adithya Sagar, Xifeng Yan, Paul Crook

Faithful Chart Summarization with ChaTS-Pi (2024.acl-long)

Challenge:	Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts.
Approach:	They propose a method for solving dialogue state tracking (DST) with large language models through function calling.
Outcome:	The proposed approach improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning.

Copied to clipboard

Syrine Krichene, Francesco Piccinno, Fangyu Liu, Julian Eisenschlos

Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation (2024.acl-long)

Challenge:	Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired.
Approach:	They propose a reference-free chart summarization metric for scoring faithfulness . they use human raters to fix and rank candidate summaries from any chart-summarization model .
Outcome:	The proposed metric scores the summarization faithfulness according to human ratings better than reference-based metrics, either learned or n-gram based.

Copied to clipboard

Cheng Niu, Xingguang Wang, Xuxin Cheng, Juntong Song, Tong Zhang

MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking (2024.acl-long)

Challenge:	Experimental results show that the model can be used to generate dialogues in new domains quickly.
Approach:	They propose to use LLMs to generate dialogue data to reduce dialogue collection and annotation costs.
Outcome:	The proposed model performs better than the baseline model trained on real data.

Copied to clipboard

Ting-Chih Chen, Chia-Wei Tang, Chris Thomas

KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction (2024.acl-long)

Challenge:	Fact-checking claims on social media platforms poses a significant challenge due to the large volume of new claims constantly being posted without sufficient methods for verification.
Approach:	They propose a model that generates claim-specific summaries from multimodal multi-document datasets using a perceiver-based model that is able to handle inputs from multiple modalities of arbitrary lengths.
Outcome:	The proposed model outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and shows strong performance on the new multi-document claims dataset.

Copied to clipboard

Zixuan Li, Yutao Zeng, Yuxin Zuo, Weicheng Ren, Wenxuan Liu, Miao Su, Yucan Guo, Yantao Liu, Lixiang Lixiang, Zhilei Hu, Long Bai, Wei Li, Yidan Liu, Pan Yang, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng

ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis (2024.acl-long)

Challenge:	None. None.. None!
Approach:	None. None.. None!
Outcome:	None. None. No. :

Copied to clipboard

Yanming Liu, Xinyue Peng, Tianyu Du, Jianwei Yin, Weihao Liu, Xuhong Zhang

On the Multi-turn Instruction Following for Conversational Web Agents (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated remarkable in-context learning capabilities in various natural language processing tasks.
Approach:	They propose a novel approach ERA-CoT which aids LLMs in understanding context by capturing relationships between entities and supports the reasoning of diverse tasks through Chain-of-Thoughts (CoT).
Outcome:	The proposed method improves on GPT3.5 and previous SOTA prompting methods by an average of 5.1% compared to previous prompting approaches.

Copied to clipboard

Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, Tat-Seng Chua

Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents (2024.acl-long)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within web-based environments.
Approach:	They propose a framework for conversational web navigation that uses multi-turn interactions with both the user and the environment.
Outcome:	The proposed framework is based on a multi-turn Mind2Web (MT-Mind2Web) it is designed to perform multi-step interactions with web-based environments .

Copied to clipboard

Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang

MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in China (2024.acl-long)

Challenge:	Existing benchmarks for LLM-based mobile agents are insufficient to evaluate their capabilities.
Approach:	They propose a benchmark to evaluate LLM-based mobile agents' planning capabilities . they expand UI operations by incorporating 103 APIs to accelerate task completion .
Outcome:	The proposed benchmarks are based on 103 collected APIs and real user queries . the data is categorized into three distinct groups: SAST, SAMT, and MAMT .

Copied to clipboard

Chen Zhang, Mingxu Tao, Quzhe Huang, Jiuheng Lin, Zhibin Chen, Yansong Feng

Challenge:	MC2 is the largest open-source corpus of minority languages in china . MC2, however, includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian .
Approach:	They propose a multilingual corpus of minority languages in China that includes four underrepresented languages . they prioritize accuracy while enhancing diversity by using a quality-centric approach .
Outcome:	The proposed model prioritizes accuracy while enhancing diversity, the authors say . MC2 includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian .

Decoder-only Streaming Transformer for Simultaneous Translation (2024.acl-long)

Copied to clipboard

Shoutao Guo, Shaolei Zhang, Yang Feng

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization (2024.acl-long)

Challenge:	Existing methods for siMT focus on the Encoder-Decoder architecture, but there are limitations in training and inference.
Approach:	They propose a model that generates translation while reading source tokens . they propose Streaming Self-Attention mechanism tailored for the Decoder-only architecture .
Outcome:	The proposed model achieves state-of-the-art performance on three translation tasks.

Copied to clipboard

Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, Minlie Huang

I am a Strange Dataset: Metalinguistic Tests for Language Models (2024.acl-long)

Challenge:	Despite the success of jailbreaking attacks, there is a lack of effort in defending against them.
Approach:	They propose to integrate goal prioritization at both training and inference stages to counteract this conflict between the goals of being helpful and ensuring safety.
Outcome:	The proposed approach reduces the Attack Success Rate (ASR) of jailbreaking from 66.4% to 3.6% for ChatGPT and reduces it from 71.0% to 6.6% for Llama2-13B.

Copied to clipboard

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space (2024.acl-long)

Challenge:	Existing datasets for metalinguistic self-reference are limited by the number of subtasks.
Approach:	They propose a dataset that aims to address metalinguistic self-reference in large language models.
Outcome:	The proposed dataset is hand-crafted by experts and validated by non-expert annotators.

Copied to clipboard

Shaolei Zhang, Tian Yu, Yang Feng

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training (2024.acl-long)

Challenge:	Large Language Models (LLMs) sometimes produce untruthful responses despite knowing the correct knowledge.
Approach:	They propose an inference-time intervention method to activate the truthfulness of Large Language Models (LLMs) by editing the features within LLM’s internal representations that govern the truthful.
Outcome:	The proposed method improves the truthfulness of 13 advanced LLMs by an average of 20% on TruthfulQA benchmark.

Copied to clipboard

Le Zhuo, Zewen Chi, Minghao Xu, Heyan Huang, Jianan Zhao, Heqi Zheng, Conghui He, Xian-Ling Mao, Wentao Zhang

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)

Challenge:	Experimental results demonstrate that ProtLLM achieves superior performance against protein-specialized baselines on protein-centric tasks and induces zero-shot and in-context learning capabilities on protein language tasks.
Approach:	They propose a cross-modal large language model (LLM) that can handle protein-centric and protein-language tasks by using a dynamic protein mounting mechanism.
Outcome:	The proposed model can predict proteins from a vast pool of candidates and can also predict natural language and biological papers.

Copied to clipboard

Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, Yang Feng

Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models (2024.acl-long)

Challenge:	Existing simultaneous translation methods focus on text-to-text and speech-totext translation.
Approach:	They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning.
Outcome:	The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model.

Copied to clipboard

Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, Gongshen Liu

Why Don’t Prompt-Based Fairness Metrics Correlate? (2024.acl-long)

Challenge:	Recent work has demonstrated the power of large language models in recalling knowledge and reasoning.
Approach:	They propose to erase shortcut neurons to mitigate the associated risks . 20% of the failures are attributed to shortcuts, they find .
Outcome:	The proposed approach reduces failures in multi-hop knowledge editing caused by shortcuts by 20% .

Copied to clipboard

Abdelrahman Zayed, Goncalo Mordido, Ioana Baldini, Sarath Chandar

NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data (2024.acl-long)

Challenge:	Existing methods to assess fairness using prompts have low correlations between fairness metrics.
Approach:	They propose a method to enhance the correlation between fairness metrics by using pre-trained language models.
Outcome:	The proposed method improves the correlation between fairness metrics by using pre-trained language models.

Copied to clipboard

Manuel Tonneau, Pedro Quinta De Castro, Karim Lasri, Ibrahim Farouq, Lakshmi Subramanian, Victor Orozco-Olvera, Samuel Fraiberger

M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset (2024.acl-long)

Challenge:	a recent study shows that hate speech detection systems are often evaluated on non-representative samples, raising concerns about overestimating performance in real-world settings.
Approach:	They propose a pretrained hate speech detection model that is annotated on a representative sample of Nigerian tweets and propose heuristics for domain-adaptive pretraining and finetuning.
Outcome:	The proposed model overestimates real-world performance by at least twofold compared to a dataset from the United States and Nigeria . the proposed model requires ten thousand Nigerian tweets flagged as hateful daily to moderate 60% of hateful content .

Copied to clipboard

Zhe Chen, Heyang Liu, Wenyi Yu, Guangzhi Sun, Hongcheng Liu, Ji Wu, Chao Zhang, Yu Wang, Yanfeng Wang

Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination (2024.acl-long)

Challenge:	Publishing open-source academic video recordings is an emerging approach to sharing knowledge online.
Approach:	They propose a multimodal, multigenre, and multipurpose audio-visual academic lecture dataset with human annotations for multimodal content recognition and understanding tasks.
Outcome:	The proposed dataset can be used for multiple audio-visual recognition and understanding tasks.

Copied to clipboard

Nakyeong Yang, Taegwan Kang, Stanley Jungkyu Choi, Honglak Lee, Kyomin Jung

Domain Adaptation for Subjective Induction Questions Answering on Products by Adversarial Disentangled Learning (2024.acl-long)

Challenge:	Existing methods to mitigate undesirable biases in instruction-following language models are not effective in accelerating instruction-based learning.
Approach:	They propose a method to eliminate bias neurons of language models in instruction-following settings by defining the bias neuron and prove its existence empirically.
Outcome:	The proposed method dramatically increases the task performance of language models under zero-shot instruction-following settings without losing the model’s knowledge.

Copied to clipboard

Yufeng Zhang, Jianxing Yu, Yanghui Rao, Libin Zheng, Qinliang Su, Huaijie Zhu, Jian Yin

Revisiting Demonstration Selection Strategies in In-Context Learning (2024.acl-long)

Challenge:	Existing methods to answer subjective questions about products are often imbalanced across product domains.
Approach:	They propose a domain-adaptive model that integrates multiple viewpoints into a good answer by integrating these heterogeneous and inconsistent viewpoints.
Outcome:	The proposed model integrates multiple viewpoints into a single answer span and is able to integrate them into the answer.

Copied to clipboard

Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao

Multimodal Table Understanding (2024.acl-long)

Challenge:	Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL).
Approach:	They propose a data- and model-dependent method to select models using in-context learning, TopK + ConE, and propose unified explanations for the effectiveness of previous methods.
Outcome:	The proposed method improves language understanding and generation tasks with different model scales.

Copied to clipboard

Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, Weiping Wang

Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding (2024.acl-long)

Challenge:	Existing approaches to understanding tables rely on textual inputs and table images are difficult to access in real-world scenarios.
Approach:	They propose a multimodal table understanding problem where the model needs to generate correct responses to various table-related requests based on the given table image.
Outcome:	The proposed model outperforms open-source MLLMs on 23 benchmarks under held-in and held-out settings.

Copied to clipboard

Huang Lei, Jiaming Guo, Guanhua He, Xishan Zhang, Rui Zhang, Shaohui Peng, Shaoli Liu, Tianshi Chen

Challenge:	Generating long-term texts using artificial intelligence has always been a challenge . however, the generated novels exhibit poor logical coherence and appeal in their plots and deficiencies in character and event depiction, ultimately compromising the overall narrative quality.
Approach:	They propose a method for extracting excelsior and expanding from novel data to generate arbitrarily long novels using large language models.
Outcome:	The proposed method produces high-quality long-form novels with a high level of logical coherence and appeal despite the use of large language models.

Few-shot Transfer Learning for Knowledge Base Question Answering: Fusing Supervised Models with In-Context Learning (2024.acl-long)

Copied to clipboard

Mayur Patidar, Riya Sawhney, Avinash Singh, Biswajit Chatterjee, Mausam ., Indrajit Bhattacharya

WatME: Towards Lossless Watermarking Through Lexical Redundancy (2024.acl-long)

Challenge:	Existing Knowledge Base Question Answering (KBQA) architectures are expensive and time-consuming to deploy.
Approach:	They propose a KBQA architecture that performs KB-retrieval using multiple source-trained retrievers and re-ranks using an LLM.
Outcome:	The proposed architecture outperforms adaptations of SoTA KBQA models when training data is limited.

Copied to clipboard

Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, Kam-Fai Wong

Challenge:	Existing methods for text watermarking rely on arbitrary vocabulary partitioning during decoding, which compromises the availability of suitable tokens and significantly degrades the quality of responses.
Approach:	They propose a method that leverages linguistic prior knowledge of lexical redundancies in LLM vocabularies to seamlessly integrate watermarks.
Outcome:	The proposed approach preserves the expressive power of large language models while preserving watermark detectability.

Text-like Encoding of Collaborative Information in Large Language Models for Recommendation (2024.acl-long)

Copied to clipboard

Yang Zhang, Keqin Bao, Ming Yan, Wenjie Wang, Fuli Feng, Xiangnan He

MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception (2024.acl-long)

Challenge:	Existing methods to adapt Large Language Models for Recommendation (LLMRec) do not represent collaborative information in a text-like format, which may not align optimally with LLMs.
Approach:	They propose a novel LLMRec method that integrates collaborative information through text-like encoding.
Outcome:	Extensive experiments show that BinLLM integrates collaborative information better with LLMs.

Copied to clipboard

Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yanfeng Wang, Yu Wang

Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning (2024.acl-long)

Challenge:	Recent advances in multimodal large language models (MLLMs) have demonstrated exceptional capabilities in visual perception and understanding, but they also suffer from hallucinations, which limit their reliability as AI systems.
Approach:	They propose a benchmark to evaluate self-awareness in perception for multimodal large language models (MLLMs) by integrating image information with knowledge quadrants, and propose MM-SAP to evaluate this capability.
Outcome:	The proposed benchmark offers detailed analysis of MLLMs with self-awareness in perception.

Copied to clipboard

Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, Jun Zhao

Multi-Aspect Controllable Text Generation with Disentangled Counterfactual Augmentation (2024.acl-long)

Challenge:	Large language models exhibit high-level commonsense reasoning abilities, especially with enhancement methods like Chain-of-Thought (CoT).
Approach:	They propose a chain-of-thought-like method to elicit models' potential abilities to generate rationales and answers that are based on attribution tracing and causal tracers to probe the internal working mechanism of the LLM.
Outcome:	The proposed method eliminates Toxic CoT problems and improves the model’s overall commonsense reasoning performance by 5.5%.

Copied to clipboard

Yi Liu, Xiangyu Liu, Xiangrong Zhu, Wei Hu

Reward-based Input Construction for Cross-document Relation Extraction (2024.acl-long)

Challenge:	Existing studies neglect attribute correlations formed by the intertwining of different attributes.
Approach:	They propose a multi-aspect controllable text generation method with disentangled counterfactual augmentation that alleviates imbalanced attribute correlations during training by disentanglement.
Outcome:	The proposed method outperforms state-of-the-art methods in imbalanced and balanced attribute correlation scenarios.

Copied to clipboard

Byeonghu Na, Suhyeon Jo, Yeongmin Kim, Il-chul Moon

Hyperspherical Multi-Prototype with Optimal Transport for Event Argument Extraction (2024.acl-long)

Challenge:	Existing methods for relation extraction are limited to a single sentence or document, but cross-document RE has emerged to address relations across multiple documents.
Approach:	They propose a sentence selector that extracts sentences based on relational evidence and rewards it with RE prediction scores.
Outcome:	The proposed method outperforms heuristic methods for relation extraction in multiple document scenarios.

Copied to clipboard

Guangjun Zhang, Hu Zhang, YuJie Wang, Ru Li, Hongye Tan, Jiye Liang

Understanding Retrieval Robustness for Retrieval-augmented Image Captioning (2024.acl-long)

Challenge:	Event Argument Extraction (EAE) aims to extract arguments for specified events from a text . previous work focused on long-distance dependencies of arguments, modeling co-occurrence relationships .
Approach:	They propose a model that takes inductive biases as targets to locate prototypes . they set multiple prototypes to represent each role to capture intra-class differences .
Outcome:	The proposed model achieves state-of-the-art on the RAMS and WikiEvents datasets.

Copied to clipboard

Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, Desmond Elliott

Semi-Supervised Spoken Language Glossification (2024.acl-long)

Challenge:	Recent retrieval-augmented models for image captioning are not perfect in practice.
Approach:	They propose to train a retrieval-augmented captioning model SmallCap by sampling retrieved captions from more diverse sets.
Outcome:	The proposed model is sensitive to tokens that appear in the majority of retrieved captions . the proposed model improves both in-domain and cross-domain performance .

Copied to clipboard

Huijie Yao, Wengang Zhou, Hao Zhou, Houqiang Li

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024.acl-long)

Challenge:	Spoken language glossification (SLG) aims to translate spoken language text into sign language gloss, i.e., written record of sign language.
Approach:	They propose a framework to translate spoken language into a sign language gloss . they use monolingual spoken language text to integrate it into training .
Outcome:	The proposed framework incorporates large-scale monolingual spoken language text into SLG training.

Copied to clipboard

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, Zhiyong Wu

InterrogateLLM: Zero-Resource Hallucination Detection in LLM-Generated Answers (2024.acl-long)

Challenge:	Existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e-book).
Approach:	They propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate curation of GUI ground data.
Outcome:	The proposed agent improves ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments.

Copied to clipboard

Yakir Yehuda, Itzik Malkiel, Oren Barkan, Jonathan Weill, Royi Ronen, Noam Koenigstein

F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods (2024.acl-long)

Challenge:	Existing methods for detecting hallucinations in large language models are limited due to their high frequency and high accuracy.
Approach:	They propose a method to detect hallucinations in large language models by repeating model-generated responses from its generated answer.
Outcome:	The proposed method achieves 87% hallucinations in a specific experiment without external knowledge.

Copied to clipboard

Yu Sun, Keyuchen Keyuchen, Shujie Wang, Peiji Li, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

Challenge:	Large language models (LLMs) have been evaluated for their instruction-following capabilities but lack references to their fundamental abilities.
Approach:	They propose a bilingual evaluation benchmark to evaluate the fundamental abilities of large language models including expression, commonsense and logic.
Outcome:	The proposed evaluation methods show higher correlation coefficients and larger distinction than other evaluators.

Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024.acl-long)

Copied to clipboard

Philipp Mondorf, Barbara Plank

Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback (2024.acl-long)

Challenge:	Recent advances in the domain of large language models (LLMs) have showcased their capability in executing deductive reasoning tasks.
Approach:	They examine inferential strategies employed by large language models through a detailed evaluation of their responses to propositional logic problems.
Outcome:	The proposed model shows that it displays reasoning patterns similar to humans, including strategies like supposition following or chain construction.

Copied to clipboard

Maria Lerner, Florian Dorner, Elliott Ash, Naman Goel

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2024.acl-long)

Challenge:	a growing body of work on learning from human feedback to align various aspects of machine learning systems with human values and preferences is focusing on the setting of fairness in content moderation.
Approach:	They propose to use human feedback to determine how two comments should be treated in content moderation to learn about human values and preferences.
Outcome:	The proposed approach is promising, as human preferences can often not be A: Some ladies like smaller men. B: Some men like smaller guys. Figure 1 shows that the proposed approach performs better for demographic intersections than a single classifier that gives equal weight to each annotation.

Copied to clipboard

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, Zhifang Sui

Numerical Reasoning

Challenge:	Existing methods for process-oriented math reward models rely on manual annotation.
Approach:	They propose a process-oriented math process reward model called Math-shepherd which assigns a reward score to each step of math problem solutions.
Outcome:	The proposed model breaks the bottleneck of manual supervision in two scenarios.

Large Language Models are not Fair Evaluators (2024.acl-long)

Copied to clipboard

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, Zhifang Sui

Improving Large Language Models in Event Relation Logical Prediction (2024.acl-long)

Challenge:	Existing evaluation frameworks that use large language models as referees are insufficient for accurately assessing their alignment with human intent.
Approach:	They propose a calibration framework to address positional bias in large language models as evaluators by manually annotating the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicun A Benchmark’s question prompt.
Outcome:	The proposed framework alleviates evaluation bias, resulting in closer alignment with human judgments.

Copied to clipboard

Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, Dongsheng Li

Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024.acl-long)

Challenge:	Event relation extraction tasks require rigorous logical reasoning and semantic comprehension, a challenge for narrative understanding and reasoning.
Approach:	They propose three approaches to endow LLMs with event relation logic to generate more coherent answers across different scenarios.
Outcome:	The proposed approach improves on a set of ERE tasks and provides insights for future work.

Copied to clipboard

Dingyi Yang, Chunru Zhan, Ziheng Wang, Biao Wang, Tiezheng Ge, Bo Zheng, Qin Jin

Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation (2024.acl-long)

Challenge:	Existing studies on dense video captioning and video story generation have made some progress, but in practical applications, we typically require synchronized narrations for ongoing visual scenes.
Approach:	They propose a task of Synchronized Video Storytelling to generate synchronized narrations for videos using a benchmark dataset with rich annotations.
Outcome:	The proposed framework can generate narrations with the guidance of the generated or predefined storyline and human evaluations validate the effectiveness.

Copied to clipboard

Wenting Chen, Linlin Shen, Jingyang Lin, Jiebo Luo, Xiang Li, Yixuan Yuan

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step (2024.acl-long)

Challenge:	Fine-grained vision-language models (VLMs) have been widely used for inter-modality local alignment between fixed patches and textual words, but they provide incomplete representations of lesions.
Approach:	They propose an Adaptive patch-word Matching model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explicit explanations.
Outcome:	The proposed model correlates chest X-ray image regions with words in medical reports and provides explanations for the generation process.

Copied to clipboard

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, Feng Zhao

Are LLM-based Evaluators Confusing NLG Quality Criteria? (2024.acl-long)

Challenge:	Existing studies evaluate the tool utilization ability of large language models based on the final output or only consider the single-step tool calling.
Approach:	They propose a new approach to evaluate the tool utilization capability of large language models (LLMs) they decompose the tool usage into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review.
Outcome:	The proposed model exhibits consistency with the outcome-oriented evaluation and provides a more fine-grained analysis of the capabilities of LLMs.

Copied to clipboard

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, Xiaojun Wan

Synergistic Interplay between Search and Large Language Models for Information Retrieval (2024.acl-long)

Challenge:	Existing studies show that LLMs confuse evaluation criteria, which reduces their reliability.
Approach:	They propose a hierarchical classification system for 11 common aspects with corresponding different evaluation criteria.
Outcome:	The proposed system is based on 11 common aspects with different evaluation criteria.

Copied to clipboard

Jiazhan Feng, Chongyang Tao, Xiubo Geng, Tao Shen, Can Xu, Guodong Long, Dongyan Zhao, Daxin Jiang

Linear Transformers with Learnable Kernel Functions are Better In-Context Models (2024.acl-long)

Challenge:	Information retrieval (IR) is an indispensable technique for locating relevant resources from vast amounts of data.
Approach:	They propose a framework that facilitates information refinement through synergy between RMs and LLMs.
Outcome:	The proposed framework improves the performance of large-scale retrieval benchmarks on web searches and low-resource retrieval tasks.

Copied to clipboard

Yaroslav Aksenov, Nikita Balagansky, Sofia Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov

Temperature-scaling surprisal estimates improve fit to human reading times – but does it do so for the “right reasons”? (2024.acl-long)

Challenge:	Current Language Models (LMs) lack essential In-Context Learning capabilities, a domain where the Transformer excels.
Approach:	They propose a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks.
Outcome:	The proposed model amplifies its In-Context Learning abilities on the Pile dataset.

Copied to clipboard

Tong Liu, Iza Škrjanec, Vera Demberg

Beyond Recognising Entailment: Formalising Natural Language Inference from an Argumentative Perspective (2024.acl-long)

Challenge:	a wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word’s negative log probability in context.
Approach:	They propose to use large language models to predict the surprisal of a word's negative log probability in context to test their predictive power.
Outcome:	The proposed model can be significantly more accurate than humans because it has more data.

Copied to clipboard

Ameer Saadat-Yazdi, Nadin Kökciyan

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (2024.acl-long)

Challenge:	Existing methods for recognizing textual entailment lack a standardized definition of inference, making it difficult to compare methods trained on different datasets.
Approach:	They propose a rigorous approach to align entailment recognition with argumentation theory by using a tool to assist humans in annotating arguments according to the PTA.
Outcome:	The proposed model is based on a human-trained dataset and provides insights into non-expert annotator training.

Copied to clipboard

Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu

CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models (2024.acl-long)

Challenge:	Existing language models that use discrete representations for unified processing of various modalities are limited to text generation and do not include multimodal output.
Approach:	They propose a multimodal language model that utilizes discrete representations for unified processing of various modalities.
Outcome:	The proposed model can be trained stably without any alterations to existing models or training paradigms.

Copied to clipboard

Zixin Chen, Hongzhan Lin, Ziyang Luo, Mingfei Cheng, Jing Ma, Guang Chen

Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024.acl-long)

Challenge:	Current methods for multimodal sarcasm target identification focus on superficial indicators in an end-to-end manner, overlooking the nuanced understanding of multimodal content.
Approach:	They propose a multimodal sarcasm target identification framework with a coarse-to-fine paradigm by augmenting sarcasm explainability with reasoning and pre-training knowledge.
Outcome:	The proposed framework outperforms state-of-the-art methods and exhibits explainability in deciphering sarcasm as well.

Copied to clipboard

Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Xiaoming Wang, Jiulong Shan, Meng Cao, Lijie Wen

Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines (2024.acl-long)

Challenge:	Existing methods to evaluate preference data without human annotations are difficult . et al., 2022b) is effective for aligning large language models with human expectations .
Approach:	They propose a method to evaluate the response preference using output probabilities under contrastive prompts.
Outcome:	The proposed method could surpass the RLHF method without human-annotated preference data.

Copied to clipboard

Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, Yonatan Belinkov

Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models (2024.acl-long)

Challenge:	Text-to-image diffusion models use a latent text prompt to guide image generation . however, the process by which the encoder produces the text representation is unknown .
Approach:	They propose a method for analyzing the text encoder of T2I models by generating images from its intermediate representations.
Outcome:	The proposed method provides valuable insights into the text encoder component in T2I pipelines.

Copied to clipboard

Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Xin Zhao, Fuzheng Zhang, Di Zhang, Kun Gai

Robust Singing Voice Transcription Serves Synthesis (2024.acl-long)

Challenge:	Existing studies overlook the multi-turn instruction following ability of large language models (LLMs) Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi- turn instruction following.
Approach:	They propose a method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis, and a context-aware preference optimization strategy to further enhance LLMs for complex queries.
Outcome:	The proposed method improves existing LLMs by up to 7.2% in multi-turn instruction following.

Copied to clipboard

Ruiqi Li, Yu Zhang, Yongqi Wang, Zhiqing Hong, Rongjie Huang, Zhou Zhao

VulLibGen: Generating Names of Vulnerability-Affected Packages via a Large Language Model (2024.acl-long)

Challenge:	Current AST methods struggle with accuracy and robustness when used for practical annotation.
Approach:	They propose a model that converts singing recordings into note sequences for automatic annotation of singing datasets.
Outcome:	The proposed model outperforms baseline models on enlarged, automatically annotated datasets.

Copied to clipboard

Tianyu Chen, Lin Li, ZhuLiuchuan ZhuLiuchuan, Zongyang Li, Xueqing Liu, Guangtai Liang, Qianxiang Wang, Tao Xie

Self-Modifying State Modeling for Simultaneous Machine Translation (2024.acl-long)

Challenge:	Existing work on affected package identification is limited by large language models . a recent study shows that 84% third-party packages contain security vulnerabilities .
Approach:	They propose a method to use LLM to generate the affected package . they propose supervised fine-tuning, retrieval augmented generation and a local search algorithm .
Outcome:	The proposed method has an average precision of 0.806 for identifying vulnerable packages in four most popular ecosystems in GitHub Advisory.

Copied to clipboard

Donglei Yu, Xiaomian Kang, Yuchen Liu, Yu Zhou, Chengqing Zong

MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation (2024.acl-long)

Challenge:	Existing methods for simultaneous machine translation fail to optimize the policy . existing methods require building a decision path to learn the policy, but they cannot explore all potential paths .
Approach:	They propose a new training paradigm that uses a read/write policy to optimize the policy . existing methods usually require building a decision path to learn a suitable policy a user makes .
Outcome:	The proposed model outperforms strong baselines and allows offline models to acquire SiMT ability with fine-tuning.

Copied to clipboard

Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, Kwan-Yee Wong

BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents (2024.acl-long)

Challenge:	Embodied agents equipped with GPT as their brains have extraordinary decision-making and generalization abilities across various tasks.
Approach:	They propose a map-based agent that introduces an online linguistic-formed map to encourage global exploration.
Outcome:	The proposed agent achieves state-of-the-art zero-shot performance on R2R and REVERIE simultaneously.

Copied to clipboard

Yifei Wang, Dizhan Xue, Shengjie Zhang, Shengsheng Qian

DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy (2024.acl-long)

Challenge:	Xi et al., 2023) show that LLM-based intelligent agents are vulnerable to backdoor attacks.
Approach:	They propose to build LLM-based intelligent agents using trained LLMs and fine-tune them on data for the agent task.
Outcome:	The proposed methods are robust even after fine-tuning on trusted data.

Copied to clipboard

Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, Rui Yan

LePaRD: A Large-Scale Dataset of Judicial Citations to Precedent (2024.acl-long)

Challenge:	Recent advances in large language models (LLMs) have revolutionized the landscape of reasoning tasks.
Approach:	They propose a new approach that rethinks the reasoning process as an evolution from indeterminacy to determinacy.
Outcome:	The proposed model surpasses all baselines on various logical reasoning benchmarks.

Copied to clipboard

Robert Mahari, Dominik Stammbach, Elliott Ash, Alex Pentland

To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering (2024.acl-long)

Challenge:	Legal passage retrieval is a practice-oriented task that seeks to predict relevant passages from precedential court decisions given the context of a legal argument.
Approach:	They present a dataset which aims to facilitate work on legal passage retrieval . they extensively evaluate various approaches and find classification-based retrieval works best .
Outcome:	The proposed dataset aims to facilitate work on legal passage retrieval . it shows that classification-based retrieval seems to work best .

Copied to clipboard

Giacomo Frisoni, Alessio Cocchieri, Alex Presepi, Gianluca Moro, Zaiqiao Meng

MERA: A Comprehensive LLM Evaluation in Russian (2024.acl-long)

Challenge:	Medical open-domain question answering requires substantial access to specialized knowledge.
Approach:	They propose a framework that generates multiple-choice questions from a set of open-book parameters and a small-scale reader that can outcompete closed-book questions by 706x using fewer parameters.
Outcome:	The proposed framework outperforms closed-book models on MedQA-USMLE, MedMCQA, and MMLU while using up to 706x fewer parameters.

Copied to clipboard

Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, Sergey Markov

SC2: Towards Enhancing Content Preservation and Style Consistency in Long Text Style Transfer (2024.acl-long)

Challenge:	Recent advances in foundation models have led to the emergence of powerful Large Language Models (LLMs), which showcase unprecedented tasksolving capabilities.
Approach:	They propose a method to evaluate FMs and LMs in fixed zero- and few-shot instruction settings that can be extended to other modalities.
Outcome:	The proposed evaluation methodology includes an open-source code base and a leaderboard with a submission system.

Copied to clipboard

Jie Zhao, Ziyu Guan, Cai Xu, Wei Zhao, Yue Jiang

Dodo: Dynamic Contextual Compression for Decoder-only LMs (2024.acl-long)

Challenge:	Existing methods for short TST are difficult to implement and can cause content degradation.
Approach:	They propose a method to vary the style polarity of text while preserving semantic content.
Outcome:	The proposed method improves over baselines and is highly efficient.

Copied to clipboard

Guanghui Qin, Corby Rosset, Ethan Chau, Nikhil Rao, Benjamin Van Durme

POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation (2024.acl-long)

Challenge:	Existing approaches to NLP are sparsifying attention patterns or approximating the attention computation with kernel methods.
Approach:	They propose a method for dynamic contextual compression for decoder-only LMs.
Outcome:	The proposed method reduces the cost of self-attention to a fraction of typical time and space.

Copied to clipboard

Shilong Pan, Zhiliang Tian, Liang Ding, Haoqi Zheng, Zhen Huang, Zhihua Wen, Dongsheng Li

NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism (2024.acl-long)

Challenge:	Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data.
Approach:	They propose a method that uses a dynamic graph to organize auxiliary languages in prompts to improve LRL translations.
Outcome:	The proposed method improves translation accuracy in low-resource languages (LRLs) using auxiliary language pairs and synthetic pseudo-parallel data.

Copied to clipboard

Miao Li, Ming-Bin Chen, Bo Tang, ShengbinHou ShengbinHou, Pengyu Wang, Haiying Deng, Zhiyu Li, Feiyu Xiong, Keming Mao, Cheng Peng, Yi Luo

MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization (2024.acl-long)

Challenge:	a novel evaluation framework assesses the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism.
Approach:	They propose to use a benchmark dataset to assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism.
Outcome:	The proposed evaluation framework is based on a dataset of 1,267 test samples in 24 news domains.

Copied to clipboard

Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, Jiajun Chen

Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training (2024.acl-long)

Challenge:	Existing models exhibit inconsistent reasoning abilities across different languages . existing models lack consistency across languages due to imbalance of training data .
Approach:	They propose a multilingual alignment-as-preference optimization framework to align reasoning processes in other languages with the dominant language.
Outcome:	The proposed framework improves multilingual reasoning across languages on three benchmarks.

Copied to clipboard

Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, Ruifeng Xu

Predicting Text Preference Via Structured Comparative Reasoning (2024.acl-long)

Challenge:	Large language models (LLMs) exhibit substantial capabilities yet face challenges such as hallucination, outdated knowledge, and untraceable reasoning processes.
Approach:	They propose a retrieval-augmented generation approach that leverages adaptive adversarial training to dynamically adjust the model’s training process in response to retrieval noises.
Outcome:	The proposed approach improves the performance of the LLaMA-2 7B model under diverse noise conditions.

Copied to clipboard

Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurzion, Alexander Rush, Jialu Liu, Michael Bendersky

CoELM: Construction-Enhanced Language Modeling (2024.acl-long)

Challenge:	Existing approaches to comparative reasoning rely on pretraining or fine-tuning models at the cost of massive human annotation and computation.
Approach:	They propose a model that prompts LLMs to generate structured intermediate comparisons by proposing aspects for comparison, followed by generating textual comparisons under each aspect.
Outcome:	The proposed model significantly reduces hallucination and improves consistency across various NLP tasks.

Copied to clipboard

Lvxiaowei Xu, Zhilin Gong, Jianhua Dai, Tianxiang Wang, Ming Cai, Jiawei Peng

Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)

Challenge:	Recent studies show that integrating constructional information can improve the performance of pre-trained language models.
Approach:	They propose a construction-Enhanced language model that embeds constructional semantics into language models for natural language generation.
Outcome:	The proposed model outperforms existing models on various benchmarks.

Copied to clipboard

Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao

On the Impact of Calibration Data in Post-training Quantization and Pruning (2024.acl-long)

Challenge:	Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video.
Approach:	They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights.
Outcome:	The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio.

Copied to clipboard

Miles Williams, Nikolaos Aletras

SymKGQA: Few-Shot Knowledge Graph Question Answering via Symbolic Program Generation and Execution (2024.acl-long)

Challenge:	Quantization and pruning are the foundations of compression for large language models . however, no prior work has investigated how calibration data impacts performance of compression methods.
Approach:	They propose an empirical study on the effect of calibration data on LLM performance.
Outcome:	The proposed methods improve performance in a post-training setting.

Copied to clipboard

Prerna Agarwal, Nishant Kumar, Srikanta Bedathur

Meta-Task Prompting Elicits Embeddings from Large Language Models (2024.acl-long)

Challenge:	Recent advances in Large Language Models have led to low-level LFs that are limited to the knowledge of underlying LLM about the LF.
Approach:	They propose a framework that generates a symbolic LF in a few-shot setting using Large Language Models.
Outcome:	The proposed framework outperforms all other few-shot and many fully-supervised KGQA approaches.

Copied to clipboard

Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, Andrew Yates

A Sentiment Consolidation Framework for Meta-Review Generation (2024.acl-long)

Challenge:	Existing methods for large language modeling are based on task-related instructions or prompts.
Approach:	They propose a method for generating high-quality sentence embeddings from Large Language Models (LLMs) using meta-task prompts.
Outcome:	The proposed method produces high-quality sentences without fine-tuning . it excels on STS benchmarks and in downstream tasks, surpassing models with similar prompts .

Copied to clipboard

Miao Li, Jey Han Lau, Eduard Hovy

Revisiting Structured Sentiment Analysis as Latent Dependency Graph Parsing (2024.acl-long)

Challenge:	Recent advances in abstractive text summarization have created plausible summaries, but it is unclear if they truly possess the capability of information consolidation to generate summary.
Approach:	They propose to prompt large language models to generate meta-reviews and use evaluation metrics to assess the quality of generated meta- reviews.
Outcome:	The proposed framework proves that human meta-reviewers follow a framework of sentiment consolidation to write meta- reviews compared with prompting them with simple instructions.

Copied to clipboard

Chengjie Zhou, Bobo Li, Hao Fei, Fei Li, Chong Teng, Donghong Ji

OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)

Challenge:	Structured Sentiment Analysis (SSA) is a problem of bi-lexical dependency graph parsing due to the internal structures of spans neglected.
Approach:	They propose to use latent spans as latent subtrees to model internal structures of spans and leverage TreeCRFs to extract the complete opinion tuple from a sentence.
Outcome:	The proposed method performs significantly better than all previous bi-lexical methods, achieving new state-of-the-art.

Copied to clipboard

Yifan Peng, Yui Sudo, Muhammad Shakeel, Shinji Watanabe

Do Large Language Models Latently Perform Multi-Hop Reasoning? (2024.acl-long)

Challenge:	Autoregressive models can be slower during inference and have potential risks of hallucination.
Approach:	They propose an encoder-only speech foundation model based on Connectionist Temporal Classification.
Outcome:	The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification.

Copied to clipboard

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, Sebastian Riedel

MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning (2024.acl-long)

Challenge:	Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is".
Approach:	They examine whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is".
Outcome:	The proposed model can latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is".

Copied to clipboard

Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, Chang Zhou

Harnessing Toulmin’s theory for zero-shot argument explication (2024.acl-long)

Challenge:	In math reasoning with large language models, fine-tuning data augmentation by query evolution and diverse reasoning paths is empirically verified effective.
Approach:	They propose to fine-tune data augmentation by query evolution and diverse reasoning paths.
Outcome:	The proposed model achieves new state-of-the-art on GSM8K and MATH.

Copied to clipboard

Ankita Gupta, Ethan Zuckerman, Brendan O’Connor

BinaryAlign: Word Alignment as Binary Sequence Labeling (2024.acl-long)

Challenge:	To better analyze informal arguments on public forums, we propose the task of argument explication, which makes explicit a text’s argumentative structure and implicit reasoning by outputting triples of propositions claim, reason warrant.
Approach:	They propose to prompt generative large language models to output explicit argument components proposed by Toulmin by prompting with the theory name.
Outcome:	The proposed method evaluates the outputs’ coverage and validity through a human study and automatic evaluation based on prior argumentation datasets and performs robustness checks over alternative LMs, prompts, and argumentation theories.

Copied to clipboard

Gaetan Latouche, Marc-André Carbonneau, Benjamin Swanson

Quantifying the Persona Effect in LLM Simulations (2024.acl-long)

Challenge:	State-of-the-art word alignment training methods require a different class depending on the availability of gold data for a particular language pair.
Approach:	They propose a novel word alignment technique based on binary sequence labeling that outperforms existing approaches in both scenarios.
Outcome:	The proposed method outperforms existing models on non-English language pairs and performs stratified error analysis over alignment error type.

Copied to clipboard

Tiancheng Hu, Nigel Collier

Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)

Challenge:	Large language models (LLMs) have shown remarkable promise in simulating human language and behavior.
Approach:	They investigate how integrating persona variables—demographic, social, and behavioral factors—impacts LLMs’ ability to simulate diverse perspectives.
Outcome:	The proposed model improves on a zero-shot model with persona prompting.

Copied to clipboard

Nishant Balepur, Abhilasha Ravichander, Rachel Rudinger

Retrieval Augmented Fact Verification by Synthesizing Contrastive Arguments (2024.acl-long)

Challenge:	Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts .
Approach:	They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone .
Outcome:	The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain.

Copied to clipboard

Zhenrui Yue, Huimin Zeng, Lanyu Shang, Yifan Liu, Yang Zhang, Dong Wang

SyllabusQA: A Course Logistics Question Answering Dataset (2024.acl-long)

Challenge:	Existing methods to verify claim credibility rely on embedded knowledge or unreliable context.
Approach:	They propose retrieval augmented fact verification through the synthesis of contrasting arguments (RAFTS) they use an embedding model to identify informative demonstrations and in-context prompts to generate the prediction and explanation.
Outcome:	The proposed method outperforms existing methods with smaller LLMs or unreliable contexts.

Copied to clipboard

Nigel Fernandez, Alexander Scarlatos, Andrew Lan

Question Answering

Challenge:	We introduce Fact-QA, an LLM-based evaluation metric to evaluate the factuality of predicted answers.
Approach:	They propose to use an open-source dataset to analyze logistics-related question-answer pairs in a logistics-based course.
Outcome:	The proposed approach performs close to humans on traditional metrics of textual similarity, but there is a significant gap between them and humans in terms of fact precision.

MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models (2024.acl-long)

Copied to clipboard

Yilin Wen, Zifeng Wang, Jimeng Sun

AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts (2024.acl-long)

Challenge:	Large language models suffer from limitations such as difficulty in incorporating new knowledge, generating hallucinations, and explaining their reasoning process.
Approach:	They propose a pipeline that leverages knowledge graphs to enhance LLMs’ inference and transparency by eliciting the mind map of LLM's, which reveals their reasoning pathways based on the ontology of knowledge.
Outcome:	The proposed pipeline enables LLMs to comprehend KG inputs and infer with a combination of implicit and external knowledge.

Copied to clipboard

Daniel Braun, Florian Matthes

Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks (2024.acl-long)

Challenge:	Annotated contracts are laborious task performed by companies, law firms, NGOs and the scientific community.
Approach:	They present a corpus of 3,764 clauses from German consumer contracts annotated by legal experts with a clause in the contract.
Outcome:	The proposed framework outperforms openly available models in detecting potentially void clauses.

Copied to clipboard

Charlotte Siska, Katerina Marazopoulou, Melissa Ailem, James Bono

Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuning (2024.acl-long)

Challenge:	Using benchmarks to evaluate Large Language Models is inconsistent with the assumption that the test prompts within a benchmark represent a random sample from some real-world distribution of interest.
Approach:	They propose to use a model's average performance across the test prompts of a benchmark to evaluate its performance.
Outcome:	The results show that the correlation between model performance across test prompts and the test prompt can change model rankings on major benchmarks.

Copied to clipboard

Eric Pasewark, Kyle Montgomery, Kefei Duan, Dawn Song, Chenguang Wang

Bridging the Preference Gap between Retrievers and LLMs (2024.acl-long)

Challenge:	Existing methods to solve compositional tasks are limited by complexity and complexity.
Approach:	They propose a method that tunes large language models to break down a problem into subproblems, solve those subproblem, and combine the results.
Outcome:	The proposed method significantly improves model performance on three representative compositional tasks: integer addition, dynamic programming, and parity.

Copied to clipboard

Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

Large Language Models Can Learn Temporal Reasoning (2024.acl-long)

Challenge:	Existing studies on retrievers and LLMs treat them as separate components . a novel bridge model is proposed to optimize the relationship between the retriever and the LLM .
Approach:	They propose a framework that chains together supervised and reinforcement learning to train a bridge model that optimizes the connection between the retriever and the LLM.
Outcome:	Empirical results show that the proposed model optimizes the connection between the retriever and the LLM.

Copied to clipboard

Siheng Xiong, Ali Payani, Ramana Kompella, Faramarz Fekri

Learning Relational Decomposition of Queries for Question Answering from Tables (2024.acl-long)

Challenge:	Temporal reasoning (TR) is a fundamental ability of large language models (LLMs) however, there is neo-standard methods to perform TR, which are not suitable for large language model applications.
Approach:	They propose a framework to enhance temporal reasoning by using a latent representation, temporal graph (TG) instead of reasoning over the original context, they adopt a temporal representation that enhances TR learning.
Outcome:	The proposed framework improves the learning of language-based TR by incorporating a latent representation, temporal graph (TG) a synthetic dataset is constructed for fine-tuning LLMs on text-to-TG translation tasks and benchmarks.

Copied to clipboard

Raphaël Mouravieff, Benjamin Piwowarski, Sylvain Lamprier

Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People (2024.acl-long)

Challenge:	Existing approaches to Table Question-Answering focus on generating answers directly from inputs, but there are limitations when executing numerical operations.
Approach:	They propose to imitate a restricted subset of SQL-like algebraic operations and use them to generate a query.
Outcome:	The proposed methods bridge the gap between semantic parsing and direct answering methods and offer valuable insights into which types of operations should be predicted by a generative architecture and which should be executed by an external algorithm.

Copied to clipboard

Dun-Ming Huang, Pol Van Rijn, Ilia Sucholutsky, Raja Marjieh, Nori Jacoby

Pareto Optimal Learning for Estimating Large Language Model Errors (2024.acl-long)

Challenge:	Existing taxonomies or text corpora suffer from experimenter bias and are not representative of real-world distributions.
Approach:	They propose an iterative method for simultaneously eliciting conversational tones and sentences . they run 50 iterations with human participants and GPT-4 and obtain a dataset of sentences and frequent conversational tone.
Outcome:	The proposed method can be used to characterize the differences between humans and LLMs.

Copied to clipboard

Theodore Zhao, Mu Wei, J. Preston, Hoifung Poon

Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models (2024.acl-long)

Challenge:	Existing approaches for improving the correctness of LLMs include prompt engineering, retrieval methods, and a generative model.
Approach:	They propose a method that generates a risk score to estimate the probability of error in an LLM by integrating multiple sources of information.
Outcome:	The proposed method is well correlated with the true LLM error rate, thus facilitating error correction.

Copied to clipboard

Victor Agostinelli, Max Wild, Matthew Raffel, Kazi Fuad, Lizhong Chen

Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM (2024.acl-long)

Challenge:	Modern large language models (LLMs) contain billions of parameters and can perform a variety of downstream tasks.
Approach:	They propose an open-source framework for fine-tuning large language models (LLMs) they address key challenges facing LLMs fine- tuned for simultaneous translation .
Outcome:	The proposed framework validates classical SimulMT concepts and practices in the context of LLMs and explores adapting LLM fine-tuned for NMT to the task of Simul-LLM.

Copied to clipboard

Bochuan Cao, Yuanpu Cao, Lu Lin, Jinghui Chen

Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) have made significant advancements but can be misused to generate harmful content.
Approach:	They propose a Robustly Aligned LLM to defend against alignment-breaking attacks by retraining existing LLMs and using adversarial or handcrafted jailbreaking prompts.
Outcome:	The proposed model reduces attack success rates from nearly 100% to around 10% or less.

Copied to clipboard

Guanming Xiong, Junwei Bao, Wen Zhao

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (2024.acl-long)

Challenge:	Knowledge base question answering (KBQA) is a challenging task, particularly in parsing intricate questions into executable logical forms.
Approach:	They propose a framework to generate logical forms through direct interaction with knowledge bases (KBs) by annotating a dataset with step-wise reasoning processes.
Outcome:	The proposed framework achieves competitive results on the WebQuestionsSP, ComplexWebQuestIONS, KQA Pro, and MetaQA datasets with a minimal number of examples (shots). Importantly, the proposed model supports manual intervention, allowing for the iterative refinement of LLM outputs.

Copied to clipboard

Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024.acl-long)

Challenge:	Existing work on tool-augmented LLMs focuses on the broad coverage of tools and the flexibility of adding new tools.
Approach:	They propose a biologically inspired method for tool-augmented LLMs that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
Outcome:	The proposed method improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and outperforms GPT-4.

Copied to clipboard

Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, Jie Fu

Aligning Large Language Models with Human Preferences through Representation Engineering (2024.acl-long)

Challenge:	Existing methods for enhancing performance through increased use of expert knowledge often result in diminishing sparsity during expert selection.
Approach:	They propose a framework that integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning.
Outcome:	The proposed framework outperforms existing methods under identical conditions concerning the number of experts.

Copied to clipboard

Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang

CODIS: Benchmarking Context-dependent Visual Comprehension for Multimodal Large Language Models (2024.acl-long)

Challenge:	Existing methods for achieving this alignment involve employing reinforcement learning from human feedback (RLHF) Existing approaches involve using RLHF to fine-tune LLMs based on human labels . however, RLRF is susceptible to instability during fine- tuning and presents challenges in implementation.
Approach:	They propose to use reinforcement learning from human feedback to fine-tune large language models with human preferences to achieve precise control of model behavior.
Outcome:	Experiments show that RAHF can be used to capture and manipulate representations to align with a broad spectrum of human preferences or values rather than being confined to a single concept or function.

Copied to clipboard

Fuwen Luo, Chi Chen, Zihao Wan, Zhaolu Kang, Qidong Yan, Yingjie Li, Xiaolong Wang, Siyu Wang, Ziyue Wang, Xiaoyue Mi, Peng Li, Ning Ma, Maosong Sun, Yang Liu

ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation (2024.acl-long)

Challenge:	Multimodal large language models have demonstrated promising results in a variety of tasks that combine vision and language.
Approach:	They propose a benchmark to assess the ability of models to use contextual information in free-form text to enhance visual comprehension.
Outcome:	The proposed model fails to extract and utilize contextual information to improve understanding of images.

Copied to clipboard

Chen Huang, Yiping Jin, Ilija Ilievski, Wenqiang Lei, Jiancheng Lv

PolCLIP: A Unified Image-Text Word Sense Disambiguation Model via Generating Multimodal Complementary Representations (2024.acl-long)

Challenge:	Empirical studies demonstrate that Araida reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.
Approach:	They propose an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections.
Outcome:	Empirical studies show that Araida reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.

Copied to clipboard

Qihao Yang, Yong Li, Xuelin Wang, Fu Lee Wang, Tianyong Hao

Prompted Aspect Key Point Analysis for Quantitative Review Summarization (2024.acl-long)

Challenge:	Existing models for word sense disambiguation lack images or senses in textual and visual datasets.
Approach:	They propose a unified image-text WSD model that uses image-sense complementarity to generate visual representations for word senses and a disambiguation-oriented image-sensor dataset to provide implicit textual representations.
Outcome:	The proposed model achieves 2.53% F1-score increase over state-of-the-art models on Textual-WSD and 2.22% HR@1 improvement on Visual-WSS.

Copied to clipboard

An Tang, Xiuzhen Zhang, Minh Dinh, Erik Cambria

Ask Again, Then Fail: Large Language Models’ Vacillations in Judgment (2024.acl-long)

Challenge:	Recent abstractive approaches generate KPs based on sentences, resulting in overlapping and hallucinated opinions.
Approach:	They propose to use supervised learning to extract short sentences as key points before matching them to review comments for quantification of KP prevalence.
Outcome:	The proposed framework achieves state-of-the-art performance on Yelp and SPACE.

Copied to clipboard

Qiming Xie, Zengzhi Wang, Yi Feng, Rui Xia

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models (2024.acl-long)

Challenge:	Existing large language models often waver in their judgments when faced with follow-up questions . this is a challenge for generating reliable responses and building user trust .
Approach:	They propose a Follow-up Questioning Mechanism and two metrics to quantify this inconsistency . they also develop a framework that teaches large language models to maintain original correct judgments .
Outcome:	The proposed framework improves the general capabilities of large language models by allowing them to maintain original correct judgments.

Copied to clipboard

Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua

Multimodal Reasoning with Multimodal Knowledge Graph (2024.acl-long)

Challenge:	Large language models are used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown.
Approach:	They propose a benchmark for evaluating large language models using a well-organized taxonomy.
Outcome:	The proposed model is based on a well-organized taxonomy and compares it with other models.

Copied to clipboard

Junlin Lee, Yequan Wang, Jing Li, Min Zhang

Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting (2024.acl-long)

Challenge:	Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs.
Approach:	They propose a multimodal reasoning method that leverages multimodal knowledge graphs to learn rich and semantic knowledge across modalities.
Outcome:	The proposed method outperforms state-of-the-art models on multimodal question answering and multimodal analogy reasoning tasks while training on only a small fraction of parameters.

Copied to clipboard

Rikui Huang, Wei Wei, Xiaoye Qu, Shengzhe Zhang, Dangyang Chen, Yu Cheng

CARE: A Clue-guided Assistant for CSRs to Read User Manuals (2024.acl-long)

Challenge:	Existing literature on temporal knowledge Graph Forecasting lacks in-depth investigation into how confidence evolves with time.
Approach:	They propose a framework to model the temporal validity of rules for Temporal Knowledge Graph Forecasting (TKGF) they propose rule-adversarial negative sampling and time-aware negative sampling strategies to facilitate TempValid learning.
Outcome:	The proposed framework outperforms state-of-the-art (SOTA) rule-based methods on six TKGF datasets.

Copied to clipboard

Weihong Du, Jia Liu, Zujie Wen, Dingnan Jin, Hongru Liang, Wenqiang Lei

Challenge:	Current solutions don't fit the online custom service scenarios well due to the lack of attention to user questions and possible responses.
Approach:	They propose to build a clue-guided assistant for customer service representations (CSRs) that can provide accurate responses and explicitly show explainable paths about how to arrive at these responses.
Outcome:	The proposed assistant can reduce CSRs' reading burden and keep high service quality, in particular with >35% decrease in time spent and keeping a >0.75 ICC score.

Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes (2024.acl-long)

Copied to clipboard

Dingzirui Wang, Longxu Dou, Xuanliang Zhang, Qingfu Zhu, Wanxiang Che

PAGED: A Benchmark for Procedural Graphs Extraction from Documents (2024.acl-long)

Challenge:	Numerical reasoning is an essential ability for NLP systems to handle numeric information.
Approach:	They propose a numerical reasoning method that generates reliable reasoning processes by decomposing the answer formula and aim to train models to generate the process with synthesized data.
Outcome:	The proposed method improves on all five datasets with an average improvement of 1.8% compared with baselines and gpt-3.5-turbo.

Copied to clipboard

Weihong Du, Wenrui Liao, Hongru Liang, Wenqiang Lei

Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content Detectors (2024.acl-long)

Challenge:	Existing methods for extraction of procedural graphs from documents are not solving the task well.
Approach:	They propose a benchmark to test automatic extraction of procedural graphs from documents . they involve three advanced large language models and enhance them with a novel self-refine strategy .
Outcome:	The proposed benchmark systematically examines the progress of current methods and explores the potential of emerging large language models (LLMs) on this task.

Copied to clipboard

Ying Zhou, Ben He, Le Sun

RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024.acl-long)

Challenge:	Recent research indicates that AI-text detection systems lack robustness and struggle to effectively differentiate perturbed texts.
Approach:	They propose to evaluate the robustness of current detection systems by using black-box text perturbation methods and adversarial learning experiments.
Outcome:	The proposed methods assess the robustness of current detection models across perturbation granularities and the impact of perturbation data augmentation on the robustity of AI-text detectors.

Copied to clipboard

Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, Tong Zhang

The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models (2024.acl-long)

Challenge:	Retrieval-augmented generation (RAG) is a main technique for alleviating hallucinations in large language models.
Approach:	They propose to integrate RAG into large language models to analyze word-level hallucinations using a corpus of 18,000 naturally generated responses from diverse LLMs.
Outcome:	The proposed model can fine tune a relatively small LLM and achieve a competitive hallucination detection performance when compared to the existing prompt-based approaches.

Copied to clipboard

Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, Ji-Rong Wen

Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)

Challenge:	a growing number of researchers are studying the hallucination issue in large language models.
Approach:	They propose a hallucination detection benchmark and a method to detect hallucines in LLMs.
Outcome:	The proposed method detects hallucinations and mitigates them using different training stages.

Copied to clipboard

Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, Dacheng Tao

Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024.acl-long)

Challenge:	Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications.
Approach:	They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model.
Outcome:	The proposed method can achieve consistent and significant performance gains across all model types and sizes.

Copied to clipboard

Yunlong Liang, Fandong Meng, Jiaan Wang, Jinan Xu, Yufeng Chen, Jie Zhou

Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024.acl-long)

Challenge:	Multi-domain learning is a good solution for solving domain tasks but it requires retraining when adding a new domain.
Approach:	They propose to exploit unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting.
Outcome:	The proposed framework exploits unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting.

Copied to clipboard

Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Jinchuan Tian, Zhenhui Ye, Luping Liu, Zehan Wang, Ziyue Jiang, Xuankai Chang, Jiatong Shi, Chao Weng, Zhou Zhao, Dong Yu

Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages (2024.acl-long)

Challenge:	Large language models (LLMs) have been used for general-purpose interfaces across multiple tasks and languages.
Approach:	They propose to use large language models as a general-purpose interface across multiple tasks and languages.
Outcome:	The proposed model performs better on 200K hours of 6-language data for voice generation applications.

Copied to clipboard

Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, Hung-yi Lee

PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails (2024.acl-long)

Challenge:	Despite the rapid development of large language models, the language capabilities of most open-source LLMs are primarily focused on English due to data constraints.
Approach:	They propose a chat vector to equip pre-trained language models with instruction following and human value alignment via simple model arithmetic.
Outcome:	The proposed method can be extended to include various languages, base models, and chat vectors.

Copied to clipboard

Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, Atul Prakash

Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLMs-Powered Assistance (2024.acl-long)

Challenge:	Recent work has shown that large language models are susceptible to automated jailbreak attacks that induce them to generate harmful content.
Approach:	They propose a two-step attack strategy that leverages a universal adversarial prefix for the Guard Model and propagates this prefix to the response.
Outcome:	The proposed attack strategy is successful against several open-source and closed-source implementations of Guard Models.

Copied to clipboard

Bo Yuan, Yulin Chen, Yin Zhang, Wei Jiang

CLOMO: Counterfactual Logical Modification with Large Language Models (2024.acl-long)

Challenge:	Existing methods for learning from noisy labels are difficult to improve . existing methods identify noisy labels and use active learning to query experts .
Approach:	They propose a collaborative learning framework to combine LLMs and small models for learning from noisy labels.
Outcome:	The proposed framework outperforms state-of-the-art baselines on synthetic and real-world noise datasets.

Copied to clipboard

Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang, Linqi Song

Exploring Hybrid Question Answering via Program-based Prompting (2024.acl-long)

Challenge:	Existing studies on evaluating model reasoning are limited in both form and content.
Approach:	They propose a task to cultivate counterfactual thought processes within large language models and an evaluation metric to evaluate their natural language output instead of modeling the task as a multiple-choice problem.
Outcome:	The proposed evaluation metric aligns well with human preference.

Copied to clipboard

Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, Ting Liu

IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages (2024.acl-long)

Challenge:	Existing approaches to question answering over heterogeneous data are limited due to large scale of information and organic coupling of heterogenous data.
Approach:	They propose a program-based prompting framework for hybrid question answering tasks . it integrates various functions to perform hybrid information-seeking over data .
Outcome:	The proposed framework surpasses baseline systems and achieves the best performance under the fewshot settings.

Copied to clipboard

Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, Partha Talukdar

Simple but Effective Compound Geometric Operations for Temporal Knowledge Graph Completion (2024.acl-long)

Challenge:	IndicGenBench is the largest benchmark for evaluating large language models on user-facing generation tasks across a diverse set of 29 Indic languages .
Approach:	They evaluate large language models on user-facing generation tasks across 29 languages . they use human curation to provide multi-way parallel evaluation data for many under-represented languages a github repository .
Outcome:	IndicGenBench is the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set of 29 Indic languages covering 13 scripts and 4 language families.

Copied to clipboard

Rui Ying, Mengting Hu, Jianfeng Wu, Yalan Xie, Xiaoyi Liu, Zhunheng Wang, Ming Jiang, Hang Gao, Linlin Zhang, Renhong Cheng

Uncertainty Aware Learning for Language Model Alignment (2024.acl-long)

Challenge:	Current methods embed factual knowledge into continuous vector space and apply geometric operations to learn potential patterns in temporal knowledge graphs.
Approach:	They propose a temporal knowledge graph completion method that uses two geometric operations to learn missing facts in temporal graphs.
Outcome:	The proposed method significantly outperforms existing temporal knowledge graph embedding models.

Copied to clipboard

Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, Dacheng Tao

Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models (2024.acl-long)

Challenge:	Existing alignment strategies that focus on diverse and high-quality data often overlook the intrinsic uncertainty of tasks, learning all data samples equally.
Approach:	They propose to introduce the sample uncertainty into the alignment of different task scenarios by a simple fashion by setting the label smoothing value of training according to the uncertainty of individual samples.
Outcome:	The proposed model outperforms standard supervised fine-tuning on high-entropy tasks and complex low-entropic tasks.

Copied to clipboard

Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan

Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey (2024.acl-long)

Challenge:	Existing approaches to user satisfaction estimation are hard to interpret and lack generalizable patterns.
Approach:	They propose to use supervised prompting to extract interpretable user satisfaction signals from natural language utterances to tailor an LLM to USE using labeled examples.
Outcome:	The proposed method extracts interpretable signals of user satisfaction from natural language utterances more effectively than embedding-based approaches.

Copied to clipboard

Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, 林一冠林一冠, Bin Xu, Ren Bowen, Chong Feng, Yang Gao, Heyan Huang

Measuring Political Bias in Large Language Models: What Is Said and How It Is Said (2024.acl-long)

Challenge:	Existing benchmark-based evaluations cannot accurately reflect the performance of real-world applications.
Approach:	They propose a reliable strategy for domains to choose more robust LLMs for real-world applications.
Outcome:	The proposed strategy addresses the challenges faced by domains to choose more robust LLMs for real-world applications.

Copied to clipboard

Yejin Bang, Delong Chen, Nayeon Lee, Pascale Fung

Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use (2024.acl-long)

Challenge:	Existing benchmarks and measures focus on gender and racial biases, but political bias exists in LLMs and can lead to polarization and other harms in downstream applications.
Approach:	They propose to analyze the content and style of LLMs generated by political issues and propose a framework that can be scalable to other topics.
Outcome:	The proposed framework is easily scalable to other topics and is explainable.

Copied to clipboard

Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, Rui Yan

Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)

Challenge:	In this paper, we demonstrate that an inherent waveform pattern in the attention allocation of large language models significantly affects their performance in tasks demanding a high degree of context awareness.
Approach:	They propose a method that compensates an attention trough with an attention peak by a process to enhance the model's awareness to various contextual positions.
Outcome:	The proposed method improves the performance of a 7B model on the largest tool-use benchmark, comparable to that of GPT-4.

Copied to clipboard

Haoyi Wu, Kewei Tu

Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages (2024.acl-long)

Challenge:	Using a key-value cache, memory consumption is a bottleneck for high-throughput language models.
Approach:	They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput.
Outcome:	The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques.

Copied to clipboard

Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, Yang Liu

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations (2024.acl-long)

Challenge:	Contemporary large language models (LLMs) are pre-trained on multilingual corpora, but their performance lags behind in most languages compared to a few resource-rich languages.
Approach:	They propose a method that leverages the internal capabilities of large language models on resource-rich languages to enhance multilingual performance.
Outcome:	The proposed method improves multilingual performance while minimizing impact on original performance in resource-rich languages.

Copied to clipboard

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, Conghui He

Browse and Concentrate: Comprehending Multimodal Content via Prior-LLM Context Fusion (2024.acl-long)

Challenge:	Currently, many benchmarks evaluate the commonsense reasoning of large language models (LLMs), but most are English-based, limiting non-English evaluations.
Approach:	They propose to use Chinese commonsense reasoning to evaluate LLMs' commonsensing ability.
Outcome:	The proposed benchmark covers both globally known and Chinese-specific commonsense reasoning abilities and can be used as a reference for future research.

Copied to clipboard

Ziyue Wang, Chi Chen, Yiqi Zhu, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu

Model Composition for Multimodal Large Language Models (2024.acl-long)

Challenge:	Multimodal Large Language Models (MLLMs) lack understanding of multi-image and interleaved inputs due to the visual features encoded by frozen encoders before being fed into the LLM backbone.
Approach:	They propose a two phase paradigm to enable in-depth multimodal context fusion prior to feeding the features into LLMs.
Outcome:	The proposed paradigm boosts the performance on 7 multi-image scenarios, contributing to increments on average accuracy by 2.13% and 7.60% against strong MLLMs baselines with 3B and 11B LLMs, respectively.

Copied to clipboard

Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)

Challenge:	Existing methods for creating versatile MLLMs rely on joint training with paired instruction data, which is resource-intensive and challenging to extend to new modalities.
Approach:	They propose a new paradigm for multimodal large language models by reusing modality encoders and merging LLM parameters.
Outcome:	The proposed model retains the modal understanding capabilities of each original model.

Copied to clipboard

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra

Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup (2024.acl-long)

Challenge:	Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding.
Approach:	They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model.
Outcome:	The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint.

Copied to clipboard

Xuxin Cheng, Ziyu Yao, Yifei Xin, Hao An, Hongxiang Li, Yaowei Li, Yuexian Zou

Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models (2024.acl-long)

Challenge:	Multimodal machine translation (MMT) aims to improve the performance of machine translation with the help of visual information.
Approach:	They propose a multimodal machine translation mixup method that integrates visual information into conventional text-only neural machine translation systems.
Outcome:	The proposed method outperforms existing models on a multi-directional dataset with fewer parameters and achieves new state-of-the-art performance.

Copied to clipboard

Changjiang Gao, Jixing Li, Jiajun Chen, Shujian Huang

MIST: Mutual Information Maximization for Short Text Clustering (2024.acl-long)

Challenge:	Existing computational metric to quantify extent of meaning composition is lacking . despite extensive neurolinguistic research, understanding how meaning is constructed is difficult .
Approach:	They propose a computational metric to quantify the degree of meaning composition . they use key-value memory interpretation of transformer feed-forward network blocks to study meaning composition.
Outcome:	Experimental results show that the metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words, suggesting multifaceted nature of meaning composition during human sentence comprehension.

Copied to clipboard

Krissanee Kamthawee, Can Udomcharoenchaikit, Sarana Nutanong

Self-chats from Large Language Models Make Small Emotional Support Chatbot Better (2024.acl-long)

Challenge:	Existing methods for clustering short texts are inadequate due to the limited amount of information provided by each text sample.
Approach:	They propose a Mutual Information Maximization Framework for Short Text Clustering which maximizes mutual information between representations on sequence and token levels.
Outcome:	The proposed framework outperforms the state-of-the-art method in terms of Accuracy or Normalized Mutual Information in most cases.

Copied to clipboard

Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, Liqiang Nie

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment (2024.acl-long)

Challenge:	Large Language Models (LLMs) have shown strong generalization abilities to excel in various tasks, including emotion support conversations.
Approach:	They propose an iterative expansion framework to prompt large teacher model to curate an expansive emotion support dialogue dataset.
Outcome:	The proposed model outperforms the teacher model in some cases . the proposed model is based on an iterative expansion framework and is available on github.com/pandazzh2020/ExTES.

Copied to clipboard

Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, Jungwook Choi

Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs (2024.acl-long)

Challenge:	Quantization-aware direct preference optimization (QDPO) improves conversational abilities of quantized LLMs . token-flipping is a critical factor for degraded text generation quality .
Approach:	They propose a method that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities.
Outcome:	The proposed method outperforms established fine-tuning techniques on two instruction-tuned LLMs in various languages and models, setting a new benchmark for conversational chatbot development.

Copied to clipboard

Tianqing Fang, Zeming Chen, Yangqiu Song, Antoine Bosselut

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing (2024.acl-long)

Challenge:	Currently, language models struggle to generate commonsense inferences for complex tasks due to data scarcity and the difficulty of reasoning over multiple pieces of information.
Approach:	They propose a dataset to generate commonsense inferences from commonsensible data . they use a commonsence knowledge graph to extract and form questions from existing commonseense knowledge graphs.
Outcome:	The proposed dataset improves the ability of language models to reason about complex events without expensive human annotations.

Copied to clipboard

Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, Yang Yang

Challenge:	Large language models (LLMs) have demonstrated remarkable capabilities across a wide spectrum of tasks, but performance and reliability in certain specialized domains still fall short of expectations.
Approach:	They propose a unified generalist framework that facilitates seamless integration of multiple expert LLMs.
Outcome:	The proposed framework outperforms existing multi-LLM collaboration paradigms across six diverse expert domains.

Learning to Plan and Generate Text with Citations (2024.acl-long)

Copied to clipboard

Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, Mirella Lapata

Exploring Precision and Recall to assess the quality and diversity of LLMs (2024.acl-long)

Challenge:	Large language models (LLMs) are increasingly useful in information-seeking scenarios, ranging from answering simple questions to generating responses to search-like queries.
Approach:	They propose to use plan-based models to improve faithfulness, grounding, and controllability of generated content and its organization.
Outcome:	The proposed models improve faithfulness, grounding, and controllability of generated content and its organization.

Copied to clipboard

Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, Alexandre Allauzen

Aligning Large Language Models by On-Policy Self-Judgment (2024.acl-long)

Challenge:	Existing benchmarks for large language models are limited to specific tasks, but they are now widely available for a wide range of tasks.
Approach:	They propose a framework for large language models such as Llama-2 and Mistral that imports precision and recall metrics from image generation to text generation.
Outcome:	The proposed framework allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.

Copied to clipboard

Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, Youngjae Yu

IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning (2024.acl-long)

Challenge:	Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model for on-policy learning.
Approach:	They propose a new alignment framework that does on-policy learning and is parameter efficient . they propose Judge-augmented Supervised Fine-Tuning to train a single model to act as a policy and a judge.
Outcome:	The proposed framework outperforms baselines in preference benchmarks and rejecting sampling by itself improves performance without additional evaluator.

Copied to clipboard

Abhinav Joshi, Shounak Paul, Akshat Sharma, Pawan Goyal, Saptarshi Ghosh, Ashutosh Modi

JumpCoder: Go Beyond Autoregressive Coder via Online Modification (2024.acl-long)

Challenge:	Legal systems worldwide struggle with exponentially growing legal cases in various courts.
Approach:	They propose a benchmark for Indian legal text understanding and reasoning task that includes domain-specific tasks that address different aspects of the legal system.
Outcome:	The proposed benchmark for Indian legal text understanding and reasoning aims to address the gap between models and the ground truth.

Copied to clipboard

Mouxiang Chen, Hao Tian, Zhongxin Liu, Xiaoxue Ren, Jianling Sun

Challenge:	Existing code large language models lack reversibility and autoregressive sequential generation is incapable of correcting previous missing statements as humans do.
Approach:	They propose a model-agnostic framework that enables human-like online modification and non-sequential generation to augment code large language models.
Outcome:	The proposed framework enables human-like modification and non-sequential generation to augment code large language models.

Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)

Copied to clipboard

Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker

Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks (2024.acl-long)

Challenge:	Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources .
Approach:	They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions .
Outcome:	The goal is to build a human-curated instruction-following dataset spanning 65 languages.

Copied to clipboard

Anwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, Tanmoy Chakraborty

Split and Rephrase with Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning capabilities.
Approach:	They propose to use large language models to generalize from labeled examples of predefined tasks to novel tasks . they use biological neurons and the Transformer architecture to study the potential for information sharing across tasks.
Outcome:	The proposed model can generalize from labeled examples of predefined tasks to novel tasks despite no examples from the target task in the context.

Copied to clipboard

Antonio David Ponce Martínez, Thierry Etchegoyhen, Jesus Javier Calleja Perez, Harritxu Gete

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024.acl-long)

Challenge:	Split and Rephrase (SPRP) tasks require modelling complex grammatical aspects to provide optimal splits and appropriate rephrasing.
Approach:	They evaluate large language models on the Split and Rephrase task . they show they can provide large improvements over the state of the art on main metrics .
Outcome:	The proposed model outperforms the state-of-the-art model on the Split and Rephrase task on the main metric, but still lacks in splitting compliance.

Copied to clipboard

Lu Ye, Ze Tao, Yong Huang, Yang Li

AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)

Challenge:	Experiments show that ChunkAttention can speed up the self-attention kernel by 3.2-4.8 compared to the start-of-the-art implementation.
Approach:	They propose a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime.
Outcome:	The proposed module can speed up the self-attention kernel by 3.2-4.8 compared to the start-of-the-art implementation, with the length of the system prompt ranging from 1024 to 4096.

Copied to clipboard

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, Jie Tang

SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models (2024.acl-long)

Challenge:	Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment.
Approach:	They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.
Outcome:	The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references.

Copied to clipboard

Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution (2024.acl-long)

Challenge:	Existing methods to address catastrophic forgetting and knowledge transfer in large language models (LLMs) ignore potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfers simultaneously.
Approach:	They propose a Shared Attentive Learning & Selection module to align the PET learning and selection modules to address catastrophic forgetting and knowledge transfer simultaneously.
Outcome:	Experiments on two CL benchmarks show that the proposed framework is superior when scaled to different model sizes, different model architectures and unseen tasks.

Copied to clipboard

Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, Jinan Xu

Cross-Lingual Knowledge Editing in Large Language Models (2024.acl-long)

Challenge:	Existing parameter-efficient fine-tuning methods such as Low-Rank Adaptation ignore the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-uning outcomes.
Approach:	They propose a parameter-efficient low-rank Adaptation method that decomposes high-rank LoRA layers into structured single-rank components and allows dynamic pruning of parameter budget .
Outcome:	The proposed method outperforms LoRA and LoRA with the same parameter budget and performance.

Copied to clipboard

Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiarong Xu, Fandong Meng

Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques (2024.acl-long)

Challenge:	Knowledge editing is a promising technique to adapt large language models to new knowledge without retraining from scratch.
Approach:	They propose to use a multilingual dataset to translate a large-scale cross-lingual synthetic dataset from English to Chinese and then to evaluate their performance in Chinese.
Outcome:	The proposed method can change model performance on several special cases without retraining from scratch.

Copied to clipboard

Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri

Learning Task Decomposition to Assist Humans in Competitive Programming (2024.acl-long)

Challenge:	Recent work on sequence labelling has explored different strategies to mitigate the lack of manually annotated data for the large majority of the world languages.
Approach:	They propose to use the mask objective to exploit the few-shot capabilities of pre-trained language models to improve their performance.
Outcome:	The proposed model-transfer outperforms data-transference and fine-tuning outperformed few-shot methods for Argument Mining task.

Copied to clipboard

Jiaxin Wen, Ruiqi Zhong, Pei Ke, Zhihong Shao, Hongning Wang, Minlie Huang

An Entropy-based Text Watermarking Detection Method (2024.acl-long)

Challenge:	Using language models (LMs) to solve complex problems, humans might struggle to understand and repair flawed ones.
Approach:	They propose to automatically decompose complex problems into simpler pieces that correspond to specific subtasks and measure their assistive value.
Outcome:	The proposed method enables non-experts to solve 33.3% more problems and speeds them up by 3.3x .

Copied to clipboard

Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li, Irwin King

Enhancing Explainable Rating Prediction through Annotated Macro Concepts (2024.acl-long)

Challenge:	Existing text watermarking algorithms for large language models (LLMs) are effective in identifying machine-generated texts, but they are not effective in low-entropy scenarios.
Approach:	They propose an Entropy-based text watermarking detection method that takes into account the influence of token entropy to better reflect the degree of watermark detection.
Outcome:	The proposed method is training-free and fully automated.

Copied to clipboard

Huachi Zhou, Shuang Zhou, Hao Chen, Ninghao Liu, Fan Yang, Xiao Huang

How to Engage your Readers? Generating Guiding Questions to Promote Active Reading (2024.acl-long)

Challenge:	Existing models learn user and item embeddings and generate reasons based on these embedds.
Approach:	They propose a concept-based explanation framework that leverages macro concepts to bridge the gap between the user/item embeddings and the recommendation reasons.
Outcome:	Extensive experiments on three datasets prove the proposed model is superior to existing models.

Copied to clipboard

Peng Cui, Vilém Zouhar, Xiaoyu Zhang, Mrinmaya Sachan

Challenge:	Using questions in written text is an effective strategy to enhance readability, but what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied.
Approach:	They present a dataset of 10K in-text questions from textbooks and scientific articles and explore various approaches to generate such questions using language models.
Outcome:	The generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers’ memorization and comprehension.

Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024.acl-long)

Copied to clipboard

Zihao Yue, Liang Zhang, Qin Jin

Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation (2024.acl-long)

Challenge:	Extensive research has shed light on the origins of multimodal hallucinations, including the inability of vision encoders to represent finegrained visual details and model reliance on inherent parametric knowledge such as language priors and statistical biases.
Approach:	They propose to use EOS to terminate generation of large multimodal models by comparing the generated text with the image to mitigate multimodal hallucinations.
Outcome:	The proposed method significantly improves the hallucination performance of Large Multimodal Models without additional data or knowledge.

Copied to clipboard

Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Boyuan Pan, Heda Wang, Yao Hu, Kan Li

More frequent verbs are associated with more diverse valency frames: Efficient principles at the lexicon-grammar interface (2024.acl-long)

Challenge:	Existing methods to improve output quality without aggregating input tokens are limited by the complexity of aggregation of responses.
Approach:	They propose to extract and integrate segment-level commonalities from candidate samples to enhance performance of LLMs in open-ended and reasoning tasks.
Outcome:	The proposed method improves performance on reasoning, code generation and mathematical reasoning tasks without requiring additional models and overlooking the knowledge present among the candidates.

Copied to clipboard

Siyu Tao, Lucia Donatelli, Michael Hahn

Quantifying Generalizations: Exploring the Divide Between Human and LLMs’ Sensitivity to Quantification (2024.acl-long)

Challenge:	Existing evidence has focused on word-internal properties, such as Zipf's observation that more frequent words are optimized in form to minimize communicative cost.
Approach:	They propose to examine the hypothesis that efficient lexicon organization is also reflected in valency, or the combinations and orders of additional words and phrases a verb selects for in a sentence.
Outcome:	The proposed hypothesis is consistent with communicative efficiency principles.

Copied to clipboard

Claudia Collacciani, Giulia Rambelli, Marianna Bolognesi

Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds (2024.acl-long)

Challenge:	Generics are expressions used to communicate abstractions about categories . they allow for exceptions, and they are a powerful way to express knowledge about the world .
Approach:	They examine how large language models interpret generics to understand their meanings . they find that the presence of a generic sentence as context influences quantifiers based on the generalization .
Outcome:	The proposed models do not exhibit a strong sensitivity to quantification, the study finds . the results suggest that the presence of a generic sentence as context influences quantifiers .

Copied to clipboard

Giulia Rambelli, Emmanuele Chersoni, Claudia Collacciani, Marianna Bolognesi

CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024.acl-long)

Challenge:	Noun-noun compounds represent an important challenge for Natural Language Understanding . correct interpretation of noun-nomin compounds is essential for many applications .
Approach:	They test whether Large Language Models can interpret the semantic relation between nouns . they also test whether they can abstract from such knowledge to predict the relation .
Outcome:	The proposed models can interpret the semantic relation between nouns and compounds using analogical comparisons.

Copied to clipboard

Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, Rui Yan

Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond (2024.acl-long)

Challenge:	CharacterEval is a benchmark for comprehensive RPCA assessment in Chinese . authors show that Chinese LLMs exhibit more promising capabilities than GPT-4 in role-playing conversation.
Approach:	They propose a Chinese benchmark for comprehensive RPCA assessment . they use a dataset of Chinese role-playing dialogues and character profiles .
Outcome:	The proposed benchmark demonstrates that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation.

Copied to clipboard

Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, Tat-Seng Chua

Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction (2024.acl-long)

Challenge:	Recent advances in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively.
Approach:	They propose to enable multimodal large language models to memorize and recall images within their parameters.
Outcome:	The proposed model performs well even with large-scale image candidate sets.

Copied to clipboard

Yice Zhang, Jie Zeng, Weiming Hu, Ziyi Wang, Shiwei Chen, Ruifeng Xu

Learning to Generate Answers with Citations via Factual Consistency Models (2024.acl-long)

Challenge:	Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review.
Approach:	They propose a self-training framework with a pseudo-label scorer to assess the match between reviews and their pseudo-labels and train a generative model on it.
Outcome:	The proposed framework can predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review, and it can significantly improve self-training.

Copied to clipboard

Rami Aly, Zhiqiang Tang, Samson Tan, George Karypis

Improving Text Embeddings with Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations.
Approach:	They propose a weakly-supervised fine-tuning method leveraging factual consistency models (FCMs) they propose to generate citations to retrieved text passages alongside LLM-produced statements .
Outcome:	The proposed method improves on the ALCE few-shot citation benchmark with various instruction-tuned LLMs with an average improvement of 34.1, 15.5, and 10.5 citation F1 points.

Copied to clipboard

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei

Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024.acl-long)

Challenge:	Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages .
Approach:	They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps.
Outcome:	The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data.

Copied to clipboard

Tianduo Wang, Shichen Li, Wei Lu

UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset (2024.acl-long)

Challenge:	Recent studies focus on enhancing large-scale language models' reasoning abilities, but the research question of how to GSM8K Performance vs. computational cost remains.
Approach:	They propose to train small-scale language models with their own outputs to avoid relying on large models' outputs.
Outcome:	The proposed approach outperforms baseline models with comparable sizes while minimizing the required compute.

Copied to clipboard

Haoyu Wang, Shuo Wang, Yukun Yan, Xujia Wang, Zhiyu Yang, Yuzhuang Xu, Zhenghao Liu, Liner Yang, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun

Document-level Claim Extraction and Decontextualisation for Fact-Checking (2024.acl-long)

Challenge:	Open-source large language models (LLMs) have gained strength across diverse fields, but the majority of studies focus on English.
Approach:	They propose a knowledge-grounded data augmentation approach to elicit more language-specific knowledge of LLMs by enhancing their ability to serve users from different countries.
Outcome:	The proposed method can prune the language-agnostic supervised fine-tuning dataset without any performance degradation.

Copied to clipboard

Zhenyun Deng, Michael Schlichtkrull, Andreas Vlachos

PairCFR: Enhancing Model Training on Paired Counterfactually Augmented Data through Contrastive Learning (2024.acl-long)

Challenge:	Existing methods for document-level claim extraction focus on identifying and extracting claims from individual sentences.
Approach:	They propose a method for document-level claim extraction for fact-checking which aims to extract check-worthy claims from documents and decontextualise them so they can be understood out of context.
Outcome:	The proposed method extracts check-worthy claims from documents and decontextualises them so they can be understood out of context.

Copied to clipboard

Xiaoqi Qiu, Yongjie Wang, Xu Guo, Zhiwei Zeng, Yu Yue, Yuhong Feng, Chunyan Miao

LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction (2024.acl-long)

Challenge:	Recent research shows that training with CAD may lead models to overly focus on modified features while ignoring other important contextual information.
Approach:	They propose to use contrastive learning to promote global feature alignment and learning counterfactual clues to improve model performance.
Outcome:	The proposed method outperforms the state-of-the-art on out-of distribution (OOD) datasets.

Copied to clipboard

Hanzhang Zhou, Junlang Qian, Zijian Feng, Lu Hui, Zixiao Zhu, Kezhi Mao

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024.acl-long)

Challenge:	In-context learning (ICL) is an emerging ability of large-scale labeled data for document-level event argument extraction (EAE).
Approach:	They propose an explicit heuristic-driven demonstration construction approach that emphasizes task heurs in document-level event argument extraction tasks.
Outcome:	The proposed method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets.

Copied to clipboard

Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, Bing Qin

mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models (2024.acl-long)

Challenge:	Large Vision-Language Models (LVLMs) suffer from multimodal hallucinations . however, the generated hallucines could influence the models’ subsequent generation .
Approach:	They propose a framework to evaluate LVLMs' behaviors when encountering generated hallucinations and a method to revise the output distribution of LVLs with the one derived from the residual visual input.
Outcome:	The proposed framework reduces the performance of open-source LVLMs by 31%, indicating that they are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions.

Copied to clipboard

Huiyuan Lai, Malvina Nissim

GunStance: Stance Detection for Gun Control and Gun Regulation (2024.acl-long)

Challenge:	Existing models show low performance for lesser resourced languages, but they can achieve surprising performance on complex reasoning tasks in natural language processing (NLP).
Approach:	They compile the first large-scale multilingual math reasoning dataset, mCoT-MATH, covering eleven diverse languages.
Outcome:	The proposed model achieves impressive consistency across languages and comparable performance to close- and open-source models even of much larger sizes.

Copied to clipboard

Nikesh Gyawali, Iustin Sirbu, Tiberiu Sosea, Sarthak Khanal, Doina Caragea, Traian Rebedea, Cornelia Caragea

Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation (2024.acl-long)

Challenge:	Social media, especially Twitter, has been a melting pot for such debates.
Approach:	They propose to annotate tweets relevant to shooting events into three classes: In-Favor, Against, and Neutral.
Outcome:	The proposed approach outperforms supervised, semi-supervised, and LLM-based zero-shot models on the dataset.

Copied to clipboard

Zdeněk Kasner, Ondrej Dusek

Don’t Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection (2024.acl-long)

Challenge:	Existing benchmarks for data-to-text generation are saturated, and there is no way to test them.
Approach:	They propose a tool for collecting structured data from public APIs to analyze the behavior of open large language models on the task of data-to-text generation.
Outcome:	The proposed model can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd.

Copied to clipboard

Min Zhang, Jianfeng He, Taoran Ji, Chang-Tien Lu

Don’t Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation (2024.acl-long)

Challenge:	Several studies have examined whether large language models exhibit bias or discrimination against individuals or groups in terms of protected attributes like race, gender, or religion.
Approach:	They evaluate LLMs' ability to detect implicit hate speech and express confidence in their responses by considering prompt patterns and mainstream uncertainty estimation methods.
Outcome:	The proposed models exhibit two extremes: (1) excessive sensitivity towards groups or topics that may cause fairness issues, resulting in misclassifying benign statements as hate speech; (2) confidence scores for each method excessively concentrate on a fixed range, remaining unchanged regardless of the dataset’s complexity.

Copied to clipboard

Giorgos Vernikos, Andrei Popescu-Belis

Generating and Evaluating Plausible Explanations for Knowledge Graph Completion (2024.acl-long)

Challenge:	Neural machine translation models estimate probabilities of target sentences given source sentences, but these estimates may not align with human judgments.
Approach:	They propose a method that synthesizes translations using a quality estimation metric . they compare it with beam search and recent reranking techniques .
Outcome:	The proposed method outperforms other methods in large language models and multilingual translation models.

Copied to clipboard

Antonio Di Mauro, Zhao Xu, Wiem Ben Rim, Timo Sztyler, Carolin Lawrence

Knowledge Representation

Challenge:	Existing XAI approaches focus on learning algorithmic explanations, but are not plausible for users.
Approach:	They propose a path-based explanation method that meets human-centric explainability constraints and enhances plausibility.
Outcome:	The proposed method meets human-centric explainability constraints and enhances plausibility.

One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation (2024.acl-long)

Copied to clipboard

Tejpalsingh Siledar, Swaroop Nath, Sankara Muddu, Rupasai Rangaraju, Swaprava Nath, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera

LANDeRMT: Dectecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation (2024.acl-long)

Challenge:	Existing evaluation methods for opinion summarizations lack adequate opinion summary evaluation datasets.
Approach:	They propose a dataset that combines 7 dimensions crucial to opinion summaries . they propose OP-I-PROMPT, a dimension-independent prompt, and OP PROMPTS, .
Outcome:	The proposed model achieves a Spearman correlation of 0.70 with human judgments, surpassing prior methods.

Copied to clipboard

Shaolin Zhu, Leiyu Pan, Bo Li, Deyi Xiong

A Joint Coreference-Aware Approach to Document-Level Target Sentiment Analysis (2024.acl-long)

Challenge:	Existing studies have shown promising results in multilingual translation with limited bilingual supervision.
Approach:	They propose a Language-Aware Neuron Detecting and Routing framework that fine tunes LLMs to Machine Translation with diverse translation training data.
Outcome:	The proposed framework selectively finetunes LLMs to MT tasks with diverse translation training data.

Copied to clipboard

Hongjie Cai, Heqing Ma, Jianfei Yu, Rui Xia

VisDiaHalBench: A Visual Dialogue Benchmark For Diagnosing Hallucination in Large Vision-Language Models (2024.acl-long)

Challenge:	Existing work on aspect-based sentiment analysis (ABSA) focuses on sentence level, document level ABSA is more practical and requires holistic document-level understanding capabilities.
Approach:	They propose a learning framework to jointly model the DTSA task and the coreference resolution task using ChatGPT.
Outcome:	The proposed framework reduces the reliance on annotated coreference information and alleviates evaluation bias caused by missing coreference targets.

Copied to clipboard

Qingxing Cao, Junhao Cheng, Xiaodan Liang, Liang Lin

AutoDSL: Automated domain-specific language design for structural representation of procedures with constraints (2024.acl-long)

Challenge:	Despite the significant success of large vision-language models, some studies have revealed that LVLMs suffer from the hallucination problem when given long-term misleading textual history.
Approach:	They propose a visual dialogue hallucination evaluation benchmark VisDiaHalBench to investigate the halluciation problem of large vision-language models when given long-term misleading textual history.
Outcome:	The proposed benchmark consists of samples with five-turn questions about an edited image and its original version.

Copied to clipboard

Yu-Zhe Shi, Haofei Hou, Zhangqian Bi, Fanxu Meng, Xiang Wei, Lecheng Ruan, Qining Wang

Multipath parsing in the brain (2024.acl-long)

Challenge:	Domain-specific Language (DSL) is an effective tool to express constraints structurally, but requires case-by-case hand-crafting.
Approach:	They propose a framework to automate domain-specific language constraint design . they propose 'autoDSL' framework to optimize syntactic and semantic constraints .
Outcome:	The framework automates constraint design across domains and abstracts semantic constraints.

Copied to clipboard

Berta Franzluebbers, Donald Dunagan, Miloš Stanojević, Jan Buys, John Hale

Syntactic Text Processing

Challenge:	a major unsolved problem in computational psycholinguistics is determining whether human comprehension considers a single analysis path 1 at a time.
Approach:	They compare syntactic surprisal from a state-of-the-art dependency parser with fMRI data . they find evidence for multipath parsing in English and Chinese data based on fm data a major unsolved problem in computational psycholinguistics is determining whether human sentence comprehension considers a single analysis path 1 at a time .
Outcome:	The proposed model shows that human parsing is multipath, with a higher r2 increase for multipath surprisal than single-path surpresal.

Search-Adaptor: Embedding Customization for Information Retrieval (2024.acl-long)

Copied to clipboard

Jinsung Yoon, Yanfei Chen, Sercan Arik, Tomas Pfister

Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)

Challenge:	Existing methods to embed text in large language models are limited to zero-shot setups and can be integrated with any LLM.
Approach:	They propose a method for customizing LLMs for information retrieval by modifying the embeddings generated by pre-trained LLM models and can be integrated with any LLM.
Outcome:	The proposed method improves performance on English, multilingual, and multimodal retrieval datasets by 5% over 14 BEIR datasets.

Copied to clipboard

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

Challenge:	Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning.
Approach:	They propose to use Proximal Policy Optimization to align large language models to human preferences.
Outcome:	The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO.

VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2024.acl-long)

Copied to clipboard

Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, Wenhu Chen

Tree Transformer’s Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects (2024.acl-long)

Challenge:	Existing metrics for conditional image generation are opaque and lack explainability . evaluators of these metrics have limited ability to evaluate image synthesis tasks .
Approach:	They propose a Visual Instruction-guided Explainable metric for evaluating conditional image models.
Outcome:	The proposed model achieves a high Spearman correlation with human evaluations, but is weaker than GPT-4o and GPT-v in evaluating synthetic images.

Copied to clipboard

Lingling Zhou, Suzan Verberne, Gijs Wijnholds

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs (2024.acl-long)

Challenge:	Prepositional phrase attachment ambiguity is structural in nature, while garden path constructions are incremental in nature.
Approach:	They pretrain and evaluate an unsupervised Transformer model that induces tree representations internally and compare it to a pretrained supervised BiLSTM model.
Outcome:	The Tree Transformer model induces tree representations internally, but its parsing ability is inferior to the supervised BiLSTM model, and it is not as sensitive to lexical cues as other large LSTM models.

Copied to clipboard

Elan Markowitz, Anil Ramakrishna, Jwala Dhamala, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing (2024.acl-long)

Challenge:	Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge.
Approach:	They propose a zero-shot reasoning algorithm that augments black-box LLMs with one or more KGs.
Outcome:	The proposed algorithm significantly improves performance on question answering and KG question answering tasks.

Copied to clipboard

Freda Shi, Kevin Gimpel, Karen Livescu

ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation (2024.acl-long)

Challenge:	Recent work has proposed a new task of textless speech constituency parsing that uses textless parsers to parse spoken word boundaries over automatically recognized spoken word borders.
Approach:	They propose a metric that compares a constituency parse tree over spoken word boundaries with a ground-truth parser tree over written words.
Outcome:	The proposed metric shows higher tolerance to syntactically plausible parses than PARSEVAL.

Copied to clipboard

Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan Reddy, Sunipa Dev

Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking (2024.acl-long)

Challenge:	Existing approaches for evaluating stereotypes have a noticeable lack of coverage of global identity groups and their associated stereotypes.
Approach:	They propose to use a dataset to evaluate nationality-based stereotypes in T2I models across 135 nationalities to assess offensive stereotypes.
Outcome:	The proposed dataset enables evaluation of known nationality-based stereotypes across 135 nationalities.

Copied to clipboard

Xiaokang Zhang, Zijun Yao, Jing Zhang, Kaifeng Yun, Jifan Yu, Juanzi Li, Jie Tang

What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)

Challenge:	Existing factuality detection methods are not effective for large language models (LLMs).
Approach:	They propose a probing model that trains on offline consistency checking results.
Outcome:	The proposed model reduces the computational burden of generating multiple responses by online consistency verification and improves on factuality detection and question answering benchmarks.

Copied to clipboard

Jiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan Cotterell

Agent Lumos: Unified and Modular Training for Open-Source Language Agents (2024.acl-long)

Challenge:	Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine.
Approach:	They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context.
Outcome:	The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt.

Copied to clipboard

Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, Bill Yuchen Lin

Challenge:	Lumos is a framework for training open-source agents on complex interactive tasks.
Approach:	They propose a framework for training open-source LLM-based agents called Lumos . Lumos features a learnable, unified and modular architecture with a planning module that learns high-level subgoal generation and a grounding module trained to translate these into the actions using various tools in the execution module.
Outcome:	The framework outperforms open-source agents on QA and web tasks.

Investigating Cultural Alignment of Large Language Models (2024.acl-long)

Copied to clipboard

Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, Mona Diab

More Victories, Less Cooperation: Assessing Cicero’s Diplomacy Play (2024.acl-long)

Challenge:	Large Language Models (LLMs) are used to represent the diversity of human experience and culturally sensitive topics.
Approach:	They propose a method leveraging anthropological reasoning to enhance cultural alignment by prompting LLMs with different pretraining data mixtures in Arabic and English.
Outcome:	The proposed method enables users to better represent the diversity of human experience and the plurality of different cultures.

Copied to clipboard

Wichayaporn Wongkamjan, Feng Gu, Yanze Wang, Ulf Hermjakob, Jonathan May, Brandon Stewart, Jonathan Kummerfeld, Denis Peskoff, Jordan Boyd-Graber

Challenge:	Diplomacy is a boardgame that offers a challenge for communicative and cooperative AI.
Approach:	They run two dozen games with Cicero and annotate in-game communication with abstract meaning representation to separate in- game tactics from general language.
Outcome:	The proposed method can outperform Cicero in communicating with humans, but it's difficult to deceive and persuade AI.

VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)

Copied to clipboard

Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, David Harwath

RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors (2024.acl-long)

Challenge:	VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation.
Approach:	They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks.
Outcome:	The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks.

Copied to clipboard

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, Chris Callison-Burch

Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles (2024.acl-long)

Challenge:	Existing methods for detecting machine-generated text are often insufficiently robust and lack benchmark datasets.
Approach:	They evaluate the out-of-domain and adversarial robustness of 8 open- and 4 closed-source detectors using RAID benchmark datasets.
Outcome:	The proposed detectors are fooled by adversarial attacks, repetition penalties, and unseen generative models.

Copied to clipboard

Julia Kruk, Michela Marchini, Rijul Magu, Caleb Ziems, David Muchlinski, Diyi Yang

On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning (2024.acl-long)

Challenge:	a dog whistle is a coded communication that carries a secondary meaning to specific audiences and is often weaponized for racial and socioeconomic discrimination.
Approach:	They propose an approach for word-sense disambiguation of dog whistles from standard speech using Large Language Models.
Outcome:	The proposed method allows disambiguation of dog whistles from standard speech using large language models.

Copied to clipboard

Franz Nowak, Anej Svete, Alexandra Butoi, Ryan Cotterell

Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends (2024.acl-long)

Challenge:	Existing theoretical treatments have shed light on some aspects of CoT, but we are still far from a concrete understanding of the concept.
Approach:	They propose to formalize CoT reasoning in a probabilistic setting to bridge this gap . they show that LMs can represent the same family of distributions over strings as probabilistic Turing machines.
Outcome:	The proposed model can represent the same family of distributions over strings as Turing machines.

Copied to clipboard

Sanjana Ramprasad, Elisa Ferracane, Zachary Lipton

LLM in a flash: Efficient Large Language Model Inference with Limited Memory (2024.acl-long)

Challenge:	Recent advances in large language models have improved summarization, but they still face a challenge of hallucination.
Approach:	They propose a taxonomy of errors to address the problem of hallucination in LLMs . they propose two prompt-based approaches for fine-grained error detection .
Outcome:	The proposed model outperforms existing metrics in identifying the novel "Contextual Inference" error type.

Copied to clipboard

Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2024.acl-long)

Challenge:	Large language models (LLMs) have high computational and memory requirements, especially for devices with limited memory.
Approach:	They propose a method that stores model parameters in flash memory but brings them on demand to DRAM . authors propose two techniques to optimize for reading data in larger, more contiguous chunks .
Outcome:	The proposed method reduces the volume of data transferred from flash and reads data in larger, more contiguous chunks.

Copied to clipboard

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Khan

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)

Challenge:	a surge of deep learning applications for video understanding have led to major advancements in video-related tasks.
Approach:	They propose a multimodal video-based conversation model that merges a video-adapted visual encoder with an LLM and a dataset that is easily scalable and robust to label noise.
Outcome:	The proposed model can understand and generate detailed conversations about videos.

Copied to clipboard

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024.acl-long)

Challenge:	Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations.
Approach:	They propose to distill knowledge from large teacher models into smaller student variants that are more efficient.
Outcome:	The proposed model outperforms existing models on standard benchmarks and dialectal data.

Copied to clipboard

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, Carole-Jean Wu

Classist Tools: Social Class Correlates with Performance in NLP (2024.acl-long)

Challenge:	Large Language Models (LLMs) have been deployed to many applications, yet their high compute and memory requirements lead to high financial and energy costs when deployed to GPU servers.
Approach:	They propose an end-to-end solution to speed-up inference of large language models . they apply layer dropout, and show that it increases the accuracy of early exit at earlier layers without adding any auxiliary layers or modules to the model.
Outcome:	The proposed method shows speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.

Copied to clipboard

Amanda Curry, Giuseppe Attanasio, Zeerak Talat, Dirk Hovy

Text Error Correction

Challenge:	despite growing concerns surrounding fairness and bias in NLP, there is a dearth of studies delving into the effects it may have on NLP systems.
Approach:	They argue that NLP systems’ performance is affected by speakers’ SES, potentially disadvantaging less-privileged socioeconomic groups.
Outcome:	The proposed model shows that NLP systems perform better on tasks with social class, ethnicity and geographical variation than those without social class.

ActionIE: Action Extraction from Scientific Literature with Programming Languages (2024.acl-long)

Copied to clipboard

Xianrui Zhong, Yufeng Du, Siru Ouyang, Ming Zhong, Tingfeng Luo, Qirong Ho, Hao Peng, Heng Ji, Jiawei Han

A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech (2024.acl-long)

Challenge:	a method that extracts experimental procedures from human language into actionable sequences in robotics language is challenging given the complexity of the instructions and context-dependent nature of the instruction.
Approach:	They propose a method that converts actions written in natural language into Python code that can be easily translated into robotics language.
Outcome:	The proposed method can extract experimental procedures from human language into actionable sequences in robotics language.

Copied to clipboard

Gaurav Verma, Rynaa Grover, Jiawei Zhou, Binny Mathew, Jordan Kraemer, Munmun Choudhury, Srijan Kumar

Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs (2024.acl-long)

Challenge:	Using data from 420k Twitter posts, we characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection.
Approach:	They develop a codebook to characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection.
Outcome:	The proposed codebook analyzes 420k tweets over 3 years and compares classifiers with hateful speech classifier classifier to detect hateful content.

Copied to clipboard

Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, Jinsong Su

COSMIC: Mutual Information for Task-Agnostic Summarization Evaluation (2024.acl-long)

Challenge:	Existing methods to compress long contexts have degraded dramatically as compression ratios increase, sometimes even falling to the closed-book level.
Approach:	They propose a query-guided compression method that preserves key information within the compressed context.
Outcome:	The proposed method can consistently perform well even at high compression ratios, and offers significant benefits in terms of inference cost and throughput.

Copied to clipboard

Maxime Darrin, Philippe Formont, Jackie Cheung, Pablo Piantanida

EUROPA: A Legal Multilingual Keyphrase Generation Dataset (2024.acl-long)

Challenge:	Existing methods for summarizing text are not well aligned with human judgments.
Approach:	They propose a task-oriented evaluation approach that assesses the quality of summarizers based on their capacity to produce summaries while preserving task outcomes.
Outcome:	The proposed method is able to predict task performance in a variety of contexts and tasks.

Copied to clipboard

Olivier Salaün, Frédéric Piedboeuf, Guillaume Le Berre, David Alfonso-Hermelo, Philippe Langlais

GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews (2024.acl-long)

Challenge:	Keyphrases are short phrases that describe a text and have been used for many applications.
Approach:	They present a dataset for multilingual keyphrase generation in the legal domain . it is derived from legal judgments from the Court of Justice of the European Union . they run multilingual models on the corpus and analyze the results .
Outcome:	The proposed dataset shows that it is better than existing models and can capture larger input context.

Copied to clipboard

Maxime Darrin, Ines Arous, Pablo Piantanida, Jackie Cheung

Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)

Challenge:	Scientific peer review is essential for the quality of academic publications.
Approach:	They propose a method that summarises scholarly reviews using a Rational Speech Act framework and novel uniqueness scores.
Outcome:	The proposed method generates more discriminative summaries than baseline methods in terms of human evaluation while achieving comparable performance with these methods in term of automatic metrics.

Copied to clipboard

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed

Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks (2024.acl-long)

Challenge:	MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings.
Approach:	They propose a family of Arabic multimodal large language models with strong vision and language capabilities.
Outcome:	The proposed models show strong performance on visual reasoning tasks and language capabilities.

Copied to clipboard

João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, Joao Magalhaes

Cheetah: Natural Language Generation for 517 African Languages (2024.acl-long)

Challenge:	Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences.
Approach:	They propose a method that integrates a Latent Diffusion Model (LDM) with an LLM to generate captions to maintain semantic coherence of the sequence.
Outcome:	The proposed method is preferred by humans in 46.6% of the cases against 26.6% for the second best method.

Copied to clipboard

Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed

TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024.acl-long)

Challenge:	Low-resource African languages pose unique challenges for natural language processing (NLG) We demonstrate the effectiveness of Cheetah through comprehensive evaluations across six generation downstream tasks.
Approach:	They develop a multilingual NLG language model for African languages called Cheetah . they demonstrate that Cheethah outperforms other models in six tasks .
Outcome:	The proposed model outperforms other models in five of six generation tasks.

Copied to clipboard

Yilun Zhao, Lyuhao Chen, Arman Cohan, Chen Zhao

KnowledgeFMath: A Knowledge-Intensive Math Reasoning Dataset in Finance Domains (2024.acl-long)

Challenge:	Long-form table question answering often generates paragraph long and complex answers . a prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context.
Approach:	They propose a modular framework that decomposes the whole process into three sub-modules . framework produces a QA-based plan first, followed by generating an answer conditioned on this plan . human evaluation results indicate the framework improves strong baselines on accuracy and truthfulness .
Outcome:	The proposed framework improves accuracy and truthfulness on the FeTaQA and QTSumm datasets.

Copied to clipboard

Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, Arman Cohan

API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs (2024.acl-long)

Challenge:	Existing benchmarks for large language models (LLMs) are only 56.6% accurate, leaving room for improvement.
Approach:	They propose a benchmark to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems using a finance-domain knowledge bank and expert-annotated solution references.
Outcome:	The proposed system achieves only 56.6% accuracy, leaving room for improvement.

Copied to clipboard

Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis Lastras

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks (2024.acl-long)

Challenge:	Existing methods to train and test large language models that involve calls to tools and APIs are lacking.
Approach:	They propose a large corpora for training and systematic testing of tool-augmented LLMs.
Outcome:	The proposed datasets mimic real-world scenarios involving API-tasks and slot filling.

Copied to clipboard

Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, Maosong Sun

Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024.acl-long)

Challenge:	LoRA-Flow uses lightweight modules to customize large language models for downstream tasks . previous work on LoRA combination relied on task-level weights for each involved LoRA .
Approach:	They propose a LoRA-Flow approach that uses dynamic weights to adjust the impact of different LoRAs.
Outcome:	The proposed method outperforms baselines with task-level weights on six generative tasks.

Copied to clipboard

Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception (2024.acl-long)

Challenge:	Unlike existing MoE approaches that rely on fixed TopK Routing, our dynamic expert selection framework dynamically allocates experts based on the confidence level in expert selection for each input.
Approach:	They propose a dynamic expert selection framework that dynamically allocates experts based on the confidence level in expert selection for each input.
Outcome:	The proposed method achieves an average improvement of 0.7% with less than 90% activated parameters and outperforms dense models in QA and machine translation tasks.

Copied to clipboard

HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi, Changhan Wang

SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents (2024.acl-long)

Challenge:	Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments.
Approach:	They propose a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages.
Outcome:	The proposed model outperforms the previous state-of-the-art by 18.5% WER and 4.7 BLEU on downstream audio-visual speech recognition and translation tasks.

Copied to clipboard

Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, Hao Zhu

\mathcal XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts (2024.acl-long)

Challenge:	Existing studies on building language agents have not addressed this social learning gap.
Approach:	They propose an interactive learning method that improves the social intelligence of language agents by using behavior cloning and self-reinforcement based training on filtered social interaction data.
Outcome:	The proposed method allows a 7B LLM to reach the social goal completion ability of an expert model (GPT-4-based agent) without the loss of more generic abilities, such as the ability to answer knowledge-based questions.

Copied to clipboard

Yifeng Ding, Jiawei Liu, Yuxiang Wei, Lingming Zhang

Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning (2024.acl-long)

Challenge:	Existing studies focus on the data perspectives of instruction tuning, leaving room for exploring advanced training schemes.
Approach:	They argue that prior works overlook the possibility of improving code instruction tuning by advancing existing training schemes.
Outcome:	The proposed model is dense because all parameters are activated to predict the next token (assuming it is a decoder-only LLM).

Copied to clipboard

Tuc Nguyen, Thai Le

Learning to Decode Collaboratively with Multiple Language Models (2024.acl-long)

Challenge:	Existing methods for fine-tuning Pretrained Language Models (PLMs) focus on training multiple adapters and continuously adding more adapters for incoming new tasks, making learning among the tasks unequal.
Approach:	They propose to mix only a select sub-set of adapters during inference via model weight averaging to optimize performance on unseen, in-domain examples.
Outcome:	The proposed methods can integrate a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but only a select sub-set of adapters during inference on unseen, in-domain examples.

Copied to clipboard

Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, David Sontag

DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models (2024.acl-long)

Challenge:	Using a latent variable model, multiple large language models can be trained to collaborate at the token level.
Approach:	They propose a method to teach multiple large language models to collaborate by interleaving their generations at the token level.
Outcome:	The proposed method improves on instruction-following, domain-specific QA, and reasoning tasks and shows that the model trained with the method exhibits several interesting collaboration patterns.

Copied to clipboard

Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, Yiqun Liu

Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning? (2024.acl-long)

Challenge:	Existing dynamic RAG methods fail to address the information needs of large language models (LLMs) despite their impressive capabilities, these models often produce text that seems coherent and plausible but factually incorrect, a problem commonly known as hallucination.
Approach:	They propose a dynamic retrieval augmented generation paradigm that actively decides when and what to retrieve during the text generation process of Large Language Models.
Outcome:	The proposed framework achieves superior performance over 4 knowledge-intensive generation datasets.

Copied to clipboard

Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, Min Zhang

CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)

Challenge:	Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections.
Approach:	They propose a co-temporal Question Answering benchmark that contains four co-time scenarios with 4,748 samples for evaluating the co-timing abilities of large language models.
Outcome:	The proposed benchmarks show that current LLMs struggle on CoTempQA tasks even when enhanced with Chain of Thought methodologies.

Copied to clipboard

Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang

LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments (2024.acl-long)

Challenge:	Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references.
Approach:	They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings.
Outcome:	The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading.

Copied to clipboard

Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen

Small But Funny: A Feedback-Driven Approach to Humor Distillation (2024.acl-long)

Challenge:	Existing benchmarks for evaluating large language models use static datasets, leading to data leakage or overlooking the complexities of multi-agent interactions.
Approach:	They propose a framework that evaluates the diverse capabilities of LLM agents in multi-agent dynamic environments.
Outcome:	The proposed framework assesses the diverse capabilities of LLM agents in multi-agent dynamic environments.

Copied to clipboard

Sahithya Ravi, Patrick Huber, Akshat Shrivastava, Vered Shwartz, Arash Einolghozati

Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) have been used to transfer knowledge from LLMs to smaller, smaller language models (SLMs).
Approach:	They propose to assign a dual role to the LLM as a “teacher” generating data, as well as evaluating the student’s performance.
Outcome:	The proposed approach narrows the performance gap between LLMs and larger models by incorporating feedback into the data.

Copied to clipboard

Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, Jun Liu

From Sights to Insights: Towards Summarization of Multimodal Clinical Documents (2024.acl-long)

Challenge:	Large Language Models (LLMs) have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language.
Approach:	They propose a model that integrates symbolic data into LLM training without loss of generality ability.
Outcome:	The proposed model performs better on symbol- and NL-centric tasks.

Copied to clipboard

Akash Ghosh, Mohit Tomar, Abhisek Tiwari, Sriparna Saha, Jatin Salve, Setu Sinha

When Phrases Meet Probabilities: Enabling Open Relation Extraction with Cooperating Large Language Models (2024.acl-long)

Challenge:	a recent WHO report highlights a drastic doctor-to-patient ratio . telehealth is one of the most impactful sectors where AI advances can bring a significant revolution .
Approach:	They propose an image-guided encoder-decoder model that uses contextual attention to create detailed visual-guides for multimodal documents.
Outcome:	The proposed model outperforms state-of-the-art models on multimodal question and dialogue summarization tasks.

Copied to clipboard

Jiaxin Wang, Lingling Zhang, Wee Sun Lee, Yujie Zhong, Liwei Kang, Jun Liu

Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation (2024.acl-long)

Challenge:	Existing clustering-based open relation extraction methods use pre-trained language models . embeddings from language models are high-dimensional and anisotropic, so there is a gap .
Approach:	They propose a framework that makes two LLMs work collaboratively to achieve clustering.
Outcome:	The proposed framework outperforms existing methods by 1.4%3.13% on different datasets.

Copied to clipboard

Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky

Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic (2024.acl-long)

Challenge:	generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are paraphrased and then used to fine-tune downstream models.
Approach:	They propose to use taboo words, hints by previous outlier solutions, and chaining on previous outliest solutions to augment text datasets as part of instructions to LLMs augmenting text dataset.
Outcome:	The proposed methods increase diversity of generated texts, but performance is highest with hints.

Copied to clipboard

Yassine El Kheir, Hamdy Mubarak, Ahmed Ali, Shammur Chowdhury

Syntactic Text Processing

Challenge:	Existing algorithms for recognizing borrowed and dialectal sounds are limited to Arabic, a dialect-rich language containing more than 22 major dialects.
Approach:	They propose a framework to recognize borrowed and dialectal sounds within phonologically diverse and dialect-rich languages that extends beyond its standard orthographic sound sets.
Outcome:	The proposed framework improves character error rate by 7% with only one and half hours of training data compared to the baseline.

Document-Level Machine Translation with Large-Scale Public Parallel Corpora (2024.acl-long)

Copied to clipboard

Proyag Pal, Alexandra Birch, Kenneth Heafield

Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length (2024.acl-long)

Challenge:	Document-level machine translation has inherent advantages over sentence-level translation due to additional information available to a model from document context.
Approach:	They propose to use document context to train context-aware models on these datasets and to use it to model document-level phenomena.
Outcome:	The proposed models improve translation quality and target document-level phenomena by incorporating contextual information from several preceding sentences.

Copied to clipboard

Nur Lan, Emmanuel Chemla, Roni Katzir

Challenge:	Neural networks offer good approximation to many tasks but fail to reach perfect generalization.
Approach:	They propose to use a formal language to test whether a theoretically correct solution is not an optimum of commonly used objectives.
Outcome:	The proposed model fails to reach the theoretically correct solution even with regularization techniques.

Context versus Prior Knowledge in Language Models (2024.acl-long)

Copied to clipboard

Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer White, Aaron Schein, Ryan Cotterell

Word Matters: What Influences Domain Adaptation in Summarization? (2024.acl-long)

Challenge:	Existing studies have investigated how often a model will rely on prior knowledge over conflicting contextual information in answering questions.
Approach:	They propose two mutual information-based metrics to measure a model’s dependency on a context and on its prior about an entity.
Outcome:	The proposed metrics show that language models can integrate prior knowledge and new information in a predictable way across different questions and contexts.

Copied to clipboard

Yinghao Li, Siyu Miao, Heyan Huang, Yang Gao

Visualization Recommendation with Prompt-based Reprogramming of Large Language Models (2024.acl-long)

Challenge:	Large Language Models (LLMs) can generalize domain datasets unseen during training but are not able to predict domain adaptation performance.
Approach:	They propose to quantify dataset learning difficulty as the learning difficulty of generative summarization, which is determined by word-based compression rate and abstraction level.
Outcome:	The proposed model can predict performance on unknown domain datasets without training, and it is based on the findings.

Copied to clipboard

Xinhang Li, Jingbo Zhou, Wei Chen, Derong Xu, Tong Xu, Enhong Chen

HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs (2024.acl-long)

Challenge:	Traditional visualization recommendations require extensive manual maintenance and yet fail to fully comprehend tabular data.
Approach:	They propose a hierarchical table prompt-based reprogramming framework that integrates tabular data into LLMs through a strategically crafted prompt learning method.
Outcome:	The proposed framework achieves state-of-the-art performance and will be made publicly available upon acceptance.

Copied to clipboard

Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, Prathosh Ap

Toward In-Context Teaching: Adapting Examples to Students’ Misconceptions (2024.acl-long)

Challenge:	Existing approaches to answer multi-hop questions are query-agnostic and the extracted facts are ambiguous as they lack context.
Approach:	They propose to use a knowledge graph to extract query-relevant information from unstructured text.
Outcome:	The proposed method achieves performance improvements on two popular datasets.

Copied to clipboard

Alexis Ross, Jacob Andreas

Bridging Word-Pair and Token-Level Metaphor Detection with Explainable Domain Mining (2024.acl-long)

Challenge:	Using computational models as pedagogical tools is becoming increasingly popular, but how effective can these models adapt as teachers to students of different types?
Approach:	They propose a suite of models and evaluation methods that combine Bayesian student models and AToM to evaluate adaptive teaching methods.
Outcome:	The proposed models outperform LLM-based and standard Bayesian teaching methods in the evaluation of simulated students across three learning domains.

Copied to clipboard

Yuan Tian, Ruike Zhang, Nan Xu, Wenji Mao

Faithful Logical Reasoning via Symbolic Chain-of-Thought (2024.acl-long)

Challenge:	Existing methods for word-pair metaphor detection provide intermediate explainable clues for detection results.
Approach:	They propose a method to bridge word-pair and token-level metaphor detection by modeling word pairs as explainable intermediate information.
Outcome:	The proposed method bridges word-pair and token-level metaphor detection by using word pairs . it provides intermediate explainable clues for the detection results, but this is a challenge .

Copied to clipboard

Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, Wynne Hsu

S2GSL: Incorporating Segment to Syntactic Enhanced Graph Structure Learning for Aspect-based Sentiment Analysis (2024.acl-long)

Challenge:	SymbCoT is a framework that integrates symbolic expressions and logic rules with CoT prompting.
Approach:	They propose a Symbolic Chain-of-Thought framework that integrates symbolic expressions and logic rules with CoT prompting.
Outcome:	The proposed framework improves on 5 standard datasets with symbolic expressions and rules . it shows that it is more faithful, flexible, and explainable than the current method .

Copied to clipboard

Bingfeng Chen, Qihan Ouyang, Yongqi Luo, Boyan Xu, Ruichu Cai, Zhifeng Hao

Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends (2024.acl-long)

Challenge:	Existing graph-based approaches to learn static structures and dynamic latent trees are lacking in incorporating semantic and syntactic information simultaneously within complex global structures.
Approach:	They propose a graph-based framework that incorporates semantic and syntactic information simultaneously within global structures.
Outcome:	The proposed framework removes irrelevant contexts and syntactic dependencies and achieves complementarity across diverse structures.

Copied to clipboard

Giuliano Martinelli, Edoardo Barba, Roberto Navigli

ESCoT: Towards Interpretable Emotional Support Dialogue Systems (2024.acl-long)

Challenge:	Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks.
Approach:	They propose a pipeline that trains a state-of-the-art Coreference Resolution system within the constraints of an academic budget and trains with up to 0.006x the memory resources.
Outcome:	The proposed framework outperforms encoder-based discriminative systems on the CoNLL-2012 benchmark, training with up to 0.006x the memory resources and obtaining 170x faster inference compared to previous state-of-the-art systems.

Copied to clipboard

Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, Qin Jin

PathReasoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answering (2024.acl-long)

Challenge:	Emotion-focused and strategy-driven chain-of-thought (ESCoT) is a new paradigm for emotional support dialogues.
Approach:	They propose an emotional support response generation scheme to improve interpretability . they generate a dataset and develop a model to generate dialogue responses with better interpretability.
Outcome:	The proposed scheme can generate dialogue responses with better interpretability.

Copied to clipboard

Fangzhi Xu, Qika Lin, Tianzhe Zhao, JiaweiHan JiaweiHan, Jun Liu

WARDEN: Multi-Directional Backdoor Watermarks for Embedding-as-a-Service Copyright Protection (2024.acl-long)

Challenge:	Existing logical reasoning tasks are challenging, especially for large language models.
Approach:	They propose a logic reasoning task model that transforms each logical sample into reasoning paths and propose an atom extension strategy supported by equivalent logical formulas to form new reasoning paths.
Outcome:	The proposed architecture achieves competitive performances on two logical reasoning benchmarks and great generalization abilities.

Copied to clipboard

Anudeex Shetty, Yue Teng, Ke He, Qiongkai Xu

Advancing Parameter Efficiency in Fine-tuning via Representation Editing (2024.acl-long)

Challenge:	Prior studies have shown that EaaS can be prone to model extraction attacks, however, this concern could be mitigated by adding backdoor watermarks to the text embeddings.
Approach:	They propose a new method that removes backdoor watermarks while maintaining the high utility of embeddings.
Outcome:	The proposed approach increases the stealthiness of watermarks and has been empirically shown to be effective against CSE attacks.

Copied to clipboard

Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang

Context Consistency between Training and Inference in Simultaneous Machine Translation (2024.acl-long)

Challenge:	Parameter Efficient Fine-Tuning (PEFT) has gained significant attention for its ability to achieve competitive results while updating only a small subset of trainable parameters.
Approach:	They propose a new approach to fine-tuning neural models that scales and biases the representation produced at each layer.
Outcome:	The proposed approach reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning and by . 32 compared with LoRA.

Copied to clipboard

Meizhi Zhong, Lemao Liu, Kehai Chen, Mingming Yang, Min Zhang

Using Natural Language Explanations to Improve Robustness of In-context Learning (2024.acl-long)

Challenge:	Simultaneous machine translation (SiMT) aims to yield a partial translation with a monotonically growing source-side context.
Approach:	They propose a training approach that encourages consistent context usage between training and inference by optimizing translation quality and latency as bi-objectives and exposing the predictions to the model during the training.
Outcome:	The proposed system outperforms existing SiMT systems with context inconsistency for the first time.

Copied to clipboard

Xuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini, Pontus Stenetorp

Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers (2024.acl-long)

Challenge:	Recent studies show that large language models excel in many tasks via in-context learning (ICL). However, ICL struggles to execute complex tasks such as arithmetic, commonsense, and symbolic reasoning.
Approach:	They propose to augment ICL with natural language explanations (NLEs) to produce further NLEs on adversarial datasets.
Outcome:	The proposed approach yields more accurate results than zero-shot-ICL and using only human-generated NLEs on eight adversarial datasets.

Copied to clipboard

Jiawen Xie, Pengyu Cheng, Xiao Liang, Yong Dai, Nan Du

ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models (2024.acl-long)

Challenge:	Existing transformer-based models struggle with long-sequence processing due to computational costs . a framework to enhance long-content processing of transformers is proposed .
Approach:	They propose a framework to enhance long-sequence processing of transformers by three steps . they demonstrate that the framework significantly outperforms prior long-quence processors .
Outcome:	The proposed framework outperforms baseline models on long-sequence summarization and reading comprehension tasks.

Copied to clipboard

Hojae Han, Jaejin Kim, Jaeseok Yoo, Youngwon Lee, Seung-won Hwang

Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels (2024.acl-long)

Challenge:	Despite the critical role of software requirements, these criteria have not been studied actively in previous code generation works.
Approach:	They propose a framework that leverages in-context learning to organize and extrapolate unexpressed requirements from textual descriptions.
Outcome:	The proposed framework generates functional requirements from textual descriptions and extrapolates unexpressed requirements from them.

Copied to clipboard

Zixia Jia, Junpeng Li, Shichuan Zhang, Anji Liu, Zilong Zheng

MULFE: A Multi-Level Benchmark for Free Text Model Editing (2024.acl-long)

Challenge:	Existing supervised learning methods rely on human annotations, but multi-label tasks pose challenges due to the specific domain knowledge and large class sets.
Approach:	They propose a framework that can be used to annotate a subset of positive classes from a multi-label dataset.
Outcome:	The proposed framework is generalized and effective across multiple tasks.

Copied to clipboard

Chenhao Wang, Pengfei Cao, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, Jun Zhao

MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024.acl-long)

Challenge:	Large Language Models (LLMs) have impressive capabilities in comprehending human language and vast parametric knowledge obtained from large corpora.
Approach:	They propose a multi-level benchmark for free text model editing to bridge the gap . they categorize probe queries into three levels of generalization .
Outcome:	The proposed method improves the generalization performance of large langugae models.

Copied to clipboard

Shengpeng Ji, Ziyue Jiang, Hanting Wang, Jialong Zuo, Zhou Zhao

Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation (2024.acl-long)

Challenge:	Existing zero-shot text-to-speech systems require a few seconds of unseen speaker voice prompts to generate high-quality voices.
Approach:	They propose a zero-shot text-to-speech system based on mobile devices . they use a discrete speech codec to integrate hierarchical information from the codec .
Outcome:	The proposed system achieves RTF of 0.09 on a single A100 GPU and has been successfully deployed on mobile devices.

Copied to clipboard

Muraleekrishna Gopinathan, Martin Masek, Jumana Abu-Khalaf, David Suter

HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position (2024.acl-long)

Challenge:	Existing speaker models learn strategies to evade evaluation metrics and obtain higher scores even for low-quality sentences.
Approach:	They propose a speaker-based instruction generator that utilises both structural and semantic knowledge of the environment to produce richer instructions.
Outcome:	The proposed model outperforms existing models and is evaluated using standard metrics.

Copied to clipboard

Kechi Zhang, Ge Li, Huangzhao Zhang, Zhi Jin

Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training (2024.acl-long)

Challenge:	Existing LLMs are constrained by their pre-trained context lengths, leading to performance issues . elucidating this limitation, we propose a training-free solution to the context length limitation in LLM applications .
Approach:	They propose a method that integrates hierarchical rotary position embedding into LLMs without extra training costs.
Outcome:	The proposed method improves performance on language modeling and long code completion tasks.

Copied to clipboard

Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, Jiaxing Zhang

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024.acl-long)

Challenge:	Large language models suffer from severe hallucinations, compromising performance in knowledge-oriented QA, dialogue, and writing.
Approach:	They propose to enhance the information searching and reflection ability of large language models by training them in position-agnostic multi-step QA tasks to improve their model's accuracy.
Outcome:	The proposed model improves in multi-doc QA and other benchmarks by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task.

Copied to clipboard

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin

When is Tree Search Useful for LLM Planning? It Depends on the Discriminator (2024.acl-long)

Challenge:	Large Language Models excel in simple tasks such as generating standalone code units, but real-world software development often involves complex code repositories with complex dependencies and extensive documentation.
Approach:	They propose a novel LLM-based agent framework that employs external tools for effective repo-level code generation.
Outcome:	The proposed framework outperforms commercial products like Github Copilot in the humanEval benchmark and shows that it is adaptable and efficient across multiple code generation tasks.

Copied to clipboard

Ziru Chen, Michael White, Ray Mooney, Ali Payani, Yu Su, Huan Sun

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)

Challenge:	Existing methods to build language agents that can plan efficiently and accurately have not met the needs of advanced planning methods to achieve such improvements.
Approach:	They propose to use iterative correction and tree search to solve multi-step problems in a language agent framework with three components: a generator, a discriminator, and a planning method.
Outcome:	The proposed methods improve performance on two tasks, text-to-SQL parsing and mathematical reasoning, while using discriminators with 90% accuracy.

Copied to clipboard

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral

Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding (2024.acl-long)

Challenge:	Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics.
Approach:	They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models.
Outcome:	The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting.

Copied to clipboard

Ruohao Guo, Wei Xu, Alan Ritter

Reducing Privacy Risks in Online Self-Disclosures with Language Models (2024.acl-long)

Challenge:	Existing large language models struggle to capture some language styles without fine-tuning.
Approach:	They propose to meta-trained LLMs based on representative lexicons to recognize new styles they have not been fine-tuned on.
Outcome:	The proposed method improves zero-shot transfer across styles on 13 established and 63 novel tasks generated with LLMs.

Copied to clipboard

Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, Wei Xu

Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models (2024.acl-long)

Challenge:	Disclosure is a social media activity that can be rewarding but also poses privacy risks.
Approach:	They propose to detect and abstract online self-disclosures using a large corpus of 4.8K annotated disclosure spans and a language model to fine-tune for detection.
Outcome:	The proposed model can detect and abstract self-disclosures with 80% accuracy, on-par with GPT-3.5.

Copied to clipboard

Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, Lifu Huang

REFINESUMM: Self-Refining MLLM for Generating a Multimodal Summarization Dataset (2024.acl-long)

Challenge:	Memory Editing (ME) has emerged as an efficient method to modify erroneous facts or inject new knowledge into Large Language Models (LLMs).
Approach:	They propose to evaluate LLMs with single edit only and parameter-modifying ME with parameter-preserving ME.
Outcome:	The proposed method can maintain LLMs’ fundamental capabilities but struggles to accurately recall edited knowledge presented in a different format.

Copied to clipboard

Vaidehi Patil, Leonardo Ribeiro, Mengwen Liu, Mohit Bansal, Markus Dreyer

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024.acl-long)

Challenge:	generating accurate and faithful multimodal summaries is challenging due to lack of appropriate multimodal datasets . large language models excel at synthesizing key information from diverse sources, but lack of adequate multimodal data sets for fine-tuning .
Approach:	They propose a dataset specifically designed for image-text multimodal summarization . they generate summaries from Wikipedia sections and corresponding images and evaluate them .
Outcome:	The proposed dataset improves summary quality by training a critic model on human annotations and using its predictions to remove low-quality summaries.

Copied to clipboard

Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, Haidar Khan

LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts (2024.acl-long)

Challenge:	Existing leaderboards are often taken at face value, but this is costly . a recent study shows that minor perturbations to the benchmark result in rankings up to 8 positions.
Approach:	They propose to use a hybrid scoring method for answer selection for large language models . they find that minor perturbations to the benchmark result in rankings changes .
Outcome:	The proposed model is a hybrid scoring method, the authors argue . the proposed model could be used to improve the performance of large language models .

Copied to clipboard

Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, Chris Kedzie

Challenge:	Existing frameworks for the automated evaluation of natural language texts are based on a large language model (LLM) that fails to agree with human judges and is not fully validated by the human judges.
Approach:	They propose a large language model (LLM) that generates a distribution over potential responses to assess multiple dimensions of interest.
Outcome:	The proposed framework predicts human judges' assessment of user satisfaction on a scale of 1–4 with an RMS error 0.5, a 2 improvement over the uncalibrated baseline.

LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition (2024.acl-long)

Copied to clipboard

Xiaomeng Zhu, Robert Frank

Evaluating Very Long-Term Conversational Memory of LLM Agents (2024.acl-long)

Challenge:	Existing studies have shown that large language models have basic, if imperfect, DE recognition abilities, but they lack the basic knowledge of the fundamental semantic properties that govern the introduction and subsequent reference to DEs.
Approach:	They propose a dataset that examines large language models’ knowledge of existence, uniqueness, plurality, and novelty.
Outcome:	The proposed dataset shows that large language models exhibit sensitivity to all of these properties except novelty, which demonstrates that they have yet to reach human-level language understanding abilities.

Copied to clipboard

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang

Prototypical Reward Network for Data-Efficient Model Alignment (2024.acl-long)

Challenge:	Existing studies on long-term open-domain dialogues focus on evaluating responses within contexts spanning no more than five chat sessions.
Approach:	They propose a machine-human pipeline to generate very long-term dialogues by leveraging LLMs and retrieval augmented generation techniques.
Outcome:	The proposed pipeline generates very long-term dialogues using LLMs and RAGs . the generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs.

Copied to clipboard

Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu

NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms (2024.acl-long)

Challenge:	Reinforcement Learning from Human Feedback (RLHF) is a reward model that fine-tunes Large Language Models (LLMs) by utilizing Prototypical Networks.
Approach:	They propose a framework utilizing Prototypical Networks to enhance reward models under limited human feedback, enabling more stable and reliable structural learning from fewer samples.
Outcome:	The proposed framework improves reward models under limited human feedback, surpassing traditional methods, especially in data-limited scenarios.

Copied to clipboard

Jonathan Zheng, Alan Ritter, Wei Xu

Impacts of Misspelled Queries on Translation and Product Search (2024.acl-long)

Challenge:	Prior work on temporal language change observed degradation when finetuning on older text and evaluating on newer data and named entities.
Approach:	They construct a benchmark to evaluate LLMs’ ability to generalize to neologisms with various natural language understanding tasks and model perplexity.
Outcome:	The proposed model performs better in downstream tasks and with later knowledge cutoff dates than models with earlier knowledge cut off dates.

Copied to clipboard

Greg Hanneman, Natawut Monaikul, Taichi Nakatani

Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs (2024.acl-long)

Challenge:	In end-to-end e-commerce, the inclusion of a dedicated spelling correction model, and the augmentation of that model’s training data with language-relevant phenomena, each improve robustness and consistency of search results.
Approach:	They first analyze the spelling-robustness of a population of machine translation systems and then apply them to a multilingual e-commerce setting to test whether spelling variations affect MT output and user behavior.
Outcome:	The proposed model reduces the number of BPE operations and improves spelling-robustness in six languages.

Copied to clipboard

Bilgehan Sel, Priya Shanmugasundaram, Mohammad Kachuee, Kun Zhou, Ruoxi Jia, Ming Jin

The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition (2024.acl-long)

Challenge:	Large Language Models (LLMs) have shown remarkable capabilities in tasks such as summarization, arithmetic reasoning, and question answering.
Approach:	They propose a framework that explores decisions’ consequences from multiple stakeholder perspectives and a SKIG framework to enhance moral reasoning in large language models.
Outcome:	The proposed framework exhibits marked improvements compared to baselines across different language models and benchmarks.

Copied to clipboard

Enshi Zhang, Rafael Trujillo, Christian Poellabauer

Transparent and Scrutable Recommendations Using Natural Language User Profiles (2024.acl-long)

Challenge:	Existing models for speech emotion recognition lack a comprehensive dataset to design accurate models.
Approach:	They propose to use a multimodal dataset to build a model that integrates pre-trained wav2vec 2.0 and BERT to learn hidden representations from fused representations of speech and text.
Outcome:	The proposed model predicts emotions on dimensions of arousal, valence, and dominance . it achieved competitive results on the MSP-PODCAST dataset .

Copied to clipboard

Jerome Ramos, Hossein A. Rahmani, Xi Wang, Xiao Fu, Aldo Lipani

Fora: A corpus and framework for the study of facilitated dialogue (2024.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) rely on implicit or explicit feedback from users to suggest new items, resulting in a lack of transparency and a user's ability to scrutinize and modify their preferences.
Approach:	They propose to use a natural language (NL) user profile to summarize a user's preferences and then use it to fine-tune a LLM using only NL profiles to make transparent and scrutable recommendations.
Outcome:	The proposed model performs on two benchmarking rating prediction datasets and is comparable to existing models.

Copied to clipboard

Hope Schroeder, Deb Roy, Jad Kabbara

Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning (2024.acl-long)

Challenge:	a new study of facilitated dialogues focuses on the sharing of personal experience . social media is a popular method of civic engagement but lacks the tools to analyze it .
Approach:	They compile 262 facilitated conversations hosted with partner organizations . they taxonomize personal sharing behaviors and facilitation strategies in the corpus .
Outcome:	The proposed framework can be used to analyze facilitated dialogues and parse spoken conversations . the data can be applied to other fields, including civic use in governance and social science .

Copied to clipboard

Yue Yu, Jiaming Shen, Tianqi Liu, Zhen Qin, Jing Nathan Yan, Jialu Liu, Chao Zhang, Michael Bendersky

What is the Best Way for ChatGPT to Translate Poetry? (2024.acl-long)

Challenge:	Recent advances in natural language processing (NLP) have witnessed the remarkable capabilities of Large Language Models (LLMs).
Approach:	They propose an Explanation-Aware Soft Ensemble framework to empower in-context learning with Large language models.
Outcome:	The proposed framework can be used to enhance in-context learning on seven natural language understanding tasks and four varying-size LLMs.

Copied to clipboard

Shanshan Wang, Derek Wong, Jingming Yao, Lidia Chao

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024.acl-long)

Challenge:	Despite promising results, our analysis reveals persistent issues in the translations generated by ChatGPT that warrant attention.
Approach:	They propose an Explanation-Assisted Poetry Machine Translation method which leverages monolingual poetry explanation as a guiding information for the translation process.
Outcome:	The proposed method outperforms traditional translation methods of ChatGPT and the existing online systems in English-Chinese poetry translation.

Copied to clipboard

Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly

DeCoT: Debiasing Chain-of-Thought for Knowledge-Intensive Tasks in Large Language Models via Causal Intervention (2024.acl-long)

Challenge:	Large language model pre-training is infeasible due to the large compute costs and duration associated with pre- training and the impending scarcity of high-quality data on the web.
Approach:	They propose to use an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as “like Wikipedia” or in “question-answer format” to jointly pre-train LLMs on real and synthetic rephrases.
Outcome:	The proposed model speeds up pre-training by 3x on the C4 dataset, and improves perplexity by 50% on average across different subsets of the Pile.

Copied to clipboard

Junda Wu, Tong Yu, Xiang Chen, Haoliang Wang, Ryan Rossi, Sungchul Kim, Anup Rao, Julian McAuley

Representation Learning with Conditional Information Flow Maximization (2024.acl-long)

Challenge:	In large language models, external knowledge is required to augment their internal knowledge through prompts, but this does not guarantee that LLMs can identify and use relevant information in the prompts to conduct chain-of-thought reasoning.
Approach:	They propose a structural causal model to formally explain the internal knowledge bias of large language models (LLMs) they review the chain-of-thought (CoT) prompting from a causal perspective and find that biased information from pretrained models can impair LLMs’ reasoning abilities.
Outcome:	The proposed model enables more accurate CoT reasoning and enhances LLM generation on knowledge-intensive tasks.

Copied to clipboard

Dou Hu, Lingwei Wei, Wei Zhou, Songlin Hu

GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024.acl-long)

Challenge:	Existing knowledge-theoretic representation learning frameworks are based on the information bottleneck principle, which preserves redundant features irrelevant to the given task.
Approach:	They propose a conditional information flow maximization framework to learn sufficient representations for the input data and target task by maximizing both input-representation and representation-label mutual information.
Outcome:	The proposed framework can extract noise-invariant sufficient representations for the input data and target task.

Copied to clipboard

Virginia Felkner, Jennifer Thompson, Jonathan May

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)

Challenge:	Current benchmarks for social biases have limitations in scope, grounding, quality and human effort required.
Approach:	They propose to use a language model to help with the development of bias benchmarks . they extend previous work to a new community and set of biases: the Jewish community and antisemitism .
Outcome:	The proposed LLM does not perform well on the Jewish community and antisemitism task.

Copied to clipboard

Martin Riddell, Ansong Ni, Arman Cohan

Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic (2024.acl-long)

Challenge:	Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks.
Approach:	They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length.
Outcome:	The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training.

Copied to clipboard

Rishabh Bhardwaj, Duc Anh Do, Soujanya Poria

Tracking the Newsworthiness of Public Documents (2024.acl-long)

Challenge:	RESTA is a tool to perform LLM realignment towards safety, which gets compromised due to downstream task fine-tuning.
Approach:	They propose to add a safety vector to the weights of a compromised model by arithmetic and demonstrate its generalizability on three existing safety evaluation benchmarks and a multilingual benchmark dataset.
Outcome:	The proposed model reduces harmfulness of the model from 18.6% to 5.1% and from 9.2% to 1.5% in parameter-efficient and full fine-tuning while maintaining most of the system's performance on the task.

Copied to clipboard

Alexander Spangher, Serdar Tumgoren, Ben Welsh, Nanyun Peng, Emilio Ferrara, Jonathan May

Challenge:	a new method to model news coverage of local government is needed . we show that newsworthiness predictions can be useful for journalists seeking to keep abreast of local governments.
Approach:	They propose a method that explicitly models when and why stories get press attention . they use an annotated corpus of news articles to build models that predict if a policy item will get covered .
Outcome:	The proposed model outperforms retrieval-based methods with limited annotated data and language use between corpora.

EWEK-QA : Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems (2024.acl-long)

Copied to clipboard

Mohammad Dehghan, Mohammad Alomrani, Sunyam Bagga, David Alfonso-Hermelo, Khalil Bibi, Abbas Ghaddar, Yingxue Zhang, Xiaoguang Li, Jianye Hao, Qun Liu, Jimmy Lin, Boxing Chen, Prasanna Parthasarathi, Mahdi Biparva, Mehdi Rezagholizadeh

Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)

Challenge:	citation-based QA systems are suffering from two shortcomings . they usually rely only on web as a source of extracted knowledge and external knowledge sources can hamper the efficiency of the system.
Approach:	They propose to use a web-based knowledge graph retrieval solution to enrich extracted knowledge fed to a citation-based QA system.
Outcome:	The proposed model outperforms open-source state-of-the-art models in 7 quantitative and human evaluation tasks.

Copied to clipboard

Shengzhi Li, Rongyu Lin, Shichao Pei

Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation (2024.acl-long)

Challenge:	Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production.
Approach:	They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets.
Outcome:	The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM.

Copied to clipboard

Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay-Yoon Lee, Mohit Iyyer, Andrew McCallum

Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor (2024.acl-long)

Challenge:	Low-resource tasks such as semi-supervised sequence generation require expert knowledge and cost.
Approach:	They propose a method for semi-supervised sequence generation where few examples are too scarce to fine tune a model.
Outcome:	The proposed method can generalize better than its teacher to unseen examples on semi-supervised sequence generation tasks.

Copied to clipboard

Sangwon Yu, Changmin Lee, Hojin Lee, Sungroh Yoon

LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP (2024.acl-long)

Challenge:	Existing methods to control text generation are inapplicable to black-box models or suffer a trade-off between control and fluency.
Approach:	They propose a new approach to control text generation that modifies context at the token level during the generation process of a backbone language model and guides subsequent text to naturally include the target attributes.
Outcome:	The proposed method can regulate the attributes of the generated text while utilizing the capability of the backbone large language models.

Copied to clipboard

Danlu Chen, Freda Shi, Aditi Agarwal, Jacobo Myerston, Taylor Berg-Kirkpatrick

Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning (2024.acl-long)

Challenge:	Existing pipelines for natural language processing only process symbolic representations of language, which are labor-intensive and noisy . a large portion of logographic data persists in a purely visual form due to the absence of transcription . this issue poses a bottleneck for researchers seeking to apply NLP to ancient logographic languages .
Approach:	They propose a benchmark for NLP analysis of ancient logographic languages using visual representations of writing.
Outcome:	The proposed pipeline outperforms existing pipelines for some tasks . the results could unlock large amounts of cultural heritage data of ancient logographic languages .

Copied to clipboard

Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, Tianyi Zhou

Confabulation: The Surprising Value of Large Language Model Hallucinations (2024.acl-long)

Challenge:	Earlier studies of instruction tuning on Large Language Models focus on creating large, varied, and high-quality datasets with responses curated by human experts.
Approach:	They propose to use a smaller and weaker model to fine tune a larger and stronger model . they find it can largely speed up the data filtering and improve performance .
Outcome:	The proposed model can filter instruction data faster and better on benchmarks.

Copied to clipboard

Peiqi Sui, Eamon Duede, Sophie Wu, Richard So

IAPT: Instance-Aware Prompt Tuning for Large Language Models (2024.acl-long)

Challenge:	'confabulations' are inherently problematic and AI research should eliminate this flaw, but confabulation is not a problem.
Approach:	They argue that measurable semantic characteristics of large language model (LLM) hallucinations mirror a human propensity to utilize increased narrativity as a cognitive resource for sense-making and communication.
Outcome:	The proposed study shows that measurable semantic characteristics of LLM confabulations mirror human propensity to utilize increased narrativity as a cognitive resource for sense-making and communication.

Copied to clipboard

Wei Zhu, Aaron Tian, Congrui Yin, Yuan Ni, Xiaoling Wang, Guotong Xie