Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document (2024.emnlp-main)

Challenge:	Various attempts to correct noisy data in the construction process have been made, but human annotation is expensive and time-consuming.
Approach:	They propose to use large language models for data annotation to imitate human annotation and classify unrelated documents from a multi-document summarization task.
Outcome:	The proposed method imitates human annotation and classifies unrelated documents from the Multi-News dataset.

Copied to clipboard

Joonho Yang, Seunghyun Yoon, ByeongJeong Kim, Hwanhee Lee

Summarization

Challenge:	Existing methods for evaluating factual consistency in abstractive summarization systems have significant limitations, especially on refinement and interpretability.
Approach:	They propose a method for detecting summary factual inconsistency based on fine-grained atomic facts decomposition and adaptive granularity expansion.
Outcome:	The proposed method outperforms existing systems on the AGGREFACT benchmark dataset and achieves state-of-the-art performance.

Prompts have evil twins (2024.emnlp-main)

Copied to clipboard

Rimon Melamed, Lucas McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, Enric Boix-Adserà

Table Question Answering for Low-resourced Indic Languages (2024.emnlp-main)

Challenge:	We find that many natural-language prompts can be replaced by corresponding unintelligible prompts that provably elicit similar behavior in language models.
Approach:	They find that natural-language prompts can be replaced by corresponding unintelligible prompts that elicit similar behavior in language models.
Outcome:	The proposed prompts are obfuscated and uninterpretable but mimic the original natural-language prompts . the problem has applications of independent interest, the authors argue .

Copied to clipboard

Vaishali Pal, Evangelos Kanoulas, Andrew Yates, Maarten Rijke

ImageInWords: Unlocking Hyper-Detailed Image Descriptions (2024.emnlp-main)

Challenge:	TableQA is the task of answering questions over tables of structured information, returning individual cells or tables as output.
Approach:	They propose a fully automatic large-scale tableQA data generation process for low-resource languages with limited budget.
Outcome:	The proposed method outperforms state-of-the-art LLMs on two Indic languages with no tableQA datasets and models on different aspects including mathematical reasoning capabilities and zero-shot cross-lingual transfer.

Copied to clipboard

Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, Radu Soricut

LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay (2024.emnlp-main)

Challenge:	generating accurate hyper-detailed image descriptions is challenging for vision-language models trained on web-scraped image-text.
Approach:	They propose a data-centric framework for generating hyper-detailed image descriptions using web-scraped image-text.
Outcome:	The proposed framework improves on human evaluations on the data, even with only 9k samples.

Copied to clipboard

Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, Hao Wang

Challenge:	Existing studies on LLM agents' social behaviors are lacking . previous studies focused on positive social behaviors, leaving research on negative social behaviors relatively scarce.
Approach:	They propose a framework that features a multi-agent system facilitating efficient communication and interaction with LLM agents.
Outcome:	The proposed framework is based on Avalon and evaluates on game success and analyzes agents’ social behaviors.

When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection (2024.emnlp-main)

Copied to clipboard

Xiangyu Zhang, Hexin Liu, Kaishuai Xu, Qiquan Zhang, Daijiao Liu, Beena Ahmed, Julien Epps

Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model (2024.emnlp-main)

Challenge:	Large language models (LLMs) are used for depression detection but their application remains unexplored.
Approach:	They propose to integrate acoustic speech information into LLMs for depression detection by integrating aural landmarks into the framework.
Outcome:	The proposed method adds critical dimensions to speech transcripts and provides insights into the unique speech patterns of individuals.

Copied to clipboard

Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia Perera, EngSiong Chng, Lina Yao

Hateful Word in Context Classification (2024.emnlp-main)

Challenge:	Existing approaches to enhance inference speed and training require complex modifications to the model.
Approach:	They propose to double the training and inference speed of Denoising Diffusion Probabilistic Models by simply redirecting the generative target to the wavelet domain.
Outcome:	The proposed method doubles the training and inference speed of Speech DDPMs by redirecting the generative target to the wavelet domain.

Copied to clipboard

Sanne Hoeken, Sina Zarrieß, Özge Alacam

Eyes Don’t Lie: Subjective Hate Annotation and Detection with Gaze (2024.emnlp-main)

Challenge:	Hate speech detection is a prevalent research field, yet word meaning is underexplored . lexical cues play a role in determining the hatefulness of words, but are not enough in focus for HSD research.
Approach:	They propose a Hateful Word in Context Classification task to determine the hatefulness of a word within a specific context.
Outcome:	The proposed task aims to determine the hatefulness of a word within a specific context, and argues that definitions prove effective overall, but not in cases where hateful connotations vary.

Copied to clipboard

Özge Alacam, Sanne Hoeken, Sina Zarrieß

NumeroLogic: Number Encoding for Enhanced LLMs’ Numerical Reasoning (2024.emnlp-main)

Challenge:	Hate speech is a complex and subjective phenomenon.
Approach:	They propose a dataset that provides gaze data collected in a hate speech annotation experiment and introduce a first gaze-integrated HSD model.
Outcome:	The proposed dataset provides gaze data from hate speech annotation experiments.

Copied to clipboard

Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle

Numerical Reasoning

Challenge:	Language models struggle with numerical and arithmetical tasks, such as multiplying 3-digit numbers.
Approach:	They propose a method to include the count of digits before each number instead of “42”.
Outcome:	The proposed format improves the reasoning process before generating the actual number.

“Thinking” Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models (2024.emnlp-main)

Copied to clipboard

Shaz Furniturewala, Surgan Jandial, Abhinav Java, Pragyan Banerjee, Simra Shahid, Sumit Bhatia, Kokil Jaidka

A Usage-centric Take on Intent Understanding in E-Commerce (2024.emnlp-main)

Challenge:	Existing debiasing techniques are typically training-based or require access to the model’s internals and output distributions, so they are inaccessible to end-users looking to adapt LLM outputs for their particular needs.
Approach:	They propose a system-based iterative framework that uses System 2 thinking processes to induce logical, reflective, and critical text generation with single, multi-step, instruction, and role-based variants.
Outcome:	The proposed framework significantly improves over other frameworks demonstrating lower mean bias in the outputs with competitive performance on the downstream tasks.

Copied to clipboard

Wendi Zhou, Tianyi Li, Pavlos Vougiouklis, Mark Steedman, Jeff Pan

Challenge:	Identifying and understanding user intents is a crucial task for E-Commerce.
Approach:	They propose to use intent understanding as a natural language reasoning task independent of product ontologies to identify and understand user intents.
Outcome:	The proposed framework can't be used to strongly align user intents with products with desirable properties and recommend useful products across diverse categories.

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2024.emnlp-main)

Copied to clipboard

Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha

Systematic Biases in LLM Simulations of Debates (2024.emnlp-main)

Challenge:	Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights.
Approach:	They compare unsupervised fine-tuning and retrieval-augmented generation approaches to learning new factual information.
Outcome:	The proposed models outperform unsupervised fine-tuning and retrieval-augmented generation (RAG) on knowledge-intensive tasks across different topics.

Copied to clipboard

Amir Taubenfeld, Yaniv Dover, Roi Reichart, Ariel Goldstein

Challenge:	Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies.
Approach:	They propose to use LLMs to simulate political debates on topics that are important aspects of people’s day-to-day lives and decision-making processes.
Outcome:	The proposed model can simulate political debates on topics that are important aspects of people’s day-to-day lives and decision-making processes.

Studying and Mitigating Biases in Sign Language Understanding Models (2024.emnlp-main)

Copied to clipboard

Katherine Atwell, Danielle Bragg, Malihe Alikhani

Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)

Challenge:	Using crowd-sourced sign language datasets to reduce performance disparities is critical to addressing potential biases and inequities.
Approach:	They use demographic information to study biases that may result from models trained on crowd-sourced sign datasets.
Outcome:	The proposed approach reduces performance disparities without decreasing accuracy.

Copied to clipboard

Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning (2024.emnlp-main)

Challenge:	Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs.
Approach:	They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses .
Outcome:	The proposed framework assesses uncertainty and confidence measures for LMs.

Copied to clipboard

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang

Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024.emnlp-main)

Challenge:	Current research emphasizes LLMs’ capacity to utilize tools in well-structured environments while overlooking their stability when confronted with the inevitable noise of the real world.
Approach:	They propose a multi-level benchmark to evaluate the robustness of large language models in tool learning by establishing five external environments with varying levels of noise.
Outcome:	The proposed model outperforms the GPT-4 model in tool learning in three critical phases: tool selection, parameter identification, and content filling.

Copied to clipboard

Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy Chen, Shafiq Joty

Scaling Properties of Speech Language Models (2024.emnlp-main)

Challenge:	Recent studies have raised concerns regarding the hallucination and flaws in their reasoning process.
Approach:	They propose a framework to learn planning-based reasoning through Direct Preference Optimization on collected trajectories, which are ranked according to synthesized process rewards.
Outcome:	The proposed model surpasses GPT-3.5-Turbo on logical reasoning benchmarks on a set of logically-based reasoning tasks.

Copied to clipboard

Santiago Cuervo, Ricard Marxer

“We Demand Justice!”: Towards Social Context Grounding of Political Texts (2024.emnlp-main)

Challenge:	Speech Language Models (SLMs) aim to learn language from raw audio without textual resources.
Approach:	They propose to use scaling properties of neural language models to estimate scale at which SLMs will be trained . they establish a strong correlation between pre-training loss and downstream syntactic and semantic performance .
Outcome:	The proposed model will have the English proficiency of text-based Large Language Models.

Copied to clipboard

Rajkumar Pujari, Chengfei Wu, Dan Goldwasser

Challenge:	Political discourse on social media often contains similar language with opposing intended meanings.
Approach:	They propose to characterize the social context required to fully understand political discourse . structured models outperform larger models on both tasks, but still lag behind human performance .
Outcome:	The proposed models outperform larger models on both tasks but lag behind human performance.

An Experimental Analysis on Evaluating Patent Citations (2024.emnlp-main)

Copied to clipboard

Rabindra Nath Nandi, Suman Maity, Brian Uzzi, Sourav Medya

Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice? (2024.emnlp-main)

Challenge:	Graph Neural Networks (GNNs)-based methods can predict patent citations using only patent text.
Approach:	They propose to use Graph Neural Networks to predict citations for patents based on their semantic similarities to generate a semantic graph of patents.
Outcome:	The proposed methods produce 94% recall for patents with high citations and outperform baselines.

Copied to clipboard

Dawei Zhu, Pinzhen Chen, Miaoran Zhang, Barry Haddow, Xiaoyu Shen, Dietrich Klakow

Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024.emnlp-main)

Challenge:	Traditionally, success in multilingual machine translation depends on large volume, diverse directions, and high quality of training data.
Approach:	They revisit the importance of large language models for translation by fine-tuning on 32 parallel sentences.
Outcome:	The proposed model can be fine-tuned on as few as 32 parallel sentences . however, the choice of direction is critical to avoid misinterpretation, the authors say .

Copied to clipboard

Le Yan, Zhen Qin, Honglei Zhuang, Rolf Jagerman, Xuanhui Wang, Michael Bendersky, Harrie Oosterhuis

Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation (2024.emnlp-main)

Challenge:	Existing approaches to generate relevance labels for large language models have not been successful in generating relevance labels.
Approach:	They propose a method to combine LLM relevance labels with ranking abilities . they take both LLM generated relevance labels and pairwise preferences .
Outcome:	The proposed method balances the ranking and labeling abilities of large language models . it takes both LLM generated relevance labels and pairwise preferences .

Copied to clipboard

Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, Tat-Seng Chua

Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation (2024.emnlp-main)

Challenge:	Non-collaborative dialogue agents are expected to engage in strategic conversations with diverse users, and this poses two main challenges for existing dialogue agents: 1) the inability to integrate user-specific characteristics into the strategic planning; 2) the difficulty of training strategic planners that can be generalized to diverse users.
Approach:	They propose to integrate a user-aware strategic planning module and a population-based training paradigm into a non-collaborative dialogue agent for securing a mutual agreement that leans favorably towards the system's objectives.
Outcome:	The proposed model can be used to achieve a mutual agreement that leans favorably towards the system's objectives.

Copied to clipboard

Saiful Salim, Rubin Yang, Alexander Cooper, Suryashree Ray, Saumya Debray, Sazzadur Rahaman

Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024.emnlp-main)

Challenge:	Large language model (LLM)-based programming assistants can also facilitate cheating in introductory computer science courses.
Approach:	They propose to use Large Language Models to detect and penalize cheating and modify problem statements to impede cheating.
Outcome:	The proposed methods reduce correctness scores by 77% and detectability by perturbations.

Copied to clipboard

Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, JingBo Zhu

On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset.
Approach:	They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR).
Outcome:	The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations.

Copied to clipboard

Abhilasha Sancheti, Haozhe An, Rachel Rudinger

EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models (2024.emnlp-main)

Challenge:	We show that models are less likely to predict romantic relationships for same-gender character pairs than different-grace character pairs.
Approach:	They perform name-replacement experiments to examine gender biases in large language models . they hypothesize that models mirror heteronormative biase and prejudice against interracial romantic relationships .
Outcome:	The results suggest that models may mirror heteronormative biases and prejudice against interracial romantic relationships in human and society.

Copied to clipboard

Maureen Seyssel, Antony D’Avirro, Adina Williams, Emmanuel Dupoux

On Fake News Detection with LLM Enhanced Semantics Mining (2024.emnlp-main)

Challenge:	EmphAssess evaluates speech-to-speech models' ability to encode and reproduce prosodic emphasis across a change of speaker and language.
Approach:	They propose a prosodic benchmark to evaluate the ability of speech-to-speech models to encode and reproduce prosodic emphasis.
Outcome:	The proposed model can encode and reproduce prosodic emphasis across speech inputs and outputs . EmphaClass classifies emphasis at the frame or word level .

Copied to clipboard

Xiaoxiao Ma, Yuchen Zhang, Kaize Ding, Jian Yang, Jia Wu, Hao Fan

On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices (2024.emnlp-main)

Challenge:	Existing methods for detecting fake news use only news embeddings to capture the lexical semantics between tokens.
Approach:	They propose a topic-based model with prompts to extract news embeddings from LLMs and a generalized page-rank model to extract local and global semantics.
Outcome:	The proposed model shows superior performance on five benchmark datasets over seven baseline methods.

Copied to clipboard

Branislav Pecher, Ivan Srba, Maria Bielikova

Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection (2024.emnlp-main)

Challenge:	Using limited labelled data, learning with limited labels is sensitive to the effects of uncontrolled randomness.
Approach:	They propose to investigate the effects of individual randomness factors while taking the interactions between them into consideration.
Outcome:	The proposed method mitigates the effects of other factors while observing how the performance varies across multiple runs.

Copied to clipboard

Zekun Li, Baolin Peng, Pengcheng He, Xifeng Yan

A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifiers (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications.
Approach:	They establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard.
Outcome:	The proposed model is overly sensitive to prompt injection attacks, focusing on the latter part of the prompt without fully understanding the context.

Copied to clipboard

Valentin Barriere, Sebastian Cifuentes

Mitigating the Alignment Tax of RLHF (2024.emnlp-main)

Challenge:	Recent research shows that named entities influence PLMs in many applications.
Approach:	They propose a method to quantify biases associated with named entities from various countries using Twitter data instead of templates or specific datasets.
Outcome:	The proposed method shows positive biases related to the language spoken in a country across all classifiers.

Copied to clipboard

Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang

Evaluating Readability and Faithfulness of Concept-based Explanations (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax.
Approach:	They propose to use a model averaging technique to find the most powerful alignment-forging Pareto front among RLHF algorithms.
Outcome:	The proposed method achieves the strongest alignment-forging Pareto front among competing methods.

Copied to clipboard

Meng Li, Haoran Jin, Ruixuan Huang, Zhihao Xu, Defu Lian, Zijia Lin, Di Zhang, Xiting Wang

Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems (2024.emnlp-main)

Challenge:	Existing methods for evaluating concepts from different perspectives lack a unified formalization.
Approach:	They propose a formal definition of concepts generalizing to diverse concept-based explanations’ settings and apply it to other types of explanations or tasks.
Outcome:	Extensive experimental analysis was carried out to determine the evaluation measures for explanation evaluation measures.

Copied to clipboard

Zhengyuan Liu, Stella Yin, Geyu Lin, Nancy Chen

MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making (2024.emnlp-main)

Challenge:	Existing large language models (LLMs) can be adopted as tutoring agents for math and language learning.
Approach:	They propose a framework to construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario.
Outcome:	The proposed framework can construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario.

Copied to clipboard

Dayuan Fu, Biqing Qi, Yihuai Gao, Che Jiang, Guanting Dong, Bowen Zhou

Challenge:	Insight is a form of long-term memory for an agent but lack of general insight can undermine its effectiveness.
Approach:	They propose an embodied agent that summarises and utilizes insight effectively across different scales and generates task-specific and high-level insight, stores it in a database, and then uses relevant insight from it.
Outcome:	The proposed agent outperforms a similar agent when planning by GPT3.5 and is more robust when faced with domain-shifting scenarios.

CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds (2024.emnlp-main)

Copied to clipboard

Min-Hsuan Yeh, Ruyuan Wan, Ting-Hao Huang

Challenge:	Existing algorithms for detecting logical fallacies in texts are expensive and require large-scale labeled datasets.
Approach:	They introduce CoCoLoFa, the largest known logical fallacy dataset, with 7,706 comments for 648 news articles labeled for fallacy presence and type.
Outcome:	The proposed dataset outperforms state-of-the-art LLMs in fallacy detection and classification.

Tokenization Is More Than Compression (2024.emnlp-main)

Copied to clipboard

Craig Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, Chris Tanner

FLIRT: Feedback Loop In-context Red Teaming (2024.emnlp-main)

Challenge:	Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens.
Approach:	They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance.
Outcome:	The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance.

Copied to clipboard

Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta

Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Corrections (2024.emnlp-main)

Challenge:	Recent work has evaluated the vulnerabilities of large generative models, such as DALL-E, ChatGPT, and GPT-4.
Approach:	They propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities against unsafe and inappropriate content generation.
Outcome:	The proposed framework evaluates a given black-box model and exposes its vulnerabilities against unsafe and inappropriate content generation.

Copied to clipboard

Lingjun Zhao, Khanh Nguyen, Hal Daumé Iii

Challenge:	Existing systems that only provide instructions generate inaccurate instructions . however, language models can still guide humans toward making sound decisions .
Approach:	They develop a system that can detect and correct errors in natural language instructions . it can also be used to narrow down search space and reduce misguidance .
Outcome:	The proposed system achieves a 13% increase in success rate and a 29% reduction in final location error distance with 80 users.

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024.emnlp-main)

Copied to clipboard

Haoyuan Wu, Haisheng Zheng, Zhuolun He, Bei Yu

GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation (2024.emnlp-main)

Challenge:	Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing tasks.
Approach:	They propose a parameter-efficient sparsity crafting method which crafts dense models into sparse models using the mixture-of-experts architecture.
Outcome:	The proposed method significantly reduces computational costs and GPU memory requirements, while maintaining the quality of approximation in function space.

Copied to clipboard

Shihao Cai, Keqin Bao, Hangyu Guo, Jizhi Zhang, Jun Song, Bo Zheng

DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities (2024.emnlp-main)

Challenge:	Existing datasets are too challenging for direct model learning or suffer from misalignment between text and images.
Approach:	They propose a pipeline that leverages GPT-4 and GPT4V to generate geometry problems with aligned text and images, facilitating model learning.
Outcome:	The proposed pipeline generates 4.9K geometry problems with aligned text and images, facilitating model learning.

Copied to clipboard

Thong Nguyen, Shubham Chatterjee, Sean MacAvaney, Iain Mackie, Jeff Dalton, Andrew Yates

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024.emnlp-main)

Challenge:	Learned Sparse Retrieval (LSR) models use vocabularies from pre-trained transformers, which often split entities into nonsensical fragments.
Approach:	They propose to enhance the LSR vocabulary with Wikipedia concepts and entities, enabling the model to resolve ambiguities more effectively and stay current with evolving knowledge.
Outcome:	The proposed model outperforms state-of-the-art models across three entity-rich document ranking datasets.

Copied to clipboard

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, Yu Wu

LongEmbed: Extending Embedding Models for Long Context Retrieval (2024.emnlp-main)

Challenge:	Existing studies on parameter-efficient fine-tuning (PEFT) for dense-architecture LLMs are lacking.
Approach:	They propose an expert-specialized fine-tuning method that tunes the experts most relevant to downstream tasks while freezing the other experts.
Outcome:	The proposed method matches or surpasses full-parameter fine-tuning.

Copied to clipboard

Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li

Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences (2024.emnlp-main)

Challenge:	Existing embedding models support only 512 input tokens, hindering their application in scenarios requiring long inputs.
Approach:	They evaluate the performance of existing embedding models by using a new benchmark and a training-free context window extension strategy.
Outcome:	The proposed model extends the input window of existing models by several folds.

Copied to clipboard

Xiangyang Liu, Junliang He, Xipeng Qiu

Overcome Noise and Bias: Segmentation-Aided Multi-Granularity Denoising and Debiasing for Enhanced Quarduples Extraction in Dialogue (2024.emnlp-main)

Challenge:	Recent studies show that large language models can perform complex reasoning tasks without labeled data and unlabeled data.
Approach:	They propose a framework for solving reasoning tasks that store answers in a streaming experience pool and orchestrate helpful questions from the pool to assist itself in answering new questions.
Outcome:	The proposed framework can self-improve as it answers reasoning questions . it stores all answered reasoning questions and their reasoning steps in a streaming experience pool .

Copied to clipboard

Xianlong Luo, Meng Yang, Yihao Wang

Integrating Plutchik’s Theory with Mixture of Experts for Enhancing Emotion Classification (2024.emnlp-main)

Challenge:	Existing methods for generating sentiment quadruples in dialogues face heightened noise and order bias challenges, leading to decreased robustness and accuracy.
Approach:	They propose a Segmentation-Aided multi-grained denoising and debiasing method to address noise and order bias challenges in ABSA.
Outcome:	The proposed method achieves word-level denoising and utterance-level demoising via topic-aware dialogue segmentation.

Copied to clipboard

Dongjun Lim, Yun-Gyung Cheong

In-context Contrastive Learning for Event Causality Identification (2024.emnlp-main)

Challenge:	Existing methods for emotion classification ignore the sentimental aspects of text, resulting in a lack of generalization and sampling bias.
Approach:	They propose a method for emotion classification using Plutchik’s Wheel of Emotions theory and a Mixture of Experts architecture to evaluate the effectiveness.
Outcome:	The proposed method improves the performance of emotion classification.

Copied to clipboard

Liang Chao, Wei Xiang, Bang Wang

What’s Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogs (2024.emnlp-main)

Challenge:	Recent prompt learning-based approaches have shown promising improvements on the ECI task . however, they are subject to the delicate design of multiple prompts and positive correlations between the main task and derivate tasks.
Approach:	They propose an event causality identification model that uses contrastive learning to enhance both positive and negative demonstrations.
Outcome:	The proposed model improves on the event-related causality identification task . it uses contrastive learning to enhance both positive and negative demonstrations .

Copied to clipboard

Anna Wegmann, Tijs Broek, Dong Nguyen

Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs (2024.emnlp-main)

Challenge:	a dataset of utterance pairs from NPR and CNN is used to classify paraphrases in dialog.
Approach:	They propose a dataset annotated for context-dependent paraphrases and develop a training for crowd-workers to classify paraphrase in dialog.
Outcome:	The proposed dataset contains 5,581 annotations on 600 utterance pairs.

Copied to clipboard

Kanishka Misra, Kyle Mahowald

Large Language Models for Data Annotation and Synthesis: A Survey (2024.emnlp-main)

Challenge:	Language models learn rare syntactic phenomena by generalization vs. memorization, a study finds . aannalysis experiments show that humans learn rare grammatical structures by generalizing from less rare phenomena.
Approach:	They iteratively trained transformer language models on a systematically manipulated corpus and evaluated their learning of a rare grammatical phenomenon.
Outcome:	The results show that language models learn rare grammatical phenomena by generalization vs. memorization . human-scale corpora are used to train the models and compare their learning to counterfactual corpors .

Copied to clipboard

Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu

Chain-of-Dictionary Prompting Elicits Translation in Large Language Models (2024.emnlp-main)

Challenge:	Existing surveys focus on LLMs' specific utility for data annotation and synthesis.
Approach:	They propose to use large language models to generate annotations from raw data . they also propose to review learning strategies for models utilizing LLM-generated annotations .
Outcome:	The proposed models can be used to improve the efficacy of machine learning models by generating and labeling raw data with relevant information.

Copied to clipboard

Hongyuan Lu, Haoran Yang, Haoyang Huang, Dongdong Zhang, Wai Lam, Furu Wei

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)

Challenge:	Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation . yet, they struggle with translating low-resource languages.
Approach:	They propose a framework that chained multilingual dictionaries to elicit translation abilities for LLMs . they show that CoD can significantly improve LLM translation by evoking more information .
Outcome:	The proposed framework improves on ChatGPT and InstructGPT's translation abilities.

Copied to clipboard

Yifan Yang, Kai Zhen, Ershad Banijamali, Athanasios Mouchtaris, Zheng Zhang

RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024.emnlp-main)

Challenge:	Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence.
Approach:	They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes.
Outcome:	The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models.

Copied to clipboard

Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, Jing Gao

BlendFilter: Advancing Retrieval-Augmented Large Language Models via Query Generation Blending and Knowledge Filtering (2024.emnlp-main)

Challenge:	Pre-trained language models have strong generalizability, but fine-tuning involves updating all parameters, rendering full fine-uning resource-intensive.
Approach:	They propose a parameter-efficient fine-tuning method that updates all pre-trained parameters during inference.
Outcome:	The proposed method outperforms baseline methods on five benchmarks across 20 datasets.

Copied to clipboard

Haoyu Wang, Ruirui Li, Haoming Jiang, Jinjin Tian, Zhengyang Wang, Chen Luo, Xianfeng Tang, Monica Cheng, Tuo Zhao, Jing Gao

HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs (2024.emnlp-main)

Challenge:	Retrieval-augmented Large Language Models struggle with complex inputs and noisy knowledge retrieval hindering model effectiveness.
Approach:	They propose a query generation method that integrates query generation blending with knowledge filtering to enhance retrieval-augmented LLMs.
Outcome:	The proposed approach surpasses state-of-the-art benchmarks on open-domain question answering benchmarks.

Copied to clipboard

Jocelyn Shen, Joel Mire, Hae Park, Cynthia Breazeal, Maarten Sap

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence (2024.emnlp-main)

Challenge:	Empathy is a foundational psychological process that drives many prosocial functions.
Approach:	They propose a theory-based taxonomy that delineates elements of narrative style that can lead to empathy with the narrator of a story.
Outcome:	The proposed taxonomy delineates elements of narrative style that can lead to empathy with the narrator of a story.

Copied to clipboard

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, Xing Sun

Challenge:	Existing studies attributed verbosity to biased labels, but new research shows that DPO can be effective in mitigating verboses.
Approach:	They propose to use a method to reduce the amount of verbosity in LLMs by using a downsampling approach.
Outcome:	The proposed approach overcomes the problem of verbosity by reducing the length reliance of the proposed algorithm.

Bridging Cultures in the Kitchen: A Framework and Benchmark for Cross-Cultural Recipe Retrieval (2024.emnlp-main)

Copied to clipboard

Tianyi Hu, Maria Maistro, Daniel Hershcovich

RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models (2024.emnlp-main)

Challenge:	Adapting recipes to cultural differences presents significant importance and challenges . bridging cultural differences is a challenge, but IR can help.
Approach:	They propose a framework that preserves the original recipe and its cultural appropriateness for the target culture.
Outcome:	The proposed framework preserves the original recipe and its cultural appropriateness for the target culture while maintaining relevance to the original.

Copied to clipboard

Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, Huaxiu Yao

CryptoTrade: A Reflective LLM-based Agent to Guide Zero-shot Cryptocurrency Trading (2024.emnlp-main)

Challenge:	Existing medical large vision language models often generate inaccurate and irrelevant answers that do not align with established medical facts.
Approach:	They propose a strategy for controlling factuality risk through calibrated selection of the number of retrieved contexts and a preference dataset to fine-tune the model.
Outcome:	The proposed model achieves an average improvement of 20.8% on three medical VQA datasets.

Copied to clipboard

Yuan Li, Bingqiao Luo, Qian Wang, Nuo Chen, Xu Liu, Bingsheng He

A Survey on In-context Learning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have been used for financial decision-making and stock market prediction for years.
Approach:	They propose to use Large Language Models to analyze on-chain and off-chain data to provide a comprehensive overview of the cryptocurrency market.
Outcome:	The proposed trading agent leverages the transparency and immutability of on-chain data, as well as the timeliness and influence of off-chain signals, providing a comprehensive overview of the cryptocurrency market.

Copied to clipboard

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, Zhifang Sui

DocHieNet: A Large and Diverse Dataset for Document Hierarchy Parsing (2024.emnlp-main)

Challenge:	In-context learning (ICL) is a new paradigm for natural language processing . large language models (LLMs) demonstrate the ability to learn from a few examples .
Approach:	They propose to explore ICL to evaluate and extrapolate the ability of large language models.
Outcome:	The proposed methods can be used to evaluate and extrapolate the ability of large language models.

Copied to clipboard

Hangdi Xing, Changxu Cheng, Feiyu Gao, Zirui Shao, Zhi Yu, Jiajun Bu, Qi Zheng, Cong Yao

Challenge:	Existing methods for document hierarchy parsing are limited due to the small scale and inconsistency of datasets.
Approach:	They propose a document hierarchy parsing dataset to compensate for the data scarcity problem and propose 'dHP' framework to grasp fine-grained text content and coarse-grounded pattern at layout element level.
Outcome:	The proposed framework grasps both fine-grained text content and coarse-grounded pattern at layout element level, enhancing the capacity of pre-trained text-layout models in handling multi-page and multi-level challenges.

AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024.emnlp-main)

Copied to clipboard

Ziyang Luo, Xin Li, Hongzhan Lin, Jing Ma, Lidong Bing

EFUF: Efficient Fine-Grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models (2024.emnlp-main)

Challenge:	proprietary large language models (LLMs) have demonstrated impressive code generation performance.
Approach:	They propose an adaptive module-based model that refines the direct response distillation process by modular decomposition and adaptive response evolution.
Outcome:	The proposed framework outperforms baseline model and code generation methods on three popular benchmarks.

Copied to clipboard

Shangyu Xing, Fei Zhao, Zhen Wu, Tuo An, Weihao Chen, Chunhui Li, Jianbing Zhang, Xinyu Dai

Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization (2024.emnlp-main)

Challenge:	Existing methods to eliminate hallucinations require expensive human annotation . hallucination in multimodal large language models poses unique challenges for current research .
Approach:	They propose a fine-grained unlearning framework that performs gradient ascent to eliminate hallucinations without paired data.
Outcome:	The proposed method reduces hallucinations while preserving quality with modest computational overhead.

Copied to clipboard

Sungbin Shin, Wonpyo Park, Jaeho Lee, Namhoon Lee

LLMs Are Zero-Shot Context-Aware Simultaneous Translators (2024.emnlp-main)

Challenge:	minimizing reconstruction error is not always ideal and can overfit calibration data.
Approach:	They propose a method to prune large language models by divide and conquer . they propose minimizing reconstruction error by more than 90% by using calibration data .
Outcome:	The proposed pruning approach generates high reconstruction errors . the proposed technique reduces reconstruction error by more than 90% .

Copied to clipboard

Roman Koshkin, Katsuhito Sudoh, Satoshi Nakamura

AgentReview: Exploring Peer Review Dynamics with LLM Agents (2024.emnlp-main)

Challenge:	Existing SiMT systems operate on a sentence level, disregarding the context established by previous sentences or the broader context implied by previous words.
Approach:	They show that open-source LLMs perform on par with or better than some state-of-the-art baselines in simultaneous machine translation tasks, zero-shot.
Outcome:	The proposed models perform on par with or better than state-of-the-art baselines in simultaneous machine translation tasks, zero-shot.

Copied to clipboard

Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, Jindong Wang

ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024.emnlp-main)

Challenge:	Existing methods of peer review analysis do not address multivariate nature of the process, account for latent variables, and are constrained by privacy concerns due to the sensitive nature of data.
Approach:	They propose a large language model based peer review simulation framework which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue.
Outcome:	The proposed framework disentangles the impacts of multiple latent factors and addresses privacy concerns.

Copied to clipboard

Kelong Mao, Chenlong Deng, Haonan Chen, Fengran Mo, Zheng Liu, Tetsuya Sakai, Zhicheng Dou

Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments (2024.emnlp-main)

Challenge:	a conversational search system requires accurate interpretation of user intent from complex multi-turn contexts.
Approach:	They propose a dual-learning approach that adapts LLMs for retrieval via contrastive learning while enhancing the complex session understanding through masked instruction tuning.
Outcome:	The proposed approach outperforms existing retrieval methods on five conversational search benchmarks.

Copied to clipboard

Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, Anna Korhonen

Learning Interpretable Legal Case Retrieval via Knowledge-Guided Case Reformulation (2024.emnlp-main)

Challenge:	Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language quality.
Approach:	They propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework which produces fairer preference decisions and improves human alignment.
Outcome:	The proposed framework produces fairer preference decisions and better aligns LLMs with humans.

Copied to clipboard

Chenlong Deng, Kelong Mao, Zhicheng Dou

Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process (2024.emnlp-main)

Challenge:	Existing methods for legal case retrieval often overlook the incorporation of legal expert knowledge, leading to unsatisfactory retrieval performance.
Approach:	They propose a legal knowledge-guided case reformulation approach based on large language models for effective and interpretable legal case retrieval.
Outcome:	The proposed model performs better on complex legal case queries than existing methods.

Copied to clipboard

Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, Yong Jiang

Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation (2024.emnlp-main)

Challenge:	Existing studies on large-scale labeled support sets are not feasible in practical scenarios.
Approach:	They introduce a language model-based determinant point process that considers uncertainty and diversity of unlabeled instances for optimal selection.
Outcome:	The proposed method can effectively select canonical examples on 9 NLU and 2 Generation datasets.

Copied to clipboard

Yuhui Zhang, Brandon McKinzie, Zhe Gan, Vaishaal Shankar, Alexander Toshev

QUDSELECT: Selective Decoding for Questions Under Discussion Parsing (2024.emnlp-main)

Challenge:	Recent advances in image tokenizers have enabled text-to-image generation using auto-regressive methods, but these methods lack pre-trained language models for text-based models.
Approach:	They adapt a pre-trained language model for auto-regressive text-to-image generation and show that pre-train language models offer limited help.
Outcome:	The proposed model is compared with a pre-trained language model and shows that it is no more effective than random initialized models.

Copied to clipboard

Ashima Suvarna, Xiao Liu, Tanmay Parekh, Kai-Wei Chang, Nanyun Peng

Challenge:	Question Under Discussion (QUD) uses implicit questions to reveal discourse relationships between sentences.
Approach:	They propose a framework that selectively decodes the QUD dependency structures considering the QUC criteria.
Outcome:	The proposed framework outperforms the state-of-the-art baseline models by 9% in human evaluation and 4% in automatic evaluation.

Mitigating Language Bias of LMMs in Social Intelligence Understanding with Virtual Counterfactual Calibration (2024.emnlp-main)

Copied to clipboard

Peng Chen, Xiao-Yu Guo, Yuan-Fang Li, Xiaowang Zhang, Zhiyong Feng

Model Balancing Helps Low-data Training and Fine-tuning (2024.emnlp-main)

Challenge:	Existing work on social intelligence using large multimodal models is under-explored due to the prevalence of text-based data in the pretraining stage.
Approach:	They propose a structure causal model to mitigate the negative language biases of large multimodal models by preserving beneficial priors.
Outcome:	The proposed model minimizes negative language bias while preserving beneficial priors while avoiding spurious correlations between LMMs' internal commonsense knowledge and the given context.

Copied to clipboard

Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment (2024.emnlp-main)

Challenge:	Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets.
Approach:	They propose a layer-wise learning rate scheduler that balances training quality across layers . they adapt it to a curated dataset to achieve alignment with specialized domains .
Outcome:	The proposed model shows that it can be used to balance training quality across layers and improve low-data training and fine-tuning for both NLP and SciML tasks.

Copied to clipboard

Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, Ahmad Beirami

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)

Challenge:	Multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages.
Approach:	They propose a method where a reward model is trained on preference data in one source language and applied to other target languages.
Outcome:	The proposed approach is effective under comprehensive evaluation settings, including human evaluation.

Copied to clipboard

Kun Luo, Minghao Qin, Zheng Liu, Shitao Xiao, Jun Zhao, Kang Liu

A New Pipeline for Knowledge Graph Reasoning Enhanced by Large Language Models Without Fine-Tuning (2024.emnlp-main)

Challenge:	Pre-trained language models have limited generalization capabilities and performance challenges.
Approach:	They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency .
Outcome:	The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency.

Copied to clipboard

Zhongwu Chen, Long Bai, Zixuan Li, Zhen Huang, Xiaolong Jin, Yong Dou

Towards Tool Use Alignment of Large Language Models (2024.emnlp-main)

Challenge:	Conventional knowledge Graph Reasoning models learn the embeddings of KG components over the structure of a KG.
Approach:	They propose a pipeline to integrate knowledge from LLMs into KGs without fine-tuning . they propose knowledge alignment, KG reasoning and entity reranking to enhance conventional models .
Outcome:	The proposed pipeline can enhance the performance of conventional KGR models in incomplete and general situations.

Copied to clipboard

Zhi-Yuan Chen, Shiqi Shen, Guangyao Shen, Gong Zhi, Xu Chen, Yankai Lin

DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models (2024.emnlp-main)

Challenge:	Existing studies on tool use with LLMs focus on enhancing tool-calling ability of LLM . e.g., LLM should not answer unsafe tool use relevant instructions or insecure tool responses to ensure reliability and harmlessness.
Approach:	They propose to use supervised fine-tuning and preference learning to align LLMs with H2A principle for tool use.
Outcome:	The proposed model demonstrates that LLMs can generate truthful and helpful responses while remaining harmless.

Copied to clipboard

Ranchi Zhao, Zhen Thai, Yifan Zhang, Shengding Hu, Jie Zhou, Yunqi Ba, Jie Cai, Zhiyuan Liu, Maosong Sun

Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are pre-trained on vast datasets composed of billions of tokens harvested from diverse text sources.
Approach:	They propose a data engineering method to refine the pretraining corpus through data rating, tagging and editing.
Outcome:	The proposed method improves the quality of the pretraining corpus by enhancing 100 billion tokens of the training corpus.

Copied to clipboard

Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, James Glass

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment (2024.emnlp-main)

Challenge:	Despite the utility and impressive capabilities of large language models, their tendency to generate hallucinations presents a significant challenge in their deployment.
Approach:	They propose a simple hallucination detection model based on the ratio of attention weights on the context versus newly generated tokens.
Outcome:	The proposed model reduces the amount of hallucinations by 9.6% in a summarization task.

Copied to clipboard

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

Challenge:	Existing algorithms for achieving optimal alignment are mostly unidirectional . a recent study suggests that large language models can be ground with evident preferences .
Approach:	They propose to ground large language models with evident preferences . they propose to use controllable preference optimization to specify different objectives .
Outcome:	The proposed models can provide responses that match various preferences among the ”3H” desiderata.

Mitigating Matthew Effect: Multi-Hypergraph Boosted Multi-Interest Self-Supervised Learning for Conversational Recommendation (2024.emnlp-main)

Copied to clipboard

Yongsen Zheng, Ruilin Xu, Guohua Wang, Liang Lin, Kwok-Yan Lam

Advancing Event Causality Identification via Heuristic Semantic Dependency Inquiry Network (2024.emnlp-main)

Challenge:	Existing methods to mitigate Matthew effect in offline recommendation systems are not effective . a number of studies have identified two root causes for the Matthew effect .
Approach:	They propose a framework to address the Matthew effect in conversational recommendation systems . they build hypergraphs to learn multi-level user interests to alleviate the Matthew effec .
Outcome:	The proposed framework achieves state-of-the-art performance on four CRS-based datasets . it improves on item-, entity-, word-oriented multiple-channel hypergraphs compared with existing methods .

Copied to clipboard

Haoran Li, Qiang Gao, Hongmei Wu, Li Huang

Knowledge Representation

Challenge:	Existing methods for ECI rely on causal features and external knowledge, but these methods fail in two dimensions: causal features between events in texts often lack explicit clues and external information may introduce bias.
Approach:	They propose a simple and effective Semantic Dependency Inquiry Network for ECI that captures semantic dependencies within the context using a unified encoder and generates a fill-in token based on comprehensive context understanding.
Outcome:	Extensive experiments show that SemDI surpasses state-of-the-art methods on three widely used benchmarks.

Exploring Union and Intersection of Visual Regions for Generating Questions, Answers, and Distractors (2024.emnlp-main)

Copied to clipboard

Wenjian Ding, Yao Zhang, Jun Wang, Adam Jatowt, Zhenglu Yang

UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation (2024.emnlp-main)

Challenge:	Existing efforts to generate image-related questions, correct answers, or challenge distractors are limited.
Approach:	They propose to put the spotlight on different image regions to diversify QADs . they propose a framework that generates each QAD based on a recurrent multimodal encoder .
Outcome:	The proposed framework puts the spotlight on different image regions to diversify QADs.

Copied to clipboard

Xiangyu Zhao, Yuehan Zhang, Wenlong Zhang, Xiao-Ming Wu

Tracking the perspectives of interacting language models (2024.emnlp-main)

Challenge:	e-commerce tasks such as multimodal retrieval and multimodal generation are largely ignored due to the diversity of the multimodal fashion domain.
Approach:	They propose a framework that integrates image generation with retrieval and text generation tasks.
Outcome:	The proposed framework outperforms state-of-the-art models across fashion tasks.

Copied to clipboard

Hayden Helm, Brandon Duderstadt, Youngser Park, Carey Priebe

MAR: Matching-Augmented Reasoning for Enhancing Visual-based Entity Question Answering (2024.emnlp-main)

Challenge:	Large language models produce high quality information at unprecedented rates . content produced by these models is propagated throughout forums that influence other models and human users .
Approach:	They propose a method for representing the perspective of individual models within a collection of LLMs.
Outcome:	The proposed method represents the perspective of individual models within a collection of LLMs in various simulated settings.

Copied to clipboard

Zhengxuan Zhang, Yin Wu, Yuyu Luo, Nan Tang

Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? (2024.emnlp-main)

Challenge:	Multimodal large language models (MLLMs) struggle with visual-based entity questions (VEQA) MLLM can identify A, but may refrain from answering due to privacy concerns.
Approach:	They propose a method that uses vector representations to analyze visual-based entity questions (VEQA) they use visual cues and textual information to integrate visual cus and visual information .
Outcome:	The proposed method significantly improves visual-based entity question answering (VEQA) it can identify faces, names, and alignments within visual objects, and then derive the answer over this matching graph.

Copied to clipboard

Zhe Yang, Yichang Zhang, Tianyu Liu, Jian Yang, Junyang Lin, Chang Zhou, Zhifang Sui

Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement (2024.emnlp-main)

Challenge:	Large language models (LLMs) have impressive capabilities, but still suffer from inconsistency issues.
Approach:	They develop a ConsisEval benchmark to evaluate LLMs' inconsistency . they find that LLM models can paradoxically fail at easier problems .
Outcome:	The proposed model achieves highest consistency score but inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc.

Copied to clipboard

Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, Sujian Li

Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation (2024.emnlp-main)

Challenge:	Recent approaches to enhance agent performance focus on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals.
Approach:	They propose a step-level framework that provides detailed step-by-step guidance to enhance agent training by using Monte Carlo methods.
Outcome:	The proposed framework outperforms strong baselines on three tasks and shows that it is effective in augmenting efficiency and its applicability to diverse models.

Copied to clipboard

Joseph Marvin Imperial, Gail Forey, Harish Tayyar Madabushi

Cross-domain NER with Generated Task-Oriented Knowledge: An Empirical Study from Information Density Perspective (2024.emnlp-main)

Challenge:	Domain experts in engineering, healthcare, and education follow strict standards for producing quality content.
Approach:	They propose a retrieval-style in-context learning-based framework to guide large language models to align with expert-defined standards.
Outcome:	The proposed framework shows that models can gain 45% to 100% increase in precise accuracy across open and commercial LLMs evaluated.

Copied to clipboard

Zhihao Zhang, Sophia Lee, Junshuang Wu, Dong Zhang, Shoushan Li, Erik Cambria, Guodong Zhou

Glue pizza and eat rocks - Exploiting Vulnerabilities in Retrieval-Augmented Generative Models (2024.emnlp-main)

Challenge:	Cross-domain Named Entity Recognition (CDNER) is crucial for Knowledge Graph (KG) construction and natural language processing (NLP)
Approach:	They propose to automatically generate task-oriented knowledge using large language models (LLMs) and then employ task-orientated pre-training (TOPT) to facilitate domain adaptation.
Outcome:	The proposed model can learn to distinguish between different entities and improve its domain adaptation.

Copied to clipboard

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, Huan Liu

Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement (2024.emnlp-main)

Challenge:	Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases.
Approach:	They propose to exploit openness of RAG models by injecting deceptive content into the retrieval database, intentionally changing the model’s behavior.
Outcome:	The proposed model can be exploited through crafted content uploads with access to the retriever.

Copied to clipboard

Yuxuan Wang, Xiaoyuan Liu

SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation (2024.emnlp-main)

Challenge:	Existing researches in Scene Graph Generation (SGG) focus on refining model architectures that are trained from scratch with datasets like Visual Genome or Open Images.
Approach:	They propose to integrate pretrained Vision-language Models into SGG to improve representation by estimating the unattainable predicates distribution.
Outcome:	The proposed method significantly improves the performance of the debiased VLMs with SGG models.

Copied to clipboard

Xiaoze Liu, Ting Sun, Tianyang Xu, Feijie Wu, Cunxiang Wang, Xiaoqian Wang, Jing Gao

MatchTime: Towards Automatic Soccer Game Commentary Generation (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have transformed machine learning but have raised significant legal concerns due to their potential to produce text that infringes on copyrights.
Approach:	They propose a lightweight, real-time defense mechanism to prevent the generation of copyrighted text by evaluating methods and testing attack strategies.
Outcome:	The proposed defense significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests.

Copied to clipboard

Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, Weidi Xie

Rethinking Token Reduction for State Space Models (2024.emnlp-main)

Challenge:	Existing data on soccer commentary are often unsatisfactory, and the quality of existing data is often poor.
Approach:	They propose to manually annotate timestamps for 49 soccer matches and then use them to create a model to correct and filter existing data.
Outcome:	The proposed model improves the viewing experience of soccer and can be trained on the curated dataset.

Copied to clipboard

Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, Yanzhi Wang

Challenge:	Existing methods for token reduction for SSMs lead to performance drops . a recent study shows that Mamba-2 improves the accuracy of the model by 5.7% to 13.1% .
Approach:	They propose a token reduction method that integrates token importance and similarity into SSMs and takes advantage of pruning and merging.
Outcome:	The proposed method improves accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods while reducing computational demands and memory requirements.

Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering (2024.emnlp-main)

Copied to clipboard

Chang Zong, Yuchen Yan, Weiming Lu, Jian Shao, Yongfeng Huang, Heng Chang, Yueting Zhuang

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic (2024.emnlp-main)

Challenge:	Recent advances with LLMs have shown promising results across various tasks, but their use in answering questions from knowledge bases remains largely unexplored.
Approach:	They propose a framework that utilizes an LLM-based agent with multiple roles for KBQA tasks.
Outcome:	The proposed framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks yielding F1 scores of 11.8% and 20.7%, respectively.

Copied to clipboard

Yuyan Zhou, Liang Song, Bingning Wang, Weipeng Chen

Event Causality Identification with Synthetic Control (2024.emnlp-main)

Challenge:	Existing methods face the trilemma of performance, data privacy, and computational costs, which hinders their application to LLMs.
Approach:	They propose a model-exclusive task arithmetic method for merging GPT-scale models which is data-agnostic and bypasses the heavy search process.
Outcome:	The proposed method achieves state-of-the-art performance on multiple tasks while minimizing the average loss difference between the merged model and each individual task model.

Copied to clipboard

Haoyu Wang, Fengze Liu, Jiayao Zhang, Dan Roth, Kyle Richardson

Challenge:	Existing approaches to event causality identification have primarily utilized linguistic patterns and multi-hop relational inference, risking false causality .
Approach:	They propose to use the Rubin Causal Model to identify event causality by generating a twin from existing corpora.
Outcome:	The proposed method can identify causal relations more robustly than previous methods, including GPT-4, which is demonstrated on a causality benchmark, COPES-hard.

Retrieved Sequence Augmentation for Protein Representation Learning (2024.emnlp-main)

Copied to clipboard

Chang Ma, Haiteng Zhao, Lin Zheng, Jiayi Xin, Qintong Li, Lijun Wu, Zhihong Deng, Yang Lu, Qi Liu, Sheng Wang, Lingpeng Kong

HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding (2024.emnlp-main)

Challenge:	Using multiple sequence alignments (MSA) to extract evolutionary knowledge is limited.
Approach:	They propose to use multiple sequence alignments to augment protein representations . they propose to employ Retrieved Sequence Augmentation to enhance protein representation learning .
Outcome:	The proposed method surpasses MSA Transformer by 5% in structural and property prediction tasks while being 373 times faster.

Copied to clipboard

Fan Yuan, Chi Qin, Xiaogang Xu, Piji Li

TopViewRS: Vision-Language Models as Top-View Spatial Reasoners (2024.emnlp-main)

Challenge:	Existing work detects hallucination by directly judging whether an object exists in an image, overlooking the association between the object and semantics.
Approach:	They propose a framework that incorporates hallucination feedback at both object and sentence semantic levels to alleviate over 15% of hallucinism.
Outcome:	The proposed framework can alleviate over 15% of hallucination even with a marginal degree of training.

Copied to clipboard

Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, Ivan Vulić

DA3: A Distribution-Aware Adversarial Attack against Language Models (2024.emnlp-main)

Challenge:	Top-view perspective is a typical way in which humans read and reason over different types of maps, but spatial reasoning capabilities of modern VLMs in this setup remain unattested and underexplored.
Approach:	They introduce a top-view spatial reasoning dataset and use it to evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity.
Outcome:	The proposed model can understand and reason over spatial relations from the top view and can be controlled at different granularities of spatial reasoning.

Copied to clipboard

Yibo Wang, Xiangjue Dong, James Caverlee, Philip Yu

Evaluating Psychological Safety of Large Language Models (2024.emnlp-main)

Challenge:	Recent attacks have shown that adversarial examples have a different data distribution than the original examples, reducing their effectiveness under detection methods.
Approach:	They propose a distribution-aware adversarial attack method that considers the distribution shifts of adversarials to improve attacks’ effectiveness under detection methods.
Outcome:	The proposed method improves the effectiveness of adversarial examples under detection methods and integrates both ASR and detectability.

Copied to clipboard

Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, Lidong Bing

An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification (2024.emnlp-main)

Challenge:	a recent study evaluated the psychological safety of large language models.
Approach:	They designed unbiased prompts to evaluate the psychological safety of large language models.
Outcome:	The proposed prompts showed that they were fine-tuned with behavioral metrics to reduce toxicity.

Copied to clipboard

Zhuowei Chen, Lianxi Wang, Yuben Wu, Xinfeng Liao, Yujia Tian, Junyang Zhong

Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering (2024.emnlp-main)

Challenge:	Existing models for textual data augmentation (DA) are highly data-hungry and struggle to perform satisfactorily under noisy conditions.
Approach:	They propose to leverage a diffusion language model to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens.
Outcome:	The proposed method captures in-domain knowledge and generates pseudo samples by reconstructing strong label-related tokens.

Copied to clipboard

Dongze Hao, Qunbo Wang, Longteng Guo, Jie Jiang, Jing Liu

PsFuture: A Pseudo-Future-based Zero-Shot Adaptive Policy for Simultaneous Machine Translation (2024.emnlp-main)

Challenge:	a framework that leverages the visual-language model to select key knowledge retrieved by DPR and answer questions improves performance of the baseline on the open-domain Knowledge-based VQA benchmark, OK-VQA.
Approach:	They propose a framework that leverages visual-language models to retrieve related knowledge . they use dense passage retrieval to retrieve knowledge related to visual-linguistics .
Outcome:	The proposed framework significantly improves the baseline on the open-domain Knowledge-based VQA benchmark, OK-VQA.

Copied to clipboard

Libo Zhao, Jing Li, Ziqian Zeng

TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging (2024.emnlp-main)

Challenge:	Simultaneous machine translation (SiMT) requires target tokens to be generated in real-time as streaming source tokens are consumed.
Approach:	They propose a zero-shot adaptive read/write policy for siMT that generates target tokens concurrently as streaming source tokens are consumed.
Outcome:	The proposed policy achieves performance on par with strong baselines and the P2F method can further enhance performance.

Copied to clipboard

Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, Fei Huang

Do We Need Language-Specific Fact-Checking Models? The Case of Chinese (2024.emnlp-main)

Challenge:	Recent studies have shown that multimodal large language models can be useful for chart understanding, but their size limits their use in resource-constrained environments.
Approach:	They propose an efficient multimodal large language model with only 3B parameters for chart understanding.
Outcome:	The proposed model outperforms several chart-understanding MLLMs with up to 13B parameters on ChartQA, Chart-to-Text, Chart to Table, OpenCQA, and ChartX.

Copied to clipboard

Caiqi Zhang, Zhijiang Guo, Andreas Vlachos

Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)

Challenge:	Existing fact-checking models in other languages lack grounding in real-world claims . current models are constrained to a single domain, like COVID-19 .
Approach:	They propose a Chinese document-level evidence retriever that can be translated into Chinese . they then construct an adversarial dataset that is more robust toward biases .
Outcome:	The proposed method outperforms translation-based methods and multilingual language models and is more robust toward biases.

Copied to clipboard

Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, Weidong Cai

CMD: a framework for Context-aware Model self-Detoxification (2024.emnlp-main)

Challenge:	Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability.
Approach:	They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning.
Outcome:	The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models.

Copied to clipboard

Zecheng Tang, Keyan Zhou, Juntao Li, Yuyang Ding, Pinzheng Wang, Yan Bowen, Renjie Hua, Min Zhang

Embedding and Gradient Say Wrong: A White-Box Method for Hallucination Detection (2024.emnlp-main)

Challenge:	Existing methods of text detoxification fail to achieve a decent balance between effectiveness and generation quality.
Approach:	They propose a text detoxification framework that pays attention to both context and detoxification process.
Outcome:	Experiments on various LLMs show that the proposed framework can yield the best performance compared to baselines.

Copied to clipboard

Xiaomeng Hu, Yiming Zhang, Ru Peng, Haozhe Zhang, Chenwei Wu, Gang Chen, Junbo Zhao

TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control (2024.emnlp-main)

Challenge:	Existing methods for hallucination detection have attracted more attention from the community.
Approach:	They propose to model the distributional distance between the regular conditional output and the unconditional output, which is generated without a given input text.
Outcome:	The proposed model achieves state-of-the-art on the hallucination benchmarks HADES and other datasets.

Copied to clipboard

Yu Zhang, Ziyue Jiang, Ruiqi Li, Changhao Pan, Jinzheng He, Rongjie Huang, Chuxin Wang, Zhou Zhao

Be Helpful but Don’t Talk too Much - Enhancing Helpfulness in Conversations through Relevance in Multi-Turn Emotional Support (2024.emnlp-main)

Challenge:	Existing models fail to generate singing voices rich in stylistic nuances for unseen singers due to multifaceted nature of singing styles.
Approach:	They propose a zero-shot SVS model for style transfer across cross-lingual speech and singing styles and multi-level style control.
Outcome:	Experimental results show that TCSinger outperforms baseline models in synthesis quality, singer similarity, and style controllability.

Copied to clipboard

Junlin Li, Bo Peng, Yu-Yin Hsu, Chu-Ren Huang

Aligning Language Models to Explicitly Handle Ambiguity (2024.emnlp-main)

Challenge:	a helpful speaker should maintain an "effect-effort" tradeoff for a conversation to help and support . a study aimed to cultivate the awareness of "optimal relevance" into the cognitive process of conversation agents .
Approach:	They integrate the "Cognitive Relevance Principle" into emotional support agents . they found that the "relevance principle" is effective in generating human-like, helpful, harmless conversations .
Outcome:	The proposed method improves human-likedness and support in multi-turn conversations . the source code will be available at https://github.com/CN-Eyetk/VLESA-ORL.git .

Copied to clipboard

Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, Taeuk Kim

Tag-grounded Visual Instruction Tuning with Retrieval Augmentation (2024.emnlp-main)

Challenge:	Large language models (LLMs) are not specifically trained to deal with ambiguous utterances . ambiguity can lead to varying interpretations of the same input based on different assumptions or background knowledge .
Approach:	They propose a pipeline that aligns large language models to manage ambiguous queries . they propose to use their own assessment of perceived ambiguity to detect and manage queries a .
Outcome:	Experimental results show that APA empowers LLMs to detect and manage ambiguous queries while retaining the ability to answer clear questions.

Copied to clipboard

Daiqing Qi, Handong Zhao, Zijun Wei, Sheng Li

GLaPE: Gold Label-agnostic Prompt Evaluation for Large Language Models (2024.emnlp-main)

Challenge:	Multimodal Large Language Models (MLLMs) have seen remarkable progress in providing general instruction-following ability, but struggle with critical problems when required to provide a detailed and accurate response to a visual instruction.
Approach:	They propose to enhance the mapping process by using retrieval-augmented tag tokens, which contain rich object-aware information such as object names and attributes.
Outcome:	The proposed model outperforms baselines that share the same language model and training data on 12 benchmarks and shows zero-shot capability when provided with specific datastores.

Copied to clipboard

Xuanchang Zhang, Zhuosheng Zhang, Hai Zhao

Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information (2024.emnlp-main)

Challenge:	Recent studies have explored leveraging the LLM itself as an optimizer to identify optimal prompts that maximize task accuracy.
Approach:	They propose a gold label-agnostic prompt evaluation method to reduce dependence on gold labels.
Outcome:	The proposed method produces more effective prompts even without gold labels.

Copied to clipboard

Runze Xia, Congchi Yin, Piji Li

Optimizing Code Retrieval: High-Quality and Scalable Dataset Annotation through Large Language Models (2024.emnlp-main)

Challenge:	Experimental results demonstrate that this method effectively disentangles the information within fMRI signals.
Approach:	They propose a task Memory Disentangling which extracts and decodes past information from fMRI signals.
Outcome:	The proposed method extracts and decodes past information from fMRI signals.

Copied to clipboard

Rui Li, Qi Liu, Liyang He, Zheng Zhang, Hao Zhang, Shengyu Ye, Junyu Lu, Zhenya Huang

Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models (2024.emnlp-main)

Challenge:	Existing methods for code retrieval struggle to balance scalability and annotation quality.
Approach:	They propose a method that integrates functions called within the repository and information on third-party APIs to enhance the annotation context.
Outcome:	The proposed method improves the annotation context by incorporating functions called within the repository and information on third-party API functionalities.

Copied to clipboard

Yongjin Yang, Jongwoo Ko, Se-Young Yun

Advancing Process Verification for Large Language Models via Tree-Based Preference Learning (2024.emnlp-main)

Challenge:	Vision-language models (VLMs) have demonstrated remarkable applicability across downstream tasks, including zero-shot image classification.
Approach:	They propose an efficient transfer learning method that integrates visual prompts and text adapters with pre-trained VLMs to achieve optimal performance for any target domain.
Outcome:	The proposed method outperforms baselines on unseen tasks.

Copied to clipboard

Mingqian He, Yongliang Shen, Wenqi Zhang, Zeqi Tan, Weiming Lu

An Inversion Attack Against Obfuscated Embedding Matrix in Language Model Inference (2024.emnlp-main)

Challenge:	Existing methods for generating step-by-step rationales fail to fully utilize the relative merits of intermediate steps, limiting the effectiveness of feedback provided.
Approach:	They propose a tree-based preference learning verifier that constructs reasoning trees via a best-first search algorithm and collects step-level paired data for preference training.
Outcome:	The proposed approach outperforms existing benchmarks on arithmetic and commonsense reasoning tasks.

Copied to clipboard

Yu Lin, Qizhi Zhang, Quanwei Cai, Jue Hong, Wu Ye, Huiqi Liu, Bing Duan

VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (2024.emnlp-main)

Challenge:	Recent studies have explored transforming user inputs to obfuscated embedded vectors, so that the data will not be eavesdropped by service providers.
Approach:	They propose to transform user inputs to obfuscated embedded vectors so that the data will not be eavesdropped by service providers.
Outcome:	The proposed inversion attack can recover user input 100% from the obfuscated vectors without a solid and deliberate security design and analysis .

Copied to clipboard

Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Bill Yuchen Lin, Wenhu Chen

LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models (2024.emnlp-main)

Challenge:	Existing video metrics are lagging behind in providing reliable scores over generated videos due to lack of large-scale human-annotated dataset.
Approach:	They propose to use VideoFeedback to train a human-annotated multi-aspect score over 37.6K synthesized videos from 11 existing video generative models.
Outcome:	The proposed model outperforms the prior best metrics by 50 points in the test.

Copied to clipboard

Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, Michael Lyu

Integrating Structural Semantic Knowledge for Enhanced Information Extraction Pre-training (2024.emnlp-main)

Challenge:	LogicAsker examines and improves the reasoning abilities of large language models such as ChatGPT and GPT-4.
Approach:	They propose a set of atomic reasoning skills grounded in propositional and predicate logic to examine and improve the reasoning abilities of large language models such as ChatGPT and GPT-4.
Outcome:	The proposed approach improves reasoning abilities in large language models such as ChatGPT and GPT-4 by up to 5%.

Copied to clipboard

Xiaoyang Yi, Yuru Bao, Jian Zhang, Yifang Qin, Faxin Lin

FuseGen: PLM Fusion for Data-generation based Zero-shot Learning (2024.emnlp-main)

Challenge:	Existing pre-training methods focus on exploiting textual knowledge, which limits scalability and versatility of resulting models.
Approach:	They propose a pre-training framework that integrates structural semantic knowledge via contrastive learning.
Outcome:	The proposed framework outperforms state-of-the-art pre-training methods across multiple tasks.

Copied to clipboard

Tianyuan Zou, Yang Liu, Peng Li, Jianqing Zhang, Jingjing Liu, Ya-Qin Zhang

I Need Help! Evaluating LLM’s Ability to Ask for Users’ Support: A Case Study on Text-to-SQL Generation (2024.emnlp-main)

Challenge:	Existing approaches to train Small Task-specific Models (STMs) using synthetic datasets are limited by the low quality of such datasets.
Approach:	They propose a data-generation based zero-shot learning framework that uses multiple PLMs to train small task-specific models.
Outcome:	The proposed framework outperforms existing methods in boosting performance across tasks.

Copied to clipboard

Cheng-Kuang Wu, Zhi Rui Tam, Chao-Chung Wu, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen

Oddballs and Misfits: Detecting Implicit Abuse in Which Identity Groups are Depicted as Deviating from the Norm (2024.emnlp-main)

Challenge:	a new study examines the proactive ability of large language models to seek user support . without external feedback, many LLMs struggle to recognize their need for user support.
Approach:	They propose metrics to evaluate the trade-off between performance improvements and user burden . they also investigate whether LLMs can determine when to request user support .
Outcome:	The proposed metrics show that without external feedback, many LLMs struggle to recognize their need for user support.

Copied to clipboard

Michael Wiegand, Josef Ruppenhofer

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting (2024.emnlp-main)

Challenge:	Abusive language is often defined as hurtful, derogatory or obscene utterances made by one person to another.
Approach:	They propose to use a dataset to detect abusive sentences in identity groups . they also report on classification experiments.
Outcome:	The proposed dataset includes 7 identity groups and includes classification experiments.

Copied to clipboard

Hyungjun Yoon, Biniyam Tolera, Taesik Gong, Kimin Lee, Sung-Ju Lee

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization (2024.emnlp-main)

Challenge:	Existing text-based prompts for large language models (LLMs) show performance degradation when handling long sensor data sequences.
Approach:	They propose a visual prompt that directs MLLMs to utilize visualized sensor data alongside descriptions of the target sensory task.
Outcome:	The proposed approach achieves 10% higher accuracy and reduces token costs by 15.8 times on nine sensory tasks involving four sensing modalities .

Copied to clipboard

Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, Jaeho Lee

CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search (2024.emnlp-main)

Challenge:	Recent advances in activation quantization methods cause outliers in tokens, causing extra overhead and speedup . a method to quantize per-tensor activation is currently challenging due to the outlier activation outlier.
Approach:	They propose a method to find a set of key-value cache which mitigates outliers in subsequent tokens when inserted as a prefix.
Outcome:	The proposed method surpasses the established baseline of per-tensor activation quantization and can be seamlessly integrated with the recent activation quantitative method.

Copied to clipboard

Fengran Mo, Abbas Ghaddar, Kelong Mao, Mehdi Rezagholizadeh, Boxing Chen, Qun Liu, Jian-Yun Nie

Towards Low-Resource Harmful Meme Detection with LMM Agents (2024.emnlp-main)

Challenge:	Recent advances in task-solving capabilities of Large Language Models (LLMs) have motivated researchers to integrate these models into existing conversational search systems.
Approach:	They propose a method that leverages the capabilities of large language models to resolve ambiguities in conversation history before query rewriting.
Outcome:	The proposed method leads to state-of-the-art results across most settings compared with closed-source LLMs.

Copied to clipboard

Jianzhao Huang, Hongzhan Lin, Liu Ziyan, Ziyang Luo, Guang Chen, Jing Ma

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values (2024.emnlp-main)

Challenge:	Existing methods for harmful meme detection are limited due to the dynamic nature of memes . eliciting knowledge-revising behavior within the LMM agent is a key factor in achieving this goal .
Approach:	They propose an agency-driven framework for low-resource harmful meme detection . they use annotated memes to leverage label information as auxiliary signals for model .
Outcome:	The proposed framework achieves superior performance than state-of-the-art methods on the low-resource harmful meme detection task.

Copied to clipboard

Zhe Hu, Yixiao Ren, Jing Li, Yu Yin

Direct Multi-Turn Preference Optimization for Language Agents (2024.emnlp-main)

Challenge:	Recent large vision language models (VLMs) have demonstrated remarkable intelligence and proficiency across diverse tasks.
Approach:	They propose a benchmark for VIsion-grounded decision-making driven by human VA.
Outcome:	The proposed model can make decisions under a vision-depicted situation using human values and human values.

Copied to clipboard

Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, Fuli Feng

Challenge:	Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss function.
Approach:	They propose a novel loss function for multi-turn agent tasks that replaces the policy constraint with the state-action occupancy measure constraint and adds length normalization to the Bradley-Terry model.
Outcome:	Experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the proposed loss function.

Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models (2024.emnlp-main)

Copied to clipboard

Leonardo Ranaldi, Andre Freitas

In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search (2024.emnlp-main)

Challenge:	Existing approaches to align reasoning abilities between Large Language Models and Smaller Language Model are supervised fine-tuning and preference optimization.
Approach:	They propose a method that elicits Smaller Language Models to self-improve their reasoning abilities via preference optimization.
Outcome:	The proposed method outperforms Instruction-tuning on commonsense and math reasoning tasks on common and math scenarios.

Copied to clipboard

Huihan Li, Yuting Ning, Zeyi Liao, Siyuan Wang, Xiang Li, Ximing Lu, Wenting Zhao, Faeze Brahman, Yejin Choi, Xiang Ren

AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation (2024.emnlp-main)

Challenge:	Logic-Induced-Knowledge-Search (LINK) is a framework for generating factually-correct yet long-tail inferential knowledge.
Approach:	They introduce a framework to obtain factually-correct yet long-tail inferential statements using variable-wise prompting grounded on symbolic rules.
Outcome:	The proposed framework is able to obtain factually-correct yet long-tail inferential statements while ensuring factual correctness.

Copied to clipboard

Wenhao Huang, Zhouhong Gu, Chenghao Peng, Jiaqing Liang, Zhixu Li, Yanghua Xiao, Liqian Wen, Zulong Chen

Challenge:	Existing methods for web scraping suffer from limited adaptability and scalability when faced with a new website.
Approach:	They propose a framework that generates web scrapers with large language models and a new executability metric to measure the performance of web scraper generation tasks.
Outcome:	The proposed framework can handle diverse web environments more efficiently.

Backward Lens: Projecting Language Model Gradients into the Vocabulary Space (2024.emnlp-main)

Copied to clipboard

Shahar Katz, Yonatan Belinkov, Mor Geva, Lior Wolf

Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (2024.emnlp-main)

Challenge:	Recent interpretability methods project weights and hidden states obtained from the forward pass to the models’ vocabularies, helping to uncover how information flows within LMs.
Approach:	They propose to cast a gradient matrix as a low-rank linear combination of forward and backward passes’ inputs and then to project these gradients into vocabulary items.
Outcome:	The proposed method can be cast as a low-rank linear combination of forward and backward passes’ inputs and project these gradients into vocabulary items.

Copied to clipboard

Jiwan Chung, Sungjae Lee, Minseo Kim, Seungju Han, Ashkan Yousefpour, Jack Hessel, Youngjae Yu

Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! (2024.emnlp-main)

Challenge:	Visual arguments rely on images to persuade viewers to do or believe something .
Approach:	They propose three tasks for evaluating visual argument understanding . they use visual premises, commonsense premises and reasoning trees to analyze visual arguments .
Outcome:	The proposed tasks evaluate visual argument understanding using a dataset of 1,611 images annotated with 5,112 visual premises (with regions), 5,574 commonsense premises, and reasoning trees connecting them into structured arguments.

Copied to clipboard

Jiwan Chung, Seungwon Lim, Jaehyun Jeon, Seungbeen Lee, Youngjae Yu

Reusing Transferable Weight Increments for Low-resource Style Generation (2024.emnlp-main)

Challenge:	Existing models lack this active understanding capacity, limiting their applicability in real-world scenarios.
Approach:	They propose a benchmark to assess the impact of multimodal inputs on lexical ambiguities.
Outcome:	The proposed benchmark assesses the impact of multimodal inputs on lexical ambiguities.

Copied to clipboard

Chunzhen Jin, Eliot Huang, Heng Chang, Yaqi Wang, Peng Cao, Osmar Zaiane

Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course (2024.emnlp-main)

Challenge:	Text style transfer (TST) is crucial in natural language processing, aiming to endow text with a new style without altering its meaning.
Approach:	They propose a framework to use style features in weight increments to transfer low-resource styles effectively.
Outcome:	The proposed framework achieves remarkable performance across different backbones, achieving particularly effective results in low-resource scenarios.

Copied to clipboard

Cheng-Han Chiang, Wei-Chih Chen, Chun-Yi Kuan, Chienchou Yang, Hung-yi Lee

Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers? (2024.emnlp-main)

Challenge:	Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research.
Approach:	They use large language models (LLMs) for automatic evaluation to evaluate a sample . they propose several recommendations for integrating LLMs into future classroom evaluations .
Outcome:	The proposed model is able to output high scores without meeting the evaluation instructions, the authors note . their model is not able for students to manipulate the model to output specific strings, they say .

Copied to clipboard

Neeladri Bhuiya, Viktor Schlegel, Stefan Winkler

Instruction Pre-Training: Language Models are Supervised Multitask Learners (2024.emnlp-main)

Challenge:	State-of-the-art Large Language Models (LLMs) are accredited with a number of different capabilities, including reading comprehension, mathematical and reasoning skills, and possessing scientific knowledge.
Approach:	They propose a benchmark to generate seemingly plausible multi-hop reasoning chains that ultimately lead to incorrect answers.
Outcome:	The proposed model circumvents the reasoning requirement but in subtle ways . it shows that it is more difficult to generate plausible alternatives .

Copied to clipboard

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models (2024.emnlp-main)

Challenge:	Unsupervised multitask pre-training has been the key to the success of language models (LMs) however, scaling it in the post-training stage trends towards better generalization.
Approach:	They propose a framework that augments massive raw corpora with instruction-response pairs to pre-train LMs.
Outcome:	The proposed framework augments massive raw corpora with instruction-response pairs to pre-train LMs.

Copied to clipboard

Renzhi Wang, Piji Li

Collaborative Performance Prediction for Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for single and batch model editing fail to apply or perform sub-optimally when faced with lifelong model editing.
Approach:	They propose an advanced Mixture of Experts (MoE) adaptor for lifelong model editing that incorporates a novel KV anchor routing method to enhance routing consistency between training and inference stage.
Outcome:	The proposed method surpasses existing models while maintaining outstanding performance in batch editing task.

Copied to clipboard

Qiyuan Zhang, Fuyuan Lyu, Xue Liu, Chen Ma

Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese (2024.emnlp-main)

Challenge:	Large language models (LLMs) are one of the most important AI research powered by largescale parameters, high computational resources, and massive training data.
Approach:	They propose a framework that leverages historical performance of large language models and other design factors to improve prediction accuracy.
Outcome:	The proposed framework surpasses scaling laws in predicting performance of large language models . it also facilitates a detailed analysis of factor importance, an area previously overlooked .

Copied to clipboard

Yuqi Chen, Sixuan Li, Ying Li, Mohammad Atari

Knowledge Verification to Nip Hallucination in the Bud (2024.emnlp-main)

Challenge:	Humans have produced written language for thousands of years, but most computational work is focused on contemporary languages and corpora.
Approach:	They propose a pipeline for historical-psychological text analysis in classical Chinese . they propose an indirect contrastive learning approach that fine-tunes pre-trained models .
Outcome:	The proposed pipeline outperforms word-embedding-based approaches across all tasks and exceeds prompting with GPT-4 in most tasks.

Copied to clipboard

Fanqi Wan, Xinting Huang, Leyang Cui, Xiaojun Quan, Wei Bi, Shuming Shi

QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios (2024.emnlp-main)

Challenge:	Recent studies have shown that large language models generate responses that sound plausible but contradict factual knowledge, a phenomenon known as hallucination.
Approach:	They propose a novel approach to align large language models to evaluate knowledge boundaries based on external knowledge to reduce hallucinations .
Outcome:	The proposed approach reduces hallucinations across six benchmarks using foundation LLMs of varying backbones and scales.

Copied to clipboard

Timo Schrader, Lukas Lange, Simon Razniewski, Annemarie Friedrich

African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification (2024.emnlp-main)

Challenge:	Existing probabilistic reasoning datasets require the model to only rank textual alternatives or use limited set of templates.
Approach:	They propose a question-answering dataset that uses probabilistic rules to express degrees of certainty.
Outcome:	The proposed model outperforms existing models on all reasoning types . it is available on Github and is expected to be used in clinical documentation .

Copied to clipboard

Gregor Geigle, Radu Timofte, Goran Glavaš

Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Models (2024.emnlp-main)

Challenge:	Recent Large Vision Language Models demonstrate impressive abilities on image understanding and reasoning tasks.
Approach:	They propose a benchmark for fine-grained object classification that is difficult to evaluate . they benchmark 12 public LVLMs on and show CLIP models exhibit better performance .
Outcome:	The proposed model improves on 12 public LVLMs on image understanding and reasoning tasks.

Copied to clipboard

Hongbang Yuan, Pengfei Cao, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, Jun Zhao

To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models (2024.emnlp-main)

Challenge:	Large language models generate hallucinated text when confronted with false premise questions . authors propose a method to mitigate false premises hallucinosity .
Approach:	They propose a method to constrain false premise attention heads during the model inference process.
Outcome:	The proposed method improves performance by constraining false premise attention heads . it yields a notable increase of nearly 20% of model performance .

Copied to clipboard

Bastien Liétard, Pascal Denis, Mikaela Keller

ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings (2024.emnlp-main)

Challenge:	Word Sense Disambiguiation and Word sense Induction are considered independent problems, but they are often neglected in practice.
Approach:	They propose an unsupervised task of learning a soft clustering amongwords that defines a set of concepts directly from data.
Outcome:	The proposed approach leverages both a local and global cross-lexicon view to induce concepts and also senses in the context of the proposed task.

Copied to clipboard

Hao Wang, Hao Li, Minlie Huang, Lei Sha

An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making (2024.emnlp-main)

Challenge:	Attaching suffixes to harmful instructions can hack the defense of Large language models (LLMs) However, due to the unreadable of adversarial suffix, it can be relatively easily penetrated by common defense methods such as perplexity filters.
Approach:	They propose an algorithm to embed adversarial suffixes into coherent and understandable text to attack Large language models (LLMs) using a Advbench dataset.
Outcome:	The proposed approach reduces the computation time of adversarial suffixes and achieves a much better attack success rate than existing techniques.

Copied to clipboard

Xiutian Zhao, Ke Wang, Wei Peng

Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? (2024.emnlp-main)

Challenge:	Recent advances in large language models have sparked interest in collaborative LLM agents.
Approach:	They propose to integrate various ordinal preferential voting mechanisms into LLMs to improve reasoning capabilities and robustness.
Outcome:	The proposed method improves reasoning capabilities and robustness of leading LLMs without complex system designs.

Copied to clipboard

Gregor Geigle, Radu Timofte, Goran Glavaš

Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment (2024.emnlp-main)

Challenge:	Large vision-language models (LVLMs) often hallucinate and produce captions that mention concepts that cannot be found in the image.
Approach:	They propose to add grounding objectives to captions that explicitly align image regions or objects to text spans to reduce hallucination.
Outcome:	The proposed evaluation protocol reduces the amount of hallucination in LVLMs by adding grounding objectives.

Copied to clipboard

Zhenyu Liu, Dongfang Li, Xinshuo Hu, Xinping Zhao, Yibin Chen, Baotian Hu, Min Zhang

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning (2024.emnlp-main)

Challenge:	Recent studies have explored the working mechanisms of In-Context Learning (ICL) however, they mainly focus on classification and simple generation tasks, limiting their broader application to more complex generation tasks in practice.
Approach:	They propose an efficient Progressive In-Context Alignment method that embeds the task function learned from demonstrations into the separator token representation.
Outcome:	The proposed method surpasses vanilla ICL and achieves comparable performance to other alignment tuning methods.

Copied to clipboard

Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang

Message Passing on Semantic-Anchor-Graphs for Fine-grained Emotion Representation Learning and Classification (2024.emnlp-main)

Challenge:	Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing.
Approach:	They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning.
Outcome:	The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability.

Copied to clipboard

Pinyi Zhang, Jingyang Chen, Junchen Shen, Zijie Zhai, Ping Li, Jie Zhang, Kai Zhang

PhiloGPT: A Philology-Oriented Large Language Model for Ancient Chinese Manuscripts with Dunhuang as Case Study (2024.emnlp-main)

Challenge:	Emotion classification is an important task with applications in education, virtual reality, and robotics.
Approach:	They propose to use token embeddings to generate a "semantic-anchor graph" using semantic anchors, sentences can be projected onto them to form a graph .
Outcome:	Empirically, the proposed system can generate meaningful semantic anchors and discriminative graph patterns for different emotion.

Copied to clipboard

Yuqing Zhang, Baoyi He, Yihan Chen, Hangqi Li, Han Yue, Shengyu Zhang, Huaiyong Dou, Junchi Yan, Zemin Liu, Yongquan Zhang, Fei Wu

Alignment-Enhanced Decoding: Defending Jailbreaks via Token-Level Adaptive Refining of Probability Distributions (2024.emnlp-main)

Challenge:	philology requires years of professional training in extensive knowledge memorization and manual textual retrieval.
Approach:	They curated the PhiloCorpus-ZH, a rich collec-tion of ancient Chinese texts spanning a millennium with 30 diverse topics, including firsthand folk copies.
Outcome:	The PhiloCorpus-ZH corpus facilitated the development of the first LLM tailored for discovering ancient Chinese manuscripts.

Copied to clipboard

Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, Sen Su

MiniConGTS: A Near Ultimate Minimalist Contrastive Grid Tagging Scheme for Aspect Sentiment Triplet Extraction (2024.emnlp-main)

Challenge:	Existing defenses against jailbreaks focus on perturbing or inspecting inputs, but ignore competing objectives, the underlying cause of alignment failures.
Approach:	They propose a novel defense that employs adaptive decoding to address the root causes of jailbreak issues.
Outcome:	The proposed defense improves safety alignment while maintaining helpfulness.

Copied to clipboard

Qiao Sun, Liujia Yang, Minghao Ma, Nanyang Ye, Qinying Gu

Evaluating Large Language Models via Linguistic Profiling (2024.emnlp-main)

Challenge:	Existing approaches within the pretraining-finetuning paradigm tend to meticulously craft complex tagging schemes and classification heads, or incorporate external semantic enhancements to enhance performance.
Approach:	They propose to integrate a minimalist tagging scheme and a novel token-level contrastive learning strategy to improve pretrained representations.
Outcome:	The proposed framework achieves comparable or superior performance compared to state-of-the-art techniques while featuring a more compact design and reduced computational overhead.

Copied to clipboard

Alessio Miaschi, Felice Dell’Orletta, Giulia Venturi

With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) undergo extensive evaluation against various benchmarks collected in established leaderboards to assess their performance across multiple tasks.
Approach:	They propose a new evaluation methodology to test LLMs' sentence generation abilities under specific linguistic constraints.
Outcome:	The proposed evaluation methodology is based on the 'linguistic profiling' approach and is not intended to be a task-oriented evaluation.

Copied to clipboard

Tyler Loakman, Yucheng Li, Chenghua Lin

KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases (2024.emnlp-main)

Challenge:	Large Language Models and Vision Language Model (VLMs) have demonstrated aptitude as potential substitutes for human participants in psycholinguistic experiments.
Approach:	They examine whether large language models and vision language models implicitly understand sound-based phenomena via orthography and imagery alone.
Outcome:	The proposed models demonstrate sound symbolism and ability to "hear" using language and vision modules.

Copied to clipboard

Jiajie Zhang, Shulin Cao, Linmei Hu, Ling Feng, Lei Hou, Juanzi Li

Understanding Higher-Order Correlations Among Semantic Components in Embeddings (2024.emnlp-main)

Challenge:	Program induction (PI) is a promising paradigm for using knowledge bases (KBs) to help large language models answer complex knowledge-intensive questions.
Approach:	They propose a plug-and-play framework that enables large language models to induce programs over any low-resourced KB.
Outcome:	Experiments show that KB-Plugin outperforms SoTA low-resourced PI methods with 25x smaller backbone LLM on large-scale and domain-specific KBs and even approaches the performance of supervised methods.

Copied to clipboard

Momose Oyama, Hiroaki Yamagiwa, Hidetoshi Shimodaira

DGLF: A Dual Graph-based Learning Framework for Multi-modal Sarcasm Detection (2024.emnlp-main)

Challenge:	Independent Component Analysis (ICA) is an effective method for visualizing and interpreting the geometric structure of embeddings.
Approach:	They quantified embeddings' non-independencies using higher-order correlations and a maximum spanning tree of semantic components.
Outcome:	The results provide deeper insights into embeddings through ICA.

Copied to clipboard

Zhihong Zhu, Kefan Shen, Zhaorun Chen, Yunyan Zhang, Yuyan Chen, Xiaoqi Jiao, Zhongwei Wan, Shaorong Xie, Wei Liu, Xian Wu, Yefeng Zheng

Evaluating D-MERIT of Partial-annotation on Information Retrieval (2024.emnlp-main)

Challenge:	Existing methods for multimodal sarcasm detection neglect high-order relationships and underestimate high-frequency messages.
Approach:	They propose a Dual Graph-based Learning Framework to capture inter-modal inconsistencies . they propose combining a hypergraph and a vanilla graph to achieve enhanced propagation .
Outcome:	The proposed model outperforms existing state-of-the-art methods on two benchmark datasets.

Copied to clipboard

Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, Yoav Goldberg

Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving (2024.emnlp-main)

Challenge:	Using partially-annotated datasets for evaluation can lead to false conclusions . a dataset containing only a subset of relevant passages might result in misleading rankings .
Approach:	They propose to use a Wikipedia passage retrieval evaluation set to contain all relevant passages for each query.
Outcome:	The proposed dataset can be downloaded from https://d-merit.github.io.

Copied to clipboard

Xin Quan, Marco Valentino, Louise Dennis, Andre Freitas

Calibrating the Confidence of Large Language Models by Eliciting Fidelity (2024.emnlp-main)

Challenge:	Existing methods for assessing the validity of explanations for NLI are time-consuming and prone to logical errors.
Approach:	They propose a framework that integrates Large Language Models and Theorem Provers to verify and refine natural language explanations through crowd-sourcing . they propose to use TPs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI.
Outcome:	The proposed framework generates and formalises explanatory sentences and suggests potential inference strategies for NLI.

Copied to clipboard

Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, Xipeng Qiu

The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models (2024.emnlp-main)

Challenge:	Large language models with RLHF and RLAIF have good alignment but exhibit overconfidence post-alignment.
Approach:	They propose a plug-and-play method to estimate the confidence of large language models.
Outcome:	The proposed method has shown good calibration performance on 6 RLHF-LMs on four MCQA datasets.

Copied to clipboard

Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen

How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics (2024.emnlp-main)

Challenge:	Reinforcement Learning from Human Feedback (RLHF) significantly enhances Natural Language Processing by aligning language models with human expectations.
Approach:	They propose to integrate feedback from humans into RLHF to improve language models by capturing human-like preferences.
Outcome:	The proposed model outperforms models trained with moderately accurate reward models on relevance, factuality, and completeness tasks.

Copied to clipboard

Adrian Cosma, Stefan Ruseti, Mihai Dascalu, Cornelia Caragea

Challenge:	Popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance.
Approach:	They propose a method for the automated creation of a challenging test set without relying on manual construction of artificial and unrealistic examples.
Outcome:	The proposed method reduces spurious correlations and improves model performance . examples labeled as having the highest difficulty show markedly decreased performance compared to the full dataset .

Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detection (2024.emnlp-main)

Copied to clipboard

Gaetan Latouche, Marc-André Carbonneau, Benjamin Swanson

CUTE: Measuring LLMs’ Understanding of Their Tokens (2024.emnlp-main)

Challenge:	Existing methods for Grammatical Error Detection (GED) rely on human annotations, but these are unavailable in many low-resource languages.
Approach:	They propose a two-stage fine-tuning pipeline to train a GED model using synthetic errors from target languages and human-annotated GED corpora from source languages.
Outcome:	The proposed method outperforms current state-of-the-art annotation-free GED methods and produces errors that are more diverse and similar to human errors.

Copied to clipboard

Lukas Edman, Helmut Schmid, Alexander Fraser

Challenge:	Large Language Models (LLMs) perform well on a wide variety of tasks, authors say . they lack direct access to characters, which can be difficult to generalize to new languages .
Approach:	They propose a benchmark to test the orthographic knowledge of Large Language Models . they find that most LLMs seem to know the spelling of their tokens - yet fail to manipulate text .
Outcome:	The proposed benchmark tests the orthographic knowledge of large language models . it finds that most LLMs seem to know the spelling of their tokens, but fail to manipulate text .

SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generation (2024.emnlp-main)

Copied to clipboard

Xinping Zhao, Dongfang Li, Yan Zhong, Boren Hu, Yibin Chen, Baotian Hu, Min Zhang

On the Role of Context in Reading Time Prediction (2024.emnlp-main)

Challenge:	Existing methods for enhancing RAG performance rely on heuristic-based augmentation . Existing approaches rely heavily on a heuriistic-driven approach, resulting in poor generalization and skews in the evidence length.
Approach:	They propose a model-based evidence extraction learning framework that optimizes a vanilla model as an evidence extractor with desired properties through self-aligned learning.
Outcome:	The proposed method reduces the evidence length by 9.25 times and improves reliability and reliability.

Copied to clipboard

Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, Ethan Wilcox

BC-Prover: Backward Chaining Prover for Formal Theorem Proving (2024.emnlp-main)

Challenge:	a new perspective on how readers integrate context during reading time prediction is presented . a recent study shows that the proportion of variance in reading times explained by context is smaller when context is represented by the orthogonalized predictor.
Approach:	They propose a technique where they project surprisal onto the orthogonal complement of frequency.
Outcome:	The proposed method shows that the proportion of variance in reading times explained by context is smaller when context is represented by the orthogonalized predictor.

Copied to clipboard

Yuhang He, Jihai Zhang, Jianzhu Bao, Fangquan Lin, Cheng Yang, Bing Qin, Ruifeng Xu, Wotao Yin

From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP (2024.emnlp-main)

Challenge:	Existing methods for interactive theorem proving in formal logic lack robustness and robustness.
Approach:	They propose a backward chaining framework guided by pseudo steps for proofstep generation that prioritizes pseudo steps.
Outcome:	The proposed framework improves on the miniF2F benchmark.

Copied to clipboard

Marius Mosbach, Vagrant Gautam, Tomás Vergara Browne, Dietrich Klakow, Mor Geva

Autoregressive Pre-Training on Pixels and Texts (2024.emnlp-main)

Challenge:	Interpretability and analysis (IA) research is a growing subfield within NLP . a criticism of this work is that it lacks actionable insights and therefore has little impact on NLP.
Approach:	They propose to quantify the impact of interpretation and analysis research on NLP . they use citation graphs and a survey to find out what is missing in IA research .
Outcome:	The proposed study shows that IA research is well-cited outside of IA and central in the NLP citation graph.

Copied to clipboard

Yekun Chai, Qingyi Liu, Jingwu Xiao, Shuohuan Wang, Yu Sun, Hua Wu

On Training Data Influence of GPT Models (2024.emnlp-main)

Challenge:	pixel-based language modeling integrates visual and textual data to improve performance of language models.
Approach:	They propose a method that integrates visual and textual data into an autoregressive framework.
Outcome:	The proposed method improves performance of pixel-based language models by incorporating visual and textual data.

Copied to clipboard

Yekun Chai, Qingyi Liu, Shuohuan Wang, Yu Sun, Qiwei Peng, Hua Wu

Understanding “Democratization” in NLP and ML Research (2024.emnlp-main)

Challenge:	generative language models have redefined performance standards across tasks . current research on the influence of training data on autoregressivity remains underexplored .
Approach:	They propose a parameterized simulation to assess the impact of training examples on the training dynamics of GPT models.
Outcome:	The proposed approach compares existing methods with existing methods across training scenarios in generative language models, spanning tasks across 14 million to 2.8 billion parameters.

Copied to clipboard

Arjun Subramonian, Vagrant Gautam, Dietrich Klakow, Zeerak Talat

Linguistic Theories

Challenge:	a large number of NLP and ML papers mention terms related to democracy . authors find that democratization is most frequently used to convey (ease of) access to or use of technologies without meaningfully engaging with theories of democratisation.
Approach:	They analyze papers using the term "democra*" to clarify how it is understood in NLP and ML . they find that democratization is most frequently used to convey (ease of) access to or use of technologies .
Outcome:	The authors analyze papers using the term "democra*" they find that democratization is most frequently used to convey (ease of) access to or use of technologies without meaningfully engaging with theories of democratisation.

DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models (2024.emnlp-main)

Copied to clipboard

Sungnyun Kim, Haofu Liao, Srikar Appalaraju, Peng Tang, Zhuowen Tu, Ravi Satzoda, R. Manmatha, Vijay Mahadevan, Stefano Soatto

Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languages (2024.emnlp-main)

Challenge:	Existing methods for visual document understanding are limited by training on a small-scale, curated document dataset, compromising generalizability of VDU models to diverse documents.
Approach:	They propose a framework that integrates external document knowledge into the data generation process.
Outcome:	The proposed framework produces high-quality annotations and surpasses direct knowledge distillation approach.

Copied to clipboard

Seonjeong Hwang, Yunsu Kim, Gary Lee

ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws (2024.emnlp-main)

Challenge:	Existing automatic question generation datasets focus on English, resulting in data gaps for other languages.
Approach:	They propose a cross-lingual transfer method that allows models to generate questions in low-resource languages.
Outcome:	The proposed method outperforms other models and achieves comparable performance across languages.

Copied to clipboard

Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, Houwen Peng

Word Alignment as Preference for Machine Translation (2024.emnlp-main)

Challenge:	Existing quality filtering methods rely on a high-quality dataset as reference . Existing methods introduce potential biases and compromise diversity .
Approach:	They propose a method that evaluates text quality based on the perplexity difference between two language models trained on the same data.
Outcome:	The proposed approach improves performance of pre-trained models without increasing training costs.

Copied to clipboard

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, Yoshimasa Tsuruoka

Improving Multi-party Dialogue Generation via Topic and Rhetorical Coherence (2024.emnlp-main)

Challenge:	Hallucination and omission are a problem in machine translation because of an LLM's size and low-resource languages.
Approach:	They propose to use word alignment as preference to optimize an LLM-based MT model to mitigate hallucination and omission problems.
Outcome:	The proposed model is able to mitigate hallucination and omission by using word alignment as preference.

Copied to clipboard

Yaxin Fan, Peifeng Li, Qiaoming Zhu

SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models (2024.emnlp-main)

Challenge:	Existing studies on multi-party dialogue generation focus on the reply-to structure of dialogue histories, but they neglect the coherence between generated responses and target utterances.
Approach:	They propose a Reinforcement Learning approach emphasizing Topic and Rhetorical Coherence to enhance the model's perception of coherence with the target utterance.
Outcome:	The proposed approach significantly outperforms the state-of-the-art baselines on two popular datasets.

Copied to clipboard

Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, Jinqiao Wang

Neuron-Level Knowledge Attribution in Large Language Models (2024.emnlp-main)

Challenge:	Existing methods fail to fully exploit the knowledge embedded in models from previous tasks . Existing techniques fail to exploit the information embedded in previous tasks, resulting in a large number of replay samples to achieve good results.
Approach:	They propose a method that uses attention weights to extract knowledge from previous tasks . they use a data replay strategy to extract the knowledge from the previous tasks.
Outcome:	The proposed method achieves comparable or even better performance with only 1/10 of replayed data used by other methods.

Copied to clipboard

Zeping Yu, Sophia Ananiadou

How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning (2024.emnlp-main)

Challenge:	Existing methods for attribution of knowledge in large language models struggle to operate at neuron level due to computational constraints.
Approach:	They propose a static method for pinpointing significant neurons using three metrics . they also propose identifying "query neurons" which activate these "value neurons"
Outcome:	The proposed method shows superior performance across three metrics compared to seven other methods . it analyzes six types of knowledge across attention and feed-forward network layers .

Copied to clipboard

Zeping Yu, Sophia Ananiadou

Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis (2024.emnlp-main)

Challenge:	In-context learning (ICL) is an emergent ability of large language models.
Approach:	They propose to use in-context learning to predict sentences with semantically-unrelated labels on 1% heads to investigate the mechanism.
Outcome:	The proposed methods reduce the majority label bias and recency bias by 22% and 17%, respectively.

Copied to clipboard

Zeping Yu, Sophia Ananiadou

Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models (2024.emnlp-main)

Challenge:	Existing studies have found that arithmetic ability is limited to a few attention heads . existing studies do not elaborate on the mechanisms of these heads or how they influence FFN layers.
Approach:	They propose a method that identifies an internal logic chain consisting of four stages from input to prediction.
Outcome:	The proposed method improves prediction probabilities by amplifying coefficient scores of FFN neurons related to predictions.

Copied to clipboard

Kushal Tatariya, Vladimir Araujo, Thomas Bauwens, Miryam De Lhoneux

GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory (2024.emnlp-main)

Challenge:	PIXEL is a vision transformer that has been pre-trained on rendered text . however, it is not able to outperform monolingual subwords like BERT .
Approach:	They propose to use PIXEL as a vision transformer to train on rendered text to explore the gap between its visual and linguistic understanding.
Outcome:	The proposed model outperforms monolingual subword models in most other contexts, but it lacks the linguistic knowledge to perform in language tasks.

Copied to clipboard

Wei Fan, Haoran Li, Zheye Deng, Weiqi Wang, Yangqiu Song

Noise, Novels, Numbers. A Framework for Detecting and Categorizing Noise in Danish and Norwegian Literature (2024.emnlp-main)

Challenge:	Existing research studies privacy by exploring various privacy attacks, defenses, and evaluations within narrowly predefined patterns.
Approach:	They propose a framework that leverages the theory of contextual integrity as a bridge to help LLMs understand the complex contexts for judicial assessing privacy violations.
Outcome:	The proposed framework bridges the theory of contextual integrity as a bridge, creating numerous synthetic scenarios grounded in relevant privacy statutes (e.g., HIPAA).

Copied to clipboard

Ali Al-Laith, Daniel Hershcovich, Jens Bjerring-Hansen, Jakob Parby, Alexander Conroy, Timothy Tangherlini

Challenge:	This study examines the literary perceptions of noise during the Scandinavian "Modern Breakthrough" period (1870-1899).
Approach:	They propose a framework for detecting and categorizing noise in literary texts from the late 19th century.
Outcome:	The proposed framework can be applied to Danish and Norwegian literature from the late 19th century.

QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models (2024.emnlp-main)

Copied to clipboard

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh

Fine-Grained Prediction of Reading Comprehension from Eye Movements (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are extremely popular, leading to a race towards reducing their inference costs.
Approach:	They propose a method that quantizes weights and activations to 4 bits to achieve better accuracy.
Outcome:	The proposed method reduces runtime costs in memory-bound models but does not address cost-bound scenarios.

Copied to clipboard

Omer Shubi, Yoav Meiri, Cfir Hadar, Yevgeni Berzak

Machine Reading Comprehension

Challenge:	a new study attempts to assess reading comprehension from eye movements in reading . eye movements provide small improvements over a text-only baseline, the authors argue .
Approach:	They propose to use eyetracking data to predict reading comprehension of a single participant . they use a battery of recent models and three new multimodal language models .
Outcome:	The proposed model can predict reading comprehension of a single participant from eye movements over a paragraph.

EfficientRAG: Efficient Retriever for Multi-Hop Question Answering (2024.emnlp-main)

Copied to clipboard

Ziyuan Zhuang, Zhiyang Zhang, Sitao Cheng, Fangkai Yang, Jia Liu, Shujian Huang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

Unsupervised Human Preference Learning (2024.emnlp-main)

Challenge:	Existing retrieval-augmented generation methods rely on multiple calls of large language models (LLMs) Large-language models lack knowledge underrepresented in training data and still face hallucinations.
Approach:	They propose an efficient retriever for multi-hop question answering that generates new queries iteratively without the need for LLM calls.
Outcome:	The proposed method surpasses existing methods on three open-domain multi-hop question-answering datasets.

Copied to clipboard

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Dilek Tur

Is Safer Better? The Impact of Guardrails on the Argumentative Strength of LLMs in Hate Speech Countering (2024.emnlp-main)

Challenge:	Existing methods for in-context learning and parameter-efficient fine-tuning fail to capture the complexity of human preferences, especially given the small, personal datasets individuals possess.
Approach:	They propose a method that uses small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization.
Outcome:	The proposed method outperforms baseline personalization methods on email and article datasets and significantly outperformed existing methods.

Copied to clipboard

Helena Bonaldi, Greta Damo, Nicolás Ocampo, Elena Cabrio, Serena Villata, Marco Guerini

Leading Whitespaces of Language Models’ Subword Vocabulary Pose a Confound for Calculating Word Probabilities (2024.emnlp-main)

Challenge:	Automated responses lack argumentative richness which characterises expert-produced counterspeech.
Approach:	They propose to automate counterspeech generation by investigating tension between helpfulness and harmlessness of LLMs and to assess whether presence of safety guardrails hinders quality of generations.
Outcome:	The proposed approach produces more cogent responses that lack argumentative richness which characterises expert-produced counterspeech.

Copied to clipboard

Byung-Doh Oh, William Schuler

LLM4Decompile: Decompiling Binary Code with Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for aggregating word-by-word conditional probabilities into word probabilities are flawed . tokens in subword vocabulary of most language models have leading whitespaces and therefore do not define stop probabilities of words.
Approach:	They propose a method to reaccount the probability of trailing whitespace into that of the current word . they show lower estimates of garden-path effects in transitive/intransitive sentences .
Outcome:	The proposed method corrects the confound in word-by-word probabilities from LMs . the proposed method lowers garden-path effects in transitive/intransitive sentences .

Copied to clipboard

Hanzhuo Tan, Qi Luo, Jing Li, Yuqun Zhang

From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)

Challenge:	Decompilation aims to convert binary code to high-level source code, but traditional tools like Ghidra often produce results that are difficult to read and execute.
Approach:	They propose an open-source LLM series trained to decompile binary code . they optimize the LLM training process and introduce the Llm4Decompile-End models .
Outcome:	The proposed models outperform GPT-4o and Ghidra on the HumanEval and ExeBench benchmarks by over 100% in terms of re-executability rate.

Copied to clipboard

Jihao Gu, Zelin Wang, Yibo Zhang, Ziji Zhang, Ping Gong

CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering (2024.emnlp-main)

Challenge:	Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information.
Approach:	They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters.
Outcome:	The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers.

Copied to clipboard

Yike Wu, Yi Huang, Nan Hu, Yuncheng Hua, Guilin Qi, Jiaoyan Chen, Jeff Pan

MTLS: Making Texts into Linguistic Symbols (2024.emnlp-main)

Challenge:	Existing knowledge rewriting methods may include irrelevant information, omit crucial details, or fail to align with the question’s semantics.
Approach:	They propose a new rewriting method CoTKR for generating reasoning traces and corresponding knowledge in an interleaved manner, thereby mitigating the limitations of single-step knowledge rewrite.
Outcome:	The proposed method mitigates the limitations of single-step knowledge rewriting and bridges the preference gap between the knowledge reactor and the question answering (QA) model.

Copied to clipboard

Wenlong Fei, Xiaohua Wang, Min Hu, Qingyu Zhang, Hongbo Li

D2R: Dual-Branch Dynamic Routing Network for Multimodal Sentiment Detection (2024.emnlp-main)

Challenge:	In linguistics, all languages can be considered as symbolic systems . most work overlooks the properties of languages as symbol systems - aaron et al., 1989).
Approach:	They propose a method to make texts into linguistic symbols to improve multilingual capability . they use a pre-training method to replace pre-trained language models with a vocabulary map .
Outcome:	The proposed method improves multilingual capabilities on multilingual tasks using BERT and RoBERTa as the backbone.

Copied to clipboard

Yifan Chen, Kuntao Li, Weixing Mai, Qiaofeng Wu, Yun Xue, Fenghuan Li

A Generic Method for Fine-grained Category Discovery in Natural Language Texts (2024.emnlp-main)

Challenge:	Existing methods for multimodal sentiment detection use the same fixed framework to classify the sentiment polarity of image-text pairs.
Approach:	They propose a multimodal dynamic interaction model that uses a fixed framework to classify the sentiment polarity of a given imagetext pair.
Outcome:	The proposed model outperforms state-of-the-art models on three publicly available datasets.

Copied to clipboard

Chang Tian, Matthew Blaschko, Wenpeng Yin, Mingzhe Xing, Yinliang Yue, Marie-Francine Moens

Toxicity Detection is NOT all you Need: Measuring the Gaps to Supporting Volunteer Content Moderators through a User-Centric Method (2024.emnlp-main)

Challenge:	Existing methods for fine-grained category discovery neglect semantic similarities of fine-grain categories.
Approach:	They propose a method that detects fine-grained clusters of semantically similar texts guided by a novel objective function.
Outcome:	The proposed method surpasses state-of-the-art methods on three benchmark tasks.

Copied to clipboard

Yang Cao, Lovely-Frances Domingo, Sarah Gilbert, Michelle Mazurek, Katie Shilton, Hal Daumé Iii

A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024.emnlp-main)

Challenge:	Existing efforts to automate content moderation have focused on identifying toxic, offensive, and hateful content . yet, it remains unclear whether improvements have addressed the needs of volunteer content moderators .
Approach:	They propose to use a model review to examine the availability of moderators' models to flag violations of various forum rules.
Outcome:	The proposed models perform poorly on a significant portion of the rules.

Copied to clipboard

Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, Jian-Yun Nie

Decompose and Compare Consistency: Measuring VLMs’ Answer Reliability via Task-Decomposition Consistency Comparison (2024.emnlp-main)

Challenge:	Existing benchmarks focus on specific predefined model abilities, such as world knowledge, reasoning, etc., making it difficult for users to determine which LLM best suits their particular needs.
Approach:	They propose to evaluate large language models from a user-centric perspective and use real-world use cases to identify their effectiveness under distinct intents.
Outcome:	The proposed benchmarks achieve a correlation between human preference and the user-reported scenarios and human intents.

Copied to clipboard

Qian Yang, Weixiang Yan, Aishwarya Agrawal

Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism (2024.emnlp-main)

Challenge:	Existing methods for estimating uncertainty using answer likelihoods or prompt-based confidence generation often suffer from overconfidence and confirmation biases.
Approach:	They propose to use Decompose and Compare Consistency (DeCC) to measure the reliability of a VLM's direct answer and indirect answers by decomposing the question into sub-questions and reasoning over the sub-answers.
Outcome:	Experiments on six vision-language tasks with three VLMs show that DeCC achieves better correlation with task accuracy compared to existing methods.

Copied to clipboard

Lang Cao

VGBench: A Comprehensive Benchmark of Vector Graphics Understanding and Generation for Large Language Models (2024.emnlp-main)

Challenge:	Recent advances in large language models have demonstrated impressive language understanding and generation capabilities, enabling them to answer a wide range of questions across various domains.
Approach:	They propose a refusal mechanism that instructs LLMs to refuse to answer challenging questions in order to avoid errors.
Outcome:	The proposed approach improves the controllability and reliability of large language models and their ability to answer questions across domains.

Copied to clipboard

Bocheng Zou, Mu Cai, Jianrui Zhang, Yong Jae Lee

What do Large Language Models Need for Machine Translation Evaluation? (2024.emnlp-main)

Challenge:	Current vision models use pixels to rasterize the visual world, but vector graphics are not the best or unique way to represent visual content.
Approach:	They propose a benchmark for vector graphics processing with capable Large Language Models . they use a set of questions to evaluate vector graphics formats and a wide range of prompting techniques .
Outcome:	The proposed benchmark compares LLMs on rasterized representations with vector graphics . it shows that LLM models show strong capability on both aspects .

Copied to clipboard

Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, Fred Blain

Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale (2024.emnlp-main)

Challenge:	Existing research shows that large language models can perform better in machine translation tasks.
Approach:	They propose to use large language models for machine translation evaluations . authors explore what translation information is needed for LLMs to evaluate MT quality .
Outcome:	The proposed model performs comparable to fine-tuned multilingual pre-trained models.

Copied to clipboard

Flavio Palo, Prateek Singhi, Bilal Fadlallah

External Knowledge-Driven Argument Mining: Leveraging Attention-Enhanced Multi-Network Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) face high computational demands at inference time due to high computational costs.
Approach:	They propose a cost-effective and high-throughput solution for large language models . PGKD distills the knowledge of LLMs into smaller, task-specific models based on teacher-student knowledge distillation .
Outcome:	PGKD outperforms BERT-based models and other knowledge distillation methods on multi-class classification datasets.

Copied to clipboard

Debela Gemechu, Chris Reed

Knowledge Representation

Challenge:	Argument mining involves the identification of argument relations (AR) between Argumentative Discourse Units (ADUs).
Approach:	They propose to leverage external resources to identify semantic paths linking ADUs . they propose to use WordNet, ConceptNet, and Wikipedia to identify these paths .
Outcome:	The proposed architecture achieves F-scores of 0.85, 0.84, 0.70, and 0.87 on four datasets.

C3PA: An Open Dataset of Expert-Annotated and Regulation-Aware Privacy Policies to Enable Scalable Regulatory Compliance Audits (2024.emnlp-main)

Copied to clipboard

Maaz Musa, Steven Winston, Garrison Allen, Jacob Schiller, Kevin Moore, Sean Quick, Johnathan Melvin, Padmini Srinivasan, Mihailis Diamantis, Rishab Nithyanand

M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning (2024.emnlp-main)

Challenge:	Privacy policies fall short of achieving compliance goals due to their inaccessibility or incomprehensibility.
Approach:	They propose to use C3PA to create an open regulation-aware dataset of expert-annotated privacy policies to aid automated audits of compliance with CCPA-related disclosure mandates.
Outcome:	The proposed dataset is uniquely suited for aiding automated audits of compliance with CCPA-related disclosure mandates from 411 unique organizations.

Copied to clipboard

Taowen Wang, Yiyang Liu, James Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu

Text Grafting: Near-Distribution Weak Supervision for Minority Classes in Text Classification (2024.emnlp-main)

Challenge:	Multimodal Large Language Models (MLLMs) exhibit remarkable performance across a wide range of domains.
Approach:	They propose a multimodal prompt tuning approach for efficient instruction tuning of MLLMs.
Outcome:	The proposed approach shows superior performance on multimodal evaluation datasets compared to state-of-the-art methods.

Copied to clipboard

Letian Peng, Yi Gu, Chengyu Dong, Zihan Wang, Jingbo Shang

Incubating Text Classifiers Following User Instruction with Nothing but LLM (2024.emnlp-main)

Challenge:	Recent work generates pseudo labels by mining texts similar to the class names from the raw corpus, but there is a high risk that LLMs cannot generate in-distribution data, leading to ungeneralizable classifiers.
Approach:	They propose to use LLMs to generate pseudo labels by mining masked templates from corpus . they then use state-of-the-art LLM to synthesize near-distribution texts falling into minority classes .
Outcome:	The proposed framework improves on the previous methods for extremely weak-supervised text classification.

Copied to clipboard

Letian Peng, Zilong Wang, Jingbo Shang

PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL (2024.emnlp-main)

Challenge:	In this paper, we aim to generate text classification data given arbitrary class definitions . Traditional supervised text classification fine-tunes models on expensive human annotation .
Approach:	They propose a framework that can generate text classification data given arbitrary class definitions . they use instruction-to-data mappings and in-context augmentation to refine the framework .
Outcome:	The proposed framework outperforms existing methods on benchmarks and training data generation by prompt engineering.

Copied to clipboard

Ruilin Luo, Liyuan Wang, Binghuai Lin, Zicheng Lin, Yujiu Yang

Conditional and Modal Reasoning in Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are powerful tools for Text-to-SQL tasks . SQL solutions have a relatively fixed pattern, allowing for categorical thinking .
Approach:	They propose that query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, thus enhancing their reasoning abilities across diverse difficulty levels and problem categories.
Outcome:	The proposed model outperforms state-of-the-art models on the Spider and BIRD datasets.

Copied to clipboard

Wesley Holliday, Matthew Mandelkern, Cedegao Zhang

Advancing Large Language Model Attribution through Self-Improving (2024.emnlp-main)

Challenge:	Inferences involving conditionals and modals play a central role in the fundamental human ability to reason about distal possibilities.
Approach:	They propose to assess the extent to which 29 LLMs are able to distinguish logically correct inferences from logical fallacious ones.
Outcome:	The LLMs that make basic mistakes with conditionals and modals display inconsistent judgments across inference patterns involving epistemic modal and conditionals, and give answers about complex conditional inferences that do not match reported human judgments.

Copied to clipboard

Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin

AlignCap: Aligning Speech Emotion Captioning to Human Preferences (2024.emnlp-main)

Challenge:	Teaching large language models to generate text with citations to evidence sources requires high-quality attribution data, which is costly and labor-intensive.
Approach:	They propose a framework for iteratively improving the attribution capability of large language models (LLMs) by attributing output to verifiable sources.
Outcome:	Experiments on three open-domain question-answering datasets show that START improves in aggregating information across multiple sources.

Copied to clipboard

Ziqi Liang, Haoxiang Shi, Hanhui Chen

Interpretability-based Tailored Knowledge Editing in Transformers (2024.emnlp-main)

Challenge:	Existing methods for speech emotion capture often produce hallucinations and lose generalization on unseen speech.
Approach:	They propose to align speech emotion captioning to human preference based on large language model (LLM) and human preference regularization to eliminate factuality and faithfulness hallucinations.
Outcome:	Experiments show that AlignCap performs better than existing methods on Zero-shot SEC task.

Copied to clipboard

Yihuai Hong, Aldo Lipani

PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling (2024.emnlp-main)

Challenge:	Existing methods for modifying in-context learning fail to analyze the instability of in-constitu learning outcomes.
Approach:	They propose a model-based knowledge editing method that considers the unique information flow of each sample and aims to correct errors without costly retraining.
Outcome:	The proposed method exploits the critical role of feed-forward MLPs in decoder-only models and reveals diverse attribute recall across transformer layers, guiding edits to specific features at different depths and mitigating over-editing issues.

Copied to clipboard

Yongchao Chen, Jacob Arkin, Yilun Hao, Yang Zhang, Nicholas Roy, Chuchu Fan

Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting (2024.emnlp-main)

Challenge:	Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task.
Approach:	They propose a method to optimize prompts for LLM-driven multi-step tasks using a human-designed feedback rule.
Outcome:	The proposed method outperforms human-engineered prompts and several other prompt optimization methods on 11 representative multi-step tasks.

Copied to clipboard

Chen Cai, Zheng Wang, Jianjun Gao, Wenyang Liu, Ye Lu, Runzhong Zhang, Kim-Hui Yap

Dissecting Fine-Tuning Unlearning in Large Language Models (2024.emnlp-main)

Challenge:	Existing VideoQA models struggle to adapt to new questions or tasks posed by newly available content.
Approach:	They propose a continual learning framework that fine-tunes a large language model for a sequence of tasks and integrates specific question constraint prompting, knowledge acquisition prompting and visual temporal awareness prompting.
Outcome:	The proposed model achieves 55.14% accuracy on both NExT-QA and DramaQA datasets and 71.24% accuracy for DramaQA.

Copied to clipboard

Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, Haiqin Yang

Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning-based unlearning are ineffective at completely erasing model-embedded knowledge, but their true effectiveness remains unclear.
Approach:	They propose to use activation patching and parameter restoration experiments to examine the limitations of fine-tuning-based unlearning methods for erasing harmful, sensitive, or copyrighted information within large language models.
Outcome:	The proposed methods alter the model’s knowledge retrieval process rather than genuinely erasing the problematic knowledge embedded in the model parameters.

Copied to clipboard

Zhengxuan Wu, Yuhao Zhang, Peng Qi, Yumo Xu, Rujun Han, Yian Zhang, Jifan Chen, Bonan Min, Zhiheng Huang

Where is the signal in tokenization space? (2024.emnlp-main)

Challenge:	Modern language models fail to follow human instructions while being faithful . a trade-off exists between instruction following and faithfulness when training LMs .
Approach:	They propose a method that relies on Reject-sampling by Self-instruct with Continued Fine-tuning to train LMs to follow human instructions while being faithful.
Outcome:	The proposed method outperforms vanilla MTL with high-quality data, but with significantly smaller data.

Copied to clipboard

Renato Geh, Honghua Zhang, Kareem Ahmed, Benjie Wang, Guy Van Den Broeck

Private Language Models via Truncated Laplacian Mechanism (2024.emnlp-main)

Challenge:	Autoregressive large language models (LLMs) generate text by predicting the next word sequentially, but tokenization is a significant challenge.
Approach:	They propose to use a deterministic, rule-based mapping from text to token sequences to compute the marginal probability over all possible tokenizations.
Outcome:	The proposed method improves on a range of LLM evaluation benchmarks for a variety of architectures, including transformers and state space models.

Copied to clipboard

Tianhao Huang, Tao Yang, Ivan Habernal, Lijie Hu, Di Wang

Estimating Knowledge in Large Language Models Without Generating a Single Token (2024.emnlp-main)

Challenge:	Existing methods for word embedding are prone to privacy leakage, resulting in weaker relaxations of DP that are inferior to the canonical DP in terms of privacy strength.
Approach:	They propose a method for private word embedding that uses a non-trivial extension of the truncated Laplacian mechanism and propose to test its effectiveness.
Outcome:	The proposed method has lower variance compared to the previous methods.

Copied to clipboard

Daniela Gottesman, Mor Geva

Consistent Autoformalization for Constructing Mathematical Libraries (2024.emnlp-main)

Challenge:	Existing methods to evaluate knowledge in large language models require querying and evaluating the model's generated responses.
Approach:	They ask whether it is possible to estimate how knowledgeable a model is about a subject entity only from its internal computation.
Outcome:	The proposed model performs well with QA accuracy and FActScore . it can be leveraged to guide decisions on how to apply further training or augment queries with retrieval.

Copied to clipboard

Lan Zhang, Xin Quan, Andre Freitas

When Context Leads but Parametric Memory Follows in Large Language Models (2024.emnlp-main)

Challenge:	Autoformalization is the task of automatically translating mathematical content written in natural language to a formal language expression.
Approach:	They propose to use three mechanisms to improve autoformalization quality . they propose to combine most-similar retrieval augmented generation, denoising steps and auto-correction with syntax error feedback to improve syntactic, terminological and semantic control.
Outcome:	The proposed mechanisms can deliver syntactically, terminologically and semantically more consistent results across different models.

Copied to clipboard

Yufei Tao, Adam Hiatt, Erik Haake, Antonie Jetter, Ameeta Agrawal

Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers (2024.emnlp-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources.
Approach:	They propose to analyze how large language models prioritize and utilize local context and parametric knowledge when answering open-ended questions in knowledge-consistent scenarios.
Outcome:	The proposed model prioritizes parametric and contextual knowledge over parametric knowledge in knowledge-consistent scenarios and their tendency to hallucinate under varying context sizes.

Copied to clipboard

Aditya Yedetore, Najoung Kim

When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2024.emnlp-main)

Challenge:	Neural networks without hierarchical biases struggle to learn linguistic rules that come naturally to humans . et al., 2018: Transformers trained on form and meaning favor hierarchically generalization more than those trained on forms alone.
Approach:	They examine whether neural networks without hierarchical biases can generalize more like humans . they find that Transformers trained on form and meaning favor hierarchic generalization .
Outcome:	The proposed neural networks perform better on syntactic evaluations when trained on form and meaning compared to those trained on forms alone.

Copied to clipboard

Tyler Chang, Catherine Arnett, Zhuowen Tu, Ben Bergen

Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use (2024.emnlp-main)

Challenge:	Multilingual language models are widely used to extend NLP systems to low-resource languages.
Approach:	They pre-train over 10,000 monolingual and multilingual language models for over 250 languages including multiple language families that are under-studied in NLP.
Outcome:	The results show that adding multilingual data improves low-resource language modeling performance, similar to increasing low-source dataset sizes by up to 33%.

Copied to clipboard

Jiajun Xi, Yinong He, Jianing Yang, Yinpei Dai, Joyce Chai

Challenge:	Existing methods for embodied agents to learn and perform tasks use low-level instructions, which may not reflect natural human communication.
Approach:	They propose to use different types of language inputs to facilitate reinforcement learning (RL) embodied agents.
Outcome:	The proposed methods show that agents trained with diverse and informative language can achieve enhanced generalization and fast adaptation to new tasks in an open world.

MiTTenS: A Dataset for Evaluating Gender Mistranslation (2024.emnlp-main)

Copied to clipboard

Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, Jasmijn Bastings

Teaching LLMs to Abstain across Languages via Multilingual Feedback (2024.emnlp-main)

Challenge:	Existing studies on gender mistranslation in translation systems have highlighted the problem . a dataset of 26 languages is presented to measure the extent of such errors .
Approach:	They propose a dataset that measures the extent of gender mistranslation in translation systems . they use handcrafted passages that target known failure patterns and synthetically generated passages .
Outcome:	The proposed dataset covers 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources.

Copied to clipboard

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Orevaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, Yulia Tsvetkov

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration (2024.emnlp-main)

Challenge:	Existing studies on LLM abstention focus on English, but they show that it can reduce the accuracy of the model by 20.5% .
Approach:	They propose to teach LLMs to abstain in the face of knowledge gaps by generating multiple feedback items in related languages.
Outcome:	Extensive experiments show that the proposed approach outperforms baselines and achieves 9.2% improvement for low-resource languages.

Copied to clipboard

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, Yulia Tsvetkov

StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements (2024.emnlp-main)

Challenge:	Existing alignment paradigms for large language models learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities.
Approach:	They propose a modular framework that "plugs" into a base LLM a pool of smaller but specialized community LMs where models collaborate in distinct modes to support three modes of pluralism: Overton, steerable, and distributional.
Outcome:	The proposed framework “plugs into” a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to support three modes of pluralism: Overton, steerable, and distributional.

Copied to clipboard

Jillian Fisher, Skyler Hallinan, Ximing Lu, Mitchell Gordon, Zaid Harchaoui, Yejin Choi

I Could’ve Asked That: Reformulating Unanswerable Questions (2024.emnlp-main)

Challenge:	Authorship obfuscation methods that ignore author-specific stylistic features are often too rigid and lead to degradation of fluency and grammaticality.
Approach:	They propose an adaptive obfuscation method that perturbs stylistic elements of text . authors release a large set of 30K high-quality, long-form texts from a diverse set of 14 authors .
Outcome:	The proposed method outperforms state-of-the-art methods on an array of domains on automatic and human evaluation.

Copied to clipboard

Wenting Zhao, Ge Gao, Claire Cardie, Alexander Rush

Challenge:	Existing large language models do not assist users in reformulating unanswerable questions . a recent study found that the models failed to reformulate questions based on assumptions that conflict with or cannot be verified with the information available in documents.
Approach:	They evaluate open-source and proprietary LLMs on couldAsk to evaluate their performance . they found that GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time .
Outcome:	The proposed model successfully reformulates questions only 26% and 12% of the time . the proposed model is not able to reformulate questions, but it can be improved .

STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions (2024.emnlp-main)

Copied to clipboard

Robert Morabito, Sangmitra Madhusudan, Tyler McDonald, Ali Emami

Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters (2024.emnlp-main)

Challenge:	Existing models that assess explicit and implicit biases are based on a single scenario . a dataset of 450 offensive progressions contains 2,700 sentences of varying severity .
Approach:	They evaluate a dataset of offensive progressions that contain 2,700 sentences . they find that even the best-performing models detect bias inconsistently .
Outcome:	The proposed dataset shows that even the best-performing models detect bias inconsistently . aligning models with human judgments on STOP can improve answer rates on sensitive tasks by 191% .

Copied to clipboard

Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, Dawn Song

Challenge:	This paper examines the political leanings of large language models (LLMs) in the 2024 election.
Approach:	They propose to use large language models to examine users' political leanings in the 2024 presidential election to determine their political preference.
Outcome:	The proposed models show that they have a political leaning and can influence political views in the 2024 presidential election.

SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning (2024.emnlp-main)

Copied to clipboard

Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, Sijia Liu

When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have highlighted the need for effective unlearning mechanisms to comply with data regulations and ethical AI practices.
Approach:	They propose a second-order optimization-based LLM unlearning framework which extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process.
Outcome:	The proposed framework outperforms first-order methods across unlearning tasks, models, and metrics.

Copied to clipboard

Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Wenlin Yao, Hassan Foroosh, Dong Yu, Fei Liu

An Analysis of Multilingual FActScore (2024.emnlp-main)

Challenge:	Using sports data, an LLM can analyze sports narratives to infer points from actions, identify related entities, attribute points accurately to players and teams, and draw conclusions.
Approach:	They propose a method to synthesize NBA basketball game narratives using real NBA basketball data and propose 'SportsGen' they find that most models fail to accurately aggregate basketball scores due to frequent scoring patterns and open-source models suffer from significant score hallucinations.
Outcome:	The proposed method can evaluate LLMs’ reasoning capabilities under complex scenarios with varying narrative lengths and density of information.

Copied to clipboard

Vu Kim, Michael Krumdick, Varshini Reddy, Franck Dernoncourt, Viet Lai

Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)

Challenge:	Recent advances in LLMs have demonstrated significant capabilities in many applications.
Approach:	They propose a dataset for FActScore on texts generated by strong multilingual LLMs and evaluate their performance in other languages.
Outcome:	The proposed dataset shows that LLMs exhibit distinct behaviors in fact extraction and fact scoring tasks.

Copied to clipboard

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo

RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering (2024.emnlp-main)

Challenge:	Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate .
Approach:	They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs.
Outcome:	The proposed language model-based evaluation model is more powerful than its predecessor.

Copied to clipboard

Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, Vittorio Castelli

PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024.emnlp-main)

Challenge:	Existing datasets for question answering based on retrieval augmented generation (RAG-QA) are either constructed using a single source corpus or consist of short extractive answers, which fall short of evaluating large language model (LLM) based RAG-QA systems on cross-domain generalization.
Approach:	They propose a dataset that integrates short extractive answers from multiple documents into a single coherent narrative.
Outcome:	The proposed dataset integrates short extractive answers from multiple documents into a single coherent narrative, covering 26K queries and large corpora across seven different domains.

Copied to clipboard

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon

Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects (2024.emnlp-main)

Challenge:	Large language models (LLMs) excel in zero-shot document ranking tasks.
Approach:	They propose a prompt-based re-ranking method that requires no further training but is only feasible for reranking a handful of candidates due to computational costs.
Outcome:	The proposed method can retrieve documents from the entire corpus without training and with a large amount of paired text data.

Copied to clipboard

Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Adelani, Daud Abolade, Noah Smith, Yulia Tsvetkov

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback (2024.emnlp-main)

Challenge:	Recent efforts to develop NLP tools for low-resource languages focus on their standard dialects.
Approach:	They propose a high-quality parallel text and speech corpus for Yoruba . they use native speakers to collect data from four regional yoruba dialects .
Outcome:	The proposed dataset shows that dialect-adaptive finetuning can narrow performance disparities . the dataset will be released publicly under an open license .

Copied to clipboard

Ju-Seung Byun, Jiyun Chun, Jihyung Kil, Andrew Perrault

Order of Magnitude Speedups for LLM Membership Inference (2024.emnlp-main)

Challenge:	Large Multimodal Models excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks.
Approach:	They propose an algorithm that alters REinforcement Learning and Supervised Fine-Tuning to refine large multimodal models with specific preferences.
Outcome:	The proposed algorithm achieves 70% win rate compared to baseline models judged by GPT-4o.

Copied to clipboard

Rongting Zhang, Martin Bertran, Aaron Roth

VIMI: Grounding Video Generation through Multi-modal Instruction (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are complex and require fine-tuning on proprietary datasets to improve performance and relevance.
Approach:	They propose a low-cost membership inference attack that leverages an ensemble of small quantile regression models to determine if a document belongs to the model’s training set.
Outcome:	The proposed approach achieves comparable or improved accuracy on fine-tuned LLMs of varying families and across multiple datasets.

Copied to clipboard

Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Tsai-Shien Chen, Kuan-Chieh Wang, Ivan Skorokhodov, Graham Neubig, Sergey Tulyakov

F2RL: Factuality and Faithfulness Reinforcement Learning Framework for Claim-Guided Evidence-Supported Counterspeech Generation (2024.emnlp-main)

Challenge:	Existing text-to-video diffusion models rely on text-only encoders for their pretraining, restricting their versatility and application in multimodal integration.
Approach:	They propose a multimodal conditional video generation framework for pretraining on augmented text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within a model.
Outcome:	The proposed model can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control.

Copied to clipboard

Haiyang Wang, Yuchen Pan, Xin Song, Xuechen Zhao, Minghao Hu, Bin Zhou

Deciphering Rumors: A Multi-Task Learning Approach with Intent-aware Hierarchical Contrastive Learning (2024.emnlp-main)

Challenge:	Existing methods for generating evidence-supported counterspeech lack clear guidance with a core claim for organizing evidence.
Approach:	They propose a Factuality and Faithfulness Reinforcement Learning framework for generating claim-guided and evidence-supported counterspeech (F2RL) they generate counter-claims based on hate speech and design a self-evaluation mechanism to select the most appropriate one.
Outcome:	The proposed framework achieves excellent performance on three benchmark datasets with strong factuality and faithfulness.

Copied to clipboard

Chang Yang, Peng Zhang, Hui Gao, Jing Zhang

Visual Prompting in LLMs for Enhancing Emotion Recognition (2024.emnlp-main)

Challenge:	Social networks are rife with noise and misleading information, presenting multifaceted challenges for rumor detection.
Approach:	They propose a new multi-task learning framework that mines latent intentions and rumor semantic features . they propose to use event-level and intent-level strategies to establish cognitive anchors .
Outcome:	The proposed framework improves the effectiveness of rumor detection and addresses the challenges present in the field.

Copied to clipboard

Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, Zhenyue Qin

IDEAW: Robust Neural Audio Watermarking with Invertible Dual-Embedding (2024.emnlp-main)

Challenge:	Existing methods for enhancing in-context emotion classification fail to include spatial relationships between different people and facial features within a single face.
Approach:	They propose a set-of-vision prompting approach that uses spatial information to mark targets precisely.
Outcome:	The proposed approach improves face count and emotion categorization while preserving the enriched image context.

Copied to clipboard

Pengcheng Li, Xulong Zhang, Jing Xiao, Jianzong Wang

Leveraging Conflicts in Social Media Posts: Unintended Offense Dataset (2024.emnlp-main)

Challenge:	Traditional methods for embedding watermarks into audio have low capacity and unsatisfactory imperceptibility.
Approach:	They propose a dual-embedding wa- termarking model for efficient locating and a model that can withstand attacks.
Outcome:	The proposed model can withstand attacks with higher capacity and more efficient locating ability compared to existing methods.

Copied to clipboard

Che Wei Tsai, Yen-Hao Huang, Tsu-Keng Liao, Didier Estrada, Retnani Latifah, Yi-Shin Chen

Outcome-Constrained Large Language Models for Countering Hate Speech (2024.emnlp-main)

Challenge:	a new study examines the impact of conflict on multi-person communication datasets on offensive language . conflict datasets often neglect contextual information and focus on intended offenses . authors propose a conflict-based data collection method to analyze inter-conflict cues in multi-user communications .
Approach:	They propose a conflict-based data collection method to utilize inter-conflict cues in multi-person communications.
Outcome:	The proposed method improves the accuracy of detecting offensive language and enriches our understanding of conflict dynamics in digital communication.

Copied to clipboard

Lingzi Hong, Pengcheng Luo, Eduardo Blanco, Xiaoying Song

Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing (2024.emnlp-main)

Challenge:	Existing research focuses on generating counterspeech with linguistic attributes such as being polite, informative, and intent-driven.
Approach:	They develop automatic counterspeech generation methods that incorporate two desired conversation outcomes into the text generation process: low conversation incivility and non-hateful hater reentry.
Outcome:	The proposed methods incorporate two desired conversation outcomes: low conversation incivility and non-hateful hater reentry.

Copied to clipboard

Changbing Yang, Garrett Nicolai, Miikka Silfverberg

Adaptive Immune-based Sound-Shape Code Substitution for Adversarial Chinese Text Attacks (2024.emnlp-main)

Challenge:	a paper addresses the data scarcity problem in automated glossing for low-resource languages . traditional manual documenting is laborintensive and a lack of data is limiting the accuracy of glossing .
Approach:	They propose to integrate token-level and sentence-level translations into models and integrate available dictionary resources into the model.
Outcome:	The proposed model improves word-level accuracy by 5% on the lowest-resource language Gitksan . the authors also show that the model improve on a simulated low-resourced language with fewer than 100 glossed sentences .

Copied to clipboard

Ao Wang, Xinghao Yang, Chen Li, Bao-di Liu, Weifeng Liu

Bootstrapped Policy Learning for Task-oriented Dialogue through Goal Shaping (2024.emnlp-main)

Challenge:	Existing text attack methods are designed for English text, but robust implementation of Chinese text is understudied.
Approach:	They propose an adaptive immune-based sound-shape code algorithm for Chinese text attacks . they leverage the Sound-Shape Code to generate natural substitutions .
Outcome:	The proposed algorithm produces high-quality Chinese adversarial examples . it can reduce duplication of population and improve search ability .

Copied to clipboard

Yangyang Zhao, Ben Niu, Mehdi Dastani, Shihan Wang

PsyGUARD: An Automated System for Suicide Detection and Risk Assessment in Psychological Counseling (2024.emnlp-main)

Challenge:	Despite the promise of reinforcement learning, achieving seamless knowledge transitions in complex dialogue environments is difficult.
Approach:	They propose a Bootstrapped Policy Learning framework which adaptively tailors progressively challenging subgoal curriculum for each complex goal through goal shaping.
Outcome:	The proposed framework has shown to be effective across four publicly available datasets with different difficulty levels.

Copied to clipboard

Huachuan Qiu, Lizhi Ma, Zhenzhong Lan

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering (2024.emnlp-main)

Challenge:	Existing systems for fine-grained suicide detection and risk assessment are lacking . a lack of domain-specific systems for this task poses a challenge to automated crisis intervention aimed at suicide prevention.
Approach:	They propose to use a fine-grained suicide detection system to assess risk in counseling . they develop a taxonomy for detecting suicide ideation and a large-scale dataset .
Outcome:	The proposed system detects suicidal ideation and assesses risk in counseling . it can provide safe, helpful, and tailored responses for further assessment .

Copied to clipboard

Jiacong Wang, Bohong Wu, Haiyong Jiang, Zhou Xun, Xin Xiao, Haoyuan Guo, Jun Xiao

DVD: Dynamic Contrastive Decoding for Knowledge Amplification in Multi-Document Question Answering (2024.emnlp-main)

Challenge:	Recent advances in Vision-Language Models and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation.
Approach:	They propose a multi-modal data construction pipeline that organizes the final output into a Python code format.
Outcome:	The proposed pipeline improves visual question answering and visual grounding benchmarks across different VLMs.

Copied to clipboard

Jing Jin, Houfeng Wang, Hao Zhang, Xiaoguang Li, Zhijiang Guo

How Do Humans Write Code? Large Models Do It the Same Way Too (2024.emnlp-main)

Challenge:	Large language models (LLMs) generate information with hallucinations due to uneven retrieval quality and irrelevant contents.
Approach:	They propose a decoding strategy which dynamically amplifies knowledge from selected documents during the generation phase.
Outcome:	The proposed method outperforms other decoding strategies on ALCE-ASQA, NQ, TQA and PopQA benchmarks.

Copied to clipboard

Long Li, Xuzheng He, Haozhe Wang, Linlin Wang, Liang He

Retrospex: Language Agent Meets Offline Reinforcement Learning Critic (2024.emnlp-main)

Challenge:	Program-of-Thought (PoT) replaces natural language-based Chain-ofThough (CoT) but introduces more reasoning errors, such as incorrect formulas or flawed logic, compared to CoT.
Approach:	They propose a method that integrates CoT and Program-of-Thought to achieve more accurate reasoning and reinforcement learning.
Outcome:	The proposed method achieves an average improvement of 6.5% on the Llama-Base model and 4.3% on the Mistral-Bass model across 8 mathematical calculation datasets.

Copied to clipboard

Yufei Xiang, Yiqun Shen, Yeqin Zhang, Nguyen Cam-Tu

Challenge:	Existing LLM agent frameworks do not fully utilize past experiences for improvement.
Approach:	They propose a LLM-based agent framework called Retrospex that analyzes past experiences in depth to improve existing agent frameworks.
Outcome:	The proposed framework analyzes past experiences in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over baselines.

Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-Context Models (2024.emnlp-main)

Copied to clipboard

Xinyu Liu, Runsong Zhao, Pengcheng Huang, Chunyang Xiao, Bei Li, Jingang Wang, Tong Xiao, JingBo Zhu

Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generation (2024.emnlp-main)

Challenge:	Existing methods, tasks and benchmarks to measure model’s effective memory length are limited.
Approach:	They propose a method called forgetting curve to measure the memorization capability of long-context models.
Outcome:	The proposed method is robust to the tested corpus and experimental settings, and can be applied to any model size.

Copied to clipboard

Yuanjie Lyu, Zihan Niu, Zheyong Xie, Chao Zhang, Tong Xu, Yang Wang, Enhong Chen

CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation (2024.emnlp-main)

Challenge:	Large language models (LLMs) often produce factual errors due to limited internal knowledge.
Approach:	They propose a retrieval-augmented generation framework that generates plan tokens to guide subsequent generation.
Outcome:	The proposed framework improves the accuracy of large language models with external knowledge sources.

Copied to clipboard

Renhao Li, Minghuan Tan, Derek Wong, Min Yang

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)

Challenge:	Recent studies have focused on constructing substantial quantities of IFT data with minimal human effort.
Approach:	They propose a multi-agent cooperation framework for the improvement of IFT responses for large language models using a debate-advise-edit-judge paradigm.
Outcome:	The proposed framework outperforms baseline models on unseen tasks and shows that it can improve instruction-following capabilities on large language models.

Copied to clipboard

Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth

Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024.emnlp-main)

Challenge:	a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias.
Approach:	They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias.
Outcome:	The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning .

Copied to clipboard

Yicheng Gao, Gonghan Xu, Zhe Wang, Arman Cohan

MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoning (2024.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for text quality evaluation.
Approach:	They propose two methods to improve the accuracy of LLM evaluators by Bayesian inference.
Outcome:	The proposed methods improve the accuracy of the win rate estimation using LLMs . the proposed methods are based on six datasets covering story generation, summarization, and instruction following tasks .

Copied to clipboard

Shuo Yin, Weihao You, Zhilong Ji, Guoqiang Zhong, Jinfeng Bai

Seeing the Forest through the Trees: Data Leakage from Partial Transformer Gradients (2024.emnlp-main)

Challenge:	a method to combine the advantages of open-source and tool-free LLMs remains to be explored.
Approach:	They propose a method to integrate open-source LLMs with external Python interpreters and augment math reasoning data.
Outcome:	The proposed method improves on GSM8K and MATH with the use of external tools.

Copied to clipboard

Weijun Li, Qiongkai Xu, Mark Dras

RWKV-CLIP: A Robust Vision-Language Representation Learner (2024.emnlp-main)

Challenge:	Recent studies show that distributed machine learning is vulnerable to gradient inversion attacks . a recent study demonstrated the possibility of reconstructing private textual training data using partial gradients .
Approach:	They propose to use partial gradients to reconstruct training data using a Transformer model.
Outcome:	The proposed method is vulnerable to gradient inversion attacks, the authors show . they show that applying differential privacy on gradients during training offers limited protection .

Copied to clipboard

Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, Jiankang Deng

KidLM: Advancing Language Models for Children – Early Insights and Future Directions (2024.emnlp-main)

Challenge:	Using large image-text datasets, large-scale image-data sets have been used for visionlanguage pre-training.
Approach:	They propose a framework that leverages Large Language Models to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags.
Outcome:	The proposed framework can combine and refine information from web-based image-text pairs, synthetic captions, and detection tags.

Copied to clipboard

Mir Tafseer Nayeem, Davood Rafiei

Using Language Models to Disambiguate Lexical Choices in Translation (2024.emnlp-main)

Challenge:	Large language models have been shown to be effective in creating educational tools for children, yet there are significant challenges in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards.
Approach:	They propose a user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children.
Outcome:	The proposed model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children’s unique preferences.

Copied to clipboard

Josh Barua, Sanjay Subramanian, Kayo Yin, Alane Suhr

How Does the Disclosure of AI Assistance Affect the Perceptions of Writing? (2024.emnlp-main)

Challenge:	In translation, a concept represented by a single word can have multiple variations in a target language.
Approach:	They evaluate language models that can be used to generate English rules for lexical selection . they find weaker models with high-quality lexicals improve accuracy .
Outcome:	The proposed model outperforms existing models on the lexical selection task in English and with native speakers.

Copied to clipboard

Zhuoyan Li, Chen Liang, Jing Peng, Ming Yin

An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records (2024.emnlp-main)

Challenge:	Recent advances in generative AI technologies like large language models have boosted the incorporation of AI assistance in writing workflows.
Approach:	They conduct an experimental study to determine whether disclosure of AI assistance in the writing process would affect people's evaluation on the quality of the writing and ranking of different writings.
Outcome:	The disclosure of AI assistance decreases the average quality ratings for argumentative essays and creative stories, and increases the quality of the writings.

Copied to clipboard

Joakim Edin, Maria Maistro, Lars Maaløe, Lasse Borgholt, Jakob Havtorn, Tuukka Ruotsalo

Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphs (2024.emnlp-main)

Challenge:	State-of-the-art explainability methods rely on human annotations, which are costly.
Approach:	They propose an approach to produce plausible and faithful explanations without annotations . they use adversarial robustness training to improve plausibility and AttInGrad .
Outcome:	The proposed method produces plausible explanations without human annotations on a medical coding task.

Copied to clipboard

Zheng Wang, Zhongyang Li, Zeren Jiang, Dandan Tu, Wei Shi

EVEDIT: Event-based Knowledge Editing for Deterministic Knowledge Propagation (2024.emnlp-main)

Challenge:	In the age of mobile internet, personal information is constantly being generated on smartphones.
Approach:	They propose a novel task of crafting personalized agents powered by large language models that leverage a user's smartphone memories to enhance downstream applications with LLM capabilities.
Outcome:	The proposed approach improves 10% over the best existing approach on a real-world dataset and improves usability.

Copied to clipboard

Jiateng Liu, Pengfei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, Ruhi Sarikaya, Kevin Small, Heng Ji

Modeling Nonnative Sentence Processing with L2 Language Models (2024.emnlp-main)

Challenge:	Existing knowledge editing approaches only operate on (subject, relation, object) triple . current methods are limited to (substance, relation) triple, causing low confidence in their answers.
Approach:	They propose a task of event-based knowledge editing that pairs facts with event descriptions to improve model confidence.
Outcome:	The proposed method improves model confidence by 55.6% while maintaining the naturalness of generation.

Copied to clipboard

Tatsuya Aoyama, Nathan Schneider

From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis (2024.emnlp-main)

Challenge:	Experimental results show that while all of the LMs’ word surprisals improve prediction of L2 reading times, there is no reliable effect of the choice of L1’s L1.
Approach:	They pretrain GPT2 on 6 different first languages, followed by English as the second language (L2).
Outcome:	The pretraining of L1 improves prediction of L2 reading times, but there is no reliable effect of the pretraining L1 on the model's performance on English speakers.

Copied to clipboard

Chuanqi Cheng, Jian Guan, Wei Wu, Rui Yan

Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs (2024.emnlp-main)

Challenge:	Existing models consisting of multiple steps of visual and language processing are limited in the visual and visual processing community . a visual reasoner is a plug-and-play approach that can be used to improve VLMs' reasoning abilities.
Approach:	They propose a least-to-most visual reasoning paradigm that divides a question into sub-questions and invokes external tools for resolving sub-questions.
Outcome:	The proposed method can improve four VLMs on four VQA benchmarks.

Copied to clipboard

Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, Zohar Karnin

Cross-Domain Audio Deepfake Detection: Dataset and Analysis (2024.emnlp-main)

Challenge:	Existing methods to assess data quality for training and testing large language models are lacking.
Approach:	They propose two approaches to assess the reliability of data for training large language models for external tool usage.
Outcome:	The proposed approaches outperform models trained on high-quality data on two popular benchmarks and an extrinsic evaluation that showcases the impact of data quality on model performance.

Copied to clipboard

Yuang Li, Min Zhang, Mengxin Ren, Xiaosong Qiao, Miaomiao Ma, Daimeng Wei, Hao Yang

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension (2024.emnlp-main)

Challenge:	Existing audio deepfake detection datasets are outdated and lack generalization capabilities.
Approach:	They construct a new cross-domain audio deepfake detection dataset comprising over 300 hours of speech data that is generated by five advanced zero-shot TTS models.
Outcome:	The proposed models achieve 4.1% and 6.5% error rates in the cross-domain ADD dataset generated by five advanced zero-shot TTS models.

Copied to clipboard

Ting Liu, Zunnan Xu, Yue Hu, Liangtao Shi, Zhiqiang Wang, Quanjun Yin

Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization (2024.emnlp-main)

Challenge:	Existing methods for Referring Expression Comprehension (REC) lack specific domain abilities for precise local visual perception and visual-language alignment.
Approach:	They propose a framework for Parameter-Efficient Transfer Learning to localize a visual region via natural language using a prior-guided prior.
Outcome:	The proposed framework achieves the best accuracy compared to the current methods with only 1.41% tunable backbone parameters.

Copied to clipboard

Miyoung Ko, Sue Park, Joonsuk Park, Minjoon Seo

Aligning Translation-Specific Understanding to General Understanding in Large Language Models (2024.emnlp-main)

Challenge:	Despite advances in large language models, how they use their knowledge for reasoning is not yet well understood.
Approach:	They propose a method that deconstructs complex real-world questions into a graph . they quantify forward discrepancy, a discrepany in LLM performance on simpler sub-problems .
Outcome:	The proposed method shows that smaller models exhibit more discrepancies than larger models . it also shows that guiding models from simpler to complex questions improves performance .

Copied to clipboard

Yichong Huang, Baohang Li, Xiaocheng Feng, Wenshuai Huo, Chengpeng Fu, Ting Liu, Bing Qin

FOOL ME IF YOU CAN! An Adversarial Dataset to Investigate the Robustness of LMs in Word Sense Disambiguation (2024.emnlp-main)

Challenge:	Large Language models (LLMs) have remarkable abilities in understanding complex texts . however, understanding misalignment leads to LLMs mistakenly translating complex concepts .
Approach:	They propose a translation process that aligns the translation-specific understanding with the general understanding to improve translation quality and reduce translation literalness.
Outcome:	The proposed translation process improves translation quality and reduces translation literalness by -25% -51%.

Copied to clipboard

Mohamad Ballout, Anne Dedert, Nohayr Abdelmoneim, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger

Concept-skill Transferability-based Data Selection for Large Vision-Language Models (2024.emnlp-main)

Challenge:	Word sense disambiguation (WSD) is a key task in natural language processing . however, these models struggle with recognizing semantic boundaries in adversarial contexts .
Approach:	They propose to use a coarse-grained WSD dataset to assess model robustness . they found that some models struggled to correctly disambiguate homonyms in adversarial contexts .
Outcome:	The proposed dataset includes four test sets to assess the robustness of language models in WSD tasks.

Copied to clipboard

Jaewoo Lee, Boyang Li, Sung Ju Hwang

LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) require instruction tuning on extensive data . training on large VL datasets can be prohibitively expensive .
Approach:	They propose a data selection technique that uses a small model as a reference model to select training data for efficient finetuning of a target LVLM.
Outcome:	The proposed method achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan.

Copied to clipboard

Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip Yu, Wenpeng Yin

Challenge:	a comparative analysis of paper (meta-)reviews by large language models (LLMs) aims to identify and distinguish LLMs from human activities .
Approach:	They present a comparative analysis to identify and distinguish LLM activities from human activities.
Outcome:	The proposed analysis aims to improve recognition of instances when someone implicitly uses LLMs for reviewing activities.

Academics Can Contribute to Domain-Specialized Language Models (2024.emnlp-main)

Copied to clipboard

Mark Dredze, Genta Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David Rosenberg, Sebastian Gehrmann

Beyond Reference: Evaluating High Quality Translations Better than Human References (2024.emnlp-main)

Challenge:	Commercially available models dominate academic leaderboards, focusing on creating and adapting general-purpose models . however, general- purpose models often underperform in specialized domains, and domain-specific models yield superior results.
Approach:	They advocate for a renewed focus on developing and evaluating domain- and task-specific models . they advocate for an adapted or adapted model that can be used to improve academic leaderboard standings .
Outcome:	The proposed model can do well on professional and linguistic examinations, college-level knowledge questions, and collections of reasoning tasks.

Copied to clipboard

Keonwoong Noh, Seokjin Oh, Woohwan Jung

Unveiling the Lexical Sensitivity of LLMs: Combinatorial Optimization for Prompt Enhancement (2024.emnlp-main)

Challenge:	Existing machine translation metrics give maximum score to reference sentence . however, these metrics overlook the possibility that candidate sentences outperform reference sentences in terms of quality.
Approach:	They propose a machine translation metrics that give an absolute score to a translated sentence based on the similarity with the reference sentence.
Outcome:	The proposed measure outperforms existing MT metrics in terms of quality and assigns positive scores to candidates that outperformed reference sentences.

Copied to clipboard

Pengwei Zhan, Zhen Xu, Qian Tan, Jie Song, Ru Xie

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages (2024.emnlp-main)

Challenge:	Large language models (LLMs) demonstrate exceptional instruct-following ability to complete downstream tasks.
Approach:	They propose a black-box combinatorial optimization framework that iteratively improves lexical choices in prompts by a search strategy related to word influence.
Outcome:	The proposed framework recovers the model's ability to instruct-follow and solve downstream tasks even when the variations are imperceptible to humans.

Copied to clipboard

Holy Lovenia, Rahmad Mahendra, Salsabil Akbar, Lester James Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, Joel Moniz, Muhammad Habibi, Frederikus Hudi, Jann Montalan, Ryan Hadiwijaya, Joanito Lopo, William Nixon, Börje Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Ryanda, Sonny Hermawan, Dan Velasco, Muhammad Kautsar, Willy Hendria, Yasmin Moslem, Noah Flynn, Muhammad Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Chia, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Aji, Sedrick Keh, Genta Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, Samuel Cahyawijaya

Induct-Learn: Short Phrase Prompting with Instruction Induction (2024.emnlp-main)

Challenge:	Southeast Asia (SEA) is home to over 1,300 indigenous languages and 671 million people . prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA .
Approach:	They propose to provide a resource center that provides standardized corpora in nearly 1,000 SEA languages across three modalities.
Outcome:	a new benchmark assesses the quality of AI models on 36 SEA languages across 13 tasks . the results highlight the importance of SEA as a culturally diverse region .

Copied to clipboard

Po-Chun Chen, Sheng-Lun Wei, Hen-Hsen Huang, Hsin-Hsi Chen

Multi-Granularity History and Entity Similarity Learning for Temporal Knowledge Graph Reasoning (2024.emnlp-main)

Challenge:	Existing methods for generating instructions from demonstrations rely on large datasets or numerous examples, which is impractical and costly in real-world scenarios.
Approach:	They propose a task-level framework that induces pseudo instructions from a few demonstrations and a short phrase, adding a CoT process into existing demonstrations.
Outcome:	The proposed framework outperforms state-of-the-art methods on two datasets and exhibits cross-model adaptability and lower cost.

Copied to clipboard

Shi Mingcong, Chunjiang Zhu, Detian Zhang, Shiting Wen, Li Qing

LUQ: Long-text Uncertainty Quantification for LLMs (2024.emnlp-main)

Challenge:	Existing models for Temporal Knowledge Graph reasoning capture repetitive history, ignoring the entity's multi-hop neighbour history which can provide valuable background knowledge for TKG reasoning.
Approach:	They propose a multi-granularity history and entity similarity learning model which captures the similarity between entities.
Outcome:	The proposed model can predict unknown facts based on historical information, but most existing models ignore multi-hop neighbour history which can provide valuable background knowledge for TKG reasoning.

Copied to clipboard

Caiqi Zhang, Fangyu Liu, Marco Basaldella, Nigel Collier

Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (2024.emnlp-main)

Challenge:	Existing research on Uncertainty Quantification (UQ) predominantly targets short text generation, however, real-world applications often necessitate much longer responses.
Approach:	They propose a method that ensembles responses from multiple models and selects the response with the lowest uncertainty.
Outcome:	The proposed method outperforms baseline methods in correlating with the model’s factuality scores (negative coefficient of -0.85 observed for Gemini Pro).

Copied to clipboard

Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten Rijke, Yixing Fan, Xueqi Cheng

Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars (2024.emnlp-main)

Challenge:	Existing methods to detect text in training corpus are limited due to their low token probabilities.
Approach:	They propose a method to calibrate token probabilities for pretraining data detection by using a divergence-based calibration method.
Outcome:	The proposed method significantly outperforms existing methods on Chinese text on English-language benchmarks and patents.

Copied to clipboard

Damien Sileo

Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach (2024.emnlp-main)

Challenge:	Existing proof generation algorithms bias reasoning toward specific proof traces and limit extensibility.
Approach:	They propose a framework with flexible context-sensitive rules binding multiple languages . they propose to use English verbalization of predicates to enhance logical reasoning .
Outcome:	The proposed framework surpasses GPT-4 in accuracy on a human-authored logic dataset by 12%.

Copied to clipboard

Maxime Poli, Emmanuel Chemla, Emmanuel Dupoux

Safely Learning with Private Data: A Federated Learning Framework for Large Language Model (2024.emnlp-main)

Challenge:	Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations.
Approach:	They show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.
Outcome:	Recent advances in speech representation modeling have shown that learning language directly from speech is feasible.

Copied to clipboard

Jia-Ying Zheng, Hainan Zhang, Lingxiang Wang, Wangjie Qiu, Hong-Wei Zheng, Zhi-Ming Zheng

Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge (2024.emnlp-main)

Challenge:	Existing large language models (LLMs) use large amounts of public data and massive parameters, but private data is often stored in isolated data silos.
Approach:	They propose a Federated Learning framework for large language models which offloads most training parameters to the server while training embedding and output layers locally.
Outcome:	The proposed framework achieves comparable metrics to centralized chatGLM model on NLU and generation tasks.

Copied to clipboard

Jiahuan Li, Yiqing Cao, Shujian Huang, Jiajun Chen

How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? (2024.emnlp-main)

Challenge:	Large language models have shown excellent performance on knowledge-intensive tasks, but pretraining data tends to contain misleading and conflicting information.
Approach:	They systematically analyze LLMs’ learning preferences for data with conflicting knowledge.
Outcome:	The proposed model outperforms human-level models on knowledge-intensive tasks by analyzing pretraining data.

Copied to clipboard

Yang Luo, Zangwei Zheng, Zirui Zhu, Yang You

How Far Can We Extract Diverse Perspectives from Large Language Models? (2024.emnlp-main)

Challenge:	MLLMs have significant capabilities for multimodal in-context learning, but their effectiveness hinges on the appropriate selection of in-constext examples.
Approach:	They propose a supervised MLLM prompt retriever that leverages a trained retriever based on MLML's confidence to select examples, which enhances multimodal in-context learning efficiency.
Outcome:	The proposed method is validated through extensive testing across three different tasks and demonstrates its effectiveness.

Copied to clipboard

Shirley Hayati, Minhwa Lee, Dheeraj Rajagopal, Dongyeop Kang

EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning (2024.emnlp-main)

Challenge:	Recent advances of large language models have gained much interest from researchers to exploit their capability of creative generation for data augmentation with less cost and higher diversity.
Approach:	They propose a criteria-based prompting technique to extract maximum diversity from LLMs.
Outcome:	The proposed method extracts diverse opinions from large language models iteratively.

Copied to clipboard

Kiran Purohit, Venktesh V, Raghuram Devalla, Krishna Yerragorla, Sourangshu Bhattacharya, Avishek Anand

An LLM Feature-based Framework for Dialogue Constructiveness Assessment (2024.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) have enabled in-context learning (ICL) a critical challenge in ICL is the selection of optimal exemplars .
Approach:	They propose an algorithm for static exemplar subset selection for reasoning tasks . they propose a method that estimates parameters without incorporating confidence information .
Outcome:	The proposed method significantly reduces the number of LLM calls to 11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%.

Copied to clipboard

Lexin Zhou, Youmna Farag, Andreas Vlachos

Relevance Is a Guiding Light: Relevance-aware Adaptive Learning for End-to-end Task-oriented Dialogue System (2024.emnlp-main)

Challenge:	Existing studies on dialogue constructiveness assessment focus on analysing conversational factors that influence individuals to take specific actions, win debates, change their perspectives or broaden their open-mindedness.
Approach:	They propose an LLM feature-based framework for dialogue constructiveness assessment that combines the strengths of feature- and neural approaches while mitigating their downsides.
Outcome:	The proposed framework outperforms standard feature-based models and neural models on three dialogue constructiveness datasets.

Copied to clipboard

Zhanpeng Chen, Zhihong Zhu, Wanshi Xu, Xianwei Zhuang, Yuexian Zou

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction (2024.emnlp-main)

Challenge:	Existing approaches to training task-oriented dialogue systems struggle with the Distractive Attributes Problem (DAP) Existing methods struggle to deal with false but similar knowledge (hard negative entities)
Approach:	They propose a two-stage training framework that eliminates hard negatives step-by-step and aligns retrieval with generation.
Outcome:	The proposed method eliminates hard negatives step-by-step and aligns retrieval with generation.

Copied to clipboard

Sergio Burdisso, Srikanth Madikeri, Petr Motlicek

Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation (2024.emnlp-main)

Challenge:	Dialog2Flow embeddings allow for modeling dialogs as continuous trajectories in a latent space with distinct action-related regions.
Approach:	They propose dialog2Flow embeddings that map dialogs to a latent space and cluster them according to their communicative and informative functions.
Outcome:	The proposed workflow embeddings show superior performance across domains.

Copied to clipboard

Raphael Tang, Crystina Zhang, Lixinyu Xu, Yao Lu, Wenyan Li, Pontus Stenetorp, Jimmy Lin, Ferhan Ture

Investigating LLMs as Voting Assistants via Contextual Augmentation: A Case Study on the European Parliament Elections 2024 (2024.emnlp-main)

Challenge:	Current diffusion models do not cover recent models, thus we curate three test sets for evaluation.
Approach:	They propose a human-calibrated measure of variability in a set of images bootstrapped from existing image-pair perceptual distances.
Outcome:	The proposed model outperforms nine baselines by 18 points in accuracy and matches graded human judgements 78% of the time.

Copied to clipboard

Ilias Chalkidis

Challenge:	In light of the recent 2024 European Parliament elections, we are investigating if Large Language Models (LLMs) can be used as Voting Advice Applications (VAAs).
Approach:	They audit MISTRAL and MIXTRAL models and evaluate their accuracy in predicting the stance of political parties based on the latest “EU and I” voting assistance questionnaire.
Outcome:	The proposed models are highly accurate with an 82% accuracy on average with a significant performance disparity across political groups (50-95%).

Adaption-of-Thought: Learning Question Difficulty Improves Large Language Models for Reasoning (2024.emnlp-main)

Copied to clipboard

Mayi Xu, Yongqi Li, Ke Sun, Tieyun Qian

LogicST: A Logical Self-Training Framework for Document-Level Relation Extraction with Incomplete Annotations (2024.emnlp-main)

Challenge:	Existing methods do not differentiate question difficulty when designing prompting methods for them.
Approach:	They propose an adaptive method to improve large language models for reasoning problems by measuring question difficulty and tailoring demonstration set construction and difficulty-adapted retrieval strategies.
Outcome:	The proposed method shows an absolute improvement of up to 5.5% on arithmetic reasoning, 7.4% on symbolic reasoning, and 2.3% on commonsense reasoning.

Copied to clipboard

Shengda Fan, Yanting Wang, Shasha Mo, Jianwei Niu

Concept Space Alignment in Multilingual LLMs (2024.emnlp-main)

Challenge:	Document-level relation extraction (DocRE) is difficult due to the vast number of entity pairs.
Approach:	They propose a neural-logic self-training framework that iteratively resolves conflicts and constructs the minimal diagnostic set for updating models.
Outcome:	The proposed framework outperforms existing methods on the document-level relation extraction (docRE) benchmark.

Copied to clipboard

Qiwei Peng, Anders Søgaard

Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model (2024.emnlp-main)

Challenge:	Multilingual large language models generalize somewhat across languages, but it is unclear whether this is a result of improved, implicit alignment, or of something else, e.g., linguistic overlap or semi-parallel subsets of training data.
Approach:	They hypothesize that implicit alignment is the reason for generalization in multilingual large language models.
Outcome:	The proposed model generalizes well across languages, but lacks linearity.

Copied to clipboard

Chenhan Yuan, Fei Huang, Ru Peng, Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian (2024.emnlp-main)

Challenge:	Existing approaches to fine tune LLMs produce unsafe responses and unreliable reasoning, but this solution introduces substantial time and space overhead due to the separate models required.
Approach:	They propose to insert extra parameters into transformer architecture to predict calibration signals along with original LLM output.
Outcome:	The proposed model reduces time and space costs while enabling seamless online deployment.

Copied to clipboard

Peng Liu, Lemei Zhang, Terje Farup, Even Lauvrak, Jon Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang

RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework (2024.emnlp-main)

Challenge:	Norwegian is under-represented within the most impressive breakthroughs in NLP tasks.
Approach:	they investigate the impact of existing Norwegian language models on Norwegian generation tasks . they pre-trained 4 Norwegian Open Language Models from parameter scales and architectures .
Outcome:	The proposed benchmark evaluates the performance of language models on Norwegian generation tasks.

Copied to clipboard

Yifan Wang, Vera Demberg

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models (2024.emnlp-main)

Challenge:	RSA-Control is a training-free controllable text generation framework . existing studies rely on fine-tuning pre-trained language models . external components could hurt coherence and accuracy of the model .
Approach:	They propose a training-free controllable text generation framework grounded in pragmatics that directs the generation process by recursively reasoning between imaginary speakers and listeners.
Outcome:	The proposed framework achieves strong attribute control while maintaining fluency and content consistency.

Copied to clipboard

Siqi Wang, Zhengyu Chen, Bei Li, Keqing He, Min Zhang, Jingang Wang

Synergizing In-context Learning with Hints for End-to-end Task-oriented Dialog Systems (2024.emnlp-main)

Challenge:	a study of large language models (LLMs) reveals the transferability and discrepancies of scaling laws between Dense and MoE models.
Approach:	They investigate the transferability and discrepancies of scaling laws between Dense Models and Mixture of Experts models.
Outcome:	The results show that the power-law scaling framework also applies to MoE Models .

Copied to clipboard

Vishal Saley, Rocktim Das, Dinesh Raghu, Mausam .

REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering (2024.emnlp-main)

Challenge:	Existing end-to-end task-oriented dialogue systems require extensive training datasets to perform well.
Approach:	They propose a system that synergizes LLMs with task-specific hints to improve alignment in low-data settings.
Outcome:	The proposed model improves alignment in low-data settings while retaining competitive performance in full-data environments.

Copied to clipboard

Yuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao, Jing Liu, Ji-Rong Wen

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)

Challenge:	Existing methods to extend knowledge scope of large language models (LLMs) lack internal parametric knowledge, resulting in misusing external knowledge.
Approach:	They propose a retrieval-augmented approach that provides LLMs with potentially relevant documents through a module.
Outcome:	The proposed approach outperforms existing methods on four open-domain QA tasks.

Copied to clipboard

Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li

On Mitigating Performance Disparities in Multilingual Speech Recognition (2024.emnlp-main)

Challenge:	Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications.
Approach:	They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) .
Outcome:	The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents.

Copied to clipboard

Monorama Swain, Anna Zee, Anders Søgaard

Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting (2024.emnlp-main)

Challenge:	Automatic Speech Recognition systems are not always equally effective for all users, and gender disparity in their performance is a significant concern.
Approach:	They compare performance of different fine-tuning algorithms for multilingual speech recognition across languages and genders.
Outcome:	The proposed algorithms improve performance and parity across languages and languages.

Copied to clipboard

Stephen Meisenbacher, Florian Matthes

To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Models (2024.emnlp-main)

Challenge:	Recent studies have focused on the integration of Differential Privacy (DP) into NLP techniques.
Approach:	They propose a method for text privatization leveraging language models to rewrite texts . they examine the usability of DP in NLP and its benefits over non-DP approaches .
Outcome:	The proposed method is a novel method for text privatization leveraging language models to rewrite texts.

Copied to clipboard

Junyan Lin, Haoran Chen, Dawei Zhu, Xiaoyu Shen

What is ”Typological Diversity” in NLP? (2024.emnlp-main)

Challenge:	Recent multimodal large language models (MLLMs) have attracted widespread attention from both industry and academia due to their potential to handle multiple modalities in a unified framework.
Approach:	They propose to classify connectors into feature-preserving and feature-compressing types and categorize tasks into three task types: coarse-grained perception, fine-grain perception, and reasoning.
Outcome:	The proposed architectures perform better on tasks with varying granularities than on external fusion architectures.

Copied to clipboard

Esther Ploeger, Wessel Poelman, Miryam De Lhoneux, Johannes Bjerva

The Computational Anatomy of Humility: Modeling Intellectual Humility in Online Public Discourse (2024.emnlp-main)

Challenge:	linguistic typology is commonly used to motivate language selections, but there are no set definitions or criteria for such claims.
Approach:	They propose to use linguistic typology to motivate language selections on the basis that a broad typological sample ought to imply generalization across a wide range of languages.
Outcome:	The proposed measures show that skewed language selection can lead to overestimated multilingual performance.

Copied to clipboard

Xiaobo Guo, Neil Potnis, Melody Yu, Nabeel Gillani, Soroush Vosoughi

Challenge:	enhancing the quality of online public discourse requires promoting foundational human virtues, such as “intellectual humility” (IH) . discourse on social media rewards forgetting our virtuous selves, embedding users within echo chambers and causing negative affect towards those who hold different beliefs.
Approach:	They propose to use a codebook to measure "intellectual humility" they manually validated the codebook and used it to develop LLM-based models .
Outcome:	The proposed model achieves a Macro-F1 score of 0.64 across labels and 0.70 when predicting IH/IA/Neutral at the coarse level.

Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness (2024.emnlp-main)

Copied to clipboard

Georgi Shopov, Stefan Gerdjikov

Benchmarking Vision Language Models for Cultural Understanding (2024.emnlp-main)

Challenge:	Existing bidirectional language models lack the ability to utilise future contexts and the pre-determined left-to-right generation order.
Approach:	They propose a class of bidirectional language models that are consistent by definition and can be efficiently used both for generation and scoring of sequences.
Outcome:	The proposed models are consistent by definition and can be efficiently used both for generation and scoring of sequences.

Copied to clipboard

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Steenkiste, Lisa Hendricks, Karolina Stanczak, Aishwarya Agrawal

Methods of Automatic Matrix Language Determination for Code-Switched Speech (2024.emnlp-main)

Challenge:	Recent multimodal vision-language models have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning.
Approach:	They propose a visual question-answering benchmark to assess VLMs' cultural understanding of various facets of culture from 11 countries across 5 continents.
Outcome:	The visual question-answering benchmark aims to assess VLMs' cultural understanding across regions.

Copied to clipboard

Olga Iakovenko, Thomas Hain

Analyzing Key Factors Influencing Emotion Prediction Performance of VLLMs in Conversational Contexts (2024.emnlp-main)

Challenge:	Code-switching (CS) is the process of speakers switching between two or more languages in spoken or written language.
Approach:	They propose to use the Matrix Language Frame theory to describe CS speech . they compare MLID of English/Mandarin and English/Spanish CS to acoustic language identity .
Outcome:	The proposed models outperform monolingual models in acoustic language identity recognition tasks.

Copied to clipboard

Jaewook Lee, Yeajin Jang, Hongjin Kim, Woojin Lee, Harksoo Kim

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models (2024.emnlp-main)

Challenge:	Recent studies show that large language models and vision large language model (VLLMs) possess EI and the ability to understand emotional stimuli in the form of text and images.
Approach:	They analyze the key elements affecting the emotion prediction performance of VLLMs in conversational contexts.
Outcome:	The proposed model performance was compared with other models in a conversational context.

Copied to clipboard

Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Sarath Chandar

Teaching Small Language Models Reasoning through Counterfactual Distillation (2024.emnlp-main)

Challenge:	Large language models (LLMs) are prohibitive to use under resource constraints due to their high latency and high latex.
Approach:	They propose to use a contextual bandit to help choose a model based on a context to improve performance.
Outcome:	The proposed model can be used to improve performance on multiple domains even without prior knowledge of the model.

Copied to clipboard

Tao Feng, Yicheng Li, Li Chenglin, Hao Chen, Fei Yu, Yin Zhang

Pretraining Language Models Using Translationese (2024.emnlp-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable performance in a wide range of downstream tasks.
Approach:	They propose a counterfactual distillation framework that leverages LLMs to generate high-quality counterfacts and utilizes multi-view CoT to enhance the diversity of reasoning samples.
Outcome:	The proposed framework enhances reasoning capabilities of large language models and is more robust to OOD data.

Copied to clipboard

Meet Doshi, Raj Dabre, Pushpak Bhattacharyya

Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval (2024.emnlp-main)

Challenge:	a recent study shows that large language models perform well in low-resource languages . a vast majority of languages don't have comparable data as compared to English .
Approach:	They propose to use Translationese as synthetic data for pre-training language models for low-resource languages.
Outcome:	The proposed method reduces performance of LMs trained on clean data in Indian languages . the proposed model performs better in English than in other languages, but is not comparable to English.

Copied to clipboard

Kyle Buettner, Adriana Kovashka

MTA4DPR: Multi-Teaching-Assistants Based Iterative Knowledge Distillation for Dense Passage Retrieval (2024.emnlp-main)

Challenge:	Existing models that account for perceptual differences in image captions are limited to use in English . culture-based tasks such as recognition, detection, and image retrieval are hindered by relying on English supervision.
Approach:	They propose and evaluate caption augmentation strategies to address these gaps . they use captions from german perception and captions that have been machine-translated or human-transcribed from English into german .
Outcome:	The proposed models achieve a mean recall improvement of +1.3, but still lack flexibility . cultural differences present in language with respect to object specificity and importance .

Copied to clipboard

Qixi Lu, Endong Xun, Gongbo Tang

Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates (2024.emnlp-main)

Challenge:	Existing studies have shown the effectiveness of knowledge distillation in DPR, but there is a performance gap between the teacher and the distilled student.
Approach:	They propose an iterative knowledge distillation method which transfers knowledge from teacher to student with help of multiple assistants in an iterated manner.
Outcome:	The proposed method achieves state-of-the-art performance among models with same parameters on multiple datasets and is competitive when compared with larger models.

Copied to clipboard

Aida Kostikova, Dominik Beese, Benjamin Paassen, Ole Pütz, Gregor Wiedemann, Steffen Eger

CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling (2024.emnlp-main)

Challenge:	Traditionally, solidarity relied on common identity and reciprocity, potentially excluding out-groups like migrants.
Approach:	They examine the frequency of (anti-)solidarity towards women and migrants in German parliamentary debates between 1867 and 2022.
Outcome:	The proposed model outperforms other models in the analysis of 2,864 text snippets and finds that solidarity with migrants outweighs anti-solidarity but frequencies and solidarity types shift over time.

Copied to clipboard

Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, Jackie Cheung

Story Embeddings — Narrative-Focused Representations of Fictional Stories (2024.emnlp-main)

Challenge:	Recent studies have found that information relevant to the next token prediction task accumulates in the hidden representations of just a few tokens.
Approach:	They propose a method that integrates attention preferences useful for a downstream task into the eviction process of hidden states.
Outcome:	The proposed method performs better on comprehension and retrieval tasks while preserving language modeling perplexity.

Copied to clipboard

Hans Ole Hatzel, Chris Biemann

C-LLM: Learn to Check Chinese Spelling Errors Character by Character (2024.emnlp-main)

Challenge:	Existing approaches to model fictional narratives have focused on the aspect of "what" rather than "how" they are being told.
Approach:	They propose a model that embeds stories such that similar stories will result in similar embeddings.
Outcome:	The proposed model shows state-of-the-art performance on multiple retrieval tasks and a narrative understanding task.

Copied to clipboard

Kunting Li, Yong Hu, Liang He, Fandong Meng, Jie Zhou

PSC: Extending Context Window of Large Language Models via Phase Shift Calibration (2024.emnlp-main)

Challenge:	Chinese Spell Checking (CSC) aims to detect and correct spelling errors in sentences.
Approach:	They propose a Chinese Spell Checking method that learns to check errors Character by Character.
Outcome:	The proposed method achieves a 2.1% enhancement in general scenarios and a significant improvement in vertical domain scenarios compared to existing methods.

Copied to clipboard

Wenqiao Zhu, Chao Xu, Lulu Wang, Jun Wu

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2024.emnlp-main)

Challenge:	Large-scale language models (LLMs) have shown impressive results across a variety of tasks.
Approach:	They propose a module for calibrating the frequencies predefined by existing methods . they conducted extensive experiments across multiple models and tasks .
Outcome:	The proposed method reduces perplexity as the context window size is varied from 16k to 32k and up to 64k.

Copied to clipboard

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, Li Yuan

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales (2024.emnlp-main)

Challenge:	Existing approaches to visual-language understanding lack unified tokenization for images and videos . lack of unified visual representations makes it difficult to learn multi-modal interactions from poor projection layers.
Approach:	They propose to unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM.
Outcome:	The proposed model outperforms Video-ChatGPT on image benchmarks and on 9 image benchmark benchmarks.

Copied to clipboard

Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, Jing Gao

Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)

Challenge:	Existing approaches to elicit confidence from large language models are limited to binary or inaccurate group-level confidence estimates.
Approach:	They propose a training framework that teaches LLMs to express more fine-grained confidence estimates.
Outcome:	The proposed training framework reduces the confidence calibration error and maintains the performance of the model.

Copied to clipboard

Richard Diehl Martinez, Zebulon Goriely, Andrew Caines, Paula Buttery, Lisa Beinborn

ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations (2024.emnlp-main)

Challenge:	Language models rely on frequency information because they maximize the likelihood of tokens during training.
Approach:	They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency.
Outcome:	The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training.

Copied to clipboard

Yunze Xiao, Yujia Hu, Kenny Choo, Roy Lee

Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models? (2024.emnlp-main)

Challenge:	Existing large language models struggle with systematically perturbed data designed to evade detection mechanisms.
Approach:	They propose a large language model with homophonic substitutions and emoji transformations to test their models' robustness against cloaking perturbations.
Outcome:	The proposed model underperforms in detecting offensive content when perturbations are applied to Chinese language datasets.

Copied to clipboard

Siyu Yuan, Cheng Jiayang, Lin Qiu, Deqing Yang

Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation (2024.emnlp-main)

Challenge:	Analogical reasoning plays a critical role in human cognition, enabling us to understand new concepts by associating them with familiar ones.
Approach:	They propose to use free-form analogies to aid students in understanding scientific concepts . they also show that analogies generated by student LMs can improve their own performance .
Outcome:	The proposed model can help students understand scientific concepts, the authors show .

Copied to clipboard

Jirui Qi, Gabriele Sarti, Raquel Fernández, Arianna Bisazza

Do Large Language Models Know How Much They Know? (2024.emnlp-main)

Challenge:	Recent research has shown that self-citing large language models (LLMs) fail to faithfully reflect their context usage throughout the generation process.
Approach:	They propose a plug-and-play approach using model internals for faithful answer attribution in RAG applications that detects context-sensitive answer tokens and pairs them with retrieved documents contributing to their prediction.
Outcome:	The proposed approach achieves citation quality and efficiency comparable to self-citation while allowing for a finer-grained control of attribution parameters.

Copied to clipboard

Gabriele Prato, Jerry Huang, Prasanna Parthasarathi, Shagun Sodhani, Sarath Chandar

Investigating Mysteries of CoT-Augmented Distillation (2024.emnlp-main)

Challenge:	Large Language Models are highly capable systems, but their capabilities and limitations are unclear.
Approach:	They develop a benchmark that challenges LLMs to recall all information they possess on specific topics.
Outcome:	The proposed model can recall excessive, insufficient, or the precise amount of information they possess on a given topic, indicating their awareness of how much they know about the given topic.

Copied to clipboard

Somin Wadhwa, Silvio Amir, Byron Wallace

SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics (2024.emnlp-main)

Challenge:	Recent studies show that eliciting chain of thought rationales from a large "teacher" model in addition to target labels yields (often substantial) improvements in model distillation.
Approach:	They ask: Why and how does this additional training signal help in model distillation?
Outcome:	The proposed method improves model performance on question answering tasks by eliciting CoT rationales from a student model in addition to target labels.

Copied to clipboard

Zhiwen You, Kanyao Han, Haotian Zhu, Bertram Ludaescher, Jana Diesner

Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP (2024.emnlp-main)

Challenge:	Recent studies have used prompt-based fine-tuning methods for text classification tasks . however, the difficulty and costs of manually selecting domain label terms for the verbalizer remain unexplored .
Approach:	They propose a framework to automatically retrieve scientific topic-related terms for low-resource text classification tasks.
Outcome:	The proposed method outperforms state-of-the-art methods on scientific text classification tasks under few and zero-shot settings.

Copied to clipboard

Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, Soheil Feizi

Learning from Natural Language Explanations for Generalizable Entity Matching (2024.emnlp-main)

Challenge:	Image-text contrastive models like CLIP struggle on compositional visio-linguistic tasks where their performance is no better than random chance.
Approach:	They propose a distillation method to enhance CLIP's compositional visio-linguistic reasoning by using a model-derived distillation objective borrowed from large text-to-image generative models like Stable-Diffusion.
Outcome:	The proposed method improves CLIP models' visio-linguistic performance on the Winoground benchmark by 7% while on the ARO dataset, it boosts performance by 3%.

Copied to clipboard

Somin Wadhwa, Adit Krishnan, Runhui Wang, Byron Wallace, Luyang Kong

Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation (2024.emnlp-main)

Challenge:	Entity matching is the task of linking records from different sources that refer to the same real-world entity.
Approach:	They propose to "distill" LLM reasoning into smaller entity matching models via natural language explanations.
Outcome:	The proposed model distillation approach achieves strong performance on out-of-domain generalization tests (10.85% F-1).

Copied to clipboard

Zhuohang Li, Jiaxin Zhang, Chao Yan, Kamalika Das, Sricharan Kumar, Murat Kantarcioglu, Bradley Malin

On the Reliability of Psychological Scales on Large Language Models (2024.emnlp-main)

Challenge:	Language models suffer from poor interpretability and transparency, as well as the intrinsic risk of hallucination and misinformation.
Approach:	They propose a statistical framework that assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge.
Outcome:	The proposed framework assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge.

Copied to clipboard

Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, Michael Lyu

Contrastive Entity Coreference and Disambiguation for Historical Texts (2024.emnlp-main)

Challenge:	Recent research has focused on examining Large Language Models’ characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics.
Approach:	They propose to examine the reliability of personality tests to LLMs by using psychological scales.
Outcome:	The proposed model can represent diverse personalities with specific prompt instructions.

Copied to clipboard

Abhishek Arora, Emily Silcock, Melissa Dell, Leander Heldring

Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)

Challenge:	Existing methods for disambiguating historical documents are not accurate for historical documents, which are replete with individuals not remembered in contemporary knowledge bases.
Approach:	They propose to train bi-encoder models for coreferencing and disambiguating individuals in historical texts and evaluate them on a historical newswire benchmark.
Outcome:	The proposed model outperforms existing models on the historical newswire benchmark and on other datasets.

Copied to clipboard

Jeonghwan Kim, Heng Ji

Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts (2024.emnlp-main)

Challenge:	Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease.
Approach:	They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability.
Outcome:	The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability.

Copied to clipboard

Sumit Asthana, Hannah Rashkin, Elizabeth Clark, Fantine Huot, Mirella Lapata

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment (2024.emnlp-main)

Challenge:	Simplifying the entire text makes it understandable but sometimes removes important details.
Approach:	They propose a simplification task for rewriting text to help readers comprehend text containing unfamiliar concepts and introduce a dataset of 22k definitions from 13 academic domains paired with a difficult concept within each definition.
Outcome:	The proposed model outperforms open-source and commercial models on the task and human judges prefer explanations over simplifications of the difficult concept.

Copied to clipboard

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, Qi Liu

Focused Large Language Models are Stable Many-Shot Learners (2024.emnlp-main)

Challenge:	Large vision-language models (LVLMs) are evolving rapidly and require data with human supervision to achieve better alignment.
Approach:	They introduce VLFeedback, the first large-scale vision-language feedback dataset . they train Silkie, an LVLM fine-tuned via direct preference optimization .
Outcome:	The proposed model outperforms its base model in helpfulness, visual faithfulness, and safety metrics and exhibits enhanced resilience against red-teaming attacks.

Copied to clipboard

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Heda Wang, Yao Hu, Kan Li

Reconsidering Sentence-Level Sign Language Translation (2024.emnlp-main)

Challenge:	In-Context Learning (ICL) enables large language models to achieve rapid task adaptation by learning from demonstrations.
Approach:	They propose a training-free method that disperses model attention from the query . they propose 'focus' search strategy that uses model perplexity to ensure sufficient attention .
Outcome:	The proposed method achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations.

Copied to clipboard

Garrett Tanzer, Maximus Shengelia, Ken Harrenstien, David Uthus

GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (2024.emnlp-main)

Challenge:	Historically, sign language machine translation is framed as a sentence-level task . however, there are known intersentential dependencies that are impossible to resolve in isolation.
Approach:	They propose a human baseline for sign language translation that substitutes a person into the machine learning task framing instead of providing the entire document as context.
Outcome:	The proposed human baseline for sign language translation shows that deaf signers can only understand key parts of the clip in light of additional discourse-level context.

Copied to clipboard

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha

Verba volant, scripta volant? Don’t worry! There are computational solutions for protoword reconstruction (2024.emnlp-main)

Challenge:	We propose a novel large-scale audio-language model with advanced audio understanding and reasoning abilities.
Approach:	They propose a general-purpose large audio-language model with advanced audio understanding and reasoning abilities that integrates an LLM with multiple types of audio representations.
Outcome:	The proposed model outperforms existing models on audio understanding tasks by 1%-84%.

Copied to clipboard

Liviu Dinu, Ana Uban, Alina Cristea, Ioan-Bogdan Iordache, Teodor-George Marchitan, Simona Georgescu, Laurentiu Zoicas

Challenge:	Existing methods for protoword reconstruction are limited to a few languages.
Approach:	They propose a new database of cognate words and etymons for the five main Romance languages and apply machine learning to it.
Outcome:	The proposed model achieves 90% accuracy in predicting protowords for Romance languages, surpassing state-of-the-art models and features.

ChatGPT Doesn’t Trust Chargers Fans: Guardrail Sensitivity in Context (2024.emnlp-main)

Copied to clipboard

Victoria Li, Yida Chen, Naomi Saphra

Personas as a Way to Model Truthfulness in Language Models (2024.emnlp-main)

Challenge:	Existing work addresses the limitations of chatbot guardrails, which limit responses to uncertain or sensitive questions.
Approach:	They generate user biographies that offer ideological and demographic information about the user.
Outcome:	The proposed model can infer a likely political ideology and modify guardrail behavior accordingly.

Copied to clipboard

Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, He He

Satyrn: A Platform for Analytics Augmented Generation (2024.emnlp-main)

Challenge:	Large language models are trained on vast amounts of text from the internet, which contains factual and misleading information.
Approach:	They hypothesize that the pretraining data is generated by groups of (un)truthful agents whose outputs share common features and form a (un-truthfully persona) this allows the model to separate truth from falsehoods and controls the truthfulness of its generation.
Outcome:	The proposed model can infer truth from falsehoods by finetuning its model on a set of facts and finetuned it on unseen topics.

Copied to clipboard

Marko Sterbentz, Cameron Barrie, Shubham Shahi, Abhratanu Dutta, Donna Hooshmand, Harper Pack, Kristian Hammond

EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning (2024.emnlp-main)

Challenge:	Large language models (LLMs) can generate fluent and coherent language, but not all information can be retrieved from text.
Approach:	They propose an approach that leverages structured data to generate fact sets that are converted to text and passed to an LLM.
Outcome:	The proposed approach generates reports in which over 86% of claims are accurate while maintaining high levels of fluency and coherence.

Copied to clipboard

Ashish Seth, Ramaneswaran Selvakumar, S Sakshi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha

EPO: Hierarchical LLM Agents with Environment Preference Optimization (2024.emnlp-main)

Challenge:	EH-MAM is a self-supervised learning approach for speech representation learning . prior methods used random masking schemes to learn speech representations .
Approach:	They propose a self-supervised approach that automatically selects hard regions during SSL training and introduces them to the model for reconstruction.
Outcome:	The proposed approach outperforms state-of-the-art models across low-resource speech recognition and SUPERB benchmarks by 5%-10%.

Copied to clipboard

Qi Zhao, Haotian Fu, Chen Sun, George Konidaris

Challenge:	Long-horizon decision-making tasks require extensive planning over multiple steps, maintaining coherence and goal orientation, which is difficult for LLMs that are typically designed for more immediate and localized predictions.
Approach:	They propose a hierarchical framework that decomposes complex tasks into manageable subgoals, utilizing separate LLMs for subgoal prediction and low-level action generation.
Outcome:	The proposed framework achieves first place on the ALFRED public leaderboard and demonstrates its potential to improve long-horizon decision-making in diverse environments.

Detection and Measurement of Syntactic Templates in Generated Text (2024.emnlp-main)

Copied to clipboard

Chantal Shaib, Yanai Elazar, Junyi Jessy Li, Byron Wallace

UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models (2024.emnlp-main)

Challenge:	Existing diversity evaluation focuses primarily on word-level features.
Approach:	They propose a method for evaluating diversity over syntactic features to characterize general repetition in large language models.
Outcome:	The proposed method shows that models produce templated text in downstream tasks at a higher rate than what is found in human-reference texts.

Copied to clipboard

Xinyu Pi, Mingyuan Wu, Jize Jiang, Haozhen Zheng, Beitong Tian, ChengXiang Zhai, Klara Nahrstedt, Zhiting Hu

Optimized Speculative Sampling for GPU Hardware Accelerators (2024.emnlp-main)

Challenge:	Vision-Language Models (VLMs) perform on par with larger models in general domain visual grounding and question-answering benchmarks.
Approach:	They propose a "Uncontextualized Uncommon Objects" benchmark to evaluate their performance on common datasets.
Outcome:	The proposed benchmark focuses on systematically testing VLMs with both large and small parameter counts on rare and specialized objects.

Copied to clipboard

Dominik Wagner, Seanie Lee, Ilja Baumann, Philipp Seeberger, Korbinian Riedhammer, Tobias Bocklet

Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts (2024.emnlp-main)

Challenge:	Large foundational speech and language models require more memory and computational resources to generate long sequences.
Approach:	They propose to optimize speculative sampling for parallel hardware accelerators by combining multiple GPU threads to reduce profiling time.
Outcome:	The proposed approach improves profiling time from 6% to 13% without compromising accuracy.

Copied to clipboard

Zhaoxuan Tan, Zheyuan Liu, Meng Jiang

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning (2024.emnlp-main)

Challenge:	Experimental results show Per-Pcs outperforms non-personalized and PEFT retrieval baselines, offering performance comparable to OPPU with significantly lower resource use across six tasks.
Approach:	They propose a framework that allows users to safely share and assemble personalized large language models using their history data.
Outcome:	Experimental results show that Per-Pcs outperforms non-personalized and PEFT retrieval baselines with significantly lower resource use across six tasks.

Copied to clipboard

Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, Meng Jiang

Unifying Multimodal Retrieval via Document Screenshot Embedding (2024.emnlp-main)

Challenge:	Experimental results demonstrate that OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark.
Approach:	They propose to integrate parametric user knowledge into the personal PEFT parameters and non-parametric knowledge from retrieval and profiles, adapting LLMs to user behavior shifts.
Outcome:	The proposed method outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark.

Copied to clipboard

Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, Jimmy Lin

Neuron Specialization: Leveraging Intrinsic Task Modularity for Multilingual Machine Translation (2024.emnlp-main)

Challenge:	Existing document retrieval pipelines require document parsing and content extraction to prepare input for indexing.
Approach:	They propose a retrieval paradigm that regards document screenshots as a unified input format . they leverage a large vision-language model to directly encode document screenshot into dense representations .
Outcome:	The proposed method outperforms existing retrieval pipelines in a text-intensive context.

Copied to clipboard

Shaomu Tan, Di Wu, Christof Monz

An Audit on the Perspectives and Challenges of Hallucinations in NLP (2024.emnlp-main)

Challenge:	Language-specific modeling methods that focus on heuristics for allocation of capacity and lack knowledge transfer capabilities are often prone to interference due to conflicting optimization demands.
Approach:	They propose a method that identifies specialized neurons to modularize feed-forward layers and updates them through sparse networks to avoid interference under multilingual translation.
Outcome:	The proposed approach achieves consistent performance gains over strong baselines with additional analyses showing reduced interference and increased knowledge transfer.

Copied to clipboard

Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, Shomir Wilson

Discovering Knowledge-Critical Subnetworks in Pretrained Language Models (2024.emnlp-main)

Challenge:	103 peer-reviewed publications on hallucination in large language models (LLMs) are characterized by a lack of agreement with the term ‘hallucination’ in the field of NLP.
Approach:	They examine 103 peer-reviewed publications on hallucination in large language models (LLMs) and conduct a survey with 171 practitioners from the field of NLP and AI to capture varying perspectives on halllucination.
Outcome:	The findings highlight the need for explicit definitions and frameworks outlining hallucination within NLP and highlight potential challenges.

Copied to clipboard

Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, Antoine Bosselut

Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models (2024.emnlp-main)

Challenge:	Pretrained language models encode implicit representations of knowledge in their parameters, but localizing these representations and disentangling them from each other remains an open problem.
Approach:	They propose a masking scheme that can be applied to weights and neurons to discover such subnetworks.
Outcome:	The proposed method can remove specific knowledge from models while minimizing adverse effects on the original model.

Copied to clipboard

Junjie Chu, Zeyang Sha, Michael Backes, Yang Zhang

Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering (2024.emnlp-main)

Challenge:	Existing GPT models allow users to interact with them for multiple rounds to optimize the task execution.
Approach:	They propose a conversation reconstruction attack targeting the contents of previous conversations between GPT models and benign users, i.e., the benign users’ input contents during their interaction with GPT.
Outcome:	The proposed attacks demonstrate that GPT-4's defense mechanisms are ineffective against these attacks.

Copied to clipboard

Armin Toroghi, Willis Guo, Mohammad Mahdi Abdollah Pour, Scott Sanner

Verifiable, Debuggable, and Repairable Commonsense Logical Reasoning via LLM-based Theory Resolution (2024.emnlp-main)

Challenge:	Existing Knowledge Graph Question Answering (KGQA) methods focus on answering factual questions, leaving questions involving commonsense reasoning unaddressed.
Approach:	They propose a commonsense KGQA methodology that axiomatically surfaces commonsensical knowledge of Large Language Models and grounding every factual reasoning step on KG triples.
Outcome:	The proposed method outperforms existing methods and reduces instances of hallucination and reasoning errors.

Copied to clipboard

Armin Toroghi, Willis Guo, Ali Pesaranghader, Scott Sanner

Understanding and Mitigating Language Confusion in LLMs (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have led to substantial interest in their application to commonsense reasoning tasks.
Approach:	They propose a logical reasoning framework that integrates commonsense knowledge with a verifiable logical framework that mitigates hallucinations and facilitates debugging.
Outcome:	The proposed framework improves on three language-based reasoning tasks and improves accuracy and reasoning correctness.

Copied to clipboard

Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, Sebastian Ruder

Can Large Language Models Learn Independent Causal Mechanisms? (2024.emnlp-main)

Challenge:	Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language.
Approach:	They develop a language confusion benchmark to evaluate LLMs' inability to consistently generate text in a user’s desired language.
Outcome:	The proposed model fails to consistently respond in the correct language, despite being prone to high temperatures and complex prompts.

Copied to clipboard

Gael Gendron, Bao Nguyen, Alex Peng, Michael Witbrock, Gillian Dobbie

MirrorStories: Reflecting Diversity through Personalized Narrative Generation with Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) perform poorly on complex reasoning tasks, such as abstract, causal, or logical reasoning.
Approach:	They propose to use two concepts from causality to learn ICMs within LLMs to improve out-of-distribution performance on abstract and causal reasoning tasks.
Outcome:	The proposed model outperforms existing models on abstract and causal reasoning tasks and is more robust to fine-tuning.

Copied to clipboard

Sarfaroz Yunusov, Hamza Sidat, Ali Emami

InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are used to create personalized “mirror stories” that reflect and resonate with individual readers’ identities.
Approach:	They propose to use Large Language Models to create personalized “mirror stories” that reflect and resonate with individual readers’ identities.
Outcome:	The proposed models outperform generic human-written and LLM-generated narratives on all metrics of engagement and textual diversity while preserving the intended moral.

Copied to clipboard

Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, Jieyu Zhao

Challenge:	Large language models (LLMs) have demonstrated the potential to mimic human social intelligence, but most studies focus on static self-report or performance-based tests.
Approach:	They propose a framework to assess LLMs' ability to understand and manage intentions by mapping their ability to infer the intentions of others in a game setting.
Outcome:	The proposed framework assesses LLMs' ability to understand and manage intentions in a game setting.

Locating Information Gaps and Narrative Inconsistencies Across Languages: A Case Study of LGBT People Portrayals on Wikipedia (2024.emnlp-main)

Copied to clipboard

Farhan Samir, Chan Young Park, Anjalie Field, Vered Shwartz, Yulia Tsvetkov

Challenge:	a recent study focuses on comparative text analyses to explain social phenomena and identify systematic biases.
Approach:	They evaluate InfoGap method to locate information gaps and inconsistencies in articles at the fact level, across languages.
Outcome:	The method identifies discrepancies in factual coverage across languages and biographical facts carrying negative connotations are more likely to be highlighted in Russian Wikipedia.

From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024.emnlp-main)

Copied to clipboard

Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, EunJeong Hwang, Vered Shwartz

Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation (2024.emnlp-main)

Challenge:	Vision-Language Models (VLMs) have shown emerging capabilities through large-scale training that have made them gain popularity in recent years.
Approach:	They propose to perform retrieval across universals and cultural visual grounding tasks to assess cultural diversity across universal and culture-specific local concepts.
Outcome:	The proposed benchmarks show that the models perform significantly across cultures, underscoring the need for enhancing multicultural understanding in vision-language models.

Copied to clipboard

Karin De Langis, Ryan Koo, Dongyeop Kang

Challenge:	Prior work explored the domain of controlled style generation, a task in which a generative language model aims to generate text with a specified style 2 . however in practice, text often contains not only a single style, but a combination of styles.
Approach:	They propose to use calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes to combine multiple styles in a reward function.
Outcome:	The proposed dynamic weighting outperforms static weighting approaches with respect style control while maintaining linguistic quality.

MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model (2024.emnlp-main)

Copied to clipboard

Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, Xuming Hu

Learning to Extract Structured Entities Using Language Models (2024.emnlp-main)

Challenge:	Existing MLLMs have a visual question answering capability but lack domain-specific information.
Approach:	They propose a framework for language model modules in MLLMs when handling projected image features and verify this hypothesis using logit lens.
Outcome:	The proposed framework will yield a 10% change in accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future.

Copied to clipboard

Haolun Wu, Ye Yuan, Liana Mikaelyan, Alexander Meulemans, Xue Liu, James Hensman, Bhaskar Mitra

Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise Comparisons (2024.emnlp-main)

Challenge:	Language Models (LMs) play a pivotal role in extracting structured information from unstructured text.
Approach:	They propose to reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives.
Outcome:	The proposed model outperforms baselines and human evaluations on the extracted entities.

Copied to clipboard

Adian Liusie, Vatsal Raina, Yassir Fathullah, Mark Gales

A Survey of AMR Applications (2024.emnlp-main)

Challenge:	LLM-as-a-judge approaches are effective but cost scales quadratically with number of candidates, which has practical limitations.
Approach:	They propose a Product of Expert (PoE) framework for efficient LLM Comparative Assessment where individual comparisons are considered experts that provide information on a pair’s score difference.
Outcome:	The proposed framework can generate score predictions that correlate well with human judgements on multiple NLG tasks with as few as 2% of comparisons.

Copied to clipboard

Shira Wein, Juri Opitz

Beyond Embeddings: The Promise of Visual Table in Visual Reasoning (2024.emnlp-main)

Challenge:	Abstract Meaning Representation (AMR) is a semantic representation that takes the form of a rooted, directed graph.
Approach:	They analyze more than 100 papers which use Abstract Meaning Representation (AMR) they highlight the range of applications for which AMR has been harnessed and techniques for incorporating it . they also highlight broader AMR engineering patterns and outline areas of future work that seem ripe for AMR incorporation.
Outcome:	The results highlight the range of applications for which AMR has been harnessed and the techniques for incorporating it into those applications.

Copied to clipboard

Yiwu Zhong, Zi-Yuan Hu, Michael Lyu, Liwei Wang

CareCorpus+: Expanding and Augmenting Caregiver Strategy Data to Support Pediatric Rehabilitation (2024.emnlp-main)

Challenge:	Visual representation learning has been a cornerstone in computer vision for decades.
Approach:	They propose a visual representation tailored for visual reasoning that provides instance-level world knowledge and detailed attributes that are essential for visual reason.
Outcome:	The proposed visual tables outperform existing models on 11 visual reasoning benchmarks.

Copied to clipboard

Shahla Farzana, Ivana Lucero, Vivian Villegas, Vera Kaelin, Mary Khetani, Natalie Parde

Challenge:	Existing studies on caregiver strategy classification in pediatric rehabilitation contexts are under-resourced and under-studied.
Approach:	They propose to manually categorized 4,037 caregiver strategies in a pediatric rehabilitation setting, and manually supplement target task data with publicly relevant child health forums.
Outcome:	The proposed method improves the quality of the dataset and shows that it performs well.

Taylor Unswift: Secured Weight Release for Large Language Models via Taylor Expansion (2024.emnlp-main)

Copied to clipboard

Guanchu Wang, Yu-Neng Chuang, Ruixiang Tang, Shaochen Zhong, Jiayi Yuan, Hongye Jin, Zirui Liu, Vipin Chaudhary, Shuai Xu, James Caverlee, Xia Hu

TimeR4 : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024.emnlp-main)

Challenge:	Existing mechanisms compromise ownership rights or raise data privacy concerns . existing mechanisms compromise security of released large language models .
Approach:	They propose a TaylorMLP to preserve the ownership of large language models by transforming the weights of LLMs into Taylor-series parameters instead of releasing original weights .
Outcome:	The proposed model preserves ownership of large language models and prevents their abuse by adjusting the generation speed and causing low-speed token generation.

Copied to clipboard

Xinying Qian, Ying Zhang, Yu Zhao, Baohang Zhou, Xuhui Sui, Li Zhang, Kehui Song

Knowledge-Centric Hallucination Detection (2024.emnlp-main)

Challenge:	Temporal Knowledge Graph Question Answering (TKGQA) aims to answer temporal questions using knowledge in Temporal knowledge graphs (TKTs).
Approach:	They propose a Time-aware retrieve-rewrite-retrieve-rerank framework to integrate temporal knowledge from TKGs into Large Language Models (LLMs) to reduce temporal hallucination, they propose rewrite module to rew questions using background knowledge stored in TKG's, then implement a retrieve-rank module to retrieve semantically and temporally relevant facts from Tkgs and rerank them according to temporal constraints.
Outcome:	The proposed approach achieves relative gains of 47.8% and 22.5% on two datasets, underscoring its effectiveness in boosting the temporal reasoning abilities of LLMs.

Copied to clipboard

Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, Zheng Zhang

Revealing the Parallel Multilingual Learning within Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown impressive capabilities but a tendency to hallucinate.
Approach:	They propose a framework that introduces claim-triplets to represent claims in LLM responses and evaluates them against a reference.
Outcome:	The proposed framework outperforms prior methods by 18.2 to 27.2 points on a benchmark spanning various NLP tasks and annotated 11k claim-triplets from 2.1k responses by seven LLMs.

Copied to clipboard

Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, JingBo Zhu

Automatic Instruction Evolving for Large Language Models (2024.emnlp-main)

Challenge:	Large language models (LLMs) can handle multilingual and cross-lingual text within a single input; however, previous studies focusing on using English as the pivot language to enhance language understanding and reasoning focus on using multiple languages.
Approach:	They propose to use parallel multilingual input to enhance the model's comprehension of the input and to examine how multilingual processing affects prediction.
Outcome:	The proposed model can handle multilingual and cross-lingual text within a single input, but previous studies focused on using English as the pivot language to enhance language understanding and reasoning.

Copied to clipboard

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, Weizhu Chen

RepEval: Effective Text Evaluation with LLM Representation (2024.emnlp-main)

Challenge:	Evol-Instruct is an end-to-end framework that evolves instruction datasets without human effort.
Approach:	They propose an end-to-end framework that evolves instruction datasets without human effort by analyzing and analyzing evolutionary strategies for the given instruction data.
Outcome:	The proposed method outperforms human-designed methods on various benchmarks including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

Copied to clipboard

Shuqian Sheng, Yi Xu, Tianhang Zhang, Zanwei Shen, Luoyi Fu, Jiaxin Ding, Lei Zhou, Xiaoying Gan, Xinbing Wang, Chenghu Zhou

Generative Models for Automatic Medical Decision Rule Extraction from Text (2024.emnlp-main)

Challenge:	Traditional metrics for automatic text evaluation are tailored to specific tasks, while LLM-based evaluation metrics are costly.
Approach:	They propose a metric that leverages projections of LLM representations for evaluation.
Outcome:	The proposed metric exhibits higher correlation with human judgments than previous methods on 14 datasets.

Copied to clipboard

Yuxin He, Buzhou Tang, Xiaoling Wang

Encoding and Controlling Global Semantics for Long-form Video Question Answering (2024.emnlp-main)

Challenge:	Medical decision rules are traditionally constructed by medical experts, which is expensive and hard to scale up.
Approach:	They propose to extract medical decision rules from text using generative models . their code will be open-source upon acceptance .
Outcome:	The proposed model outperforms state-of-the-art models on a Chinese benchmark and achieves 67% tree accuracy.

Copied to clipboard

Thong Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu

Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)

Challenge:	Existing methods to find answers for long videos fail to reason over the whole sequence of video, leading to sub-optimal performance.
Approach:	They propose a state space layer to integrate global semantics into video . they use a gating unit to enable controllability over the flow of global semantic into visual representations.
Outcome:	The proposed framework is able to integrate global semantics into visual representations.

Copied to clipboard

Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, Jiliang Tang

Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs (2024.emnlp-main)

Challenge:	Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents.
Approach:	They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction.
Outcome:	The proposed methods are validated using the objective of existing jailbreak attacks.

Copied to clipboard

Cheng Gao, Chaojun Xiao, Zhenghao Liu, Huimin Chen, Zhiyuan Liu, Maosong Sun

Does Large Language Model Contain Task-Specific Neurons? (2024.emnlp-main)

Challenge:	Existing studies focus on case-to-case retrieval using lengthy queries, which does not match real-world scenarios.
Approach:	They propose a method to construct query-candidate pairs and build the largest LCR dataset to date, LEAD.
Outcome:	Experimental results show that the method can provide ample training signals for LCR models.

Copied to clipboard

Ran Song, Shizhu He, Shuting Jiang, Yantuan Xian, Shengxiang Gao, Kang Liu, Zhengtao Yu

Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models (2024.emnlp-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable capabilities in comprehensively handling various types of natural language processing (NLP) tasks.
Approach:	They propose a method for task-specific neuron localization based on Causal Gradient Variation with Special Tokens (CGVST) this method identifies task- specific neurons by concentrating on the most significant tokens during task processing, eliminating redundant tokens and minimizing interference from non-essential neurons.
Outcome:	The proposed method can locate task-specific neurons across eight public tasks.

Copied to clipboard

Philipp Mondorf, Barbara Plank

Advancing Test-Time Adaptation in Wild Acoustic Test Settings (2024.emnlp-main)

Challenge:	logical puzzles that involve determining identity of characters require a variety of reasoning skills.
Approach:	They propose a benchmark for suppositional reasoning based on knights and knaves puzzles . they show lower-performing models exhibit a diverse range of reasoning errors .
Outcome:	The proposed benchmark demonstrates that models struggle with suppositional reasoning . lower performing models struggle to grasp the concept of truth and lies, the study finds .

Copied to clipboard

Hongfu Liu, Hengguan Huang, Ye Wang

Learning to Retrieve Iteratively for In-Context Learning (2024.emnlp-main)

Challenge:	Existing wild vision TTA methods fail to handle speech data due to the unique characteristics of high-entropy speech frames, which are unreliably filtered out even when containing crucial semantic content.
Approach:	They propose a method for acoustic foundation models to perform confidence-based adaptation in wild acustic test settings.
Outcome:	The proposed method outperforms baselines under Gaussian noise, environmental sounds, accent variations, and sung speech in the wild.

Copied to clipboard

Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, Benjamin Van Durme

Taxonomy-guided Semantic Indexing for Academic Paper Search (2024.emnlp-main)

Challenge:	In-context learning is a powerful tool for learning large language models.
Approach:	They propose an iterative retrieval framework that empowers retrievers to make iterable decisions through policy optimization.
Outcome:	The proposed framework outperforms existing methods on semantic parsing datasets with 4M additional parameters for state encoding.

Copied to clipboard

SeongKu Kang, Yunyi Zhang, Pengcheng Jiang, Dongha Lee, Jiawei Han, Hwanjo Yu

Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts (2024.emnlp-main)

Challenge:	Academic paper search often struggles to match underlying academic concepts between queries and documents.
Approach:	They propose a framework that extracts key concepts from papers and organizes them as a semantic index guided by an academic taxonomy.
Outcome:	The proposed framework can be flexibly employed to enhance existing retrieval frameworks.

Copied to clipboard

Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Libo Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models (2024.emnlp-main)

Challenge:	Program of Thoughts (PoT) is an approach characterized by its executable intermediate steps, which ensure the accuracy of the logical calculations in the reasoning process.
Approach:	They propose a task and model agnostic approach which harnesses strength and diversity from various languages to achieve better performance across all tasks.
Outcome:	The proposed approach outperforms Python Self-Consistency in almost all tasks and models and achieves comparable or superior performance on ChatGPT.

Copied to clipboard

Hongfu Liu, Yuxi Xie, Ye Wang, Michael Shieh

Incomplete Utterance Rewriting with Editing Operation Guidance and Utterance Augmentation (2024.emnlp-main)

Challenge:	Recent efforts have identified adversarial suffixes capable of jailbreaking LLMs . however, GCG struggles with computational inefficiency, limiting further studies .
Approach:	They propose a two-stage transfer learning framework which decouples the search process into behavior-agnostic pre-searching and behavior-relevant post-search.
Outcome:	The proposed approach outperforms baseline on Llama2-chat-7b with ASRs of 43.9 (+ 22.2) and 39.0 (+ 19.5) on valid and test sets.

Copied to clipboard

Zhiyu Cao, Peifeng Li, Yaxin Fan, Qiaoming Zhu

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in LLMs (2024.emnlp-main)

Challenge:	Existing generation methods on Incomplete Utterance Rewriting (IUR) can generate coherent utterances, but they often include irrelevant and redundant tokens in rewritten utteras .
Approach:	They propose a multi-task learning framework that uses editing operation labels to guide generation model to focus on critical tokens in dialogue context.
Outcome:	The proposed model outperforms state-of-the-art models on open-domain and task-oriented dialogues on three datasets.

Copied to clipboard

Yiyuan Li, Shichao Sun, Pengfei Liu

Aligning Large Language Models with Diverse Political Viewpoints (2024.emnlp-main)

Challenge:	Existing methods to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic.
Approach:	They propose a benchmark for fuzzy reasoning that incorporates generalized quantifiers.
Outcome:	The proposed benchmark shows that existing methods do not improve on FRoG . strong mathematical reasoning skills are not indicative of success, the authors show .

Copied to clipboard

Dominik Stammbach, Philine Widmer, Eunjung Cho, Caglar Gulcehre, Elliott Ash

“You Gotta be a Doctor, Lin” : An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations (2024.emnlp-main)

Challenge:	Large language models such as ChatGPT exhibit striking political biases . a recent study shows that chatbots exhibit progressive, liberal, and proenvironmental biase .
Approach:	They propose to align large language models with 100,000 comments from candidates running for national parliament in Switzerland.
Outcome:	The proposed model generates more accurate political viewpoints from Swiss parties compared to commercial models such as ChatGPT.

Copied to clipboard

Huy Nghiem, John Prindle, Jieyu Zhao, Hal Daumé Iii

Extending Context Window of Large Language Models from a Distributional Perspective (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated racial and gender biases in various applications.
Approach:	They use Large Language Models to simulate hiring decisions and salary recommendations for candidates with 320 first names that strongly signal their race and gender, across over 750,000 prompts.
Outcome:	The proposed models favor candidates with White female-sounding names over other demographic groups across 40 occupations.

Copied to clipboard

Yingsheng Wu, Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin

Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions (2024.emnlp-main)

Challenge:	Existing scaling methods for extending context window rely on empirical approaches and lack understanding of the internal distribution within RoPE resulting in suboptimal performance.
Approach:	They propose to optimize the context window extending task from the view of rotary angle distribution by minimizing disturbance between rotary angles to maintain consistency with the pre-training phase.
Outcome:	The proposed approach reduces by up to 72% of the distributional disturbance when extending LLaMA2’s context window to 8k, and reduces it by up 32% when extending to 16k.

Copied to clipboard

Hakyung Sung, Kristopher Kyle

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration (2024.emnlp-main)

Challenge:	Argument structure constructions (ASCs) are lexicogrammatical patterns at the clausal level.
Approach:	They evaluate the effectiveness of pre-trained language models in identifying argument structure constructions . they use supervised training with RoBERTa and prompt-guided annotation with GPT-4 .
Outcome:	The proposed model outperforms the gold-standard model on three methods . the results show that the model performs better on gold-standardized data .

Copied to clipboard

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, Jiashi Feng

Position Engineering: Boosting Large Language Models through Positional Information Manipulation (2024.emnlp-main)

Challenge:	Large language models (LLMs) have advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities.
Approach:	They propose a competition-based benchmark framework specifically designed to assess LLMs within multi-agent environments.
Outcome:	The proposed framework enhances the LLMs’ abilities in navigating complex social and cognitive dimensions by over threefold between the strongest and weakest LLM models.

Copied to clipboard

Zhiyuan He, Huiqiang Jiang, Zilong Wang, Yuqing Yang, Luna Qiu, Lili Qiu

Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated significant strides towards achieving artificial general intelligence.
Approach:	They propose a technique termed position engineering which alters the positional information in the prompt without modifying the text itself.
Outcome:	The proposed technique significantly improves on the baseline in retrieval-augmented generation and in-context learning scenarios.

Copied to clipboard

Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, Benyou Wang

ADELIE: Aligning Large Language Models on Information Extraction (2024.emnlp-main)

Challenge:	Multimodal large language models (MLLMs) lack visual knowledge in medical applications due to data privacy concerns and high annotation costs.
Approach:	They refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) to denoise and reformat the data.
Outcome:	The proposed model significantly improves the MMMU Health & Medicine track and shows that it can be used in multimodal scenarios.

Copied to clipboard

Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li

Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons (2024.emnlp-main)

Challenge:	Large language models (LLMs) struggle to follow complex instructions of IE tasks due to not being aligned with humans.
Approach:	They propose an aligned large language moDEL that effectively solves various IE tasks including closed IE, open IE and on-demand IE.
Outcome:	The proposed model achieves state-of-the-art (SoTA) performance among open-source models.

Copied to clipboard

Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, Daniel Zeng

Lexically Grounded Subword Segmentation (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models have underscored their exceptional reasoning prowess with natural language understanding across a broad spectrum of tasks.
Approach:	They examine whether Large Language Models actively recall or retrieve their internal repositories of factual knowledge when faced with reasoning tasks.
Outcome:	The proposed model improves reasoning performance while suppressing it leads to notable degradation.

Copied to clipboard

Jindřich Libovický, Jindřich Helcl

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024.emnlp-main)

Challenge:	Statistical word segmentation algorithms have remained a thorn in the side of many researchers.
Approach:	They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space.
Outcome:	The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks.

Copied to clipboard

Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang

Do Text-to-Vis Benchmarks Test Real Use of Visualisations? (2024.emnlp-main)

Challenge:	Modern Large Language Models (LLMs) are expensive and time-consuming.
Approach:	They propose a new technique of context-aware dynamic draft tree into drafting modeling.
Outcome:	The proposed method achieves speedup ratios of up to 5x, which is 1.3x that of EAGLE.

Copied to clipboard

Hy Nguyen, Xuefei He, Andrew Reeson, Cecile Paris, Josiah Poon, Jonathan Kummerfeld

Challenge:	Existing benchmarks for visualisations are limited and do not reflect real-world use.
Approach:	They analysed visualisation code from Python, R, Javascript, and Vega to find similarities and differences between real-world and benchmark datasets.
Outcome:	The results show that benchmark datasets do not test the same distribution of chart types, attributes, and actions as real-world examples.

Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs (2024.emnlp-main)

Copied to clipboard

Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, Fei Wu

Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) struggle when it comes to specialized domains due to limited domain-specific knowledge.
Approach:	They propose an adaptive method that automatically identifies valuable words from a given domain vocabulary.
Outcome:	The proposed method has been validated on three Chinese datasets and performed on general tasks.

Copied to clipboard

Jingyu Hu, Weiru Liu, Mengnan Du

Multi-Dialect Vietnamese: Task, Dataset, Baseline Models and Challenges (2024.emnlp-main)

Challenge:	Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models in processing tabular data.
Approach:	They propose a method that uses clustering and evolutionary strategies to curate a representative sample set from training data.
Outcome:	The proposed method significantly improves fairness across various metrics, showing its efficacy in real-world scenarios.

Copied to clipboard

Nguyen Dinh, Thanh Dang, Luan Thanh Nguyen, Kiet Nguyen

Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment (2024.emnlp-main)

Challenge:	Vietnamese is a low-resource language, but each province has its own distinct pronunciation variations.
Approach:	They propose a dataset that captures the rich diversity of 63 provincial dialects spoken in Vietnam.
Outcome:	The proposed dataset captures the rich diversity of 63 provincial dialects spoken across Vietnam.

Copied to clipboard

Vyas Raina, Adian Liusie, Mark Gales

Rethinking the Reversal Curse of LLMs: a Prescription from Human Knowledge Reversal (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations . however, no study has examined the vulnerability of judge-LLM to adversarial manipulation .
Approach:	They propose a simple surrogate attack where a surrogated model is attacked and the learned attack phrase transferred to unknown judge-LLMs.
Outcome:	The proposed algorithm shows that judge-LLMs can be significantly more susceptible to adversarial attacks when used for absolute scoring, rather than comparative assessment.

Copied to clipboard

Zhicong Lu, Li Jin, Peiguang Li, Yu Tian, Linhao Zhang, Sirui Wang, Guangluan Xu, Changyuan Tian, Xunliang Cai

More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs (2024.emnlp-main)

Challenge:	Existing methods for large language models (LLMs) are limited by their aggressive sample permutation and lack a detailed understanding of the underlying reasons for the reversal curse.
Approach:	They propose a method which enhances bidirectional entity correlation modeling and pairwise relationship reasoning to overcome the reversal curse.
Outcome:	The proposed method overcomes the reversal curse by augmenting the samples with entity order-reversals and semantically preserved question-answer pairs.

Copied to clipboard

Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Chao Wu, Changlong Sun, Kun Kuang, Fei Wu

Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Models (2024.emnlp-main)

Challenge:	a recent study shows that performance on general tasks decreases after Large Language Models are fine-tuned on domain-specific tasks.
Approach:	They propose a general capability integration approach to integrate general capabilities and domain knowledge within a single instance.
Outcome:	The proposed method improves performance on domain-specific tasks by integrating general capabilities and domain knowledge.

Copied to clipboard

Vyas Raina, Rao Ma, Charles McGhee, Kate Knill, Mark Gales

GENRA: Enhancing Zero-shot Retrieval with Rank Aggregation (2024.emnlp-main)

Challenge:	'special' tokens in large speech foundation models such as Whisper are used to guide their language generation process, but can be exploited by adversarial attacks to manipulate the model's behavior.
Approach:	They propose a method to learn a universal acoustic realization of Whisper's \|endoftext\|> token, which encourages the model to ignore the speech and only transcribe the special token, effectively muting the model.
Outcome:	The proposed method can mute Whisper models for over 97% of speech samples and can be used to bypass speech moderation systems and protect private speech data.

Copied to clipboard

Georgios Katsimpras, Georgios Paliouras

XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have been shown to perform zero-shot document retrieval, a process that typically consists of two steps: retrieving relevant documents, and re-ranking them based on their relevance to the query.
Approach:	They propose a new approach to zero-shot document retrieval that incorporates rank aggregation to improve retrieval effectiveness.
Outcome:	The proposed approach improves existing methods on benchmark datasets and shows that it can perform zero-shot retrieval.

Copied to clipboard

Zichen Chen, Jianda Chen, Ambuj Singh, Misha Sra

Divide and Conquer Radiology Report Generation via Observation Level Fine-grained Pretraining and Prompt Tuning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have achieved remarkable success in natural language tasks, yet understanding their reasoning processes remains a significant challenge.
Approach:	They propose a dataset that includes 24204 instances where each instance interprets the LLM’s reasoning behavior using knowledge graphs and graph attention networks (GAT).
Outcome:	The proposed explanation framework reduces hallucinations and improves grounded explanation generation in large language models.

Copied to clipboard

Yuanpin Zhou, Huogen Wang

SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information (2024.emnlp-main)

Challenge:	Recent advances in image captioning and vision-language pretraining have made it difficult for radiologists to generate coherent and accurate reports.
Approach:	They propose a model which breaks down full-text radiology reports into concise observation descriptions and encodes observation predictions into a decoding stage.
Outcome:	The proposed model achieves significant improvements across all metrics, underscoring its capability to generate semantically coherent and clinically accurate radiology reports.

Copied to clipboard

Jiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su, Xiaoye Qu, Yu Cheng

UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models (2024.emnlp-main)

Challenge:	Existing studies focus on the text modality or are limited to specific tasks.
Approach:	They propose a framework to teach Large Vision-Language Models to selectively utilize retrieved information and improve their robustness against irrelevant or misleading references.
Outcome:	The proposed framework improves LVLMs’ ability to utilize retrieved multimodal references and their robustness against irrelevant or misleading information.

Copied to clipboard

Zhanyue Qin, Haochuan Wang, Deyuan Liu, Ziyang Song, Cunhang Fan, Zhao Lv, Jinlin Wu, Zhen Lei, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, Dianbo Sui

Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (2024.emnlp-main)

Challenge:	Existing LLMs demonstrate powerful capabilities between tasks, but can they make sequential decisions?
Approach:	They propose to evaluate sequential decision-making capability of large language models (LLMs) using novel metrics based Monte Carlo methods.
Outcome:	The proposed benchmark improves sequential decision-making performance compared to the vanilla LLM player.

Copied to clipboard

Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, Yu Su

MORPHEUS: Modeling Role from Personalized Dialogue History by Exploring and Utilizing Latent Space (2024.emnlp-main)

Challenge:	Large language models (LLMs) are generalist agents capable of operating within complex environments.
Approach:	They propose a class of tools that can serve as a middleware layer shielding LLMs from environmental complexity.
Outcome:	The proposed tool can shield the LLM from environmental complexity in two representative complex environments.

Copied to clipboard

Yihong Tang, Bo Wang, Dongming Zhao, Jinxiaojia Jinxiaojia, Zhangjijun Zhangjijun, Ruifang He, Yuexian Hou

KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server (2024.emnlp-main)

Challenge:	Personalized Dialogue Generation relies on external role data, which can be scarce and raise privacy concerns.
Approach:	They propose a framework to extract role information from dialogue history . they use persona codebook to represent roles in latent space and posterior distribution of role information .
Outcome:	The proposed framework can generalize across roles, even for unseen roles.

Copied to clipboard

WenHao Wang, Xiaoyu Liang, Rui Ye, Jingyi Chai, Siheng Chen, Yanfeng Wang

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)

Challenge:	Existing methods to train large language models on private data are not effective because they rely on a local model for generation, resulting in a performance decline, or expose private data to API servers.
Approach:	They propose a client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy.
Outcome:	The proposed framework improves model performance and privacy while learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from server.

Copied to clipboard

Xuan Gong, Tianshi Ming, Xinpeng Wang, Zhihua Wei

Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models (2024.emnlp-main)

Challenge:	Despite the success of Large Vision-Language Models, they suffer from hallucination.
Approach:	They propose a training-free strategy that "Dive into" the attention of LVLMs to "Reduce" object hallucination by using classification tokens of ViT.
Outcome:	The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP .

Copied to clipboard

Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, Jun Zhao

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)

Challenge:	Recent studies have shown that large language models may possess preliminary planning capabilities.
Approach:	They examine the look-ahead planning mechanism in large language models from the perspectives of information flow and internal representations.
Outcome:	The proposed model can decode the decision from the output of MHSA in the middle layers at the last token.

Copied to clipboard

Wenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin, Li Yue, Ming Zhou

An Empirical Study of Multilingual Reasoning Distillation for Question Answering (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive.
Approach:	They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them.
Outcome:	The proposed approach saves significant resources and accelerates convergence and performance.

Copied to clipboard

Patomporn Payoungkhamdee, Peerat Limkonchotiwat, Jinheon Baek, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong

Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? (2024.emnlp-main)

Challenge:	Existing efforts to distill reasoning capabilities have focused mainly on English, leaving multilingual distillation underexplored.
Approach:	They propose a method that incorporates incorrect rationales as additional guidance to improve multilingual reasoning in large language models.
Outcome:	Empirical results show that d-CoT-nR significantly surpasses the baseline, improving accuracy in unseen languages and correctness in step-by-step reasoning.

Copied to clipboard

Gal Yona, Roee Aharoni, Mor Geva

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024.emnlp-main)

Challenge:	Despite their unprecedented capabilities, large language models (LLMs) often output erroneous information, which may lead users to overly rely on their false output.
Approach:	They formalize faithful response uncertainty based on the gap between the model’s intrinsic confidence in the assertions it makes and the decisiveness by which they are conveyed.
Outcome:	The proposed model is poor at faithfully conveying uncertainty on knowledge-intensive questions.

Copied to clipboard

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig

Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning (2024.emnlp-main)

Challenge:	Pre-training Large Language Models (LLMs) on textual corpora embeds substantial factual knowledge in their parameters, which is essential for excelling in various downstream applications.
Approach:	They propose to use supervised fine-tuning to align large language models to new factual information that is not acquired through pre-training.
Outcome:	The proposed model is trained to generate facts that are not grounded in pre-existing knowledge, but hallucinates when examples with new knowledge are learned.

Copied to clipboard

Ming Shan Hee, Aditi Kumaresan, Roy Lee

MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding (2024.emnlp-main)

Challenge:	Recent research has developed models targeting specific modalities but lacks transferability between formats.
Approach:	They conduct extensive experiments using few-shot in-context learning with large language models to explore the transferability of hate speech detection between modalities.
Outcome:	The proposed model outperforms vision-language demonstrations in few-shot learning settings.

Copied to clipboard

Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, Chen Luo, Qingyu Yin, Bing Yin, Long Chen, Yangqiu Song

ECON: On the Detection and Resolution of Evidence Conflicts (2024.emnlp-main)

Challenge:	Existing methods for acquiring large-scale intentions generate product-centric intentions without product images and incur high costs for scalability.
Approach:	They propose a multimodal framework that allows Large Vision-Language Models to infer purchase intentions from multimodal product metadata and prioritize human-centric ones.
Outcome:	The proposed framework shows that it is robust to different prompts and superior to previous methods.

Copied to clipboard

Cheng Jiayang, Chunkit Chan, Qianqian Zhuang, Lin Qiu, Tianhang Zhang, Tengxiao Liu, Yangqiu Song, Yue Zhang, Pengfei Liu, Zheng Zhang

“Image, Tell me your story!” Predicting the original meta-context of visual misinformation (2024.emnlp-main)

Challenge:	Recent studies have shown that AI generated content is more likely to dominate search results, making it difficult to detect when compared to human-produced content.
Approach:	They propose a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios.
Outcome:	The proposed method enables the detection of conflicting information in real-world scenarios and shows that weaker models struggle with similar answer conflicts while stronger models show robust performance.

Copied to clipboard

Jonathan Tonglet, Marie-Francine Moens, Iryna Gurevych

Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning (2024.emnlp-main)

Challenge:	a surge of visual misinformation poses a growing threat to our society.
Approach:	They propose to use images to contextualize images to establish original meta-context . they use a framework that collects evidence and questions from the open web to ground the image .
Outcome:	The proposed approach helps human fact-checkers identify misinformation . it uses image contextualization to establish the original meta-context of the image .

Copied to clipboard

Zhili Shen, Pavlos Vougiouklis, Chenxin Diao, Kaustubh Vyas, Yuanyi Ji, Jeff Pan

Mixture-of-Subspaces in Low-Rank Adaptation (2024.emnlp-main)

Challenge:	Existing methods for text-to-SQL semantic parsing are limited to retrieving schemata based on a single query.
Approach:	They propose a text-to-sql semantic parser that uses abstract syntax trees to select few-shot examples for retrieval-augmented generation.
Outcome:	The proposed model can generate approximated versions of SQL queries in parallel, and shows improvements over state-of-the-art benchmarks.

Copied to clipboard

Taiqiang Wu, Jiahao Wang, Zhe Zhao, Ngai Wong

PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data (2024.emnlp-main)

Challenge:	Using a subspace-inspired Low-Rank Adaptation method, large language models can be optimized for downstream tasks using parameter-efficient finetuning.
Approach:	They propose a subspace-inspired Low-Rank Adaptation method that decomposes LoRA weights into two subspaces and merges them into the frozen original weight.
Outcome:	The proposed method outperforms LoRA on commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation tasks.

Copied to clipboard

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, Sunayana Sitaram

LawBench: Benchmarking Legal Knowledge of Large Language Models (2024.emnlp-main)

Challenge:	Evaluation of multilingual Large Language Models is challenging due to a variety of factors including the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and lack of local, cultural nuances in translated benchmarks.
Approach:	They evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations.
Outcome:	The proposed models perform best in most Indic languages, while the agreement drops for direct assessment especially for Bengali and Odia.

Copied to clipboard

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, Vincent Ng

Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards (2024.emnlp-main)

Challenge:	LegalBench evaluated 20 LLMs in 162 legal tasks in 20 countries and jurisdictions.
Approach:	They present a comprehensive evaluation of 21 popular Large Language Models and the first comparative analysis of the empirical results.
Outcome:	The proposed benchmarks are based on the Bloom’s cognitive taxonomy and are compared to 21 popular LLMs.

Copied to clipboard

Furkan Şahinuç, Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych

Efficient Vision-Language pre-training via domain-specific learning for human activities (2024.emnlp-main)

Challenge:	Existing leaderboards are incomplete and some contain incorrect information.
Approach:	They propose a manually-curated Scientific Leaderboard dataset that overcomes these problems . they propose three experimental settings where TDM triples are fully defined, partially defined, or undefined .
Outcome:	The proposed system overcomes the shortcomings of existing leaderboard datasets . it can be used to evaluate and compare scientific methods, but it requires manual labor .

Copied to clipboard

Adrian Bulat, Yassine Ouali, Ricardo Guerrero, Brais Martinez, Georgios Tzimiropoulos

Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training (2024.emnlp-main)

Challenge:	Current vision-language models owe their success to large-scale pretraining on web-collected data.
Approach:	They propose a domain-aligned pretraining strategy that aligns the downstream tasks to the downstream domain without additional data collection.
Outcome:	The proposed method outperforms existing models on large-scale vision-language training datasets while preserving generalist knowledge.

Copied to clipboard

Wenbo Li, Guohao Li, Zhibin Lan, Xue Xu, Wanru Zhuang, Jiachen Liu, Xinyan Xiao, Jinsong Su

Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works (2024.emnlp-main)

Challenge:	Existing text-to-image models struggle to generate images with legible visual texts . current models lack support for Chinese texts, misspelling, and lack of diversity .
Approach:	They propose to empower backbone models to generate visual texts in Chinese and English . they propose to augment conventional training objective with glyph-aware training losses .
Outcome:	The proposed methods can generate visual texts in English and Chinese while maintaining image generation quality.

Copied to clipboard

Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, Deqing Yang

Shimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang

Challenge:	Recent advances in large language models (LLMs) have catalyzed numerous AI applications, among which role-playing agents (RPAs) are particularly popular.
Approach:	They propose to evaluate LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development.
Outcome:	The proposed model outperforms existing models and literature summarization methods and proves its ability to understand fictional characters in downstream tasks.

Copied to clipboard

AdaSwitch: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown impressive language capabilities, but most of them have very unbalanced performance across different languages.
Approach:	They propose to use question translation data to enhance LLMs' multilingual capabilities by using mechanistic interpretability methods.
Outcome:	The proposed method improves multilingual alignment even with unannotated answers in English and a wide range of languages even with instruction-tuned LLMs.

Copied to clipboard

Hao Sun, Jiayi Wu, Hengyi Cai, Xiaochi Wei, Yue Feng, Bo Wang, Shuaiqiang Wang, Yan Zhang, Dawei Yin

CoBa: Convergence Balancer for Multitask Finetuning of Large Language Models (2024.emnlp-main)

Challenge:	Recent advances in large language models have been remarkable . users face a choice between using cloud-based LLMs for generation quality or local-based ones for lower computational cost .
Approach:	They propose a new LLM utilization paradigm that facilitates collaborative operation . they evaluate AdaSwitch across 7 benchmarks and compare it to other LLMs .
Outcome:	The proposed model improves performance of local and cloud agents across 7 benchmarks . it achieves competitive results compared to the cloud agent while utilizing less computational overhead.

Copied to clipboard

Zi Gong, Hang Yu, Cong Liao, Bingchang Liu, Chaoyu Chen, Jianguo Li

mDPO: Conditional Preference Optimization for Multimodal Large Language Models (2024.emnlp-main)

Challenge:	Existing multi-task learning approaches for large language models fall short due to computational intensive or lack of simultaneous task convergence.
Approach:	They propose a new multi-task learning approach that dynamically adjusts task weights during the training process, ensuring that the validation loss of all tasks progresses towards convergence at an even pace.
Outcome:	The proposed approach improves the performance of large language models by up to 13% compared to the second-best approaches.

Copied to clipboard

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models (2024.emnlp-main)

Challenge:	Recent studies have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement.
Approach:	They propose a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference.
Outcome:	The proposed method significantly improves performance on two multimodal LLMs of different sizes and three widely used benchmarks.

Copied to clipboard

Fei Wang, Ninareh Mehrabi, Palash Goyal, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)

Challenge:	Recent studies have explored using LLMs for efficient data collection.
Approach:	They propose a method that takes into account the characteristics of the desired dataset and monitors the status of the generated data.
Outcome:	The proposed method improves safety and quality of three representative large language models against safety issues without sacrificing model utility.

Copied to clipboard

Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas

Attribute or Abstain: Large Language Models as Long Document Assistants (2024.emnlp-main)

Challenge:	In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API.
Approach:	They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API.
Outcome:	The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting.

Copied to clipboard

Jan Buchmann, Xiao Liu, Iryna Gurevych

FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models (2024.emnlp-main)

Challenge:	Existing approaches to attribution have only been evaluated in RAG settings, where initial retrieval confounds performance.
Approach:	They propose to use a benchmark to evaluate attribution on long document tasks . they find that citations and additional retrieval perform best for large models .
Outcome:	The proposed approach performs best on large and fine-tuned models, while additional retrieval can help for small, prompted models.

Copied to clipboard

Xiaochen Wang, Jiaqi Wang, Houping Xiao, Jinghui Chen, Fenglong Ma

Retrieved In-Context Principles from Previous Mistakes (2024.emnlp-main)

Challenge:	Existing foundation models are limited in access to diverse modalities and privacy regulations restrict the development of comprehensive foundation models.
Approach:	They propose a knowledge injection approach to extract and inject healthcare knowledge into medical foundation models to enhance their ability to handle multiple tasks and modalities.
Outcome:	The proposed method preserves privacy and enhances the model’s ability to handle complex medical tasks involving multiple modalities.

Copied to clipboard

Hao Sun, Yong Jiang, Bo Wang, Yingyan Hou, Yan Zhang, Pengjun Xie, Fei Huang

EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control (2024.emnlp-main)

Challenge:	Recent advances in in-context learning (ICL) have limited customization and inadequate error coverage.
Approach:	They propose a method to retrieve in-context principles from mistakes to improve model performance.
Outcome:	The proposed framework enhances model performance when applied to various prompting strategies.

Copied to clipboard

Haozhe Chen, Run Chen, Julia Hirschberg

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024.emnlp-main)

Challenge:	EmoKnob framework allows fine-grained emotion control in speech synthesis with few-shot demonstrative samples of arbitrary emotion.
Approach:	They propose a framework that allows fine-grained emotion control in speech synthesis . they propose two methods to apply emotion control on emotions described by open-ended text .
Outcome:	The proposed framework allows fine-grained emotion control in speech synthesis with few-shot demonstrative samples of arbitrary emotion.

Copied to clipboard

Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, Mao Yang

An L* Algorithm for Deterministic Weighted Regular Languages (2024.emnlp-main)

Challenge:	Recent research has focused on pushing weight-only quantization to extremely low-bit due to numerical representation limitations.
Approach:	They propose a vector-based quantization approach that pushes LLMs to extremely low-bit . they propose scalar-based weight quantization that reduces memory requirements and optimizes storage costs .
Outcome:	The proposed method reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on mistral-7B, 4.41-7.34, on llaMA-3 on QA tasks on average.

Copied to clipboard

Clemente Pasti, Talu Karagöz, Franz Nowak, Anej Svete, Reda Boumasmoud, Ryan Cotterell

Challenge:	Angluin (1987) introduced the active learning scheme L , where the learner interacts with an oracle by asking it queries.
Approach:	They propose a weighted variant of Angluin's (1987) L* algorithm for learning finite state automatas from black-box models.
Outcome:	The proposed algorithm learns aminimal automaton for the target language.

Towards Verifiable Text Generation with Evolving Memory and Self-Reflection (2024.emnlp-main)

Copied to clipboard

Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, Dawei Yin

Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification (2024.emnlp-main)

Challenge:	Large language models (LLMs) often produce factually incorrect information, also known as hallucination.
Approach:	They propose a framework for verifiable text generation with evolving memory and self-reflection that incorporates long-term memory to retain documents and recent documents.
Outcome:	The proposed framework outperforms baselines on five datasets across three knowledge-intensive tasks.

Copied to clipboard

Pritish Sahu, Karan Sikka, Ajay Divakaran

Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributes (2024.emnlp-main)

Challenge:	Large Visual Language Models (LVLMs) suffer from hallucinations due to limited training data, lack of * Equal contribution precise grounding, and over-reliance on language priors.
Approach:	They propose a framework to detect and mitigate hallucinations through claim verification using program-of-thought prompting and Python code to generate a graph.
Outcome:	The proposed framework improves over baseline LVLMs and existing methods across several benchmarks.

Copied to clipboard

Yusuke Hirota, Jerone Andrews, Dora Zhao, Orestis Papakyriakopoulos, Apostolos Modas, Yuta Nakashima, Alice Xiang

RealVul: Can We Detect Vulnerabilities in Web Applications with LLM? (2024.emnlp-main)

Challenge:	Traditional approaches only target labeled attributes, ignoring biases from unlabeled ones.
Approach:	They propose a method that ensures protected group independence from all attributes and mitigates inpainting biases through data filtering.
Outcome:	The proposed approach achieves an average reduction of 46.1% in leakage-based bias metrics for multi-label classification and 74.8% for image captioning.

Copied to clipboard

Di Cao, Yong Liao, Xiuwei Shang

Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel (2024.emnlp-main)

Challenge:	a lack of research specifically focused on vulnerabilities in the PHP language hinders the model’s ability to effectively capture the characteristics of specific vulnerabilities.
Approach:	They propose a framework that can isolate potential vulnerability triggers while streamlining code and eliminating unnecessary semantic information.
Outcome:	The proposed framework can isolate potential vulnerability triggers while streamlining the code and eliminating unnecessary semantic information.

Copied to clipboard

Brendan King, Jeffrey Flanigan

Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)

Challenge:	a task-oriented dialogue system requires turn-level annotations for interacting with their APIs.
Approach:	They propose an unsupervised approach that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent.
Outcome:	The proposed method doubles the success rate of a strong GPT-3.5 benchmark.

Copied to clipboard

Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, Benyou Wang

WPO: Enhancing RLHF with Weighted Preference Optimization (2024.emnlp-main)

Challenge:	Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs.
Approach:	They propose a framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bia,*Authority Bia and Beauty Bia's on LLM and human judges.
Outcome:	The proposed framework investigates Misinformation Oversight Bias, Gender Bia,*Authority Bia and **Beauty Bia' on LLM and human judges.

Copied to clipboard

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

Walking in Others’ Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias (2024.emnlp-main)

Challenge:	Off-policy preference optimization suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization.
Approach:	They propose a method to simulate on-policy learning with off-police preference data.
Outcome:	The proposed method outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 and MT-bench.

Copied to clipboard

Rongwu Xu, Zian Zhou, Tianwei Zhang, Zehan Qi, Su Yao, Ke Xu, Wei Xu, Han Qiu

MetaReflection: Learning Instructions for Language Agents using Past Reflections (2024.emnlp-main)

Challenge:	Existing prompting methods that require white-box access to the model or substantial training fail to simultaneously lessen toxicity and bias.
Approach:	They propose a strategy that encourages LLMs to integrate diverse human perspectives and self-regulate their responses by incorporating diverse human viewpoints.
Outcome:	The proposed approach can significantly diminish toxicity (up to 89%) and bias (up 73%) in LLMs’ responses.

Copied to clipboard

Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, Sherry Shi

Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have gained popularity due to their ability to generate human-like text and solve complex tasks.
Approach:	They propose an offline reinforcement learning technique that augments a semantic memory based on experiential learnings from past trials.
Outcome:	The proposed technique boosts Language agents’ performance by 4 % to 16.82 % over the raw GPT-4 baseline and performs on par with existing state-of-the-art prompt optimization techniques while requiring fewer LLM calls.

Copied to clipboard

Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

On Eliciting Syntax from Language Models via Hashing (2024.emnlp-main)

Challenge:	Existing models for dialog tutoring fail to detect student errors and tailor their feedback to them.
Approach:	They propose to build dialog tutoring models to scaffold students' problem-solving and verify student solutions by using automatic and human evaluation.
Outcome:	The proposed model improves the quality of the tutor response generation by detecting student errors and adjusting the feedback to the errors.

Copied to clipboard

Yiran Wang, Masao Utiyama

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios (2024.emnlp-main)

Challenge:	Unsupervised parsing aims to infer syntactic structure from raw text . despite its importance, advancements in this task have been slow .
Approach:	They propose to use unsupervised parsing to infer syntactic structure from raw text . they upgrade the bit-level CKY to first-order to encode lexicon and syntax .
Outcome:	The proposed method shows competitive performance on various datasets.

Copied to clipboard

Zetian Ouyang, Yishuai Qiu, Linlin Wang, Gerard De Melo, Ya Zhang, Yanfeng Wang, Liang He

The Best Defense is Attack: Repairing Semantics in Textual Adversarial Examples (2024.emnlp-main)

Challenge:	Chinese medical large language models (LLMs) are underperforming on this benchmark, especially where medical reasoning and factual consistency are vital.
Approach:	They propose a benchmark with 14 expert-guided clinical scenarios to assess the medical ability of large language models across 7 pivot dimensions.
Outcome:	The proposed benchmark has been validated in several ways.

Copied to clipboard

Heng Yang, Ke Li

CSSL: Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource Languages (2024.emnlp-main)

Challenge:	Recent studies have revealed the vulnerability of pre-trained language models to adversarial attacks.
Approach:	They propose a novel approach to repair adversarial examples using an adversarial detector.
Outcome:	The proposed approach is effective in various adversarial attack scenarios.

Copied to clipboard

Pretam Ray, Jivnesh Sandhan, Amrith Krishna, Pawan Goyal

Perceptions of Linguistic Uncertainty by Language Models and Humans (2024.emnlp-main)

Challenge:	Neural dependency parsing has achieved remarkable performance for low resource morphologically rich languages.
Approach:	They propose a self-supervised learning method to make the model robust to word order variations.
Outcome:	The proposed model shows a substantial gain of 3.03/2.95 points in 7 relatively free word order languages when compared to the best performing baseline.

Copied to clipboard

Catarina Belém, Markelle Kelly, Mark Steyvers, Sameer Singh, Padhraic Smyth

Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM (2024.emnlp-main)

Challenge:	Prior work has shown that humans are well-attuned to the use of uncertainty expressions, exhibiting population-level agreement in mapping these expressions to numerical responses.
Approach:	They propose to map linguistic expressions of uncertainty to numerical responses by using a theory of mind approach to understand the uncertainty of another agent.
Outcome:	The proposed model can map expressions to probabilistic responses in a human-like manner, but different behavior depending on whether a statement is actually true or false.

Copied to clipboard

Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, Tagyoung Chung

Zero-shot Cross-domain Dialogue State Tracking via Context-aware Auto-prompting and Instruction-following Contrastive Decoding (2024.emnlp-main)

Challenge:	Contrastive decoding (CD) improves the next-token distribution of a large expert language model (LM) using a small amateur LM.
Approach:	They propose a new unsupervised decoding method called Asymptotic Probability Decoding (APD) that extrapolates the probability curves from the LMs of different sizes to infer the asymptototic probabilities from an infinitely large LM.
Outcome:	The proposed method improves the next-token distribution of a large expert language model using a small amateur LM.

Copied to clipboard

Xiaoyu Dong, Yujie Feng, Zexin Lu, Guangyuan Shi, Xiao-Ming Wu

Knowledge Conflicts for LLMs: A Survey (2024.emnlp-main)

Challenge:	Previous studies have implemented slot-based input improvements, such as schema-driven descriptions and question-answering formats, but still suffer from negative transfer for seen slots and inefficient transfer for unseen slots due to the significant source-target domain gap.
Approach:	They propose a framework that generates dynamic, context-aware slot queries to improve model transferability by penalizing deviations from the provided instructions.
Outcome:	Experiments on two datasets show that the proposed model performs better than existing models on the restaurant domain.

Copied to clipboard

Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, Wei Xu

MisinfoEval: Generative AI in the Era of “Alternative Facts” (2024.emnlp-main)

Challenge:	This survey examines knowledge conflicts for large language models (LLMs) this survey aims to shed light on strategies for improving the robustness of LLMs .
Approach:	They focus on three categories of knowledge conflicts: context-memory, inter-context, and intra-membry conflict.
Outcome:	The findings highlight the challenges faced by large language models when blending contextual and parametric knowledge.

Copied to clipboard

Saadia Gabriel, Liang Lyu, James Siderius, Marzyeh Ghassemi, Jacob Andreas, Asuman Ozdaglar

MEANT: Multimodal Encoder for Antecedent Information (2024.emnlp-main)

Challenge:	Existing efforts to address misinformation on social media platforms are hampered by user biases and scalability challenges.
Approach:	They propose a framework for generating and comprehensively evaluating large language model based misinformation interventions using a simulated social media environment and personalized explanations tailored to users' beliefs.
Outcome:	The proposed framework improves accuracy at reliability labeling by up to 41.72% and personalized explanations appeal to users' pre-existing values.

Copied to clipboard

Benjamin Irving, Annika Schoene

A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)

Challenge:	Multimodal data is an ideal candidate for multimodal evaluation, but information can exist across time.
Approach:	They propose a multimodal encoder for anticedent information and a dataset that consists of price, Tweets, and graphical data.
Outcome:	The MEANT model improves performance on baselines by 15% and the textual information affects performance far more than visual information on time-dependent tasks.

Copied to clipboard

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, Wai Lam

AGRaME: Any-Granularity Ranking with Multi-Vector Embeddings (2024.emnlp-main)

Challenge:	Decoding methods are essential for converting language models from next-token predictors into practical task solvers.
Approach:	They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent .
Outcome:	The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization.

Copied to clipboard

Revanth Gangi Reddy, Omar Attia, Yunyao Li, Heng Ji, Saloni Potdar

FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2024.emnlp-main)

Challenge:	Existing ranking algorithms restrict granularity to full passages or require a specific dense index for each desired level of granules.
Approach:	They propose a multi-vector ranking approach that leverages multi-vctor embeddings to rank at varying levels of granularity while maintaining encoding at a single (coarser) level of grail.
Outcome:	The proposed method surpasses prompt-driven citation generation by incorporating proposition-level ranking to post-hoc citation addition.

Copied to clipboard

Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, Heng Ji

Exploring Nested Named Entity Recognition with Large Language Models: Methods, Challenges, and Insights (2024.emnlp-main)

Challenge:	Existing listwise LLMs lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers.
Approach:	They propose a listwise LLM reranking approach that leverages the first generated identifier to obtain a ranked ordering of the candidates.
Outcome:	The proposed approach accelerates inference by 50% while maintaining robust ranking performance with gains across BEIR benchmark.

Copied to clipboard

Hongjin Kim, Jai-Eun Kim, Harksoo Kim

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods (2024.emnlp-main)

Challenge:	Named entity recognition (NER) is a challenging task in natural language processing . nested NER requires sophisticated techniques to identify entities within entities .
Approach:	They investigate the application of Large Language Models (LLMs) to nested NER . they find methodologies from previous work are less effective .
Outcome:	The proposed methods outperform BERT-based models in nested NER tasks . however, they do not outperformed the existing models on the GENIA dataset .

Copied to clipboard

Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, Bhuwan Dhingra

“Flex Tape Can’t Fix That”: Bias and Misinformation in Edited Language Models (2024.emnlp-main)

Challenge:	ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Approach:	They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Outcome:	The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach.

Copied to clipboard

Karina Halevy, Anna Sotnikova, Badr AlKhamissi, Syrielle Montariol, Antoine Bosselut

Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective (2024.emnlp-main)

Challenge:	Weight-based model editing methods can unintentionally alter unrelated parametric knowledge representations, potentially increasing the risk of harm.
Approach:	They propose a benchmark dataset for measuring bias amplification of model editing methods for demographic traits such as race, geographic origin, and gender.
Outcome:	The proposed methods can unintentionally alter unrelated parametric knowledge representations, potentially increasing the risk of harm.

Copied to clipboard

Yujian Liu, Yang Zhang, Tommi Jaakkola, Shiyu Chang

Challenge:	Existing and new datasets show that our approach achieves competitive performance in all of the criteria.
Approach:	They propose a new task of LLM targeted unlearning where unlearning targets only the information about the unlearning target, rather than everything in the unlearned documents.
Outcome:	The proposed method achieves competitive performance on existing and new datasets without optimizing for the aforementioned criteria.

LIONs: An Empirically Optimized Approach to Align Language Models (2024.emnlp-main)

Copied to clipboard

Xiao Yu, Qingyang Wu, Yu Li, Zhou Yu

Jellyfish: Instruction-Tuning Local Large Language Models for Data Preprocessing (2024.emnlp-main)

Challenge:	Recent studies have focused on aligning large language models with pre-trained datasets.
Approach:	They conduct a rigorous analysis of a three-stage training pipeline using sequence packing, loss masking and increasing the preference dataset size in DPO to improve the performance of language models.
Outcome:	The proposed models outperform the official instruct models tuned with closed-source data and algorithms.

Copied to clipboard

Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada

A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery (2024.emnlp-main)

Challenge:	Until 2021, most efforts were concentrated on one or two specific tasks such as error detection (ED) and data imputation (DI).
Approach:	They propose to instruction tune local LLMs as universal DP task solvers that operate on a local, single, and low-priced GPU, ensuring data security and enabling further customization.
Outcome:	The proposed models deliver competitiveness and generalizability to unseen tasks while barely compromising the base models’ abilities in NLP tasks.

Copied to clipboard

Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, Jiawei Han

MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024.emnlp-main)

Challenge:	Existing surveys on scientific LLMs focus on one or two fields or a single modality.
Approach:	They survey 260 scientific LLMs and examine their architectures and pre-training techniques . they also discuss commonalities and differences between LLM architectures .
Outcome:	The proposed model architectures and evaluation techniques are used to improve scientific discovery.

Copied to clipboard

Liyan Tang, Philippe Laban, Greg Durrett

Beyond Label Attention: Transparency in Language Models for Automated Medical Coding via Dictionary Learning (2024.emnlp-main)

Challenge:	Current methods for fact-checking are based on verifying each piece of a model against potential evidence using an LLM.
Approach:	They propose a method that builds small fact-checking models that have GPT-4-level performance but 400x lower cost.
Outcome:	The proposed model outperforms other models and reaches GPT-4 accuracy.

Copied to clipboard

John Wu, David Wu, Jimeng Sun

MOSEL: Inference Serving Using Dynamic Modality Selection (2024.emnlp-main)

Challenge:	Current efforts in interpretability of medical coding rely heavily on label attention mechanisms, which often leads to the highlighting of extraneous tokens irrelevant to the ICD code.
Approach:	They propose to leverage dictionary learning to extract sparsely activated representations from dense language models embedded in superposition to facilitate accurate interpretability.
Outcome:	The proposed model extracts sparsely activated representations from dense language models in superposition, even when the highlighted tokens are medically irrelevant.

Copied to clipboard

Bodun Hu, Le Xu, Jeongyoon Moon, Neeraja Yadwadkar, Aditya Akella

Challenge:	Recent advances in machine learning have enabled deep learning to exceed human capabilities in various tasks.
Approach:	They propose a new form of dynamism, modality selection, where modality picks modalities from inference inputs while maintaining the model quality.
Outcome:	The proposed system improves system throughput by 3.6 and job completion times by 11 compared to modality-agnostic approaches.

From RAG to Riches: Retrieval Interlaced with Sequence Generation (2024.emnlp-main)

Copied to clipboard

Palak Jain, Livio Baldini Soares, Tom Kwiatkowski

Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition (2024.emnlp-main)

Challenge:	RICHES interleaves retrieval with sequence generation tasks . traditional approaches chain LLM generation with separate retrieval model .
Approach:	They propose a novel approach that interleaves retrieval with sequence generation tasks . they propose attributed evidence, multi-hop retrievals and interleave thoughts to plan on what to retrieve next .
Outcome:	The proposed approach can work with any Instruction-tuned model, without additional training.

Copied to clipboard

Hsuan Su, Hua Farn, Fan-Yun Sun, Shang-Tse Chen, Hung-yi Lee

Learning to Correct for QA Reasoning with Black-box LLMs (2024.emnlp-main)

Challenge:	Existing methods for speech recognition suffer from the synthetic-to-real gap . existing methods suffer from this distributional shift due to acoustic mismatches .
Approach:	They propose to use task arithmetic to fine-tune an ASR model on synthetic data to mitigate the synthetic-to-real gap.
Outcome:	The proposed method shows an improvement of 10.03% over baselines on the SLURP dataset.

Copied to clipboard

Jaehyung Kim, Dongyoung Kim, Yiming Yang

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024.emnlp-main)

Challenge:	Existing approaches to improve reasoning capability of large language models rely on accessibility or require significantly increased train- and inference-time costs.
Approach:	They propose a method to improve QA reasoning of large language models in a black-box setting by using a trained adaptation model to perform a seq2seq mapping from the often-imperfect reasonings of the original LLM to the correct or improved reasonings.
Outcome:	The proposed approach significantly improves reasoning accuracy across various QA benchmarks compared to the best-performing adaptation baselines.

Copied to clipboard

Ori Yoran, Samuel Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant

PostMark: A Robust Blackbox Watermark for Large Language Models (2024.emnlp-main)

Challenge:	Current language models and retrieval-augmented LMs are limited in their ability to perform tasks on the web.
Approach:	They propose a benchmark to evaluate language agents built on top of language models . they propose 'AssistantBench' which includes 214 tasks that can be automatically evaluated .
Outcome:	The proposed agent outperforms existing agents in a new benchmark for language agents on the web.

Copied to clipboard

Yapei Chang, Kalpesh Krishna, Amir Houmansadr, John Wieting, Mohit Iyyer

Assessing “Implicit” Retrieval Robustness of Large Language Models (2024.emnlp-main)

Challenge:	Existing methods to detect LLM-generated text require access to the underlying LLM’s logits, which LLM providers are loath to share due to fears of model distillation.
Approach:	They develop a post-hoc watermarking procedure that inserts an input-dependent set of words into the text after the decoding process has completed.
Outcome:	The proposed method is more robust to paraphrasing attacks than existing methods.

Copied to clipboard

Xiaoyu Shen, Rexhina Blloshmi, Dawei Zhu, Jiahuan Pei, Wei Zhang

On the Relationship between Truth and Political Bias in Language Models (2024.emnlp-main)

Challenge:	Retrieval-augmented generation (RAG) is a framework to enhance large language models with external knowledge, but its effectiveness is constrained by the retrieval robustness of the model.
Approach:	They propose to use gold and distracting context to fine-tune models to handle relevant or irrelevant retrieved context in an end-to-end manner.
Outcome:	The proposed model performs better when gold and distracting context are used, while still extracting correct answers when retrieval is accurate.

Copied to clipboard

Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, Jad Kabbara

Can Active Label Correction Improve LLM-based Modular AI Systems? (2024.emnlp-main)

Challenge:	Language model alignment research often attempts to ensure that models are helpful and harmless, but can obscure how improving one aspect might impact the other.
Approach:	They analyze the relationship between truthfulness and political bias in language models.
Outcome:	The results show that optimizing models for truthfulness results in a left-leaning political bias.

Copied to clipboard

Karan Taneja, Ashok Goel

Challenge:	Large language models (LLMs) are powerful zero or few-shot learners and can generalize to a wide range of tasks without any model fine-tuning.
Approach:	They propose to use LLM annotations to train smaller task-specific improved models that can replace LLMs.
Outcome:	The proposed method can improve oracle performance with feedback on 17-24% fewer examples than the number of noisy examples in the dataset across three different NLP tasks.

Statistical Uncertainty in Word Embeddings: GloVe-V (2024.emnlp-main)

Copied to clipboard

Andrea Vallebueno, Cassandra Handan-Nader, Christopher Manning, Daniel Ho

Annotation alignment: Comparing LLM and human annotations of conversational safety (2024.emnlp-main)

Challenge:	Static word embeddings are ubiquitous in computational social science applications . however, assessing the statistical uncertainty in downstream conclusions remains challenging .
Approach:	They propose a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for one of the most widely used word embedding models.
Outcome:	The proposed method enables hypothesis testing in key word embedding tasks.

Copied to clipboard

Rajiv Movva, Pang Wei Koh, Emma Pierson

DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions (2024.emnlp-main)

Challenge:	We examine whether LLMs and humans agree when annotating the safety of user-chatbot conversations.
Approach:	They leverage a recent DICES dataset in which 350 conversations are each rated for safety by 112 annotators spanning 10 race-gender groups.
Outcome:	The LLMs annotators are compared to human annotator demographic groups and can predict when one group finds a conversation unsafe .

Copied to clipboard

Nigel Fernandez, Alexander Scarlatos, Wanyong Feng, Simon Woodhead, Andrew Lan

The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention (2024.emnlp-main)

Challenge:	a new variational approach to distractors in multiple-choice questions is needed . high-quality distractors are crucial to the assessment and pedagogical value of MCQs . a variational method that learns the error behind distractors is more effective .
Approach:	They propose a variational approach that learns an interpretable representation of errors behind distractors in math MCQs.
Outcome:	The proposed method outperforms state-of-the-art approaches on distractors in math MCQs.

Copied to clipboard

Yixin Wan, Di Wu, Haoran Wang, Kai-Wei Chang

CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models (2024.emnlp-main)

Challenge:	Prompt-based “diversity interventions” are commonly adopted to improve the diversity of Text-to-Image models depicting individuals with diverse racial or gender traits.
Approach:	They propose a benchmark to quantify the trade-off between using diversity interventions and preserving demographic factuality in T2I models.
Outcome:	The proposed model significantly improves the demographic factuality under diversity interventions while preserving diversity.

Copied to clipboard

Yuetai Li, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Dinuka Sahabandu, Bhaskar Ramasubramanian, Radha Poovendran

Enhancing Reinforcement Learning with Dense Rewards from Language Model Critic (2024.emnlp-main)

Challenge:	Generative large language models (LLMs) have remarkable performance in generation tasks, but datasets used to train or fine-tune these models are often not disclosed to users.
Approach:	They develop an inference time defense called CleanGen to mitigate backdoor attacks for generation tasks in large language models.
Outcome:	The proposed inference time defense achieves lower attack success rates (ASR) compared to baseline defenses for all five backdoor attacks.

Copied to clipboard

Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, Lei Meng

Words Matter: Reducing Stigma in Online Conversations about Substance Use with Large Language Models (2024.emnlp-main)

Challenge:	Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences, but the sparsity of these signals can lead to inefficient and unstable learning.
Approach:	They propose a framework that utilizes the critique capability of Large Language Models to produce intermediate-step rewards during RL training.
Outcome:	The proposed framework improves sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.

Copied to clipboard

Layla Bouzoubaa, Elham Aghakhani, Rezvaneh Rezapour

Efficient Sequential Decision Making with Large Language Models (2024.emnlp-main)

Challenge:	Only 7% of people living with an SUD receive any form of treatment, with stigma reported as a major barrier.
Approach:	They propose a computational framework for analyzing stigma and de-stigmatizing online content and delving into the linguistic features that propagate stigma towards PWUS.
Outcome:	The proposed model transforms stigmatizing language into more empathetic language and analyzes over 1.2 million posts on social media .

Copied to clipboard

Dingyang Chen, Qi Zhang, Yinglun Zhu

SignCLIP: Connecting Text and Sign Language by Contrastive Learning (2024.emnlp-main)

Challenge:	Existing approaches to retrain or finetune large language models (LLMs) for decision making suffer from computational burden of gradient updates.
Approach:	They propose a model selection algorithm that leverages online model selection algorithms to efficiently incorporate LLMs agents into sequential decision making.
Outcome:	The proposed approach outperforms both traditional decision making algorithms and vanilla LLM agents on a large-scale Amazon dataset.

Copied to clipboard

Zifan Jiang, Gerard Sant, Amit Moryossef, Mathias Müller, Rico Sennrich, Sarah Ebling

APPLS: Evaluating Evaluation Metrics for Plain Language Summarization (2024.emnlp-main)

Challenge:	SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs without optimizing for a specific task or sign language of limited size.
Approach:	They propose a method for learning visual representations for sign language processing from large-scale video-text pairs without directly optimizing for a specific task or sign language.
Outcome:	The proposed model can learn from multilingual video-text pairs without optimizing for a specific task or sign language of limited size.

Copied to clipboard

Yue Guo, Tal August, Gondy Leroy, Trevor Cohen, Lucy Wang

Ontologically Faithful Generation of Non-Player Character Dialogues (2024.emnlp-main)

Challenge:	Existing evaluation metrics for plain language summarization (PLS) lack a dedicated assessment metric and the suitability of text generation evaluation metrics is unclear due to unique transformations.
Approach:	They propose a granular meta-evaluation testbed to evaluate PLS metrics . they identify four PLS criteria and define perturbations that sensitive metrics should be able to detect .
Outcome:	The proposed testbed assesses performance of 14 existing metrics including scores, features, and prompt-based evaluations.

Copied to clipboard

Nathaniel Weir, Ryan Thomas, Randolph d’Amore, Kellie Hill, Benjamin Van Durme, Harsh Jhamtani

LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives (2024.emnlp-main)

Challenge:	a key challenge in creating NPC dialogues is that they should serve coherent narratives.
Approach:	They propose to use supervised and in-context learning techniques to generate trees of dialogue between video game characters that accurately reflect quest and entity specifications.
Outcome:	The proposed model performs well but room for improvement.

Copied to clipboard

Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, Sara Hooker

Challenge:	Historically, high-quality labeled data has been costly to curate due to scarcity of available data and financial cost.
Approach:	They characterize the impact of passive inheritance of model properties by studying how the source of synthetic data shapes models’ internal biases, calibration and preferences, and their generations’ textual attributes.
Outcome:	The proposed model inheritance can increase lexical diversity or reduce toxicity.

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs (2024.emnlp-main)

Copied to clipboard

Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, Ekaterina Artemova, Vladislav Mikhailov

Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction (2024.emnlp-main)

Challenge:	Existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena.
Approach:	They propose to use a Russian benchmark of linguistic minimal pairs to evaluate grammatical knowledge of language models.
Outcome:	The proposed benchmark includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon.

Copied to clipboard

Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, Yangqiu Song

Toward Compositional Behavior in Neural Models: A Survey of Current Views (2024.emnlp-main)

Challenge:	Existing approaches to condensing textual information into concise and structured tables are limited in their applicability in broader contexts.
Approach:	They propose a benchmark dataset for generating summary tables of competitions based on real-time commentary texts that incorporates large-scale textual information into concise and structured tables.
Outcome:	The proposed method exhibits strong generalization abilities, surpassing previous approaches on several other text-to-table datasets.

Copied to clipboard

Kate McCurdy, Paul Soulos, Paul Smolensky, Roland Fernandez, Jianfeng Gao

Challenge:	Compositionality is a core property of natural language, and it is regarded as a key goal for modern NLP systems.
Approach:	They propose a conceptual framework to address compositionality in NLP . they propose to use this framework to survey researchers active in this area .
Outcome:	The proposed framework finds consensus on key points and suggests that scale alone is unlikely to achieve the desired behavior.

Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (2024.emnlp-main)

Copied to clipboard

Krista Opsahl-Ong, Michael Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, Omar Khattab

Reverse-Engineering the Reader (2024.emnlp-main)

Challenge:	Language Model Programs (LMs) require crafting prompts that are jointly effective for all modules.
Approach:	They propose a novel algorithm for optimizing language model (LM) prompts for all modules by using program- and data-aware techniques and stochastic mini-batch evaluation functions.
Outcome:	The proposed algorithm outperforms baseline optimizers on five of seven diverse LM programs by as high as 13% accuracy.

Copied to clipboard

Samuel Kiegeland, Ethan Wilcox, Afra Amini, David Reich, Ryan Cotterell

Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation (2024.emnlp-main)

Challenge:	Existing studies have sought to determine to what extent language models can serve as useful models of human cognition by aligning them to human psychometric data.
Approach:	They propose a method to fine-tune a language model to implicitly optimize parameters of a linear regressor that directly predicts humans’ reading times of in-context linguistic units.
Outcome:	The proposed technique improves language models’ psychometric predictive power but also its perplexity on held-out test data.

Copied to clipboard

Di Wu, Jia-Chen Gu, Fan Yin, Nanyun Peng, Kai-Wei Chang

Structure Guided Prompt: Instructing Large Language Model in Multi-Step Reasoning by Exploring Graph Structure of the Text (2024.emnlp-main)

Challenge:	Existing studies show that RALMs generate baseless information or contradicts with the retrieved context.
Approach:	They propose a lightweight monitor that leverages fine-grained decoding dynamics to synchronously detect unfaithful sentences.
Outcome:	Empirical results show that SynCheck outperforms baseline faithfulness detection and FOD outperformed traditional strategies in terms of faithfulness.

Copied to clipboard

Kewei Cheng, Nesreen Ahmed, Theodore Willke, Yizhou Sun

Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) excel at straightforward reasoning tasks, but struggle when faced with complex multi-step reasoning.
Approach:	They propose a framework that converts unstructured text into a graph and instructs LLMs to navigate this graph using task-specific strategies.
Outcome:	The proposed framework improves the multi-step reasoning capabilities of Large Language Models in a zero-shot setting.

Copied to clipboard

David Schulte, Felix Hamborg, Alan Akbik

The effects of distance on NPI illusive effects in BERT (2024.emnlp-main)

Challenge:	Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage .
Approach:	They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model.
Outcome:	The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance.

Copied to clipboard

So Lee, Mai Vu

Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)

Challenge:	Recent studies have examined the syntactic capabilities of pre-trained language models, such as BERT.
Approach:	They examine the syntactic capabilities of pre-trained language models by using psycholinguistic stimuli.
Outcome:	The proposed model is highly sensitive to hierarchical or linear information compared to hierarical layers .

Copied to clipboard

Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme

Textual Inference

Challenge:	Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains.
Approach:	They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference.
Outcome:	The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy.

Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US (2024.emnlp-main)

Copied to clipboard

Christabel Acquaye, Haozhe An, Rachel Rudinger

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding (2024.emnlp-main)

Challenge:	Recent work has highlighted the culturally-contingent nature of commonsense knowledge . a multi-stage process is used to evaluate the commonsence of English LLMs .
Approach:	They propose a test set of 525 multiple-choice questions to evaluate commonsense knowledge of English LLMs in Ghana and the u.s. They use existing commonsensible datasets to rewrite them in a multi-stage process.
Outcome:	The proposed model improves on the culturally-contingent commonsense knowledge of English LLMs in Ghana and the United States.

Copied to clipboard

Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Wang

Challenge:	Existing models for GUI understanding ignore a key GUI-referring task: screen reading based on user-indicated points.
Approach:	They propose a Tree-of-Lens agent that constructs a Hierarchical Layout Tree based on user input points and a GUI screenshot.
Outcome:	The proposed agent can interpret the Screen Point-and-Read task on mobile, web, and operating systems.

Ranking Manipulation for Conversational Search Engines (2024.emnlp-main)

Copied to clipboard

Samuel Pfrommer, Yatong Bai, Tanmay Gautam, Somayeh Sojoudi

Fast Forwarding Low-Rank Training (2024.emnlp-main)

Challenge:	Recent research demonstrates that Large Language Models are highly vulnerable to jailbreaking and prompt injection attacks.
Approach:	They propose a tree-of-attacks-based jailbreaking technique which promotes low-ranked products . they propose enabling LLMs to be jailed and prompt injections to disrupt safety .
Outcome:	The proposed technique promotes low-ranked products in conversational search engines.

Copied to clipboard

Adir Rahamim, Naomi Saphra, Sara Kangaslahti, Yonatan Belinkov

Precise Model Benchmarking with Only a Few Observations (2024.emnlp-main)

Challenge:	Modern optimizers provide a spectacular array of tweaks to stabilize training trajectories and accelerate Stochastic Gradient Descent (SGD).
Approach:	They propose a fast-forward approach to accelerate large segments of SGD training . they alternate between Adam SGD for burn-in and accelerating by line search .
Outcome:	The proposed approach speeds up training without compromising model performance.

Copied to clipboard

Riccardo Fogliato, Pratik Patil, Nil-Jana Akpinar, Mathew Monfort

Challenge:	Accurate evaluation of large language models is crucial for identifying their strengths and weaknesses.
Approach:	They propose an empirical Bayes estimator that balances direct and regression estimates for each subgroup separately, improving the precision of subgroup-level estimates of model performance.
Outcome:	The proposed model reduces the mean squared error by up to 50% on multiple datasets.

Attribute Diversity Determines the Systematicity Gap in VQA (2024.emnlp-main)

Copied to clipboard

Ian Berlot-Attwell, Kumar Agrawal, Annabelle Carrell, Yash Sharma, Naomi Saphra

Challenge:	a systematicity gap exists between neural networks generalizing to new combinations of familiar concepts . conventionally trained neural networks struggle to generalize systematically .
Approach:	They propose to train a visual question answering model with CLEVR-HOPE as a diagnostic dataset to test this hypothesis.
Outcome:	The systematicity gap is reduced by increasing the diversity of training data, the authors show . the authors suggest that the more distinct attribute type combinations are seen during training, the more systematic the model will be.

ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models (2024.emnlp-main)

Copied to clipboard

Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel Weld, Joseph Chee Chang, Kyle Lo

Development of Cognitive Intelligence in Pre-trained Language Models (2024.emnlp-main)

Challenge:	Using language models (LMs) can generate literature review tables by decomposing it into separate schema and value generation steps.
Approach:	They propose a framework that leverages language models to perform literature review table generation by decomposing it into separate schema and value generation steps.
Outcome:	The proposed framework decomposes the task into two sub-tasks: schema generation and value generation.

Copied to clipboard

Raj Shah, Khushi Bhardwaj, Sashank Varma

Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding (2024.emnlp-main)

Challenge:	Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). Prior research into emergental cognitive abilities of PLMs has been path-independent to model training.
Approach:	They use four task categories to examine the alignment of ten popular families of PLMs and evaluate their performance to the developmental trajectories of children's thinking.
Outcome:	The results show that the models are more aligned to children's thinking than previous studies.

Copied to clipboard

Chong Zhang, Yi Tu, Yixi Zhao, Chenshu Yuan, Huan Chen, Yue Zhang, Mingxu Chai, Ya Guo, Huijia Zhu, Qi Zhang, Tao Gui

Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)

Challenge:	Existing models of layout reading order do not convey the complete reading order information in the layout.
Approach:	They propose to model layout reading order as ordering relations over layout elements . they propose a reading-order-relation-enhancing pipeline to improve model performance .
Outcome:	The proposed model outperforms existing models on a visual-rich document dataset and on eight cross-domain VrD-IE/QA tasks without targeted optimization.

Copied to clipboard

Sam Blouir, Jimmy Smith, Antonios Anastasopoulos, Amarda Shehu

Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models? (2024.emnlp-main)

Challenge:	Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings.
Approach:	They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks.
Outcome:	The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks.

Copied to clipboard

Pinzhen Chen, Simon Yu, Zhicheng Guo, Barry Haddow

Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (2024.emnlp-main)

Challenge:	Existing practices of fine-tuning and evaluating multilingual large language models may not align with this objective due to a heavy reliance on translation.
Approach:	They propose to use translated or native instruction data to fine-tune multilingual large language models.
Outcome:	The proposed model can be fine tuned and evaluated in multilingual large language models . the results show that native or translated data can be used to compare model performance .

Copied to clipboard

Sheridan Feucht, David Atkinson, Byron Wallace, David Bau

TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering (2024.emnlp-main)

Challenge:	Current language models process text as sequences of tokens that roughly correspond to words . individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise .
Approach:	They propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers.
Outcome:	The proposed method "reads out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers.

Copied to clipboard

Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Darrell, Roei Herzig

Evaluating the Effectiveness of Large Language Models in Establishing Conversational Grounding (2024.emnlp-main)

Challenge:	Existing methods that can find relevant information, extract it, and answer video questions in a single pass are not able to adapt if insufficient or incorrect information is collected.
Approach:	They propose a modular multi-LMM agent framework that can find relevant information, extract it, and answer the question simultaneously.
Outcome:	The proposed model improves performance on several VideoQA benchmarks without fine-tuning on specific datasets.

Copied to clipboard

Biswesh Mohapatra, Manav Kapadnis, Laurent Romary, Justine Cassell

Unlocking Memorization in Large Language Models with Dynamic Soft Prompting (2024.emnlp-main)

Challenge:	despite its importance, there has been limited research on conversational grounding in recent years . pre-trained language models have been costly and time-consuming to evaluate .
Approach:	They evaluate the performance of large language models in various aspects of conversational grounding . they propose ways to enhance the capabilities of the models that lag in this aspect .
Outcome:	The proposed model performance is based on pre-trained language models and a large pre-training dataset.

Copied to clipboard

Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, Yanfu Zhang

If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions (2024.emnlp-main)

Challenge:	Pretrained large language models excel in a variety of natural language processing tasks . however, they pose significant security risks due to their tendency to memorize training data .
Approach:	They propose a method to estimate LLM memorization using dynamic, prefix-dependent soft prompts.
Outcome:	The proposed method can achieve maximum relative improvement of 135.3% and 39.8% over baseline compared to state-of-the-art methods.

Copied to clipboard

Reza Esfandiarpoor, Cristina Menghini, Stephen Bach

Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction (2024.emnlp-main)

Challenge:	Recent studies assume that VLMs prioritize visual attributes to represent concepts.
Approach:	They propose a novel approach to characterize features important for VLMs using reinforcement learning.
Outcome:	The proposed approach characterizes features that are important for VLMs . it shows that spurious descriptions have a major role in VLM representations despite providing no helpful information.

Copied to clipboard

Bowen Zhang, Harold Soh

MQuinE: a Cure for “Z-paradox” in Knowledge Graph Embedding (2024.emnlp-main)

Challenge:	Existing methods for knowledge graph creation (KGC) are limited in their ability to scale up to text common in many real-world applications.
Approach:	They propose a framework for knowledge graph creation from input text using a pre-defined schema and a trained component that retrieves schema elements relevant to the input text.
Outcome:	The proposed framework extract-define-canonicalize extracts high-quality triplets with a succinct self-generated schema without any parameter tuning and with significantly larger schemas compared to prior works.

Copied to clipboard

Yang Liu, Huang Fang, Yunfeng Cai, Mingming Sun

Can Transformers Learn n-gram Language Models? (2024.emnlp-main)

Challenge:	Existing knowledge graph embedding models suffer from Z-paradox, a deficiency in expressiveness . Embedding-based models map each entity and relation into a vector or matrix .
Approach:	They propose a new knowledge graph embedding model that does not suffer from Z-paradox while preserves strong expressiveness to model various relation patterns with theoretical justification.
Outcome:	The proposed model outperforms existing models on link prediction tasks while maintaining strong expressiveness.

Copied to clipboard

Anej Svete, Nadav Borenstein, Mike Zhou, Isabelle Augenstein, Ryan Cotterell

StablePrompt : Automatic Prompt Tuning using Reinforcement Learning for Large Language Model (2024.emnlp-main)

Challenge:	Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them.
Approach:	They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter .
Outcome:	The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter.

Copied to clipboard

Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, Junmo Kim

Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems (2024.emnlp-main)

Challenge:	Recent advances in large language models have made it difficult to find appropriate prompts for tasks with multiple input-output formats.
Approach:	They propose a prompt tuning method based on reinforcement learning (RL) they propose an anchor model and an extension for generating input-dependent prompts.
Outcome:	The proposed method outperforms existing methods on a variety of tasks and achieves State-of-the-art performance across diverse types and sizes of LLMs.

Copied to clipboard

Philippe Laban, Alexander Fabbri, Caiming Xiong, Chien-Sheng Wu

Summarization

Challenge:	Recent advances in efficient attention mechanisms have led to the expansion of the context length of large language models.
Approach:	They propose a procedure to synthesize Haystacks of documents and generate a summary that identifies relevant insights and precisely cites the source documents.
Outcome:	The proposed evaluation can score summaries on Coverage and Citation . the proposed evaluation lags human performance estimates by 10+ points on SummHay .

Multi-pass Decoding for Grammatical Error Correction (2024.emnlp-main)

Copied to clipboard

Xiaoying Wang, Lingling Mu, Jingyi Zhang, Hongfei Xu

Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations (2024.emnlp-main)

Challenge:	Seq2edit models decode only once without aware of subsequent tokens.
Approach:	They propose to iteratively refine the correction results of seq2seq models via Multi-Pass Decoding (MPD) to improve performance, but MPD increases inference costs . they propose to merge the source input and previous round correction result into one sequence.
Outcome:	Experiments on the CoNLL-14 and BEA-19 test set show that the proposed approach improves over baselines.

Copied to clipboard

Yucheng Jiang, Yijia Shao, Dekun Ma, Sina Semnani, Monica Lam

SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine Translation (2024.emnlp-main)

Challenge:	Recent advances in language models (LMs) and retrieval-augmented generation (RAG) have led to more capable chatbots and generative search engines.
Approach:	They propose to emulate the educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers by watching and steering the discourse among several LM agents.
Outcome:	The proposed system outperforms baseline methods on discourse trace and report quality and is preferred by 70% of participants over a search engine and 78% over sabota.

Copied to clipboard

Chenming Tang, Zhixiang Wang, Yunfang Wu

Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge (2024.emnlp-main)

Challenge:	In-context learning improves performance of large language models (LLMs) performance of ICL highly depends on quality of demonstrations .
Approach:	They propose a syntactic-augmented COverage-based In-context example selection strategy that leverages syntastic knowledge beyond word matching to select better examples for machine translation.
Outcome:	The proposed strategy obtains the highest average COMET score among learning-free methods.

Copied to clipboard

Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, Yang Liu, Zilong Zheng

STORYSUMM: Evaluating Faithfulness in Story Summarization (2024.emnlp-main)

Challenge:	despite advances in multimodal large language models, the challenge of interpreting long-form videos remains a challenge . despite advancements in video-language benchmarks, the inefficiency in temporal grounding and limited pre-trained context window size remains .
Approach:	They propose a framework that bootstraps MLLMs with advanced temporal grounding capabilities and broadens their contextual scope.
Outcome:	The proposed framework significantly enhances the temporal capabilities of existing MLLMs.

Copied to clipboard

Melanie Subbiah, Faisal Ladhak, Akankshya Mishra, Griffin Adams, Lydia Chilton, Kathleen McKeown

Summarization

Challenge:	Existing methods for evaluating abstractive summarization are lacking in faithfulness evaluation.
Approach:	They propose a dataset that measures faithfulness of LLM summaries with localized errors and faithfulness labels for evaluation methods.
Outcome:	The proposed method does not achieve more than 70% accuracy on this task.

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts (2024.emnlp-main)

Copied to clipboard

Haofei Yu, Zhengyang Qi, Lawrence Jang, Russ Salakhutdinov, Louis-Philippe Morency, Paul Pu Liang

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer (2024.emnlp-main)

Challenge:	Multimodal models focus on the correspondence between images and text, but this only covers a subset of real-world interactions.
Approach:	They propose an approach to enhance multimodal models by training separate expert models for each type of interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both . modality is used to capture overlaps in semantic content between images and text, making a strong multi-view redundancies assumption.
Outcome:	The proposed approach improves on a sarcasm detection and humor detection task.

Copied to clipboard

Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, Kyusong Lee

Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have expanded their capabilities to multimodal contexts, including comprehensive video understanding.
Approach:	They propose to store and retrieve relevant video frames for specific queries and a Divide-and-Conquer loop capable of autonomous reasoning.
Outcome:	The proposed model efficiently stores and retrieves relevant video frames for specific queries, preserving the detailed content of videos.

Copied to clipboard

Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg

CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions (2024.emnlp-main)

Challenge:	Extractive Machine Reading Comprehension (MRC) is a challenging field in the field of Natural Language Processing.
Approach:	They propose a Question-Attended Span Extraction module to address the limitations of generative approaches for extractive machine reading comprehension (MRC) . module significantly enhances performance of pre-trained generative language models, enabling them to surpass the extractive capabilities of advanced Large Language Models (LLMs)
Outcome:	The QASE module surpasses state-of-the-art models in few-shot settings.

Copied to clipboard

Jun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng, Yunjie Liao, Min Zhang

ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers (2024.emnlp-main)

Challenge:	Current studies have focused on fine-tuning, but the use of instruction tuning is not as effective as fine-cuning.
Approach:	They propose a commonality-aware instruction tuning strategy to cluster instruction datasets into distinct groups with three proposed metrics Task, Embedding and Length.
Outcome:	The proposed strategy boosts an average improvement of 2.1% on the general domain and 5.2% on the special domain.

Copied to clipboard

Yuzhe Gu, Enmao Diao

Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models (2024.emnlp-main)

Challenge:	Existing neural speech codecs trade model complexity for reconstruction performance . ESC is a lightweight, parameter-efficient speech coder .
Approach:	They propose an efficient speech codec based on a cross-scale residual vector quantization scheme and transformers that can achieve high-fidelity speech reconstruction with significantly lower model complexity.
Outcome:	The proposed codec achieves high-fidelity speech reconstruction with significantly lower model complexity.

Copied to clipboard

Jaeseong Lee, Seung-won Hwang, Wonpyo Park, Mingi Ji

Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood (2024.emnlp-main)

Challenge:	Existing methods to convert pretrained dense models to MoEs are limited to ReLU-based models with natural sparsity.
Approach:	They propose a G-MoEfication approach for arbitrary dense models where activation sparsity assumptions no longer hold.
Outcome:	The proposed method reduces the inference cost associated with dense models by sparsely activating experts.

Copied to clipboard

Yang Xu, Yu Wang, Hao An, Zhichen Liu, Yongyuan Li

Psycholinguistics

Challenge:	Existing methods for detecting modelgenerated texts from human texts are limited by the fact that absolute likelihood values of texts are bound to certain linguistic and cognitive constraints.
Approach:	They propose to use relative likelihood values instead of absolute ones to extract useful features from the spectrum-view of likelihood for the human-model text detection task.
Outcome:	The proposed method can reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies.

Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning (2024.emnlp-main)

Copied to clipboard

Jiahui Li, Hanlin Zhang, Fengda Zhang, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou

Fine-grained Pluggable Gradient Ascent for Knowledge Unlearning in Language Models (2024.emnlp-main)

Challenge:	Recent research has developed algorithms for reinforcement learning from human feedback and AI-generated feedback.
Approach:	They propose a method for reinforcement learning from human feedback and AI-generated feedback that incorporates weighting, ranking, and constraining to handle disparate rewards.
Outcome:	The proposed method reduces disparity and enhances stability among rewards . empirical results show that the proposed method is efficient and straightforward .

Copied to clipboard

XiaoHua Feng, Chaochao Chen, Yuyuan Li, Zibin Lin

ARM: An Alignment-and-Replacement Module for Chinese Spelling Check Based on LLMs (2024.emnlp-main)

Challenge:	Existing studies on knowledge unlearning focus on computer vision but extend their exploration to other fields.
Approach:	They propose an adaptive objective that calculates gradients with fine-grained control specifically targeting sensitive tokens.
Outcome:	The proposed method improves the general ability of language models while achieving knowledge unlearning.

Copied to clipboard

Changchun Liu, Kai Zhang, Junzhe Jiang, Zirui Liu, Hanqing Tao, Min Gao, Enhong Chen

On the In-context Generation of Language Models (2024.emnlp-main)

Challenge:	Chinese Spelling Check (CSC) aims to identify and correct spelling errors in Chinese texts, where enhanced semantic understanding of a sentence can significantly improve correction accuracy.
Approach:	They propose a plug-and-play Alignment-and -Replacement module that enhances existing Chinese CSC models without retraining or fine-tuning.
Outcome:	The proposed module improves existing models while reducing retraining and fine-tuning.

Copied to clipboard

Zhongtao Jiang, Yuanzhe Zhang, Kun Luo, Xiaowei Yuan, Jun Zhao, Kang Liu

Atomic Inference for NLI with Generated Facts as Atoms (2024.emnlp-main)

Challenge:	Large language models (LLMs) have the ability of in-context generation (ICG) when given an in-text prompt, they can implicitly recognize the pattern of the examples and complete the prompt in the desired way.
Approach:	They propose a plausible latent variable model to model the distribution of pretrained corpora and formalize ICG as a problem of next topic prediction.
Outcome:	The proposed model can model the distribution of pretrained corpora and then formalize ICG as a problem of next topic prediction.

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, Marek Rei

Towards Robust Speech Representation Learning for Thousands of Languages (2024.emnlp-main)

Challenge:	Existing models that can provide accurate explanations are not interpretable, i.e. they do not reflect the inner workings of the model.
Approach:	They propose to use LLM-generated facts as atoms to make interpretable models that can be used to make accurate predictions for each component part of an input.
Outcome:	The proposed method outperforms existing methods on natural language understanding tasks with a multi-stage fact generation process and a training regime that incorporates the facts.

Copied to clipboard

William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, Shinji Watanabe

I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses (2024.emnlp-main)

Challenge:	XEUS is a cross-lingual encoder for universal speech that can be trained on 1 million hours of data across 4057 languages.
Approach:	They propose a Cross-lingual Encoder for Universal Speech that can be trained on 1 million hours of data across 4057 languages and a newly created corpus of 7400+ hours from 4057 .
Outcome:	The proposed model outperforms state-of-the-art models on several benchmarks and outperfies MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively.

Copied to clipboard

Xuan Ren, Biao Wu, Lingqiao Liu

PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment (2024.emnlp-main)

Challenge:	Recent research has demonstrated that a large language model (LLM) can generate training data for another LLM, or for creating supplementary training materials, such as rationales.
Approach:	They conduct an in-depth investigation to understand why fine-tuning an LLM with responses generated by a LLM often yields better results than using responses generated from humans.
Outcome:	The proposed approach can be used to transfer knowledge from a larger model to a smaller one, or for creating supplementary training materials, such as rationales.

Copied to clipboard

Jiahuan Li, Shujian Huang, Aarron Ching, Xinyu Dai, Jiajun Chen

An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance (2024.emnlp-main)

Challenge:	Large language models exhibit reasonable multilingual abilities, despite predominantly English-centric pretraining.
Approach:	They propose a framework that establishes multilingual alignment prior to language model pretraining and preserves this alignment using a code-switching strategy during pretraining.
Outcome:	Experiments in a synthetic English to English-Clone setting show that PreAlign outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-linguistic knowledge application.

Copied to clipboard

Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, Graham Neubig

When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models (2024.emnlp-main)

Challenge:	a new task is to translate images to make them culturally relevant . currently, translation systems focus on translating words and images .
Approach:	They propose a task of translating images to make them culturally relevant . they build pipelines comprising state-of-the-art generative models to do the task .
Outcome:	The proposed pipelines can translate only 5% of translated images for some countries and no translation is successful for others.

Copied to clipboard

Ting-Yun Chang, Jesse Thomason, Robin Jia

Challenge:	In-context learning is a method for adapting large language models to tasks with instructions or examples.
Approach:	They propose a method to decompose the output of large language models into components . they propose component reweighting, which learns to linearly re-scale component activations from a few labeled examples.
Outcome:	The proposed method improves by 6.0% accuracy points over 24 examples given 24 examples on Llama-2-7B.

Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference (2024.emnlp-main)

Copied to clipboard

Jianxing Yu, Shiqi Wang, Han Yin, Zhenlong Sun, Ruobing Xie, Bo Zhang, Yanghui Rao

Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions (2024.emnlp-main)

Challenge:	a new method to detect clickbait posts on the Web is needed to detect such posts.
Approach:	They propose a method to detect clickbait posts on the Web using latent factors . they use features in multiple modalities to characterize the posts and causal inference to eliminate noise .
Outcome:	The proposed method can detect clickbait posts on popular social media platforms with good generalization ability.

Copied to clipboard

Jinsung Yoon, Rajarishi Sinha, Sercan Arik, Tomas Pfister

KNN-Instruct: Automatic Instruction Construction with K Nearest Neighbor Deduction (2024.emnlp-main)

Challenge:	Embeddings from Large Language Models (LLMs) have emerged as critical components in information retrieval applications.
Approach:	They propose a tuning framework for the customization of LLM embeddings.
Outcome:	The proposed framework reduces embedding dimensions while maintaining comparable performance levels.

Copied to clipboard

Jianshang Kou, Benfeng Xu, Chiwei Zhu, Zhendong Mao

Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation (2024.emnlp-main)

Challenge:	Existing methods for generating synthetic instructions for large language models suffer from stale distribution and scalability.
Approach:	They propose a method which incorporates KNN deduction to produce meaningful new instructions by summarizing and learning from existing ones.
Outcome:	The proposed method outperforms all 7B models on the LMSYS leaderboard.

Copied to clipboard

Zhen Lin, Shubhendu Trivedi, Jimeng Sun

MixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularity (2024.emnlp-main)

Challenge:	Currently, the most commonly used confidence score is the likelihood of the generated sequence . different tokens should be weighted differently depending on the context.
Approach:	They propose to assign different weights to various tokens using attention values elicited from the base LLM.
Outcome:	The proposed model improves the confidence of the predicted sequence probability by assigning weights to tokens based on attention values elicited from the base model.

Copied to clipboard

Fengyu Cai, Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, Iryna Gurevych, Heinz Koeppl

CARER - ClinicAl Reasoning-Enhanced Representation for Temporal Health Risk Prediction (2024.emnlp-main)

Challenge:	Recent studies show the importance of document retrieval in the scientific domain.
Approach:	They propose a zero-shot approach to measure query-document similarity using atomic components in queries and documents to combine them into a united score.
Outcome:	The proposed approach outperforms previous document retrieval methods by 24.7%, 9.8%, and 6.9% on nDCG@5 with unsupervised, supervised, and LLM-based retrievers.

Copied to clipboard

Tuan Nguyen, Thanh Huynh, Minh Hieu Phan, Quoc Viet Hung Nguyen, Phi Le Nguyen

“In-Dialogues We Learn”: Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning (2024.emnlp-main)

Challenge:	Existing deep learning methods require large datasets to achieve high generalizability.
Approach:	They propose a framework that enhances deep learning models with clinical rationales derived from medically proficient Large Language Models.
Outcome:	The proposed framework outperforms state-of-the-art models on two tasks using two popular EHR datasets by up to 11.2%.

Copied to clipboard

Chuanqi Cheng, Quan Tu, Wei Wu, Shuo Shang, Cunli Mao, Zhengtao Yu, Rui Yan

Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective (2024.emnlp-main)

Challenge:	Existing approaches to personalized dialogue generate pre-defined profiles that are time-consuming and labor-intensive to create.
Approach:	They propose a framework that leverages dialogue history to characterize personas without pre-defined profiles.
Outcome:	The proposed framework improves BLEU and ROUGE scores on three datasets and human evaluations further validate the proposed method.

Copied to clipboard

Hanqi Yan, Yanzheng Xiang, Guangyi Chen, Yifei Wang, Lin Gui, Yulan He

Challenge:	Recent studies focus on monosemanticity on its basic units.
Approach:	They propose to revisit monosemanticity from the feature decorrelation perspective and advocate for its encouragement.
Outcome:	The proposed method improves representation diversity and activation sparsity and improves preference alignment performance.

Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding (2024.emnlp-main)

Copied to clipboard

Xin Liu, Farima Fatahi Bayat, Lu Wang

Reasoning Robustness of LLMs to Adversarial Typographical Errors (2024.emnlp-main)

Challenge:	Existing methods to calibrate language models are limited in inference-time efficiency or fail to provide informative signals.
Approach:	They propose an activation-based calibration method, ActCab, which trains a linear layer on top of the LM’s last-layer activations.
Outcome:	The proposed method improves on five popular QA benchmarks and reduces the average expected calibration error (ECE) score by up to 39%.

Copied to clipboard

Esther Gan, Yiran Zhao, Liying Cheng, Mao Yancan, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, Michael Shieh

InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting.
Approach:	They develop an algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking.
Outcome:	The proposed algorithm detects typographical errors in large and closed-source LLMs and shows that they are robust to them.

Copied to clipboard

Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, Xipeng Qiu

Belief Revision: The Adaptability of Large Language Models Reasoning (2024.emnlp-main)

Challenge:	Existing methods for enhancing harmlessness and helpfulness of large language models (LLMs) involve complex and resource-intensive training processes.
Approach:	They propose a method that decouples harmlessness from helpfulness during inference phase.
Outcome:	The proposed method significantly reduces the attack success rate (ASR) of harmful instructions and jailbreak instructions while maintaining almost unchanged performance in downstream tasks.

Copied to clipboard

Bryan Wilie, Samuel Cahyawijaya, Etsuko Ishii, Junxian He, Pascale Fung

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models (2024.emnlp-main)

Challenge:	Existing evaluations assume language models operate with consistent information.
Approach:	They propose a dataset to test LMs' belief revision ability when presented with new evidence.
Outcome:	The proposed framework improves language models’ adaptiveness to changing information, highlighting a critical trade-off.

Copied to clipboard

Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, Dejing Dou

Bio-RFX: Refining Biomedical Extraction via Advanced Relation Classification and Structural Constraints (2024.emnlp-main)

Challenge:	Existing frameworks for learning Large Language Models (LLMs) require adaptive data processing and low-rank adjustment to improve accuracy and fine-tuning speed.
Approach:	They propose a fisher information-based adaptive federated curriculum learning framework with two novel methods to improve FL fine-tuning process.
Outcome:	The proposed framework improves performance and fine-tuning speed compared with baseline approaches.

Copied to clipboard

Minjia Wang, Fangzhou Liu, Xiuxing Li, Bowen Dong, Zhenyu Li, Tengyu Pan, Jianyong Wang

Decoding Matters: Addressing Amplification Bias and Homogeneity Issue in Recommendations for Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for extracting structured data from unstructured texts neglect unique features of the biomedical literature, such as ambiguous entities and nested proper nouns.
Approach:	They propose a model that leverages sentence-level relation classification before entity extraction to tackle entity ambiguity.
Outcome:	The proposed model outperforms baselines in both NER and RE tasks and has competitive performance compared to the state-of-the-art fine-tuned baselines for RE.

Copied to clipboard

Keqin Bao, Jizhi Zhang, Yang Zhang, Xinyue Huo, Chong Chen, Fuli Feng

LLMs Are Prone to Fallacies in Causal Inference (2024.emnlp-main)

Challenge:	Existing approaches to adapt Large Language Models (LLMs) for recommendation encounter significant challenges such as amplification bias and homogeneity.
Approach:	They propose a new decoding approach called Debiasing-Diversifying Decoding (D3) that disables length normalization for ghost tokens to alleviate amplification bias and incorporates a text-free assistant model to encourage tokens less frequently generated by LLMs for counteracting recommendation homogeneity.
Outcome:	Extensive experiments on real-world datasets demonstrate the proposed approach’s effectiveness in enhancing accuracy and diversity.

Copied to clipboard

Nitish Joshi, Abulhair Saparov, Yixin Wang, He He

Challenge:	Recent work shows that causal facts can be extracted from LLMs through prompting . but it is unclear if this success is limited to explicitly-mentioned causal facts in pretraining data .
Approach:	They fine tune LLMs on synthetic data and test whether they can infer causal relations . they find that LLM can correctly deduce absence of causal relations from temporal and spatial relations if order is randomized .
Outcome:	The proposed model outperforms existing methods on causal inference tasks.

Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (2024.emnlp-main)

Copied to clipboard

Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang

The Lou Dataset - Exploring the Impact of Gender-Fair Language in German Text Classification (2024.emnlp-main)

Challenge:	Existing methods for improving LLMs in simulations are limited due to privacy concerns and limited domain knowledge.
Approach:	They propose a pipeline that elicits qualitative feedback from a domain-expert and transforms it into a set of principles that govern an LLM-prompted roleplay.
Outcome:	The proposed pipeline shows a 30% improvement in response quality and principle following for the downstream task.

Copied to clipboard

Andreas Waldis, Joel Birrer, Anne Lauscher, Iryna Gurevych

When Generative Adversarial Networks Meet Sequence Labeling Challenges (2024.emnlp-main)

Challenge:	Gender-fair language fosters inclusion by addressing all genders or using neutral forms.
Approach:	They present a dataset that provides high-quality reformulations for German text classification . they find substantial label flips, reduced prediction certainty, and altered attention patterns .
Outcome:	The proposed dataset provides high-quality reformulations for German text classification . it finds label flips, reduced prediction certainty, and significantly altered attention patterns .

Copied to clipboard

Yu Tong, Ge Chen, Guokai Zheng, Rui Li, Jiang Dazhi

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering (2024.emnlp-main)

Challenge:	Existing approaches for sequence labeling use a feature extractor and sequence tagger . a recent study shows that SLGAN is versatile and highly effective .
Approach:	They propose a framework that harnesses the capabilities of Generative Adversarial Networks to address sequence labeling challenges.
Outcome:	The proposed framework exhibits strong adaptability to various sequence labeling tasks.

Copied to clipboard

Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee

Speechworthy Instruction-tuned Language Models (2024.emnlp-main)

Challenge:	Existing methods for enhancing QA performance of Large Language Models (LLMs) have limitations, including duplicated entities or relations, reduced evidence density, and failure to highlight crucial evidence.
Approach:	They propose an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented Large Language Models (LLMs) that incorporates external knowledge into LLMs to improve QA performance.
Outcome:	The proposed framework improves LLM’s zero-shot QA performance especially when noisy facts are retrieved.

Copied to clipboard

Hyundong Cho, Nicolaas Jedema, Leonardo Ribeiro, Karishma Sharma, Pedro Szekely, Alessandro Moschitti, Ruben Janssen, Jonathan May

Data, Data Everywhere: A Guide for Pretraining Dataset Construction (2024.emnlp-main)

Challenge:	Current instruction tuned language models are trained on textual preference data and therefore not aligned to speech domain.
Approach:	They propose to use radio-industry best practices to prompt and learn speech-based preference data to improve speech-suitability of popular instruction tuned language models.
Outcome:	The proposed methods achieve the best win rates in head-to-head comparisons, resulting in preferred or tied to the base model in 76.2% of comparisons on average.

Copied to clipboard

Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together (2024.emnlp-main)

Challenge:	Recent language models have impressive capabilities on a number of evaluation areas.
Approach:	They conduct systematic analysis of pretraining set construction to identify which methods yield the greatest gains in model accuracy.
Outcome:	The proposed method can be used to refine and improve a pretraining set.

Copied to clipboard

Dilara Soylu, Christopher Potts, Omar Khattab

Demystifying Verbatim Memorization in Large Language Models (2024.emnlp-main)

Challenge:	Recent work shows the potential of building more powerful Natural Language Processing systems by composing multiple skills of LMs into pipelines.
Approach:	They propose to combine weight and prompt optimization strategies to optimize a modular LM pipeline.
Outcome:	The proposed strategies outperform optimizing weights and prompts alone by 60% and 6% on average across LMs and tasks.

Copied to clipboard

Jing Huang, Diyi Yang, Christopher Potts

AmbigNLG: Addressing Task Ambiguity in Instruction for NLG (2024.emnlp-main)

Challenge:	Existing studies have shown that Large Language Models (LLMs) memorize long sequences verbatim, with serious copyright and privacy implications.
Approach:	They develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences.
Outcome:	The proposed framework creates a control model M () and a treatment model M with injected sequences.

Copied to clipboard

Ayana Niwa, Hayate Iso

Distributional Properties of Subword Regularization (2024.emnlp-main)

Challenge:	AmbigNLG is a novel task designed to tackle task ambiguity in instructions for NLG . ambiguous instructions often impede the performance of Large Language Models (LLMs) .
Approach:	They propose an ambiguity taxonomy that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications.
Outcome:	The proposed task improves alignment of generated text with user expectations, achieving 15.02-point increase in ROUGE scores.

Copied to clipboard

Marco Cognetta, Vilém Zouhar, Naoaki Okazaki

DataTales: A Benchmark for Real-World Intelligent Data Narration (2024.emnlp-main)

Challenge:	Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training.
Approach:	They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word.
Outcome:	The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus.

Copied to clipboard

Yajing Yang, Qian Liu, Min-Yen Kan

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)

Challenge:	Existing benchmarks fail to capture the requisite analytical complexity for practical applications.
Approach:	They propose a benchmark to assess the proficiency of language models in data narration.
Outcome:	The proposed model combines financial reports with market data to demonstrate proficiency in data narration.

Copied to clipboard

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization (2024.emnlp-main)

Challenge:	Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings.
Approach:	They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM.
Outcome:	The proposed model significantly speeds up inference time and out-of-domain speedup across various languages.

Copied to clipboard

Yangfan Ye, Xiachong Feng, Xiaocheng Feng, Weitao Ma, Libo Qin, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024.emnlp-main)

Challenge:	Current studies focus on single-language or single-document tasks for news summarization . lack of a benchmark inhibits researchers from adequately studying this invaluable problem.
Approach:	They propose a novel task that unifies Multi-lingual, Cross-lingual and Multi-document Summarization into one task.
Outcome:	The proposed task encapsulates the real-world requirements all-in-one and is validated by extensive analysis.

Copied to clipboard

Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah Smith, Luke Zettlemoyer

More Insightful Feedback for Tutoring: Enhancing Generation Mechanisms and Automatic Evaluation (2024.emnlp-main)

Challenge:	Multilingual language models often underperform monolingual ones due to inter-language competition for model parameters.
Approach:	They propose Cross-lingual Expert Language Models (X-ELM) which mitigates inter-language competition by independently training language models on subsets of the multilingual corpus.
Outcome:	The proposed model outperforms jointly trained multilingual models across all 16 considered languages and transfer the gains to downstream tasks.

Copied to clipboard

Wencke Liermann, Jin-Xia Huang, Yohan Lee, Kong Joo Lee

Challenge:	Incorrect student answers can be valuable learning opportunities provided that the student understands where they went wrong and why.
Approach:	They propose to use a KL regularization term to achieve more targeted input representations and a preference optimization step to encourage student answer-adaptive feedback generation.
Outcome:	The proposed model outperforms existing models in 3.3 METEOR points.

Stable Language Model Pre-training by Reducing Embedding Variability (2024.emnlp-main)

Copied to clipboard

Woojin Chung, Jiwoo Hong, Na Min An, James Thorne, Se-Young Yun

What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (2024.emnlp-main)

Challenge:	Stable pre-training is essential for achieving better-performing language models, but tracking pre-train stability is impractical due to high computational costs.
Approach:	They propose to use Token Embedding Variability as a proxy to estimate pre-training stability.
Outcome:	The proposed method improves stability and lowers perplexities even at deeper layer counts.

Copied to clipboard

Kavya Manohar, Leena Pillai

Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets (2024.emnlp-main)

Challenge:	Existing text normalization routines that target Indic scripts are flawed when applied to multilingual automatic speech recognition models.
Approach:	They propose to develop text normalization routines that leverage native linguistic expertise to ensure more robust and accurate evaluations of multilingual automatic speech recognition models.
Outcome:	The proposed normalization routines can be leveraged to improve performance metrics for Indic languages.

Copied to clipboard

Benjamin Schiller, Johannes Daxenberger, Andreas Waldis, Iryna Gurevych

Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas (2024.emnlp-main)

Challenge:	Topic-dependent argument mining is a task that requires expert knowledge to recognize retrieved arguments.
Approach:	They investigate the effect of TDAM dataset composition on model performance by using carefully composed training samples and reducing the training sample size by almost 90%.
Outcome:	The proposed model can achieve 95% of the maximum performance on three different datasets.

Copied to clipboard

Seungjong Sun, Eungu Lee, Seo Baek, Seunghyun Hwang, Wonbyung Lee, Dongyan Nan, Bernard Jansen, Jang Kim

ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator (2024.emnlp-main)

Challenge:	Large language models (LLMs) exhibit a high degree of alignment with human behavior based on their robust capabilities for natural language understanding and generation.
Approach:	They developed a dataset of 5K fictional avatar images for assignment as visual personas to large language models (LLMs) and analyzed their negotiation behaviors based on the visual traits depicted in these images.
Outcome:	The proposed model exhibited aggressive negotiation behaviors when the opponent’s image appeared less aggressive than their own, and less aggressive negotiation behavior when the opposing image appeared more aggressive.

Copied to clipboard

Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, Lei Sha

Dynamic Multi-granularity Attribution Network for Aspect-based Sentiment Analysis (2024.emnlp-main)

Challenge:	Large language models (LLMs) are proven to benefit a lot from retrieval-augmented generation (RAG) due to noisy and fabricating content, it is inevitable that RAG systems are vulnerable to these noises and prone to respond incorrectly.
Approach:	They propose to optimize retrieval-augmented generation (RGG) with an Adversarial Tuning Multi-agent system (ATM) ATM steers the Generator to have a robust perspective of useful documents for question answering with the help of an auxiliary Attacker agent.
Outcome:	The proposed system improves the retrieval-augmented generator with an auxiliary Attacker agent and can discriminate useful documents amongst fabrications.

Copied to clipboard

Yanjiang Chen, Kai Zhang, Feng Hu, Xianquan Wang, Ruikang Li, Qi Liu

Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization (2024.emnlp-main)

Challenge:	Existing methods for predicting sentiment polarity of aspects are susceptible to interference caused by irrelevant contexts and lack sentiment knowledge at a data-specific level.
Approach:	They propose a novel Aspect-based sentiment analysis method that leverages attention scores to model the relationships between aspects and contexts.
Outcome:	The proposed method is able to predict sentiments from a set of five benchmark datasets.

Copied to clipboard

Shahed Masoudian, Markus Frohmann, Navid Rekabsaz, Markus Schedl

Large Language Models Know What is Key Visual Entity: An LLM-assisted Multimodal Retrieval for VQA (2024.emnlp-main)

Challenge:	Existing methods for debiasing depend on attribute labels and target attributes.
Approach:	They propose a method that uses class-wise variance of embeddings to reduce the effects of debiasing on a downstream task.
Outcome:	The proposed method outperforms baselines that rely on attribute labels while maintaining performance on the target task.

Copied to clipboard

Pu Jian, Donglei Yu, Jiajun Zhang

Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights (2024.emnlp-main)

Challenge:	Existing visual language models struggle to capture longtail knowledge in the real world due to redundant visual information.
Approach:	They propose a method leveraging the reasoning capability of a large language model to identify key visual entities.
Outcome:	The proposed method outperforms other strong visual language model-based systems in two knowledge-intensive VQA benchmarks and performs comparably to models with 1-2 orders larger parameters.

Copied to clipboard

Hao Yang, Lizhen Qu, Ehsan Shareghi, Reza Haf

Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations (2024.emnlp-main)

Challenge:	Large Multimodal Models have demonstrated a strong capability to understand multimodal information and interact with human users.
Approach:	They propose a speech-specific risk taxonomy to assess LMMs' ability to detect high-risk interactions in multimodal settings.
Outcome:	The proposed model is based on a speech-specific risk taxonomy covering 8 risk categories . it shows that the models are ineffective in detecting paralinguistic-specific risks in speech .

Copied to clipboard

Milan Bhan, Jean-Noël Vittaut, Nicolas Chesneau, Marie-Jeanne Lesot

What are the Generator Preferences for End-to-end Task-Oriented Dialog System? (2024.emnlp-main)

Challenge:	Autoregressive Large Language Models (LLMs) have demonstrated "emergent abilities" such as in-context learning, instruction following and reasoning.
Approach:	They propose a method that generates rationales from post hoc explanation methods applied to small language models to improve their own performance.
Outcome:	The proposed method improves on four SLMs and five datasets with strong reasoning abilities.

Copied to clipboard

Wanshi Xu, Xianwei Zhuang, Zhanpeng Chen, Zhihong Zhu, Xuxin Cheng, Yuexian Zou

Paraphrase Types Elicit Prompt Engineering Capabilities (2024.emnlp-main)

Challenge:	Existing methods to improve the accuracy of entity retrieval are not effective.
Approach:	They propose a framework that improves the performance of task-oriented dialogue systems by obtaining fine-grained matching information between contexts and entities and extracting the entity attribute shift matrix as preference signals.
Outcome:	The proposed framework outperforms existing methods and improves the quality of the dialogue.

Copied to clipboard

Jan Philip Wahle, Terry Ruas, Yang Xu, Bela Gipp

VLEU: a Method for Automatic Evaluation for Generalizability of Text-to-Image Models (2024.emnlp-main)

Challenge:	Until now, it has been unknown how variations in the linguistic expression of prompts affect language models.
Approach:	They evaluate which linguistic features influence models through paraphrase types . they found that changes in morphology and lexicon showed promise in improving prompts .
Outcome:	The results show that paraphrases can improve language models' performance . the authors show that changes in morphology and lexicon can improve prompts .

Copied to clipboard

Jingtao Cao, Zhang Zheng, Hongru Wang, Kam-Fai Wong

Towards Online Continuous Sign Language Recognition and Translation (2024.emnlp-main)

Challenge:	Existing metrics, such as CLIP, measure the semantic alignment between single prompts and their corresponding images, but they fail to evaluate a model’s generalizability across a broad spectrum of textual inputs.
Approach:	They propose a metric that leverages the power of Large Language Models to sample from the visual text domain and assess its generalizability.
Outcome:	The proposed metric evaluates the generalizability of T2I models and provides valuable insights during the finetuning process.

Copied to clipboard

Ronglai Zuo, Fangyun Wei, Brian Mak

Mitigate Extrinsic Social Bias in Pre-trained Language Models via Continuous Prompts Adjustment (2024.emnlp-main)

Challenge:	Existing studies on continuous sign language recognition (CSLR) use offline models with high latency and memory usage.
Approach:	They develop a sign dictionary and train an isolated sign language recognition model on the dictionary.
Outcome:	The proposed model achieves state-of-the-art on three popular benchmarks across task settings.

Copied to clipboard

Yiwei Dai, Hengrui Gu, Ying Wang, Xin Wang

Split and Merge: Aligning Position Biases in LLM-based Evaluators (2024.emnlp-main)

Challenge:	Existing methods of extrinsic bias mitigation rely on manual word lists for sensitive groups . however, these word lists are limited by length and scope, resulting in poor performance.
Approach:	They propose a method which generates continuous token lists from the entire vocabulary space and uses them to bridge the gap between outputs and targets in fairness learning process.
Outcome:	The proposed method outperforms baseline methods on three NLU tasks.

Copied to clipboard

Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, Yang Liu

Integrating Argumentation and Hate-Speech-based Techniques for Countering Misinformation (2024.emnlp-main)

Challenge:	Large language models (LLMs) have shown promise as automated evaluators for assessing the quality of answers generated by AI systems.
Approach:	They propose an alignment-based system that calibrates position bias in a lightweight yet effective manner by taking into account both length and semantics and combining them into a single prompt.
Outcome:	Extensive experiments with six LLMs on 11,520 answer pairs show that PORTIA significantly improves consistency and consistency rates with humans.

Copied to clipboard

Sougata Saha, Rohini Srihari

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment (2024.emnlp-main)

Challenge:	scalable strategies to combat online misinformation are short-term and insufficient, authors say . current reactive approaches, like content flagging and banning, do little to change perception of misinformants . human evaluations show that our framework generates expert-like responses .
Approach:	They propose a framework that generates persuasive responses from hate-speech counter-responses . human evaluations show that the framework generates expert-like responses .
Outcome:	The proposed framework generates expert-like responses and is 14% more engaging, 21% more natural, and 18% more factual than the best available alternatives.

Copied to clipboard

Wenda Xu, Jiachen Li, William Yang Wang, Lei Li

One2Set + Large Language Model: Best Partners for Keyphrase Generation (2024.emnlp-main)

Challenge:	Existing offline DAP methods for aligning large language models with human preference are computationally expensive due to their two-stage training pipeline that consists of a reward modeling phase.
Approach:	They propose to align large language models to human desiderata from offline preference datasets by using an online approach.
Outcome:	The proposed approach improves performance across a wide range of tasks when training with the same amount of preference data.

Copied to clipboard

Liangying Shao, Liang Zhang, Minlong Peng, Guoqi Ma, Hao Yue, Mingming Sun, Jinsong Su

Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering (2024.emnlp-main)

Challenge:	Existing selection methods make redundant selections, causing poor recall and accuracy.
Approach:	They propose a framework to generate keyphrases from a one2set-based model and an LLM as selector.
Outcome:	The proposed framework surpasses state-of-the-art models in absent keyphrase prediction.

Copied to clipboard

Yifei Yuan, Yang Deng, Anders Søgaard, Mohammad Aliannejadi

ORPO: Monolithic Preference Optimization without Reference Model (2024.emnlp-main)

Challenge:	Product-related question answering (PQA) involves utilizing product-related resources to provide precise answers to users.
Approach:	They propose a task of multilingual cross-market product-based question answering that combines product-related questions with product-specific questions from a multilingual marketplace.
Outcome:	The proposed task provides answers to product-related questions in a multilingual marketplace even in fewer languages.

Copied to clipboard

Jiwoo Hong, Noah Lee, James Thorne

A Multi-Perspective Analysis of Memorization in Large Language Models (2024.emnlp-main)

Challenge:	Pre-trained language models with vast training corpora have shown remarkable abilities in diverse natural language processing tasks.
Approach:	They propose a model-free monolithic odds ratio preference optimization algorithm, ORPO, to improve preference alignment.
Outcome:	The proposed algorithm outperforms state-of-the-art language models with more than 7B and 13B parameters on the ultrafeedback alone.

Copied to clipboard

Bowen Chen, Namgi Han, Yusuke Miyao

Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) can generate the same sequences contained in the pre-train corpus, known as memorization.
Approach:	They analyze the relationship between memorization and outputs from Large Language Models (LLMs) they show a sudden drop and increase in the frequency of input tokens when generating memorized/unmemorized sequences .
Outcome:	The proposed model can generate the same sequences contained in the pre-train corpus, and it can predict unmemorized tokens.

Copied to clipboard

Nicolò Penzo, Maryam Sajedinia, Bruno Lepri, Sara Tonelli, Marco Guerini

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024.emnlp-main)

Challenge:	Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs.
Approach:	They propose a methodological pipeline to investigate model performance across structural attributes of conversations.
Outcome:	The proposed method analyzes the performance of an LLM to classify multi-party conversations . it shows that response selection relies more on the textual content of conversations compared to addressee recognition .

Copied to clipboard

Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, Iryna Gurevych

Unveiling the Role of Pretraining in Direct Speech Translation (2024.emnlp-main)

Challenge:	Recent prompting techniques have improved LLMs’ performance on various reasoning tasks, but there is little understanding of what triggers reasoning abilities in LLM in the inference stage.
Approach:	They propose a method that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution.
Outcome:	The proposed method boosts multiple LLMs by 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral.

Copied to clipboard

Belen Alastruey, Gerard Gállego, Marta Costa-jussà

PCQPR: Proactive Conversational Question Planning with Reflection (2024.emnlp-main)

Challenge:	Existing approaches to train direct speech-to-text translation systems are pretraining the encoder on automatic speech recognition, thus losing efficiency in the training process.
Approach:	They propose to change the decoder cross-attention to integrate source information from earlier steps in training.
Outcome:	The proposed model can achieve comparable performance to the pretrained model while reducing training time.

Copied to clipboard

Shasha Guo, Lizi Liao, Jing Zhang, Cuiping Li, Hong Chen

CodeAgent: Autonomous Communicative Agents for Code Review (2024.emnlp-main)

Challenge:	Current CQG methods focus on immediate context without strategic consideration of the specified conversational outcome.
Approach:	They propose a method that uses a planning algorithm inspired by Monte Carlo Tree Search to generate contextually relevant questions.
Outcome:	The proposed approach surpasses existing methods in e-learning and customer service fields . it generates contextually appropriate questions strategically devised to reach a specified outcome .

Copied to clipboard

Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, Tegawendé Bissyandé

TroL: Traversal of Layers for Large Language and Vision Models (2024.emnlp-main)

Challenge:	Existing methods for code review rely on single input-output generative models and thus lack the collaborative nature of code review.
Approach:	They propose a multi-agent Large Language Model (LLM) system for code review automation that incorporates a supervisory agent to ensure that all the agents’ contributions address the initial review question.
Outcome:	The proposed system detects inconsistencies between code changes and commit messages, identify vulnerabilities, validates code style adherence, and suggests code revisions.

Copied to clipboard

Byung-Kwan Lee, Sangyun Chung, Chae Won Kim, Beomchan Park, Yong Man Ro

MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language (2024.emnlp-main)

Challenge:	Existing open-source LLVMs that perform comparably to closed-source models such as GPT-4V are often considered too large, having a larger number of layers.
Approach:	They propose a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers, which enables the reuse of layers in a token-wise manner.
Outcome:	The proposed model outperforms open-source models with larger model sizes and outperformed closed-source LLVMs with substantial models.

Copied to clipboard

Shun Wang, Ge Zhang, Han Wu, Tyler Loakman, Wenhao Huang, Chenghua Lin

Revisiting Supertagging for faster HPSG parsing (2024.emnlp-main)

Challenge:	Existing evaluation methods focus on fluency and factual reliability, while neglecting figurative quality.
Approach:	They propose a set of human evaluation metrics focused on the translation of figurative language and a parallel metaphor corpus generated by post-editing.
Outcome:	The proposed evaluation protocol estimates four aspects of MT: Metaphorical Equivalence, Emotion, Authenticity, and Quality.

Copied to clipboard

Olga Zamaraeva, Carlos Gómez-Rodríguez

Improve Dense Passage Retrieval with Entailment Tuning (2024.emnlp-main)

Challenge:	a new supertagger for HPSG-based treebanks is used to improve parsing speed and accuracy.
Approach:	They propose to integrate the best supertagger into an HPSG-based parser and compare it to an existing system.
Outcome:	The proposed system achieves 97.26% accuracy on 950 sentences from WSJ23 and 93.88% on the out-of-domain technical essay The Cathedral and the Bazaar.

Copied to clipboard

Lu Dai, Hao Liu, Hui Xiong

ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for dense retrieval have demonstrated remarkable performance in IR tasks.
Approach:	They propose a method to improve the embedding of dense retrievers by using existence claim as a bridge.
Outcome:	The proposed method can be plugged into current dense retrieval methods and the results are published in the journal Nature.

Copied to clipboard

Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana

TEMA: Token Embeddings Mapping for Enriching Low-Resource Language Models (2024.emnlp-main)

Challenge:	Currently, tool-augmented large language models (LLMs) only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100.
Approach:	They propose a multi-level diagnostic process to assess the LLM's hallucinations through two perspectives: depth and breadth.
Outcome:	The proposed diagnostic process assesses the hallucinations of large language models through two perspectives: depth and breadth.

Copied to clipboard

Rodolfo Zevallos, Núria Bel, Mireia Farrús

DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting (2024.emnlp-main)

Challenge:	Low-resource languages, that is, languages that do not have a massive amount of text, risk being almost excluded from the possibility of having good NLP applications.
Approach:	They propose an algorithm that maps token embeddings of a richly pre-trained model to a poorly trained model and creates a more complex model.
Outcome:	The proposed model reduces perplexity and is competitive or better for the most semantic tasks.

Copied to clipboard

Xuanming Zhang, Anthony Diaz, Zixun Chen, Qingyang Wu, Kun Qian, Erik Voss, Zhou Yu

Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback (2024.emnlp-main)

Challenge:	Existing automated writing evaluation systems only detect incoherence in writing . a recent study has found that incorporating specific reasons for incohence improves the quality of rewrites .
Approach:	They propose a benchmark that includes expert annotations for detecting incoherence in L2 English writing, identifying the underlying reasons, and rewriting the incoerent sentences.
Outcome:	The proposed benchmark improves coherence in L2 English writing by fine-tuning models . the authors find that incorporating specific reasons improves quality of rewrites .

Copied to clipboard

Fatemeh Pesaran Zadeh, Juyeon Kim, Jin-Hwa Kim, Gunhee Kim

PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)

Challenge:	Existing datasets do not cover full range of chart types, such as 3D, volumetric, and gridded charts.
Approach:	They propose a hierarchical pipeline and a new dataset for chart generation that leverages the relationships within rich datasets.
Outcome:	The proposed method outperforms open-source models and is comparable to state-of-the-art proprietary models in data visualization tasks.

Copied to clipboard

Christoph Leiter, Steffen Eger

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning (2024.emnlp-main)

Challenge:	Large language models (LLMs) are useful for low-resource scenarios and time-restricted applications.
Approach:	They propose a large-scale evaluation tool for large language models that uses prompts . they evaluate 720 prompt templates for open-source LLM-based metrics on MT and summarization datasets a 6.6M evaluations.
Outcome:	The proposed model evaluates 720 prompt templates on machine translation and summarization datasets.

Copied to clipboard

Shuai Zhao, Meihuizi Jia, Anh Tuan Luu, Fengjun Pan, Jinming Wen

Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models (2024.emnlp-main)

Challenge:	In-context learning has shown high efficacy in several NLP tasks, especially in few-shot settings.
Approach:	They propose a backdoor attack method that poisons demonstration examples and poisons the demonstration context, preserving the model's generality.
Outcome:	The proposed method can make models behave in alignment with predefined intentions without fine-tuning the model.

Copied to clipboard

Javier Chiyah-Garcia, Alessandro Suglia, Arash Eshghi

Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models (2024.emnlp-main)

Challenge:	In dialogue, the addressee may misunderstand the speaker and respond erroneously.
Approach:	They collect, analyse, and publicly release a dataset of multi-modal TPR sequences in dialogue . they evaluate several state-of-the-art Vision and Language Models across multiple settings .
Outcome:	The proposed model underperforms in a human-robot interaction task compared to humans . the proposed model can benefit from specialised losses targeting relevant tokens .

Copied to clipboard

Xinrong Zhang, Yingfa Chen, Shengding Hu, Xu Han, Zihang Xu, Yuanwei Xu, Weilin Zhao, Maosong Sun, Zhiyuan Liu

Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations (2024.emnlp-main)

Challenge:	Large language models (LLMs) are increasingly permeating daily lives and require real-time interactions that mirror human conversations.
Approach:	They propose to use time-division-multiplexing to process queries and responses pseudo-simultaneously.
Outcome:	The proposed model can listen to users while generating output and adjust to provide instant feedback.

Copied to clipboard

Matthias Lindemann, Alexander Koller, Ivan Titov

Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)

Challenge:	Inductive biases play a critical role in NLP, especially in learning from limited data and generalizing systematically outside of the training distribution.
Approach:	They propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform syntactic transformations of dependency trees given a description of the transformation.
Outcome:	The proposed model can perform syntactic transformations and generalize semantic parsing with attention heads that keep track of which syntaktic transformation needs to be applied to which token.

Copied to clipboard

Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains.
Approach:	They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies.
Outcome:	The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking.

Copied to clipboard

Tu Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues

Red Teaming Language Models for Processing Contradictory Dialogues (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are rapidly developing and are becoming more and more useful in scientific tasks.
Approach:	They propose to use LLM-as-a-judge to grade LLMs on SciEx to assess their ability on scientific tasks.
Outcome:	The proposed benchmarks show that the LLMs perform decently on free-form exams, achieving 0.948 Pearson correlation with expert grading.

Copied to clipboard

Xiaofei Wen, Bangzheng Li, Tenghao Huang, Muhao Chen

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (2024.emnlp-main)

Challenge:	a recent study shows that language models are prone to self-contradiction during dialogues.
Approach:	They propose a red teaming framework that detects and attempts to explain dialogues, then modifies existing contradictory content using the explanation.
Outcome:	The proposed task improves the ability to detect contradictory dialogues and provides valid explanations.

Copied to clipboard

Sander Land, Max Bartolo

Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs (2024.emnlp-main)

Challenge:	Disconnect between tokenizer creation and model training in language models allows for specific inputs, such as the infamous SolidGoldMagikarp token, to induce unwanted model behaviour.
Approach:	They propose to use tokenizer analysis, model weight-based indicators, and prompting techniques to detect problematic tokens in large language models.
Outcome:	The proposed methods show that tokenizers are under-trained across a diverse set of models and provide insights into improving the efficiency and safety of language models.

Copied to clipboard

Houman Mehrafarin, Arash Eshghi, Ioannis Konstas

Pragmatic Norms Are All You Need – Why The Symbol Grounding Problem Does Not Apply to LLMs (2024.emnlp-main)

Challenge:	Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions.
Approach:	They investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle.
Outcome:	The proposed models leverage word/phrase overlaps across sections of test input, models’ inherent knowledge during pre-training or fine-tuning, and names of entities.

Copied to clipboard

Reto Gubelmann

Linguistic Theories

Challenge:	'Symbol grounding problem' is a philosophical problem that arises when questionable theories of meaning are presupposed.
Approach:	They argue that LLMs are vulnerable to Harnad’s symbol grounding problem (SGP), as it has been claimed recently . they trace the origins of the SGP to the computational theory of mind .
Outcome:	The proposed model-theoretic semantics does not give rise to the SGP, as it has been claimed in the literature.

Major Entity Identification: A Generalizable Alternative to Coreference Resolution (2024.emnlp-main)

Copied to clipboard

Kawshik Sundar, Shubham Toshniwal, Makarand Tapaswi, Vineet Gandhi

Enhancing High-order Interaction Awareness in LLM-based Recommender Model (2024.emnlp-main)

Challenge:	Prior work identified annotation differences as one of the main reasons for the limited generalization gap in coreference resolution models.
Approach:	They propose an alternative referential task where the target entities are assumed to be specified in the input and the task is limited to the frequent entities.
Outcome:	The proposed model generalizes well across domains on multiple datasets with supervised models and LLM-based few-shot prompting.

Copied to clipboard

Xinfeng Wang, Jin Cui, Fumiyo Fukumoto, Yoshimi Suzuki

What Are the Odds? Language Models Are Capable of Probabilistic Reasoning (2024.emnlp-main)

Challenge:	Existing approaches to model user-item interactions do not account for high-order interactions.
Approach:	They propose to enhance whole-word embeddings to enhance LLMs’ interpretation of graph-constructed interactions for recommendations without requiring graph pre-training.
Outcome:	The proposed model outperforms state-of-the-art methods in direct recommendations.

Copied to clipboard

Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, Daniel McDuff

MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction (2024.emnlp-main)

Challenge:	Language models (LMs) are capable of remarkably complex linguistic tasks, but numerical reasoning is an area in which they struggle.
Approach:	They evaluate the probabilistic reasoning capabilities of language models using idealized and real-world statistical distributions.
Outcome:	The proposed model can make inferences about distributions, even if assumptions are incorrect or misspecified.

Copied to clipboard

Han Jiang, Junwen Duan, Zhe Qu, Jianxin Wang

LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models (2024.emnlp-main)

Challenge:	Existing methods to extract text snippets from input text to support model predictions without explicit rationale annotation have limited their ability to capture meaningful internal correlations between aspects.
Approach:	They propose a multi-aspect rationale extractor that extracts text snippets to support model predictions without explicit rationale annotation.
Outcome:	The proposed method achieves state-of-the-art on two unsupervised rationale extraction benchmarks.

Copied to clipboard

Hayder Elesedy, Pedro Esperanca, Silviu Vlad Oprea, Mete Ozay

“A good pun is its own reword”: Can Large Language Models Understand Puns? (2024.emnlp-main)

Challenge:	Existing model-based guardrails have not been designed for resource-constrained computational portable devices such as mobile phones.
Approach:	They propose a parameter-efficient guardrail adaptation method that relies on knowledge sharing between LLMs and guardrail models to adapt to content moderation tasks.
Outcome:	The proposed method outperforms existing guardrail methods with lower parameter overhead and higher accuracy on the generative task.

Copied to clipboard

Zhijun Xu, Siyu Yuan, Lingjie Chen, Deqing Yang

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)

Challenge:	Existing studies on the understanding of puns in large language models (LLMs) have not explored the use of pun in creative writing and humor creation.
Approach:	They propose to use pun recognition, explanation and generation tasks to evaluate the capabilities of large language models (LLMs) they adopt automated evaluation metrics from prior research and introduce new evaluation methods and metrics that align more closely with human cognition.
Outcome:	The proposed methods align more closely with human cognition than previous evaluation metrics.

Copied to clipboard

Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, Jun Liu

Dependency Graph Parsing as Sequence Labeling (2024.emnlp-main)

Challenge:	Existing metrics fail to align well with human judgments when evaluating QG questions.
Approach:	They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions.
Outcome:	The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency .

Copied to clipboard

Ana Ezquerro, David Vilares, Carlos Gómez-Rodríguez

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data (2024.emnlp-main)

Challenge:	Various linearizations have been proposed to cast syntactic dependency parsing as sequence labeling, but they cannot handle reentrancy or cycles.
Approach:	They propose unbounded linearizations that can be used to cast dependency parsing as sequence labeling.
Outcome:	The proposed linearizations can cast syntactic dependency parsing as a sequence labeling task.

Copied to clipboard

Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, Etienne Bernard

Towards a Greek Proverb Atlas: Computational Spatial Exploration and Attribution of Greek Proverbs (2024.emnlp-main)

Challenge:	Named Entity Recognition (NER) is a core component of natural language processing, present in a variety of applications such as medical coding, financial news analysis, or legal documents parsing.
Approach:	They propose to use Large Language Models (LLMs) to create NuNER, a compact language representation model specialized in the Named Entity Recognition task.
Outcome:	The proposed model outperforms similar-sized foundation models in the few-shot regime and is based on a human-annotated dataset.

Copied to clipboard

John Pavlopoulos, Panos Louridas, Panagiotis Filos

Challenge:	a recent study focuses on Greek proverbs, which carry wisdom and are still used today . it is the first large-scale machine-actionable dataset of Greek prowords quantifying their spatial distribution across different locations.
Approach:	They propose to use a publicly-available and machine-actionable dataset of Greek proverbs to quantify their spatial distribution across different locations.
Outcome:	The proposed dataset is a publicly-available and machine-actionable dataset of Greek proverb variants.

Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications (2024.emnlp-main)

Copied to clipboard

Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, Jian Wu

Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss (2024.emnlp-main)

Challenge:	Recent studies have focused on how large language models process multiple languages, but internal mechanisms of LLMs remain insufficiently explored.
Approach:	They propose to convert dense LLMs into fine-grained MoE architectures and analyze their activation patterns using expert activation frequency heatmaps.
Outcome:	The proposed method outperforms random expert pruning and exceeds models in some languages.

Copied to clipboard

Bowen Zhang, Chunping Li

Rationalizing Transformer Predictions via End-To-End Differentiable Self-Training (2024.emnlp-main)

Challenge:	despite its efficiency, Sentence-BERT ignores the progressive nature of semantic relationships, despite a promising approach . contrastive learning methods have improved performance on renowned STS benchmarks, but they fail to leverage fine-grained information.
Approach:	They propose a regression framework that categorizes text pairs as either semantically similar or dissimilar . they propose two loss functions: Translated ReLU and Smooth K2 Loss to bridge this gap .
Outcome:	The proposed method achieves convincing performance across seven established STS benchmarks.

Copied to clipboard

Marc Brinner, Sina Zarrieß

Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation (2024.emnlp-main)

Challenge:	Neural networks are increasingly prevalent across a wide range of applications, driving significant advancements in fields such as natural language processing, computer vision, and beyond.
Approach:	They propose an end-to-end differentiable training paradigm for stable training of a rationalized transformer classifier.
Outcome:	The proposed model is capable of classifying a sample and scoring input tokens without any explicit supervision and produces class-wise rationales without instabilities.

Copied to clipboard

Markus Frohmann, Igor Sterner, Ivan Vulić, Benjamin Minixhofer, Markus Schedl

Applying Contrastive Learning to Code Vulnerability Type Classification (2024.emnlp-main)

Challenge:	Sentence segmentation is a key task in many NLP systems, but no prior method has achieved all of the features needed to segment a text into sentences.
Approach:	They propose a new model that uses punctuation to enhance robustness and adaptability.
Outcome:	The proposed model outperforms baselines across 8 corpora across diverse domains and languages and is available under the MIT license.

Copied to clipboard

Chen Ji, Su Yang, Hongyu Sun, Yuqing Zhang

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts (2024.emnlp-main)

Challenge:	Recent approaches to classification of vulnerabilities ignore their relationships and treat each class in isolation, resulting in non-scalable code vector representations.
Approach:	They propose a hierarchical contrastive learning framework to bring vector representations of related CWEs closer together and use max-pooling to enable the model to handle longer vulnerability code inputs.
Outcome:	The proposed framework outperforms state-of-the-art methods by 2.97%-17.90% on accuracy and 0.98%-22.27% on weighted-F1 with even better performance on higher-quality datasets.

Copied to clipboard

Ruida Wang, Jipeng Zhang, Yizhen Jia, Rui Pan, Shizhe Diao, Renjie Pi, Tong Zhang

Multi-Level Cross-Modal Alignment for Speech Relation Extraction (2024.emnlp-main)

Challenge:	a framework for formal proof writing using formal languages like Lean4 is needed to prove mathematical theorems using formal language.
Approach:	They propose a framework that trains a general-purpose LLM to be a Lean4 expert.
Outcome:	The proposed framework achieves cumulative accuracies of 36.48% and 33.61% on MiniF2F-Valid and Test datasets.

Copied to clipboard

Liang Zhang, Zhen Yang, Biao Fu, Ziyao Lu, Liangying Shao, Shiyu Liu, Fandong Meng, Jie Zhou, Xiaoli Wang, Jinsong Su

Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Existing studies use synthetic speech to train and evaluate SpeechRE models, hindering their development . modality gap issue limits performance of existing models, limiting future researches .
Approach:	They propose to use speech data to train and evaluate SpeechRE models by using real speech . they propose to train a cross-modal alignment model to bridge the modality gap .
Outcome:	The proposed model can train to bridge the modality gap between speech encoder and text decoder . the proposed model is based on two real SpeechRE datasets .

Copied to clipboard

Christopher Schröder, Gerhard Heyer

PANDA: Persona Attributes Navigation for Detecting and Alleviating Overuse Problem in Large Language Models (2024.emnlp-main)

Challenge:	Existing methods to train models without labeled data are lacking in supervised tasks . a lack of labeles is the main obstacle to real-world applications .
Approach:	They propose a semi-supervised approach that uses a model to obtain pseudo-labels for unlabeled data.
Outcome:	The proposed method outperforms the reproduced methods on four text classification benchmarks.

Copied to clipboard

Jinsung Kim, Seonmin Koo, Heuiseok Lim

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (2024.emnlp-main)

Challenge:	In persona-grounded dialogue, it is required to respond fluently and ground attributes according to the current conversation topic properly.
Approach:	They propose a framework to quantify the persona overuse problem of LLMs by establishing clear standards and verifying various LLM based on them.
Outcome:	The proposed framework aims to quantify the persona overuse problem of LLMs by establishing clear standards and verifying various LLM based on them.

Copied to clipboard

null Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker

Subword Segmentation in LLMs: Looking at Inflection and Consistency (2024.emnlp-main)

Challenge:	Existing approaches to safety alignment focus on homogeneous monolingual settings . preference training and safety measures often overfit to harms common in Western-centric datasets .
Approach:	They propose to use human annotated red teaming prompts to identify global and local harms.
Outcome:	The proposed approach can address and optimize for a non-homogeneous set of languages and cultural preferences while minimizing both global and local harms.

Copied to clipboard

Marion Marco, Alexander Fraser

Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments (2024.emnlp-main)

Challenge:	Subword segmentation is not linguistically guided and is not currently well understood in LLMs.
Approach:	They group words according to their segmentation properties and compare how well a model can solve a linguistic task for these groups using two criteria: adherence to morpheme boundaries and segmentation consistency of inflected forms of a lemma.
Outcome:	The results show that the criterion of segmentation consistency can predict the model’s ability to recognize and generate the lemma from an inflected form, providing evidence that subword segmentation is relevant.

Copied to clipboard

Omar Sharif, Joseph Gatto, Madhusudan Basak, Sarah Masud Preum

Let Me Teach You: Pedagogical Foundations of Feedback for Language Models (2024.emnlp-main)

Challenge:	Existing work on event-specific argument extraction is limited to contiguous spans of text . Existing approaches to event-centric information extraction are limited to explicit arguments .
Approach:	They propose two key argument types that cannot be modeled by existing EE frameworks . implicit and scattered arguments are crucial to elicit full breadth of information required for proper event modeling.
Outcome:	The proposed dataset includes 7,464 argument annotations from online health discourse.

Copied to clipboard

Beatriz Borges, Niket Tandon, Tanja Käser, Antoine Bosselut

Unknown Claims: Generation of Fact-Checking Training Examples from Unstructured and Structured Data (2024.emnlp-main)

Challenge:	Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning Large Language Models to human preferences.
Approach:	They propose a feedback framework for Large Language Models that outlines various characteristics of the feedback space and a taxonomy based on these variables.
Outcome:	The proposed framework provides a general mapping of the feedback space and provides examples for mapping to future research.

Copied to clipboard

Jean-Flavien Bussotti, Luca Ragazzi, Giacomo Frisoni, Gianluca Moro, Paolo Papotti

TL-CL: Task And Language Incremental Continual Learning (2024.emnlp-main)

Challenge:	Existing methods for fact-checking are labor-intensive and time-consuming.
Approach:	They propose a framework that generates training instances for FC systems automatically using textual and tabular content.
Outcome:	The proposed framework generates training instances for FC systems using textual and tabular content.

Copied to clipboard

Shrey Satapara, P. K. Srijith

Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)

Challenge:	a multilingual model is periodically updated to accommodate new tasks in previously learned languages or new languages for established tasks.
Approach:	They propose an adapter-based parameter-efficient fine-tuning strategy for continual learning in multilingual models.
Outcome:	The proposed approach outperforms other parameter-efficient approaches without access to historical data for replay.

Copied to clipboard

Daniel Jeong, Saurabh Garg, Zachary Lipton, Michael Oberst

Empowering Multi-step Reasoning across Languages via Program-Aided Language Models (2024.emnlp-main)

Challenge:	Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks.
Approach:	They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases .
Outcome:	The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases.

Copied to clipboard

Leonardo Ranaldi, Giulia Pucci, Barry Haddow, Alexandra Birch

Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)

Challenge:	In-context learning methods elicit Large Language Models to solve tasks using provided demonstrations without parameter updates.
Approach:	They propose a method for aligning reasoning programs across languages using a double-step cross-lingual prompting mechanism.
Outcome:	The proposed method outperforms existing prompting methods and reduces interaction time.

Copied to clipboard

Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, Qi Liu

ControlMath: Controllable Data Generation Promotes Math Generalist Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction.
Approach:	They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance.
Outcome:	The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies.

Copied to clipboard

Nuo Chen, Ning Wu, Jianhui Chang, Linjun Shou, Jia Li

Where Am I From? Identifying Origin of LLM-generated Content (2024.emnlp-main)

Challenge:	Currently, mathematical reasoning is one of the most challenging areas for closed-source LLMs.
Approach:	They propose an iterative method involving an equation-generator module and two LLM-based agents that generate diverse equations and transform them into math word problems.
Outcome:	The proposed method enables the generation of diverse math problems, not limited to specific domains or distributions.

Copied to clipboard

Liying Li, Yihan Bai, Minhao Cheng

Challenge:	Generative models have produced high-quality content, but they pose security risks . a new framework for deep learning systems enables the tracing of AI-generated content back to its source .
Approach:	They propose a digital forensics framework that embeds a secret watermark into the generated output and a "depth watermark" this watermark strengthens the link between content and generator, enabling accurate tracing while maintaining the quality of the generated content.
Outcome:	The proposed framework ensures accurate tracing while maintaining quality of generated content.

ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment (2024.emnlp-main)

Copied to clipboard

Tarek Naous, Michael Ryan, Anton Lavrouk, Mohit Chandra, Wei Xu

GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text (2024.emnlp-main)

Challenge:	Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-lingual analyses.
Approach:	They propose to use a multilingual multi-domain dataset to benchmark multilingual and monolingual models for multilingual readability assessment.
Outcome:	The proposed model trains better in supervised, unsupervised, and few-shot prompting settings and identifies shortcomings in state-of-the-art unsupervised methods.

Copied to clipboard

Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, Lori Levin

GDTB: Genre Diverse Data for English Shallow Discourse Parsing across Modalities, Text Types, and Domains (2024.emnlp-main)

Challenge:	Existing resources for standardized, easily accessible IGT data limit their applicability to linguistic research.
Approach:	They compile the largest existing corpus of interlinear glossed text data from a variety of sources and use it to generate annotated text.
Outcome:	The proposed model outperforms SOTA models on monolingual corpora by 6.6%.

Copied to clipboard

Yang Janet Liu, Tatsuya Aoyama, Wesley Scivetti, Yilun Zhu, Shabnam Behzad, Lauren Levine, Jessica Lin, Devika Tiwari, Amir Zeldes

RA2FD: Distilling Faithfulness into Efficient Dialogue Systems (2024.emnlp-main)

Challenge:	Existing shallow discourse parsing systems focus on the Wall Street Journal corpus, but the data is limited to the news domain and is 35 years old.
Approach:	They propose to use the Wall Street Journal corpus as a benchmark for PDTB-style shallow discourse parsing.
Outcome:	The proposed dataset is compatible with PDTB, but suffers from degradation out-of-domain.

Copied to clipboard

Zhiyuan Zhu, Yusheng Liao, Chenxin Xu, Yunfeng Guan, Yanfeng Wang, Yu Wang

Subjective Topic meets LLMs: Unleashing Comprehensive, Reflective and Creative Thinking through the Negation of Negation (2024.emnlp-main)

Challenge:	Retrieval Augmented Generation (RAG) is effective but inference inefficient, while Retrieral Free Generations (RFG) are more efficient but sacrifice faithfulness.
Approach:	They propose a retrieval-free model training scheme that uses a teacher-student framework to distill the faithfulness capacity of a student's knowledge-infused responses.
Outcome:	The proposed model surpasses the previous SOTA RFG model on knowledge-grounded dialogue datasets by an average of 33% while improving inference efficiency.

Copied to clipboard

Fangrui Lv, Kaixiong Gong, Jian Liang, Xinyu Pang, Changshui Zhang

Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently (2024.emnlp-main)

Challenge:	Large language models (LLMs) exhibit powerful reasoning capacity, but their evaluation still lacks comprehensiveness.
Approach:	They propose a framework grounded in the principle of the Negation of Negation (NeoN) to unleash the potential comprehensive, reflective, and creative thinking abilities of LLMs.
Outcome:	The proposed framework unleashes the potential comprehensive, reflective, and creative thinking abilities of large language models.

Copied to clipboard

Kanishka Misra, Allyson Ettinger, Kyle Mahowald

Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking (2024.emnlp-main)

Challenge:	Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction.
Approach:	They propose to use in-context examples and instructions to improve LMs' robustness in performing property inheritance.
Outcome:	The proposed model can perform non-trivial property inheritance on in-context examples and instructions, but it is inconsistent with the task.

Copied to clipboard

Jun Bai, Zhuofan Chen, Zhenzi Li, Hanhua Hong, Jianfei Zhang, Chen Li, Chenghua Lin, Wenge Rong

Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism (2024.emnlp-main)

Challenge:	Existing methods for text ranking are based on intuition, but their estimated transferability may not align well with the objectives of text ranking.
Approach:	They propose to compute expected rank as transferability, explicitly reflecting the model’s ranking capability.
Outcome:	The proposed method shows significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption.

Copied to clipboard

Anhao Zhao, Fanghua Ye, Jinlan Fu, Xiaoyu Shen

Self-Powered LLM Modality Expansion for Large Speech-Text Models (2024.emnlp-main)

Challenge:	Large language models exhibit remarkable in-context learning (ICL) capabilities, but the underlying working mechanism of ICL remains unclear.
Approach:	They propose a Two-Dimensional Coordinate System that unifies both views into a systematic framework that explains the behavior of ICL through two orthogonal variables: whether similar examples are presented in the demonstrations and whether LLMs can recognize the task.
Outcome:	The proposed method can interpret ICL for generation tasks effectively.

Copied to clipboard

Tengfei Yu, Xuebo Liu, Zhiyi Hou, Liang Ding, Dacheng Tao, Min Zhang

ABSEval: An Agent-based Framework for Script Evaluation (2024.emnlp-main)

Challenge:	Large language models exhibit remarkable performance across diverse tasks . however, these methods require significant resource demands and tend to overfit specific tasks.
Approach:	They propose a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning.
Outcome:	The proposed model mitigates speech anchor bias and improves the fusion of speech and text modalities in large language models.

Copied to clipboard

Sirui Liang, Baoli Zhang, Jun Zhao, Kang Liu

Challenge:	Existing studies on script evaluation of large language models (LLMs) have not evaluated scripts generated by LLMs due to their logical structure, sequential organization, and open-ended nature.
Approach:	They propose to use a script evaluation dataset to evaluate LLM scripts . they propose to develop an agent-based script evaluation framework ABSEval to evaluate scripts.
Outcome:	The proposed framework provides superior accuracy and relevance, aligning closely with human evaluation.

Latent Concept-based Explanation of NLP Models (2024.emnlp-main)

Copied to clipboard

Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, Hassan Sajjad

Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher (2024.emnlp-main)

Challenge:	Existing attempts to explain deep learning models rely on input features, such as the words . however, such explanations are often less informative due to the discrete nature of words and lack of contextual verbosity.
Approach:	They propose a method that generates explanations for predictions based on latent concepts . they map the representations of salient input words into the training latent space .
Outcome:	The proposed method generates explanations for predictions based on latent concepts . it maps representations of salient input words into training latent space .

Copied to clipboard

Hyunjong Ok, Jegwang Ryu, Jaeho Lee

Challenge:	Large language models (LLMs) have demonstrated their tremendous capability to generate human-like text sentences that convey rich knowledge in various problem domains.
Approach:	They propose an algorithm to aggregate small-scale LLM and LLM predictions on initial tokens so that the generated tokens can more accurately condition the subsequent token generation by small-level LLM only.
Outcome:	The proposed method improves on the limited supervision scenario on a wide range of models and datasets.

Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research (2024.emnlp-main)

Copied to clipboard

Yida Mu, Mali Jin, Xingyi Song, Nikolaos Aletras

The Mystery of the Pathological Path-star Task for Language Models (2024.emnlp-main)

Challenge:	Social media data exhibits distinctive characteristics such as rapid and continual topic evolution.
Approach:	They propose new protocols and best practices for improving dataset development from social media data and its usage.
Outcome:	The proposed protocols and best practices improve the performance of social media datasets and their usage.

Copied to clipboard

Arvid Frydenlund

Voices in a Crowd: Searching for clusters of unique perspectives (2024.emnlp-main)

Challenge:	Language models have become increasingly capable of solving a variety of complex tasks.
Approach:	They propose a path-star task where multiple arms radiate from a single starting node and each node is unique.
Outcome:	The proposed task is learnable using teacher-forcing in alternative settings and improves results across a variety of model types.

Copied to clipboard

Nikolas Vitsakis, Amit Parekh, Ioannis Konstas

Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent (2024.emnlp-main)

Challenge:	Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata.
Approach:	They propose a framework that trains models without encoding annotator metadata and creates clusters of similar opinions, that are called voices.
Outcome:	The proposed framework captures minority perspectives based on demographic factors in two distinct datasets while also capturing majority perspectives.

Copied to clipboard

Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Hao Peng, Liehuang Zhu

SLANG: New Concept Comprehension of Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have revolutionized open-domain dialogue agents but face challenges in multi-character role-playing (MCRP) scenarios.
Approach:	They propose a framework for efficient multi-character role-playing that employs a dynamic low-rank adapter strategy and distinct LoRA blocks for each character.
Outcome:	Neeko employs a dynamic low-rank adapter (LoRA) strategy, enabling it to adapt seamlessly to diverse characters.

Copied to clipboard

Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Xueqi Cheng

Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models (2024.emnlp-main)

Challenge:	Dynamic nature of language limits the adaptability of Large Language Models (LLMs) Traditionally, LLMs are trained on static data, which limits their adaptability .
Approach:	They propose a benchmark to integrate novel data and assess LLMs’ ability to comprehend emerging concepts, alongside a causal inference-based approach to enhance LLM comprehension of new phrases and their colloquial context.
Outcome:	The proposed model outperforms baseline models in terms of precision and relevance in the comprehension of Internet slang and memes.

Copied to clipboard

Michael Lan, Philip Torr, Fazl Barez

Why Does New Knowledge Create Messy Ripple Effects in LLMs? (2024.emnlp-main)

Challenge:	Recent work aims to reverse engineer transformer models into human-readable representations . transformers exhibit strong capabilities on linguistic tasks, but their complex architectures make them difficult to interpret.
Approach:	They extend transformer models into human-readable representations that implement algorithmic functions by analyzing sequence continuation tasks.
Outcome:	The proposed model can be reverse-engineered into human-readable representations that implement algorithmic functions.

Copied to clipboard

Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, Heng Ji

Lifelong Event Detection via Optimal Transport (2024.emnlp-main)

Challenge:	Existing research has focused on post-training knowledge editing (KE) for language models to ensure that knowledge remains accurate and up-to-date.
Approach:	They propose to use a GradSim indicator to detect when and why updated knowledge ripples in language models.
Outcome:	The proposed indicator GradSim shows that LMs that fail to handle ripple effects have low GradSIM.

Copied to clipboard

Viet Dao, Van-Cuong Pham, Quyen Tran, Thanh-Thien Le, Linh Ngo, Thien Nguyen

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)

Challenge:	Continual event detection (CED) is a challenging task due to catastrophic forgetting, where learning new tasks hampers performance on previous ones.
Approach:	They propose a method that leverages optimal transport principles to align the optimization of a classification module with the intrinsic nature of each class, as defined by their pre-trained language modeling.
Outcome:	The proposed method outperforms state-of-the-art methods on MAVEN and ACE datasets and is a pioneering solution in continual event detection.

Copied to clipboard

Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot

Challenge:	Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories?
Approach:	They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories.
Outcome:	The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories.

FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)

Copied to clipboard

KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza

Domain adapted machine translation: What does catastrophic forgetting forget and why? (2024.emnlp-main)

Challenge:	Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”.
Approach:	They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way.
Outcome:	The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way.

Copied to clipboard

Danielle Saunders, Steve DeNeefe

Enhancing AI Assisted Writing with One-Shot Implicit Negative Feedback (2024.emnlp-main)

Challenge:	Neural Machine Translation (NMT) models can be specialized by domain adaptation, often fine-tuning on a dataset of interest.
Approach:	They propose a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the relationship between the data and the in-domain vocabulary coverage.
Outcome:	The proposed model can be specialized by fine-tuning on a domain of interest, but can fail to achieve the predicted quality of the target domain.

Copied to clipboard

Benjamin Towle, Ke Zhou

Challenge:	Various systems have been proposed to draft and automate replies for users . yet, the heterogeneity of the inputs and architectures often renders it difficult to combine insights from user behaviour in one system to improve performance in another.
Approach:	They propose an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process.
Outcome:	The proposed approach improves Rouge-L, generating the correct intent and generating an 86% win-rate on the multiWOZ and Schema-Guided Dialog datasets.

Atomic Self-Consistency for Better Long Form Generations (2024.emnlp-main)

Copied to clipboard

Raghuveer Thirukovalluru, Yukun Huang, Bhuwan Dhingra

Challenge:	Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the accuracy of the information in responses.
Approach:	They propose a technique that improves the recall of relevant information in an LLM.
Outcome:	The proposed technique improves the recall of relevant information in an LLM.

“Global is Good, Local is Bad?”: Understanding Brand Bias in LLMs (2024.emnlp-main)

Copied to clipboard

Mahammed Kamruzzaman, Hieu Nguyen, Gene Kim

Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach (2024.emnlp-main)

Challenge:	a recent study examined social biases in LLMs but brand bias has received little attention.
Approach:	They examine the behavior of LLMs in the market place by analyzing a brand-based dataset . they find a consistent pattern of brand bias in this space .
Outcome:	The proposed model favors established global brands while marginalizing local ones . the proposed model could boost local brand preference in LLM outputs in specific contexts .

Copied to clipboard

Siqi Li, Danni Liu, Jan Niehues

ACE: A LLM-based Negotiation Coaching System (2024.emnlp-main)

Challenge:	Incorrect translation of rare words can severely degrade the accuracy of ST models .
Approach:	They propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in ST models by incorporating retrieved examples into ST models.
Outcome:	The proposed approach outperforms other modalities and exhibits higher robustness to unseen speakers.

Copied to clipboard

Ryan Shea, Aymen Kallala, Xin Liu, Michael Morris, Zhou Yu

Challenge:	The rapid progress of LLMs has led to the development of more sophisticated AI tutoring systems.
Approach:	They develop an LLM-based assistant for coaching negotiation that provides users with targeted feedback for improvement.
Outcome:	The proposed system improves negotiation performance significantly compared to a system that doesn’t provide feedback and one which uses an alternative method.

TransferTOD: A Generalizable Chinese Multi-Domain Task-Oriented Dialogue System with Transfer Capabilities (2024.emnlp-main)

Copied to clipboard

Ming Zhang, Caishuang Huang, Yilong Wu, Shichun Liu, Huiyuan Zheng, Yurui Dong, Yujiong Shen, Shihan Dou, Jun Zhao, Junjie Ye, Qi Zhang, Tao Gui, Xuanjing Huang

PATIENT-𝜓: Using Large Language Models to Simulate Patients for Training Mental Health Professionals (2024.emnlp-main)

Challenge:	Current datasets cater to user-led systems and are limited to predefined specific scenarios and slots.
Approach:	They propose to use a Chinese dialogue dataset to train a model that authentically simulates human-computer dialogues in 30 popular life service scenarios.
Outcome:	The proposed model achieves a joint accuracy of 75.09% in out-of-domain evaluations . it also achieves notable abilities in slot filling and questioning .

Copied to clipboard

Ruiyi Wang, Stephanie Milani, Jamie Chiu, Jiayin Zhi, Shaun Eack, Travis Labrum, Samuel Murphy, Nev Jones, Kate Hardy, Hong Shen, Fei Fang, Zhiyu Chen

DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction (2024.emnlp-main)

Challenge:	Mental illness remains one of the most critical public health issues.
Approach:	They propose a patient simulation framework for cognitive behavior therapy training that uses large language models to act as a simulated therapy patient.
Outcome:	The proposed framework improves the skill acquisition and confidence of mental health trainees beyond textbooks, videos, and role-play with non-patients.

Copied to clipboard

Xueren Ge, Abhishek Satpathy, Ronald Williams, John Stankovic, Homa Alemzadeh

ModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities (2024.emnlp-main)

Challenge:	Prior work focused on hierarchical label structures but neglected to incorporate external knowledge from medical guidelines.
Approach:	They propose to incorporate external knowledge from medical guidelines into domain knowledge enhanced classification for diagnosis prediction.
Outcome:	The proposed system outperforms state-of-the-art label-wise attention networks and transformer models on a real-world emergency medical services dataset and a public electronic health record dataset.

Copied to clipboard

Yukun Jiang, Zheng Li, Xinyue Shen, Yugeng Liu, Michael Backes, Yang Zhang

Large Language Models Can Self-Correct with Key Condition Verification (2024.emnlp-main)

Challenge:	Large vision-language models have been widely used but stereotypical biases are unexplored.
Approach:	They propose a framework to SCAN stereotypical bias within large vision-language models . they examine stereotype biases with respect to gender and race in three scenarios .
Outcome:	The proposed framework can reduce stereotypical biases in large vision-language models . the currently popular models show significant stereotype biase .

Copied to clipboard

Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, Meng Jiang

Learning to Write Rationally: How Information Is Distributed in Non-native Speakers’ Essays (2024.emnlp-main)

Challenge:	Existing methods to correct reasoning without external feedback have not been used in large language models.
Approach:	They propose an iterative verify-then-correct framework to progressively identify and correct (probably) false responses, named ProCo.
Outcome:	The proposed method improves the accuracy of LLMs on three reasoning tasks.

Copied to clipboard

Zixin Tang, Janet Van Hell

Challenge:	a study of second language learners with different native language backgrounds shows that people distribute information evenly in language production.
Approach:	They compare essays written by second language learners with different native language backgrounds to examine how they distribute information in non-native L2 production.
Outcome:	The authors found that writers with higher L2 proficiency can reduce uncertainty of language production while still conveying informative content.

Defending Against Social Engineering Attacks in the Age of LLMs (2024.emnlp-main)

Copied to clipboard

Lin Ai, Tharindu Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael Davinroy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, Arslan Basharat, Anthony Hoogs, Joshua Garland, Huan Liu, Julia Hirschberg

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models (2024.emnlp-main)

Challenge:	Existing research has developed frameworks to understand human-to-human CSE attacks.
Approach:	They propose a modular defense pipeline that improves detection at both the message and conversation levels.
Outcome:	The proposed model can be exploited to facilitate chat-based social engineering attacks and generate high-quality CSE content, but their detection capabilities are suboptimal, leading to increased operational costs for defense.

Copied to clipboard

Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training (2024.emnlp-main)

Challenge:	federated fine-tuning of ODFMs is limited due to their limited size and system heterogeneity . emerging foundation models (FMs) have remarkable zero/few shot learning capabilities .
Approach:	They propose a federated fine-tuning method that leverages system and data heterogeneity at the edge.
Outcome:	a proposed method for federated fine-tuning improves performance on ODFMs . it allows heterogeneous LoRA ranks across clients for their individual system resources .

Copied to clipboard

Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Target-Aware Language Modeling via Granular Data Sampling (2024.emnlp-main)

Challenge:	Existing speculative decoding methods require additional model structure and training processes to assist the model for draft token generation.
Approach:	They propose a make some noise training framework that introduces some noise at the input for the model to learn the denoising task.
Outcome:	The proposed model improves inference speed by 2.3-2.7x times without compromising model performance.

Copied to clipboard

Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra

SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness (2024.emnlp-main)

Challenge:	Language model pretraining is the cornerstone of universal language models (LMs), creating generalpurpose representations to excel across a variety of downstream tasks.
Approach:	They propose to use multi-granular tokens to sample large-scale language models for domain-specific use cases.
Outcome:	The proposed model outperforms random sampled samples on eight benchmarks with 1% of the data and performs on par with the full RefinedWeb data.

Copied to clipboard

Tanmay Parekh, Jeffrey Kwan, Jiarui Yu, Sparsh Johri, Hyosang Ahn, Sreya Muppalla, Kai-Wei Chang, Wei Wang, Nanyun Peng

CoGen: Learning from Feedback with Coupled Comprehension and Generation (2024.emnlp-main)

Challenge:	Prior studies focused on English posts to provide early warnings for epidemic prediction, but these work focused on non-English posts.
Approach:	They propose a multilingual event extraction framework for extracting epidemic event information for any disease and language using 5.1K tweets in four languages.
Outcome:	The proposed framework can provide epidemic warnings for COVID-19 in its earliest stages in Dec 2019 (3 weeks before global discussions) and aggregate community epidemic discussions like symptoms and cure measures, aiding misinformation detection and public attention monitoring.

Copied to clipboard

Mustafa Omer Gul, Yoav Artzi

Challenge:	Existing studies on coupling comprehension and generation in computational systems show that the ability to finish incomplete partner utterances in dialogue is closely related to comprehension and vice versa.
Approach:	They propose techniques to tightly integrate comprehension and generation capabilities with focus on continually learning from interaction with users.
Outcome:	The proposed models improve performance by 26% and 17% over time, while the non-coupled system is more human-like.

UNICORN: A Unified Causal Video-Oriented Language-Modeling Framework for Temporal Video-Language Tasks (2024.emnlp-main)

Copied to clipboard

Yuanhao Xiong, Yixin Nie, Haotian Liu, Boxin Wang, Jun Chen, Rong Jin, Cho-Jui Hsieh, Lorenzo Torresani, Jie Lei

Story Morals: Surfacing value-driven narrative schemas using large language models (2024.emnlp-main)

Challenge:	Recent advances in large multimodal models have encouraged the development of large multi-modal models . however, it is unclear how to extend these models to the more complex video domain .
Approach:	They propose a visual instruction tuning framework to address temporal video-language tasks . they collect a dataset and fine-tune the framework on instruction-following data .
Outcome:	The proposed model can perform better on established temporal video-language tasks without training objectives and intensive pre-training.

Copied to clipboard

David Hobson, Haiqi Zhou, Derek Ruths, Andrew Piper

OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants (2024.emnlp-main)

Challenge:	Using large language models, we extract and validate story morals across a diverse set of narrative genres.
Approach:	They propose a task of narrative schema labelling based on the concept of "story morals" they use large language models to extract and validate story morals across a diverse set of genres .
Outcome:	The proposed method extracts and validates story morals across folktales, novels, movies and TV, personal stories from social media and the news using automated metrics and human assessments.

Copied to clipboard

Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn, Laura Petry, Olga Koumoundouros, Jayne Bottarini, Peichen Liu, Eric Rice, Swabha Swayamdipta

AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies (2024.emnlp-main)

Challenge:	a large-scale analysis of millions of tweets on homelessness is challenging to understand at scale.
Approach:	They propose a framing typology: Online Attitudes Towards Homelessness (OATH) They use large language models to analyze millions of tweets to find patterns in public attitudes .
Outcome:	The proposed model speeds up annotations while incurring a 3 point performance reduction compared to existing classifiers .

Copied to clipboard

Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Murari Tiyyala, Nicholas Andrews, Daniel Khashabi

SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents (2024.emnlp-main)

Challenge:	Analogical reasoning is an important part of human communication, says a new study . a benchmark to determine analogical reasoning ability in language models is needed .
Approach:	They propose to benchmark analogical reasoning ability in language models by collecting 340 analogies from human writings.
Outcome:	The proposed benchmark aims to determine analogical reasoning ability in language models.

Copied to clipboard

Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Latecki, Eduard Dragut

Analysis of Plan-based Retrieval for Grounded Text Generation (2024.emnlp-main)

Challenge:	Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data.
Approach:	They propose to use a scientific entity and relation extraction dataset to capture interactions between entities in full texts.
Outcome:	The proposed dataset captures the intricate use and interactions among entities in full texts and provides an out-of-distribution test set to offer a more realistic evaluation.

Copied to clipboard

Ameya Godbole, Nicholas Monath, Seungyeon Kim, Ankit Singh Rawat, Andrew McCallum, Manzil Zaheer

Detecting Errors through Ensembling Prompts (DEEP): An End-to-End LLM Framework for Detecting Factual Errors (2024.emnlp-main)

Challenge:	Large, parametric language models (LLMs) produce fluent text for many applications . hallucinations are generation of text that is factually correct and semantically plausible .
Approach:	They propose to use learning-tuned LLMs to infuse models with retrieval mechanisms to reduce hallucinations.
Outcome:	The proposed approach reduces the frequency of hallucinations by reducing the coverage of relevant facts and generating more informative responses while providing higher attribution rates.

Copied to clipboard

Alex Chandler, Devesh Surve, Hui Su

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs (2024.emnlp-main)

Challenge:	Existing methods for detecting factual errors in text summarization are inadequate for the task.
Approach:	They propose an end-to-end large language model framework for detecting factual errors in text summarization.
Outcome:	The proposed framework achieves state-of-the-art (SOTA) balanced accuracy on the AggreFact-XSUM FTSOTA, TofuEval Summary-Level, and HaluEVAL Summarization benchmarks.

Copied to clipboard

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, Sara Hooker

Boosting Logical Fallacy Reasoning in LLMs via Logical Structure Tree (2024.emnlp-main)

Challenge:	Preference optimization is a widely adopted post-training technique to align large language models with human preferences.
Approach:	They propose a method for generating multilingual feedback data to balance data coverage.
Outcome:	The proposed method achieves 54.4% win-rate against current state-of-the-art multilingual LLM in its parameter class and 69.5% win- rate or higher against widely used models like Gemma, Mistral and Llama 3.

Copied to clipboard

Yuanyuan Lei, Ruihong Huang

Chain and Causal Attention for Efficient Entity Tracking (2024.emnlp-main)

Challenge:	Logical fallacy is the use of invalid or flawed reasoning in the construction of a statement.
Approach:	They propose to build a logical structure tree to represent hierarchical logic flow among relation connectives and their arguments in a statement.
Outcome:	The proposed model significantly improves accuracy and recall for fallacy detection and fallacy classification.

Copied to clipboard

Erwan Fagnou, Paul Caillon, Blaise Delattre, Alexandre Allauzen

BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models (2024.emnlp-main)

Challenge:	Existing approaches to handle entity tracking require at least log2 (n+1) layers to handle n state changes.
Approach:	They propose an efficient enhancement to the standard attention mechanism to handle long-term dependencies with a single layer.
Outcome:	The proposed model can handle entity tracking with n state changes with a single layer.

Copied to clipboard

Yi Zeng, Weiyu Sun, Tran Huynh, Dawn Song, Bo Li, Ruoxi Jia

A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution (2024.emnlp-main)

Challenge:	Safety backdoors in large language models can be triggered while evading detection during normal interactions.
Approach:	They propose a bi-level optimization method that uses a key insight: backdoor triggers induce a uniform drift in the model’s embedding space . inner level identifies universal perturbations to the decoder’s embedded spaces that steer the model towards defender-defined unwanted behaviors; outer level fine-tunes the model to reinforce safe behaviors against these perturbations.
Outcome:	The proposed mitigation method reduces the success rate of safety backdoor attacks from over 95% to 1% for general harmful behaviors and from 47% to 0% for Sleeper Agents, without compromising the model’s usefulness.

Copied to clipboard

Zhengmian Hu, Tong Zheng, Heng Huang

FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition (2024.emnlp-main)

Challenge:	Authorship attribution relies on manual features and fails to capture long-range correlations, limiting their effectiveness.
Approach:	They propose to use Bayesian methods to calculate the probability that a text entails previous writings of an author.
Outcome:	The proposed model can achieve 85% accuracy on the IMDb and blog datasets.

Copied to clipboard

Xiaoqiang Wang, Lingfei Wu, Tengfei Ma, Bang Liu

OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation (2024.emnlp-main)

Challenge:	Large language models (LLMs) are evaluated by overall performance on various text understanding and generation tasks.
Approach:	They propose a framework for Fine-grAined and Cognition-grounded LLMs’ Capability Evaluation that dissociates the language-related capabilities from cognition-related ones.
Outcome:	The proposed framework dissociates the language-related capabilities from cognition-related ones and breaks down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems.

Copied to clipboard

Tanvir Mahmud, Diana Marculescu

Language Concept Erasure for Language-invariant Dense Retrieval (2024.emnlp-main)

Challenge:	Existing methods for audio separation are limited due to over-separation, under-serparation and dependence on predefined training sources.
Approach:	They propose a framework that leverages large language models (LLMs) for automated audio separation, eliminating the need for manual intervention and overcoming source limitations.
Outcome:	The proposed framework outperforms existing methods in separating new, unseen, and variable sources in real-world mixtures, and is available on github.

Copied to clipboard

Zhiqi Huang, Puxuan Yu, Shauli Ravfogel, James Allan

Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)

Challenge:	Multilingual models aim for language-invariant representations but still encode language identity.
Approach:	They propose a multi-task learning framework that induces language invariance in multilingual retrieval by reducing language-specific signals in the embedding space.
Outcome:	The proposed learning framework improves language-invariant dense retrieval over baselines on English retrieval data and general multilingual corpora.

Copied to clipboard

Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, Yuandong Tian

Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks (2024.emnlp-main)

Challenge:	Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences.
Approach:	They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring.
Outcome:	The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers.

Copied to clipboard

Yue Zhou, Henry Zou, Barbara Di Eugenio, Yang Zhang

Turn Waste into Worth: Rectifying Top-k Router of MoE (2024.emnlp-main)

Challenge:	Existing research has shown that large language models have difficulty discerning the veracity of their intrinsic answers.
Approach:	They propose a jailbreak attack method that generates an aligned language model for malicious output.
Outcome:	The proposed method achieves competitive performance with more harmful outputs.

Copied to clipboard

Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, Xipeng Qiu

Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination (2024.emnlp-main)

Challenge:	Top-k router suffers from redundancy computation and memory costs due to unbalanced routing . some experts are overflow, where exceeding tokens are dropped, while others are empty, which are padded with zeros, negatively impacting model performance.
Approach:	They propose a top-k router that is unbalanced and uses a multi-gPU system to handle dropped tokens and padding.
Outcome:	The proposed model surpasses the top-1 router by 4.7% in terms of performance . the top-k router suffers from redundancy computation and memory costs .

Copied to clipboard

Pittawat Taveekitworachai, Febri Abdullah, Ruck Thawonmas

CommVQA: Situating Visual Question Answering in Communicative Contexts (2024.emnlp-main)

Challenge:	a series of investigations into an interesting phenomenon where performance increases in large language models when providing a prompt that causes and exploits hallucination.
Approach:	They propose a null-shot prompting approach that intentionally instructs LLMs to look at and utilize information from a nil section.
Outcome:	The proposed approach causes and exploits hallucination in large language models on a range of tasks including arithmetic reasoning, commonsense reasoning, and reading comprehension.

Copied to clipboard

Nandita Naik, Christopher Potts, Elisa Kreiss

Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding (2024.emnlp-main)

Challenge:	Current visual question answering models are trained on image-question pairs in isolation, but the questions people ask are dependent on their informational needs and prior knowledge about the image content.
Approach:	They propose a visual question-answer-as-question dataset that contains 1000 images and 8,949 question-announcer pairs to evaluate how situating images within naturalistic contexts shapes visual questions.
Outcome:	The proposed dataset contains 1000 images and 8,949 question-answer pairs.

Copied to clipboard

Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang, Kaihuo Zhang, Zhiyuan Liu, Maosong Sun

Challenge:	Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) drafting efficiency has become a bottleneck in the final speedup of speculative drafting, therefore generating longer drafts at less cost can lead to better speedup.
Approach:	They propose a method that uses existing model to drafting and target LLM to verify draft in a low-cost parallel manner.
Outcome:	The proposed method can achieve speedups of up to 2.4 over speculative decoding and 3.9 over vanilla decoding without fine-tuning draft and target models.

1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators? (2024.emnlp-main)

Copied to clipboard

Yue Huang, Chenrui Fan, Yuan Li, Siyuan Wu, Tianyi Zhou, Xiangliang Zhang, Lichao Sun

How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have been recognized for their impressive capabilities in natural language processing (NLP).
Approach:	They propose a method to enhance the multilingual performance of Large Language Models by aggregating knowledge from diverse languages.
Outcome:	The proposed method reduces the performance disparity across languages and offers valuable insights for further exploration.

Copied to clipboard

Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, Vasant Honavar

Style-Specific Neurons for Steering LLMs in Text Style Transfer (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning large language models are not suitable for task-dependent tasks.
Approach:	They propose a generalized self-imitation learning framework which aligns large language models with offline demonstration data.
Outcome:	The proposed framework outperforms baselines in many challenging benchmarks . it is available on github.com/tengxiao1/GSIL .

Copied to clipboard

Wen Lai, Viktor Hangya, Alexander Fraser

Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers (2024.emnlp-main)

Challenge:	Existing LLMs tend to prioritize preserving original meaning over enhancing stylistic differences in TST.
Approach:	They propose a novel approach to steering LLMs using style-specific neurons in TST.
Outcome:	Empirical results show that the proposed method improves the fluency of the generated text.

Copied to clipboard

Tianhua Zhang, Kun Li, Hongyin Luo, Xixin Wu, James Glass, Helen Meng

Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction (2024.emnlp-main)

Challenge:	Existing methods to incorporate retriever’s preference during the training of query rewriting models rely on extensive annotations such as in-domain rewrites and/or relevant passage labels, limiting their generalization and adaptation capabilities.
Approach:	They propose a framework for training query rewriting models with limited rewrite annotations from seed datasets and completely no passage label.
Outcome:	The proposed approach decontexualizes conversational queries into self-contained questions suitable for off-the-shelf retrievers.

Copied to clipboard

Sizhe Zhou, Yu Meng, Bowen Jin, Jiawei Han

DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models (2024.emnlp-main)

Challenge:	Existing Relation extraction models require extensive annotated training data, which is costly and labor-intensive to collect.
Approach:	They propose a new zero-shot RE task where only relation definitions are provided instead of seen-unseen relation instances.
Outcome:	The proposed task significantly improves cost-effective zero-shot performance by large margins.

Copied to clipboard

Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu

Leveraging Context-Aware Prompting for Commit Message Generation (2024.emnlp-main)

Challenge:	DA-Code is a code generation benchmark designed to assess LLMs on agent-based data science tasks.
Approach:	They propose a code generation benchmark specifically designed for LLMs on agent-based data science tasks.
Outcome:	The benchmark performs better than existing frameworks, but lacks accuracy . it is based on real-world data, and includes examples that cover a wide range of tasks .

Copied to clipboard

Zhihua Jiang, Jianwei Chen, Dongning Rao, Guanghui Ye

Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination (2024.emnlp-main)

Challenge:	Existing methods for writing comprehensive commit messages focus on the changed lines or nearest context lines, but excessive contexts can lead to noise.
Approach:	They propose a code model COMMIT that can generate automatic commit messages by combining a dataset with a context-aware prompt.
Outcome:	The proposed model surpasses all existing models including pre-trained language models for code and large language models such as Code-LlaMa.

Copied to clipboard

Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, Dan Klein

Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning (2024.emnlp-main)

Challenge:	a large-scale study of linguistic bias exhibited by ChatGPT covers 10 dialects of English . standard varieties of English, especially SAE, dominate available training data .
Approach:	They use ChatGPT to generate models that default to "standard" varieties of English . they also use a feature annotation and native speaker evaluation to analyze the responses .
Outcome:	The proposed models default to "standard" varieties of English, but non-"standard" ones exhibit stereotyping, demeaning content, lack of comprehension, condescending responses.

Copied to clipboard

Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, Hui Xue’

A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models (2024.emnlp-main)

Challenge:	Existing methods to correct outdated or erroneous knowledge in large language models (LLMs) are slow and cumbersome, resulting in catastrophic knowledge forgetting and degradation of model performance.
Approach:	They propose a RetriEval-augmented ContInuous Prompt lEarning method that converts knowledge statements into short and informative continuous prompts, prefixed to the LLM’s input query embedding.
Outcome:	The proposed method improves the performance of large language models (LLMs) while maintaining the overall performance of the model.

Copied to clipboard

Zhihao Wang, Shiyu Liu, Jianheng Huang, Wang Zheng, YiXuan Liao, Xiaoxin Chen, Junfeng Yao, Jinsong Su

Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages (2024.emnlp-main)

Challenge:	Version updates are an indispensable requirement for Large Language Models . a large learning rate in the first stage and a complete learning decay process are crucial for version updates of LLMs.
Approach:	They propose a learning rate path switching training paradigm for version updates of Large Language Models.
Outcome:	The proposed paradigm reduces training cost to 58% when training four versions of LLMs compared to PTFS and CPT .

Copied to clipboard

Jimin Sohn, Haeji Jung, Alex Cheng, Jooeon Kang, Yilin Du, David Mortensen

An Analysis and Mitigation of the Reversal Curse (2024.emnlp-main)

Challenge:	Existing zero-shot cross-lingual NER approaches require substantial prior knowledge of the target language, which is impractical for low-resource languages.
Approach:	They propose a phonemic representation based on the International Phonetic Alphabet (IPA) to bridge the gap between representations of different languages.
Outcome:	The proposed method outperforms baseline models in low-resource languages with highest average F1 score and lowest standard deviation.

Copied to clipboard

Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, Rui Yan

Exploring the Practicality of Generative Retrieval on Dynamic Corpora (2024.emnlp-main)

Challenge:	Recent research observes a phenomenon in large language models called the "reversal curse" when dealing with two entities, LLMs excel in handling sequences in the form of "aRb" but when asked "who is Mary Lee Pfeiffer's son?" the LLM exhibits considerable confusion and fails to provide a as the answer .
Approach:	They conduct the first-ever study of how the reversal curse happens in large language models . they find that LLMs excel in handling sequences in the form of "aRb" but struggle to provide a satisfactory answer when asked "who is Mary Lee Pfeiffer's son?"
Outcome:	The proposed study shows that the reversal curse can stem from specific training objectives . the study also shows that a reverse query can be difficult to understand .

Copied to clipboard

Chaeeun Kim, Soyoung Yoon, Hyunji Lee, Joel Jang, Sohee Yang, Minjoon Seo

OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting (2024.emnlp-main)

Challenge:	a lack of comprehensive comparison between GR and Dual Encoders in IR systems is limiting . GR is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6) and storage footprint (x4)
Approach:	They propose to use autoregressive language models to perform information retrieval (IR) their results highlight the potential of GR for future use in practical IR systems .
Outcome:	The proposed model is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, efficient inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to the most common model, Dual Encoder (DE).

Copied to clipboard

Xukai Liu, Ye Liu, Kai Zhang, Kehang Wang, Qi Liu, Enhong Chen

Don’t Just Say “I don’t know”! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations (2024.emnlp-main)

Challenge:	Entity Linking (EL) is the process of associating ambiguous textual mentions to specific entities in a knowledge base.
Approach:	They propose a framework that utilizes the few-shot learning capabilities of Large Language Models without the need for fine-tuning to improve the accuracy of EL.
Outcome:	The framework outperforms current state-of-the-art methods in a few-shot entity linking task.

Copied to clipboard

Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, Tat-Seng Chua

Fewer is More: Boosting Math Reasoning with Reinforced Context Pruning (2024.emnlp-main)

Challenge:	Existing studies investigate ways to refuse to answer unknown questions . Large Language Models (LLMs) display a significant level of overconfidence when answering questions that they are aware of.
Approach:	They propose a self-alignment method to utilize Large Language Models to enhance its response-ability to unknown questions.
Outcome:	The proposed method is superior to baseline methods on four types of unknown questions.

Copied to clipboard

Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Fan Yang, Mao Yang

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown impressive capabilities, yet they struggle with math reasoning.
Approach:	They propose a coarse-to-fine pruner that prunes unimportant tokens to fit the context window.
Outcome:	The proposed approach outperforms prompting baselines across various LLMs and 5 math datasets and achieves 4.55% absolute improvements without any fine-tuning.

Copied to clipboard

Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David Clifton

Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction (2024.emnlp-main)

Challenge:	Existing studies focus on evaluating large language models in close-ended QA tasks, but many clinical decisions involve answering open-ended questions without pre-set options.
Approach:	They construct a benchmark to better understand large language models in the clinic . they use existing datasets to evaluate LLMs in clinical situations .
Outcome:	The proposed model outperforms human experts in multiple medical tasks.

Copied to clipboard

Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, Songlin Hu

Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective (2024.emnlp-main)

Challenge:	Existing approaches focus on improving attack success rates while overlooking the need for comprehensive test case coverage.
Approach:	They propose a top-down approach to automated red teaming that scales up the diversity of test cases using an extensible, fine-grained risk taxonomy.
Outcome:	The proposed approach scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy and leverages reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner.

Copied to clipboard

Van-Cuong Pham, Thien Nguyen

DynamicER: Resolving Emerging Mentions to Dynamic Entities for RAG (2024.emnlp-main)

Challenge:	Existing methods to edit LLMs' activations are limited by their magnitude and direction consistency.
Approach:	They propose a method that edits activations to alter their magnitudes and directions to preserve activation norms.
Outcome:	The proposed method preserves activation norm and improves safety benchmarks.

Copied to clipboard

Jinyoung Kim, Dayoon Ko, Gunhee Kim

Knowledge Representation

Challenge:	Existing entity linking models struggle to link new expressions to entities in the dynamic nature of human language.
Approach:	They propose a task to resolve emerging mentions to dynamic entities and a benchmark to evaluate their model's adaptability to new expressions.
Outcome:	The proposed method outperforms baselines on QA task with resolved mentions and improves retrieval-augmented generation performance.

Preserving Generalization of Language models in Few-shot Continual Relation Extraction (2024.emnlp-main)

Copied to clipboard

Quyen Tran, Nguyen Thanh, Nguyen Anh, Nam Hai, Trung Le, Linh Ngo, Thien Nguyen

A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)

Challenge:	Existing methods for Few-shot Continual Relations Extraction (FCRE) are limited in labeled training data and models must learn from a few new samples to solve new tasks.
Approach:	They propose a method that leverages often-discarded language model heads to integrate knowledge from new relations with limited labeled data while avoiding catastrophic forgetting.
Outcome:	The proposed method circumvents catastrophic forgetting and preserves prior knowledge from pre-trained backbones while maintaining accuracy of existing classifications.

Copied to clipboard

Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang

Consecutive Batch Model Editing with HooK Layers (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains.
Approach:	They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks .
Outcome:	The proposed evaluations are reproducible, reliable, and robust.

Copied to clipboard

Shuaiyi Li, Yang Deng, Deng Cai, Hongyuan Lu, Liang Chen, Wai Lam

Challenge:	Existing models that retrain are time- and resource-consuming, but they lack the memory to support sequential and batch editing.
Approach:	They propose a model editing method that supports sequential and batch editing . they use a small amount of memory to store several hook layers that remain unchanged over time .
Outcome:	The proposed method is memory-friendly and can store hook layers that remain unchanged over time.

Topic-Oriented Open Relation Extraction with A Priori Seed Generation (2024.emnlp-main)

Copied to clipboard

Linyi Ding, Jinfeng Xiao, Sizhe Zhou, Chaoqi Yang, Jiawei Han

Xiangci Li, Jessica Ouyang

Challenge:	Existing methods for open relation extraction give sub-optimal results on specific topics.
Approach:	They propose a method that leverages the built-in knowledge of large language models to maintain a dynamic seed relation dictionary for the topic.
Outcome:	The proposed approach empowers better topic-oriented control over the generated relations and improves ORE performance along the five dimensions, especially on specialized and narrow topics.

Copied to clipboard

Curriculum Consistency Learning for Conditional Sentence Generation (2024.emnlp-main)

Challenge:	Academic research paper authors must perform literature review to compare work with prior work . authors must compose coherent story that connects prior work and current work based on author's understanding of field .
Approach:	They propose to use automatic related work generation (RWG) to generate papers . authors summarize key approaches and define tasks in a zoo of historical works .
Outcome:	a new study summarises key approaches and defines the tasks and discusses the challenges of RWG.

Copied to clipboard

Liangxin Liu, Xuebo Liu, Lian Lian, Shengjun Cheng, Jun Rao, Tengfei Yu, Hexuan Deng, Min Zhang

A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)

Challenge:	Consistency learning (CL) has proven to be a valuable technique for improving the robustness of conditional sentence generation models.
Approach:	They propose a strategy that guides models to learn consistency in alignment with their current capacity to differentiate between features.
Outcome:	The proposed strategy delivers +2.0 accuracy point improvement compared with vanilla IT and +0.7 COMET scores over traditional CL methods in MT tasks.

Copied to clipboard

Leonardo Bertolazzi, Albert Gatt, Raffaella Bernardi

Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision (2024.emnlp-main)

Challenge:	syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences.
Approach:	They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results.
Outcome:	The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning.

Copied to clipboard

Fan Jiang, Tom Drummond, Trevor Cohn

MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages (2024.emnlp-main)

Challenge:	Cross-lingual open domain question answering requires multiple models, requiring substantial annotated datasets and auxiliary resources to bridge between languages.
Approach:	They propose a selfsupervised method that exploits Wikipedia's cross-lingual link structure . they show that the method outperforms comparable methods on supervised and zero-shot settings .
Outcome:	The proposed method outperforms comparable methods on supervised and zero-shot language adaptation settings.

Copied to clipboard

Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri

Improving Knowledge Graph Completion with Structure-Aware Supervised Contrastive Learning (2024.emnlp-main)

Challenge:	Existing speech FMs fall short of full compliance with open-source principles . existing models do not have model weights, code, and training data publicly available .
Approach:	They propose to use a CC-BY license to create open-source speech FMs for EU languages . they collect suitable training data by surveying automatic speech recognition datasets .
Outcome:	The proposed model can be used in the 24 official languages of the European Union.

Copied to clipboard

Jiashi Lin, Lifang Wang, Xinyu Lu, Zhongtian Hu, Wei Zhang, Wenxuan Lu

Contribution of Linguistic Typology to Universal Dependency Parsing: An Empirical Investigation (2024.emnlp-main)

Challenge:	Existing contrastive methods focus on individual triples, overlooking the broader structural connectivities and topologies of KGs.
Approach:	They propose a new contrastive learning framework that incorporates four tasks specifically tailored to KG data: Vertex-level CL, Neighbor-level Cl, Path-levelCL, and Relation composition level CL.
Outcome:	The proposed framework achieves SOTA performance under standard supervised and low-resource settings.

Copied to clipboard

Ali Basirat, Navid Hemmati

TRoTR: A Framework for Evaluating the Re-contextualization of Text Reuse (2024.emnlp-main)

Challenge:	UD is a global initiative to create a standard annotation for the dependency syntax of human languages.
Approach:	They propose a typologically motivated transformation of UD that emphasizes information packaging over lexical semantics.
Outcome:	The proposed scheme differs from previous attempts to create a universal annotation for human languages.

Copied to clipboard

Francesco Periti, Pierluigi Cassotti, Stefano Montanelli, Nina Tahmasebi, Dominik Schlechtweg

Challenge:	Existing methods for detecting text reuse focus on recontextualization . current approaches focus on text reuse across a diachronic corpus .
Approach:	They propose a framework that relies on topic relatedness for evaluating the diachronic change of context in which text is reused.
Outcome:	The proposed framework evaluates biblical text reuse human-annotated with topic relatedness . it exhibits greater sensitivity to textual similarity than topic relatedity, the authors show .

Structured Optimal Brain Pruning for Large Language Models (2024.emnlp-main)

Copied to clipboard

Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, Yong Liu

Automatically Generated Definitions and their utility for Modeling Word Meaning (2024.emnlp-main)

Challenge:	Existing pruning methods for Large Language Models rely on unstructured pruning or require special hardware to accelerate computation.
Approach:	They propose a retraining-free structured pruning method called SoBP . they evaluate the effectiveness of SoBP across 14 models from 3 LLM families .
Outcome:	The proposed method outperforms current state-of-the-art pruning methods on 8 datasets.

Copied to clipboard

Francesco Periti, David Alfter, Nina Tahmasebi

Challenge:	Modern language models generate semantic representations for words based on context and context based models.
Approach:	They propose to use dictionary-like sense definitions to generate sentence embeddings . they evaluate the quality of the generated definitions on existing English benchmarks based on the results of their study .
Outcome:	The proposed model sets new state-of-the-art results on lexical semantics tasks compared to baselines .

How Do Your Code LLMs perform? Empowering Code Instruction Tuning with Really Good Data (2024.emnlp-main)

Copied to clipboard

Yejie Wang, Keqing He, Dayuan Fu, Zhuoma GongQue, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang, Xunliang Cai, Weiran Xu

MAIR: A Massive Benchmark for Evaluating Instructed Retrieval (2024.emnlp-main)

Challenge:	Recent research has shown that code pre-trained models improve coding capabilities.
Approach:	They propose a code data pruning strategy to identify which datasets are high-quality code instruction data.
Outcome:	The proposed model achieves state-of-the-art performance using fewer training data.

Copied to clipboard

Weiwei Sun, Zhengliang Shi, Wu Long, Lingyong Yan, Xinyu Ma, Yiding Liu, Min Cao, Dawei Yin, Zhaochun Ren

Rethinking the Evaluation of In-Context Learning for LLMs (2024.emnlp-main)

Challenge:	Existing IR benchmarks focus on a limited scope of tasks, making them insufficient for evaluating the latest IR models.
Approach:	They propose a multi-task instruction-tuned IR benchmark that includes 126 distinct IR tasks across 6 domains.
Outcome:	The proposed model performs better on instruction-tuned models than non-instruction-tunned models on MAIR.

Copied to clipboard

Guoxin Yu, Lemao Liu, Mo Yu, Yue Yu, Xiang Ao

Cluster-Norm for Unsupervised Probing of Knowledge (2024.emnlp-main)

Challenge:	Existing studies evaluate In-context learning methods based on task performance . however, this evaluation protocol overlooks the significant cost associated with the demonstration configuration process .
Approach:	They propose a two-dimensional evaluation paradigm that considers both configuration costs and task performance.
Outcome:	The proposed evaluation paradigm can be applied to any ICL method as a plugin.

Copied to clipboard

Walter Laurito, Sharan Maiya, Grégoire Dhimoïla, Owen Yeung, Kaarel Hänni

Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries (2024.emnlp-main)

Challenge:	Empirical evidence suggests that simulated internal beliefs or knowledge can be extracted from language models but such methods require labels, which in some domains may not be readily provided due to human biases or because humans simply do not know the correct label.
Approach:	They propose a method to minimize the impact of unrelated features in activation space by clustering and normalizing activations of contrast pairs before applying unsupervised probing techniques.
Outcome:	Empirical evidence suggests that simulated internal beliefs or knowledge can be extracted from language model activations without human labels.

Copied to clipboard

Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson

Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration (2024.emnlp-main)

Challenge:	Large language models (LLMs) can solve complex multi-step problems, but little is known about how these computations are implemented internally.
Approach:	They propose a "back-patching" analysis method to solve multi-hop queries . they propose resolving the bridge entity into the bridge and the second hop into the target entity into latent steps.
Outcome:	The proposed method solves multi-hop queries that require two information extraction steps . it shows that the later layers lack the necessary knowledge to correctly generate the answer .

Copied to clipboard

Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng

Where am I? Large Language Models Wandering between Semantics and Structures in Long Contexts (2024.emnlp-main)

Challenge:	Large language models (LLMs) are difficult to interpret due to their black-box nature and randomness.
Approach:	They propose a new method which enhances influence functions by addressing fitting errors by eliminating knowledge bias present in the base model before fine-tuning.
Outcome:	The proposed method outperforms existing methods and achieves an average AUC of 91.64%.

Copied to clipboard

Seonmin Koo, Jinsung Kim, YoungJoon Jang, Chanjun Park, Heuiseok Lim

KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students (2024.emnlp-main)

Challenge:	Existing evaluations of the open-domain question answering task focus solely on whether the model provides the correct answer.
Approach:	They propose to examine the phenomenon of discrepancies in abilities across two distinct tasks—QA and evidence selection—when performed simultaneously.
Outcome:	The proposed framework and resources examines the ability of large language models to perform two distinct tasks simultaneously, from the perspective of task alignment.

Copied to clipboard

Matthew Shu, Nishant Balepur, Shi Feng, Jordan Boyd-Graber

Large Language Models Can Be Contextual Privacy Protection Learners (2024.emnlp-main)

Challenge:	Existing student models use study data like student's past responses to predict the probability a student can recall a flashcard.
Approach:	They propose to use student models to predict recall of flashcards to build a content-aware student model that uses deep knowledge tracing, retrieval, and BERT to predict student recall.
Outcome:	The proposed content-aware student model outperforms existing student models in AUC and calibration error and is more efficient than SOTA.

Copied to clipboard

Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Quanquan Gu, Haifeng Chen, Wei Wang, Wei Cheng

A SMART Mnemonic Sounds like “Glue Tonic”: Mixing LLMs with Student Feedback to Make Mnemonic Learning Stick (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable linguistic comprehension and generation capability, but when applied to specialized industries, they face challenges such as hallucination, insufficient domain knowledge, and failing to incorporate the latest domain knowledge.
Approach:	They propose a paradigm for fine-tuning LLMs that effectively injects domain-specific knowledge while safeguarding inference-time data privacy.
Outcome:	The proposed model protects private data while enhancing the model's knowledge.

Copied to clipboard

Nishant Balepur, Matthew Shu, Alexander Hoyle, Alison Robey, Shi Feng, Seraphina Goldfarb-Tarrant, Jordan Boyd-Graber

Challenge:	a new study shows that mnemonics are not effective at matching student learning to a standardized learning model.
Approach:	They build a keyword mnemonic generator that finds mnemonics students favor in a flashcard app . they use expressed and observed preferences to find out what students think is helpful .
Outcome:	The proposed mnemonics outperform existing models in keyword mnemonics . the human writer outperformed both models in terms of keyword simplicity and explanation quality .

Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models (2024.emnlp-main)

Copied to clipboard

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, Reza Haf

MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction (2024.emnlp-main)

Challenge:	Large language models are fine-tuned on diverse datasets to develop a range of skills . each skill has unique characteristics, and datasets are heterogeneous and imbalanced . a general, model-agnostic, reinforcement learning framework is proposed to optimize data usage .
Approach:	They propose a general, model-agnostic, reinforcement learning framework that optimizes data usage automatically during the fine-tuning process.
Outcome:	The proposed framework optimizes data usage automatically during the fine-tuning process.

Copied to clipboard

Jun-Hyung Park, Yeachan Kim, Mingyu Lee, Hyuntae Park, SangKeun Lee

First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning (2024.emnlp-main)

Challenge:	Existing methods for chemical representation learning often lead to overfitting and limited scalability due to early convergence.
Approach:	They propose a framework to train Transformers on SMILES sequences to learn from structural examples and integrate external materials embedding to enrich molecular representations.
Outcome:	The proposed model outperforms state-of-the-art models on molecular property prediction tasks.

Copied to clipboard

Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Keisuke Sakaguchi, Kentaro Inui

Tools Fail: Detecting Silent Errors in Faulty Tools (2024.emnlp-main)

Challenge:	Explicit multi-step reasoning is widely adopted to improve the performance of language models.
Approach:	They propose a systematic reasoning strategy that LMs use to solve multi-step reasoning tasks.
Outcome:	The proposed strategy improves the performance of language models by combining heuristics with rational strategies.

Copied to clipboard

Jimin Sun, So Yeon Min, Yingshan Chang, Yonatan Bisk

Challenge:	a failure in one tool can trigger a cascade of errors, leading to complete task failure.
Approach:	They propose a framework for tools more broadly which explores a model’s ability to detect “silent” tool errors and reflect on how to plan.
Outcome:	The proposed approach shows that the model can detect "silent" tool errors and plan.

Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity (2024.emnlp-main)

Copied to clipboard

Bowen Zhang, Chunping Li

Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing (2024.emnlp-main)

Challenge:	Semantic Textual Similarity (STS) is a key indicator of the encoding capabilities of embedding models.
Approach:	They propose to use Pearson’s correlation coefficient as a loss function to refine model performance beyond contrastive learning to achieve a Spearman’s ceiling.
Outcome:	The proposed method surpasses state-of-the-art strategies with minimal amount of fine-grained annotated samples.

Copied to clipboard

Deokhyung Kang, Seonjeong Hwang, Yunsu Kim, Gary Lee

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)

Challenge:	Existing approaches to extend semantic parsing (SP) beyond English are challenging due to the complex slot alignment step after translation.
Approach:	They propose a method to enhance cross-lingual transfer for SP by utilizing mPLMs.
Outcome:	The proposed method synthesizes target language utterances from source meaning representations while maintaining high slot value alignment rates.

Copied to clipboard

Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, Arash Eshghi

Are LLMs Good Zero-Shot Fallacy Classifiers? (2024.emnlp-main)

Challenge:	a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval.
Approach:	They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling.
Outcome:	The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension.

Copied to clipboard

Fengjun Pan, Xiaobao Wu, Zongrui Li, Anh Tuan Luu

The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis (2024.emnlp-main)

Challenge:	Existing fallacy classifiers lack sufficient labeled data for training, limiting their out-of-distribution (OOD) generalization abilities.
Approach:	They propose to use Large Language Models (LLMs) for zero-shot fallacy classification.
Outcome:	The proposed schemes outperform existing classifiers in OOD inference scenarios and opendomain tasks.

Copied to clipboard

Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, Yulan He

More DWUGs: Extending and Evaluating Word Usage Graph Datasets in Multiple Languages (2024.emnlp-main)

Challenge:	In-context learning (ICL) is a capability that enables large language models to excel in proficiency through demonstration examples.
Approach:	They present a survey on the interpretation and analysis of in-context learning . they focus on theoretical and empirical perspectives on the concept .
Outcome:	The proposed model can perform tasks with minimal examples without re-training and has demonstrated proficiency across various tasks with a minimal set of task-oriented examples.

Copied to clipboard

Dominik Schlechtweg, Pierluigi Cassotti, Bill Noble, David Alfter, Sabine Schulte Im Walde, Nina Tahmasebi

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)

Challenge:	Word Usage Graphs (WUGs) represent word sense clusters from simple pairwise word use judgments.
Approach:	They propose to use a weighted graph to represent human semantic proximity judgments for pairs of word uses to infer word sense clusters from simple pairwise word use judgments.
Outcome:	The proposed approach can be applied in a Word Sense Induction (WSI) setting or for Word sense disambiguation (WSD) it is the first and to date largest manually annotated, diachronic WUG dataset.

Copied to clipboard

Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, Masashi Sugiyama

ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos (2024.emnlp-main)

Challenge:	Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning.
Approach:	They propose a method to fine-tune CLIP without introducing any overhead of extra parameters.
Outcome:	The proposed method improves CLIP by 7.27% average harmonic mean accuracy.

Copied to clipboard

Arpan Phukan, Manish Gupta, Asif Ekbal

Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation (2024.emnlp-main)

Challenge:	Existing studies on question generation from videos are mostly focused on generating questions about common objects and attributes.
Approach:	They propose a model architecture combining Transformers, rich context signals and a combination of cross-entropy and contrastive loss function to encourage entity-centric question generation.
Outcome:	The proposed system yields BLEU, ROUGE, CIDEr, and METEOR scores of 71.3, 78.6, 7.31, and 81.9.

Copied to clipboard

Elaf Alhazmi, Quan Sheng, Wei Emma Zhang, Munazza Zaib, Ahoud Alhazmi

Challenge:	Objective questions such as fill-in-the-blank and multiple-choice require examinees to select one valid answer from a set of invalid options.
Approach:	They examine distractor generation tasks, datasets, methods, and evaluation metrics for English objective questions.
Outcome:	The proposed task is based on fill-in-the-blank and multiple choice questions and is widely utilized in educational settings across various domains and subjects.

Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG (2024.emnlp-main)

Copied to clipboard

William Merrill, Noah Smith, Yanai Elazar

ASL STEM Wiki: Dataset and Benchmark for Interpreting STEM Articles (2024.emnlp-main)

Challenge:	a new study examines how novel language models generate training text . large LMs and constrained decoding strategies both decrease novelty .
Approach:	They develop a novel search tool inspired by genomic data to find n-grams in training data.
Outcome:	The proposed tool can search for n-grams over a corpus in constant time w.r.t. large LMs and more constrained decoding strategies both decrease novelty.

Copied to clipboard

Kayo Yin, Chinmay Singh, Fyodor Minakov, Vanessa Milan, Hal Daumé Iii, Cyril Zhang, Alex Lu, Danielle Bragg

Can Automatic Metrics Assess High-Quality Translations? (2024.emnlp-main)

Challenge:	Deaf and hard-of-hearing students face significant barriers in accessing STEM education due to the scarcity of STEM resources in signed languages.
Approach:	They develop models to identify fingerspelled words in American Sign Language (ASL) given an English sentence and a video, the model detects which English phrase is fingerspelled in the clip.
Outcome:	ASL STEM Wiki is the first continuous signing dataset focused on STEM . it detects fingerspelled words and queries them for appropriate signs to suggest to interpreters.

Copied to clipboard

Sweta Agrawal, António Farinhas, Ricardo Rei, Andre Martins

Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation (2024.emnlp-main)

Challenge:	a recent human evaluation study found that translations produced by current MT systems achieve very high-quality scores when judged by humans on a direct assessment scale of 0 to 100.
Approach:	They stress-test the ability of current translation quality metrics to detect correct translations . they show that current metrics often over or underestimate translation quality .
Outcome:	The proposed method overestimates translation quality, the authors show . they show that current metrics often overestimate translation quality .

Copied to clipboard

Sweta Agrawal, José De Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno Guerreiro, Andre Martins

DC-Instruct: An Effective Framework for Generative Multi-intent Spoken Language Understanding (2024.emnlp-main)

Challenge:	Existing algorithms for machine translation do not match human preferences, but they can be expensive to obtain and curate at a large scale.
Approach:	They propose an approach that leverages the best of both worlds by collecting sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems.
Outcome:	The proposed approach improves translation quality on WMT23 and FLORES benchmarks.

Copied to clipboard

Bowen Xing, Lizi Liao, Minlie Huang, Ivor Tsang

KnowTuning: Knowledge-aware Fine-tuning for Large Language Models (2024.emnlp-main)

Challenge:	Existing prompt learning frameworks lack explicit modeling of dual-task dependencies and oversight of task-specific semantic differences among utterances.
Approach:	They propose a generative framework based on Dual-task Inter-dependent Instructions (DII) and Supervised Contrastive Instructions that leverages utterance semantics differences by guiding LLMs to determine whether a pair of utterrances share the same or similar labels.
Outcome:	The proposed framework outperforms existing models and state-of-the-art methods on public benchmark datasets and shows that it improves SLU reasoning.

Copied to clipboard

Yougang Lyu, Lingyong Yan, Shuaiqiang Wang, Haibo Shi, Dawei Yin, Pengjie Ren, Zhumin Chen, Maarten Rijke, Zhaochun Ren

SecCoder: Towards Generalizable and Robust Secure Code Generation (2024.emnlp-main)

Challenge:	Large language models (LLMs) are a default solution for many natural language processing tasks.
Approach:	They propose a knowledge-aware fine-tuning method to improve LLMs' knowledge awareness . they propose augmentation and comparison stages to improve accuracy and reliability .
Outcome:	The proposed method generates more facts with less factual error rate under fine-grained facts evaluation.

Copied to clipboard

Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, Jianwei Yin

Nash CoT: Multi-Path Inference with Preference Equilibrium (2024.emnlp-main)

Challenge:	Existing secure code generation methods have limited generalizability to unseen test cases and poor robustness against the attacked model, leading to safety failures in code generation.
Approach:	They propose a generalizable and robust secure code generation method SecCoder by using in-context learning and the safe demonstration.
Outcome:	The proposed method achieves a significant security improvement of 7.20% on unseen test cases and better robustness against the attacked model.

Copied to clipboard

Ziqi Zhang, Cunxiang Wang, Xiao Xiong, Yue Zhang, Donglin Wang

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention (2024.emnlp-main)

Challenge:	Multi-path inference is an improvement on multi-path reasoning, but there is no optimal setting for the number of inference paths.
Approach:	They propose to use question-related role templates to guide LLMs into relevant roles to reduce the dependence on the number of inference paths.
Outcome:	The proposed system can achieve comparable or better results than self-consistency with the same number of paths.

Copied to clipboard

Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou

Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector (2024.emnlp-main)

Challenge:	Existing studies have found that low-rank pre-training often compromises effectiveness.
Approach:	They propose to apply low-dimensional module only to the attention layer to improve both effectiveness and efficiency.
Outcome:	The proposed model saves 12.4% time while improving test perplexity and on downstream tasks compared with vanilla Transformer.

Copied to clipboard

Xiaoxue Cheng, Junyi Li, Xin Zhao, Hongzhi Zhang, Fuzheng Zhang, Di Zhang, Kun Gai, Ji-Rong Wen

Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding (2024.emnlp-main)

Challenge:	Existing studies on hallucination detection rely heavily on closed-source LLMs such as GPT-4.
Approach:	They propose an LLM-based agent framework called HaluAgent that integrates LLMs, multi-functional toolbox and a memory mechanism for hallucination detection.
Outcome:	The proposed framework integrates the LLM, multi-functional toolbox, and can detect hallucinations on Chinese and English datasets.

Copied to clipboard

Wei Li, Zhen Huang, Xinmei Tian, Le Lu, Houqiang Li, Xu Shen, Jieping Ye

LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History (2024.emnlp-main)

Challenge:	Despite promising progress, vision-language models still exhibit significant challenges in understanding visio-linguistic concepts beyond object terms.
Approach:	They propose a framework that encourages the model to pay greater attention to composition words denoting relationships and attributes within the text.
Outcome:	The proposed framework improves the ability to discern intricate details and construct more sophisticated interpretations of combined visual and linguistic elements.

Copied to clipboard

Akash Gupta, Ivaxi Sheth, Vyas Raina, Mark Gales, Mario Fritz

Social Bias Probing: Fairness Benchmarking for Language Models (2024.emnlp-main)

Challenge:	Recent advances in Natural Language Processing (NLP) have led to the widespread deployment of large language models (LLMs) across various applications.
Approach:	They propose to formalize the study of task-switches in conversational LLMs by analyzing conversational history.
Outcome:	The proposed study formalizes and investigates the sensitivity of large language models to taskswitch scenarios in conversational LLMs.

Copied to clipboard

Marta Marchiori Manerba, Karolina Stanczak, Riccardo Guidotti, Isabelle Augenstein

Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models (2024.emnlp-main)

Challenge:	Existing methods for evaluating social biases in language models have been limited to binary association tests on small datasets.
Approach:	They propose a framework for probing language models for social biases by assessing disparate treatment . they use a large-scale benchmark to examine the diversity of identities and stereotypes .
Outcome:	The proposed framework expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes.

Copied to clipboard

Wenhao Yu, Hongming Zhang, Xiaoman Pan, Peixin Cao, Kaixin Ma, Jian Li, Hongwei Wang, Dong Yu

DynaThink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models (2024.emnlp-main)

Challenge:	Standard RALMs often neglect their intrinsic knowledge due to the interference from retrieved information.
Approach:	They propose a new approach to improve robustness of RALMs by generating sequential reading notes for each retrieved document.
Outcome:	The proposed approach outperforms standard RALMs on four open-domain QA benchmarks.

Copied to clipboard

Jiabao Pan, Yan Zhang, Chen Zhang, Zuozhu Liu, Hongwei Wang, Haizhou Li

Revisiting Automated Evaluation for Long-form Table Question Answering (2024.emnlp-main)

Challenge:	Large language models (LLMs) have emerged as prominent foundation models for diverse applications due to their outstanding ability to understand and generate humanlike text.
Approach:	They propose a dynamic decision-making framework that categorizes tasks into two distinct pathways: 'Fast' and 'Slow' they propose 'self-consistency' strategy to replace the straight-forward decoding method used in COT prompting .
Outcome:	The proposed method achieves more than 3% increase in accuracy with lower cost on five popular reasoning benchmarks.

Copied to clipboard

Yuqi Wang, Lyuhao Chen, Songcheng Cai, Zhijian Xu, Yilun Zhao

Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems (2024.emnlp-main)

Challenge:	Existing automated metrics for long-form table question answering (LFTQA) are poorly correlated with human judgments and fail to distinguish between factually accurate responses and those that are factual incorrect.
Approach:	They propose to use a meta-evaluation dataset to assess the effectiveness of LLM-based LFTQA systems.
Outcome:	The proposed meta-evaluation dataset includes 2,988 human-annotated examples.

Copied to clipboard

Italo Silva, Hanqi Yan, Lin Gui, Yulan He

Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoning (2024.emnlp-main)

Challenge:	Existing evaluation metrics that reflect the performance of causal event extraction tasks are poorly reflecting the inherent ambiguity of cause and effect boundaries.
Approach:	They propose to use a weak-to-strong supervision method to train an evaluation model while still achieving high performance in training an RL model.
Outcome:	The proposed method achieves high agreement with human-annotated data while still achieving high performance in training an RL model.

Copied to clipboard

Zhihan Zhang, Tao Ge, Zhenwen Liang, Wenhao Yu, Dian Yu, Mengzhao Jia, Dong Yu, Meng Jiang

FinDVer: Explainable Claim Verification over Long and Hybrid-content Financial Documents (2024.emnlp-main)

Challenge:	Existing studies focus on broadening the training set with data augmentation techniques to maximize such benefits.
Approach:	They propose a method that embeds problem reflection into each training instance.
Outcome:	The proposed method enhances performance in standard and complex scenarios that require reflective thinking.

Copied to clipboard

Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, Arman Cohan

Extracting Prompts by Inverting LLM Outputs (2024.emnlp-main)

Challenge:	FinDVer is a benchmark to evaluate the explainable claim verification capabilities of LLMs . financial documents are typically long, intricate and dense, and they include both quantita and numerical reasoning.
Approach:	They propose a benchmark to evaluate the explainable claim verification capabilities of LLMs . they assess 25 LLM systems under long-context and RAG settings .
Outcome:	The proposed benchmark can be used to evaluate the explainable claim verification capabilities of LLMs in financial documents.

Copied to clipboard

Collin Zhang, John Morris, Vitaly Shmatikov

BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs (2024.emnlp-main)

Challenge:	Unlike previous methods, output2prompt only needs outputs of normal user queries.
Approach:	They propose a black-box method that extracts the model's prompt without accessing its logits and without adversarial or jailbreaking queries.
Outcome:	The proposed method extracts the prompt that generated the outputs without accessing the model's logits and without adversarial or jailbreaking queries.

Copied to clipboard

Zhiting Fan, Ruizhe Chen, Ruiling Xu, Zuozhu Liu

VHASR: A Multimodal Speech Recognition System With Vision Hotwords (2024.emnlp-main)

Challenge:	Existing evaluation approaches rely on fixed-form outputs and cannot adapt to flexible open-text generation scenarios.
Approach:	They propose a plug-and-play tool to detect social bias in open-text LLMs.
Outcome:	Extensive experiments show that BiasAlert outperforms state-of-the-art methods in detecting bias in open-text generation scenarios.

Copied to clipboard

Jiliang Hu, Zuchao Li, Ping Wang, Haojun Ai, Lefei Zhang, Hai Zhao

A Probability–Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors (2024.emnlp-main)

Challenge:	Existing models that incorporate audio-related image information do not improve speech recognition performance.
Approach:	They propose a novel approach utilizing audio-related image information and set up a multimodal speech recognition system that uses vision as hotwords to enhance the model’s speech recognition capability.
Outcome:	The proposed model outperforms unimodal ASR model and achieves SOTA among existing image-based multimodal ASL models.

Copied to clipboard

Naaman Tan, Josef Valvoda, Tianyu Liu, Anej Svete, Yanxia Qin, Min-Yen Kan, Ryan Cotterell

Bridging Local Details and Global Context in Text-Attributed Graphs (2024.emnlp-main)

Challenge:	a relationship exists between the quality of a string and its probability, p(y), under a language model, and the quality and quality of the string.
Approach:	They examine the probability-quality relationship in language models aligned to human preferences through reinforcement learning through human feedback.
Outcome:	The proposed method improves the quality of text sampled from a language model by skewing the model towards high-probability strings.

Copied to clipboard

Yaoke Wang, Yun Zhu, Wenqiao Zhang, Yueting Zhuang, Liyunfei Liyunfei, Siliang Tang

Building Resources for Emakhuwa: Machine Translation and News Classification Benchmarks (2024.emnlp-main)

Challenge:	Existing studies focus on combining different information levels but overlook interconnections, i.e., contextual textual information among nodes.
Approach:	They propose a framework that bridges local and global perspectives by leveraging contextual textual information.
Outcome:	The proposed framework achieves state-of-the-art performance while reducing tokens significantly.

Copied to clipboard

Felermino Ali, Henrique Lopes Cardoso, Rui Sousa-Silva

RepMatch: Quantifying Cross-Instance Similarities in Representation Space (2024.emnlp-main)

Challenge:	Emakhuwa is the most widely spoken language in Mozambique but has received limited attention in NLP research.
Approach:	They propose a comprehensive collection of NLP resources for Emakhuwa, Mozambique's most widely spoken language.
Outcome:	The proposed models show good performance in news topic classification and promising results in machine translation.

Copied to clipboard

Mohammad Modarres, Sina Abbasi, Mohammad Taher Pilehvar

Commonsense Knowledge Editing Based on Free-Text in LLMs (2024.emnlp-main)

Challenge:	Recent advances in dataset analysis have enabled more sophisticated approaches to analyzing and characterizing training data instances.
Approach:	They propose a method that characterizes data through the lens of similarity.
Outcome:	The proposed method can compare datasets, identify more representative subsets, and uncover heuristics underlying the construction of some challenge datasets.

Copied to clipboard

Xiusheng Huang, Yequan Wang, Jun Zhao, Kang Liu

A Closer Look at Multidimensional Online Political Incivility (2024.emnlp-main)

Challenge:	Existing knowledge editing methods focus on editing triple-based facts such as entity-relation pairs and events (multiple triplets).
Approach:	They propose a Knowledge Localization for Free-Text method which uses a Dynamics-aware Module to locate the parameter positions corresponding to commonsense knowledge and a knowledge editing module to update knowledge.
Outcome:	The proposed method exploits the potential of the MLP and Attention layers and edits commonsense knowledge based on free-text.

Copied to clipboard

Sagi Pendzel, Nir Lotan, Alon Zoizner, Einat Minkov

Linguistic Theories

Challenge:	80% of the uncivil tweets are authored by 20% of the users, where users who are politically engaged are more inclined to use uncival language.
Approach:	They analysed 13K political tweets in the U.S. using crowd sourcing and classified them by their respective categories.
Outcome:	The proposed method enables us to characterise the distribution of incivility across users and geopolitical regions.

Leveraging BERT and TFIDF Features for Short Text Clustering via Alignment-Promoting Co-Training (2024.emnlp-main)

Copied to clipboard

Zetong Li, Qinliang Su, Shijing Si, Jianxing Yu

Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation (2024.emnlp-main)

Challenge:	Existing clustering methods rely on keyword information, but they lack this information.
Approach:	They propose a CO-Training C**lustering framework to make use of BERT and TFIDF features.
Outcome:	The proposed framework outperforms existing SOTA methods on eight datasets.

Copied to clipboard

Bar Iluz, Yanai Elazar, Asaf Yehudai, Gabriel Stanovsky

Unsupervised Named Entity Disambiguation for Low Resource Domains (2024.emnlp-main)

Challenge:	In this study, we examine three considerations for intrinsic debiasing in neural machine translation models.
Approach:	They propose to measure the extrinsic bias of neural machine translation models by embedding them in a neural embeddable space and using different tokens to debias them.
Outcome:	The proposed methods over-rely on gender stereotypes and over-represent them in their models.

Copied to clipboard

Debarghya Datta, Soumajit Pramanik

SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)

Challenge:	Existing approaches to Named Entity Disambiguation (NED) are inefficient for domain specific tasks such as searching, question answering and information extraction.
Approach:	They propose a unsupervised approach leveraging the concept of Group Steiner Trees which can identify the most relevant candidate for entity disambiguation using contextual similarities across candidate entities for all the mentions present in a document.
Outcome:	The proposed approach outperforms the state-of-the-art methods by more than 40% in terms of Precision@1 and Hit@5 across various domain-specific datasets.

Copied to clipboard

Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets

MoCoKGC: Momentum Contrast Entity Encoding for Knowledge Graph Completion (2024.emnlp-main)

Challenge:	High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters.
Approach:	They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters.
Outcome:	The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches.

Copied to clipboard

Qingyang Li, Yanru Zhong, Yuchu Qin

ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities (2024.emnlp-main)

Challenge:	Existing approaches to knowledge graph completion have not integrated the structural attributes of knowledge graphs with the textual descriptions of entities to generate robust entity encodings.
Approach:	They propose to integrate structural information from knowledge graphs with textual descriptions of entities to generate robust entity encodings.
Outcome:	The proposed model improves on the standard evaluation metric, Mean Reciprocal Rank (MRR), while surpassing the current best model on the Wikidata5M dataset.

Copied to clipboard

Ying Su, Zhan Ling, Haochen Shi, Cheng Jiayang, Yauwai Yim, Yangqiu Song

Shortcuts Arising from Contrast: Towards Effective and Lightweight Clean-Label Attacks in Prompt-Based Learning (2024.emnlp-main)

Challenge:	Large language models (LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning ability.
Approach:	They propose to evaluate the planning ability of large language models and multi-modal counterfactual vision language models (VLMs) using a multi-factual household activity simulator and a chatGPT task description to evaluate their reasoning ability.
Outcome:	The proposed benchmark evaluates the planning ability of multi-modal and counterfactual vision language models on a household activity simulator and a chatGPT task description.

Copied to clipboard

Xiaopeng Xie, Ming Yan, Xiwen Zhou, Chenlong Zhao, Suli Wang, Yong Zhang, Joey Zhou

GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)

Challenge:	Prompt-based learning paradigms are vulnerable to backdoor attacks, requiring false activations and false data augmentation.
Approach:	They propose a method that uses triggers to create stronger shortcuts by leveraging activation values and data selection strategies to create the shortcuts.
Outcome:	The proposed method is based on the concept that a backdoor acts as a shortcut and can achieve high effectiveness and stealthiness at low poisoning rates.

Copied to clipboard

Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, Virginia Smith

RaTEScore: A Metric for Radiology Report Generation (2024.emnlp-main)

Challenge:	Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads.
Approach:	They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates.
Outcome:	The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements.

Copied to clipboard

Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

HalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning (2024.emnlp-main)

Challenge:	Existing metrics to evaluate the quality of medical reports are limited due to the complexity of clinical free-form texts.
Approach:	They propose a new metric to assess the quality of medical reports generated by AI models.
Outcome:	The proposed metric is based on a medical NER dataset and trained on NER models . it aligns more closely with human preference than existing metrics, the authors show .

Copied to clipboard

Shayan Ali Akbar, Md Mosharaf Hossain, Tess Wood, Si-Chi Chin, Erica Salinas, Victor Alvarez, Erwin Cornejo

Learning to Rank Salient Content for Query-focused Summarization (2024.emnlp-main)

Challenge:	HalluMeasure is a new LLM-based hallucination detection mechanism that decomposes an LLM response into atomic claims and evaluates each claim against the provided reference context.
Approach:	They propose a new LLM-based hallucination detection mechanism that decomposes an LLM response into atomic claims and evaluates each atomic claim against the provided reference context.
Outcome:	The proposed model can detect 3 major categories of hallucinations and 10 more specific subtypes which help to identify reasons behind the hallucinian errors.

Copied to clipboard

Sajad Sotudeh, Nazli Goharian

Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions (2024.emnlp-main)

Challenge:	Query-focused summarization (QFS) is gaining prominence in research community.
Approach:	They propose to integrate Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization.
Outcome:	The proposed model outperforms the state-of-the-art on QMSum benchmark and SQuALITY benchmark while offering a lower training overhead.

Copied to clipboard

Qian Ruan, Ilia Kuznetsov, Iryna Gurevych

LitSearch: A Retrieval Benchmark for Scientific Literature Search (2024.emnlp-main)

Challenge:	Generative large language models (LLMs) have brought advances in text generation, but their potential for enhancing classification tasks remains underexplored.
Approach:	They propose a framework for thoroughly investigating fine-tuning LLMs for classification . they instantiate this framework in edit intent classification (EIC) a challenging and underexplored classification task.
Outcome:	The proposed framework is applied to edit intent classification (EIC) The proposed methods are generalizable on five further classification tasks.

Copied to clipboard

Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao

Open-world Multi-label Text Classification with Extremely Weak Supervision (2024.emnlp-main)

Challenge:	Literature search questions pose significant challenges for modern retrieval systems . a lack of domain expertise and reasoning through lengthy papers is a challenge .
Approach:	They propose a retrieval benchmark for literature search queries using inline citations from papers and questions about recently published papers.
Outcome:	The proposed retrieval benchmarks outperform state-of-the-art retrieval models and reranking pipelines.

Copied to clipboard

Xintong Li, Jinya Jiang, Ria Dharmani, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang

LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law (2024.emnlp-main)

Challenge:	Similar single-label XWS settings cannot be easily adapted for multi-l label classification.
Approach:	They propose a novel method for open-world multi-label text classification under extremely weak supervision where the user provides a brief description without any labels or ground-truth label space.
Outcome:	The proposed method exhibits a remarkable increase in ground-truth label space coverage on various datasets.

Copied to clipboard

Toni Liu, Nicolas Boulle, Raphaël Sarfati, Christopher Earls

Challenge:	LLaMA-2 language model is capable of in-context time series extrapolation without specific prompting or fine-tuning, revealing an in-constitution version of a neural scaling law.
Approach:	They propose an algorithm for extracting probability density functions of multi-digit numbers directly from Large language models (LLMs) LLaMA-2 is a language model trained on text and can extrapolate dynamical system time series without prompting or engineering .
Outcome:	The proposed model can extrapolate dynamical systems without prompting or engineering . it also achieves an in-context version of a neural scaling law .

AKEW: Assessing Knowledge Editing in the Wild (2024.emnlp-main)

Copied to clipboard

Xiaobao Wu, Liangming Pan, William Yang Wang, Anh Tuan Luu

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation (2024.emnlp-main)

Challenge:	Recent Large Language Models (LLMs) have revolutionized the NLP field but their knowledge could become incorrect or outdated over time.
Approach:	They propose a new practical benchmark for knowledge editing that covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets.
Outcome:	The proposed method covers structured facts, unstructured texts as facts, and extracted triplets.

Copied to clipboard

Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, Pang Wei Koh

Dense X Retrieval: What Retrieval Granularity Should We Use? (2024.emnlp-main)

Challenge:	Existing studies focus on literal copying, but current methods reduce literal copy but not non-literal copying.
Approach:	They propose a benchmark to measure literal and non-literal copying in LMs . they use copyrighted fiction books as text sources to assess literal copying .
Outcome:	The proposed model measures literal and non-literal copying in copyrighted texts . large models show significantly more copying, with literal copying rates increasing .

Copied to clipboard

Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, Dong Yu

Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach (2024.emnlp-main)

Challenge:	a learned dense retrieval model is often overlooked when using a corpus for inference, resulting in a design choice of retrieval unit . granularity of retrievals is important for both retrieval and downstream tasks .
Approach:	They propose a retrieval unit for dense retrieval that uses propositions to index corpus . propositions are defined as atomic expressions within text, each encapsulating a distinct factoid .
Outcome:	The proposed retrieval unit outperforms passage-level units on retrieval and downstream tasks.

Copied to clipboard

Yanchen Liu, Mingyu Ma, Wenna Qin, Azure Zhou, Jiaao Chen, Weiyan Shi, Wei Wang, Diyi Yang

Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models (2024.emnlp-main)

Challenge:	Existing studies on susceptibility to misinformation rely on self-reported beliefs, which can be subject to bias, expensive to collect, and challenging to scale for downstream applications.
Approach:	They propose a computational approach to efficiently model users’ latent susceptibility levels by using demographic factors and political ideology as inputs.
Outcome:	The proposed model shows that political leanings and other psychological factors exhibit varying degrees of association with susceptibility to COVID-19 misinformation.

Copied to clipboard

Zheng Zhao, Yftah Ziser, Shay Cohen

XDetox: Text Detoxification with Token-Level Toxicity Explanations (2024.emnlp-main)

Challenge:	Pre-trained large language models retain task-specific knowledge, but where and to what extent they retain it remains unexplored.
Approach:	They investigate the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across over 60 NLP tasks.
Outcome:	The results show that pre-trained models retain task-specific knowledge . some tasks are already encoded in pre-train models, but others benefit from instruction tuning.

Copied to clipboard

Beomseok Lee, Hyunwoo Kim, Keon Kim, Yong Suk Choi

Optimizing Chinese Lexical Simplification Across Word Types: A Hybrid Approach (2024.emnlp-main)

Challenge:	Existing methods for mitigating toxic content are black-box approaches, which results in limitations in modifying toxic tokens.
Approach:	They propose a method that integrates token-level toxicity explanations with the masking and infilling detoxification processes.
Outcome:	The proposed method outperforms baseline methods in fluency and toxicity reduction.

Copied to clipboard

ZiHao Xiao, Jiefu Gong, Shijin Wang, Wei Song

Control Large Language Models via Divide and Conquer (2024.emnlp-main)

Challenge:	Expensive large language models outperform small models in simplifying complex content words and Chinese idioms from the dictionary.
Approach:	They propose to use a retrieval-based interpretation augmentation strategy to refine small models to simplify complex content words and Chinese idioms.
Outcome:	The proposed framework outperforms large language models in Chinese Lexical Simplification (CLS) and improves on OOD models.

Copied to clipboard

Bingxuan Li, Yiwei Wang, Tao Meng, Kai-Wei Chang, Nanyun Peng

Joint Pre-Encoding Representation and Structure Embedding for Efficient and Low-Resource Knowledge Graph Completion (2024.emnlp-main)

Challenge:	Lexically Constrained Generation (LCG) is a crucial task of text generation.
Approach:	They propose a Divide and Conquer Generation strategy to enhance LLMs' performance in Lexically Constrained Generation with prompt-based controlling.
Outcome:	The proposed strategy shows 90% improvement on the most challenging LCG task.

Copied to clipboard

Chenyu Qiu, Pengjiang Qian, Chuang Wang, Jian Yao, Li Liu, Fang Wei, Eddie Eddie

Improving Discriminative Capability of Reward Models in RLHF Using Contrastive Learning (2024.emnlp-main)

Challenge:	Existing knowledge graph completion models require longer training and inference times as well as increased memory usage.
Approach:	They propose to encode textual descriptions into semantic representations before training and integrate structural embedding with pre-encoded semantic description to improve model's prediction performance on 1-N relations.
Outcome:	The proposed model increases inference speed by 30x and reduces training memory by approximately 60% on the WN18RR and UMLS datasets.

Copied to clipboard

Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang

RoCEL: Advancing Table Entity Linking through Distinctive Row and Column Contexts (2024.emnlp-main)

Challenge:	Current methods rely on ranking losses to teach reward model to assess preferences, but they are susceptible to noise and ambiguous data, often failing to deeply understand human intentions.
Approach:	They propose a method that incorporates contrastive learning into the reward modeling process to enhance generalization and stabilize the reinforcement learning training process.
Outcome:	The proposed method enhances generalization of the reward model, stabilizes the reinforcement learning training process, and improves the final alignment with human preferences.

Copied to clipboard

Yuanzheng Wang, Yixing Fan, Jiafeng Guo, Ruqing Zhang, Xueqi Cheng

Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for table entity linking ignore row and column contexts . existing methods for TEL focus on understanding sequential text contexts, making it difficult to adapt to the row and columns structure of tables.
Approach:	They propose to leverage row and column contexts to enhance the semantics of mentions in entity disambiguation.
Outcome:	The proposed method outperforms the state-of-the-art (SOTA) baseline by 1.5% on the in-domain dataset and 3.7% on average across three out-of domain datasets.

Copied to clipboard

Zi’ou Zheng, Christopher Malon, Martin Min, Xiaodan Zhu

Efficient Overshadowed Entity Disambiguation by Mitigating Shortcut Learning (2024.emnlp-main)

Challenge:	Large language models (LLMs) are essential for performing complex multi-step reasoning tasks, such as multi-hop reasoning tasks.
Approach:	They propose to use large language models to derive structured intermediate proof steps to improve their performance by using examples.
Outcome:	The proposed models can derive correct proof steps with in-context learning.

Copied to clipboard

Panuthep Tasawong, Peerat Limkonchotiwat, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong

AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction (2024.emnlp-main)

Challenge:	Entity disambiguation (ED) is crucial in natural language processing tasks such as question-answering and information extraction.
Approach:	They propose a method to reduce computational overhead on overshadowed entities by addressing shortcut learning.
Outcome:	The proposed method achieves state-of-the-art performance without compromising inference speed.

Copied to clipboard

Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Pan, Kam-Fai Wong

Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment (2024.emnlp-main)

Challenge:	Existing state-of-the-art Large Language Models (LLMs) still cannot perform well in this situation even with the help of in-context learning and finetuning.
Approach:	They propose a benchmark to evaluate LLMs’ ability to plan and execute multiple APIs from various sources in order to complete the user’s task.
Outcome:	The proposed benchmarks show that the existing state-of-the-art LLMs still cannot perform well in this situation even with in-context learning and finetuning.

Copied to clipboard

Zhipeng Chen, Kun Zhou, Xin Zhao, Jingyuan Wang, Ji-Rong Wen

AudioVSR: Enhancing Video Speech Recognition with Audio Data (2024.emnlp-main)

Challenge:	Experimental results show that large language models are struggling to align with human preference in complex tasks and scenarios.
Approach:	They propose a low-redundant alignment method that selects the top-10% most updated parameters in LLMs for alignment training.
Outcome:	The proposed method improves on 10 datasets and shows that it is redundant . it can be used to train LLMs on QA and ECQA datasets, but it is not feasible to test it on a large dataset.

Copied to clipboard

Xiaoda Yang, Xize Cheng, Jiaqi Duan, Hongshun Qiu, Minjie Hong, Minghui Fang, Shengpeng Ji, Jialong Zuo, Zhiqing Hong, Zhimeng Zhang, Tao Jin

ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? (2024.emnlp-main)

Challenge:	Recent work has shown poor performance with non-Indo-European languages . previous work primarily utilizes video information to build VSR models .
Approach:	They propose a generative model for data inflation that integrates synthetic data with authentic visual data to enhance the VSR model.
Outcome:	The proposed model improves on the audio-visual alignment problem in audio-video tasks.

Copied to clipboard

Siddhant Waghjale, Vishruth Veerendranath, Zhiruo Wang, Daniel Fried

Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level (2024.emnlp-main)

Challenge:	Current methods for optimizing program efficiency improve performance measured by execution time, but they often come at the cost of severely decreasing the functional correctness.
Approach:	They propose a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing.
Outcome:	The proposed approach improves performance while maintaining correctness while adding execution information.

Copied to clipboard

Zhaopeng Feng, Ruizhe Chen, Yan Zhang, Zijie Meng, Zuozhu Liu

Re-ReST: Reflection-Reinforced Self-Training for Language Agents (2024.emnlp-main)

Challenge:	General-purpose Large Language Models (LLMs) like GPT-4 have exhibited strong translation abilities.
Approach:	They propose to use a model-agnostic model to refine the performance of general-purpose large-language models for machine translation (MT) by utilizing Gemma-2B/7B as the backbone.
Outcome:	The proposed model-agnostic and cost-effective tool improves the performance of general-purpose large-language models for machine translation (MT) by integrating it with any general-use LLM.

Copied to clipboard

Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, Nanyun Peng

Effective Synthetic Data and Test-Time Adaptation for OCR Correction (2024.emnlp-main)

Challenge:	Existing methods to fine tune language agents with reasoning-action trajectories require high-quality model-generated samples, which are hard to obtain for challenging language agent tasks.
Approach:	They propose a method to employ reflection during inference without ground-truth feedback to improve agents more autonomously.
Outcome:	The proposed method improves self-training performance on open-source language agents by 7.6% and 14.1% respectively.

Copied to clipboard

Shuhao Guan, Cheng Xu, Moule Lin, Derek Greene

SRF: Enhancing Document-Level Relation Extraction with a Novel Secondary Reasoning Framework (2024.emnlp-main)

Challenge:	Recent research has framed the post-OCR task as a Seq2Seq Neural Machine Translation (NMT) task.
Approach:	They propose a method for constructing post-OCR synthetic data with different noise levels using weak supervision.
Outcome:	The proposed method reduces CER by 68.67% without relying on manual annotations.

Copied to clipboard

Fu Zhang, Qi Miao, Jingwei Cheng, Hongsen Yu, Yi Yan, Xin Li, Yongxue Wu

FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024.emnlp-main)

Challenge:	Existing methods for document-level relation extraction ignore bidirectional mention interaction when generating relational features for entity pairs.
Approach:	They propose a document-level relation extraction model that incorporates bidirectional mention fusion and a simple yet effective evidence extraction module for relation prediction.
Outcome:	The proposed model achieves SOTA performance and the proposed method is effective and general when integrated into existing models.

Copied to clipboard

Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang

Exploring the Learning Capabilities of Language Models using LEVERWORLDS (2024.emnlp-main)

Challenge:	Referring Expression Comprehension (REC) is a cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding.
Approach:	They propose to use a new reference expression comprehension (REC) dataset to evaluate the capabilities of language understanding, image comprehension, and language-to-image grounding.
Outcome:	The proposed model is able to reject scenarios where the target object is not visible in the image, a key aspect often overlooked in existing models and approaches.

Copied to clipboard

Eitan Wagner, Amir Feder, Omri Abend

CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models (2024.emnlp-main)

Challenge:	Existing models of stochastic learning involve learning general structure rules and specific properties of the instance.
Approach:	They propose a framework that allows the generation of physics-inspired worlds that follow a similar generative process with different distributions and their instances can be expressed in natural language.
Outcome:	The proposed framework allows the generation of physics-inspired worlds that follow a similar generative process with different distributions and their instances can be expressed in natural language.

Copied to clipboard

Eitan Wagner, Yuli Slavutsky, Omri Abend

DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding (2024.emnlp-main)

Challenge:	Language model scores are often treated as probabilities, but their reliability as probability estimators has mainly been studied through calibration, overlooking other aspects.
Approach:	They propose a framework to assess model reliability across interchangeable completion and conditioning orders by performing statistical tests on real and synthetic data to eliminate training effects.
Outcome:	The proposed framework assesses the consistency of model predictions across interchangeable completion and conditioning orders on real and synthetic data to eliminate training effects.

Copied to clipboard

Manan Suri, Puneet Mathur, Franck Dernoncourt, Rajiv Jain, Vlad Morariu, Ramit Sawhney, Preslav Nakov, Dinesh Manocha

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging (2024.emnlp-main)

Challenge:	Document structure editing involves manipulating localized textual, visual, and layout components in document images based on user’s requests.
Approach:	They propose a framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs) by localizing edit regions of interest and disambiguating user edit requests into edit commands.
Outcome:	The proposed framework outperforms baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12%) tasks.

Copied to clipboard

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

Understanding Slang with LLMs: Modelling Cross-Cultural Nuances through Paraphrasing (2024.emnlp-main)

Challenge:	Modern large language models (LLMs) showcase impressive capabilities across various tasks with aligning their behavior with human preferences.
Approach:	They propose a framework that integrates domain-specific knowledge into a general reward model by model merging.
Outcome:	The proposed framework improves performance across different benchmarks and provides detailed analysis showing the effects of model merging.

Copied to clipboard

Ifeoluwa Wuraola, Nina Dethlefs, Daniel Marciniak

Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024.emnlp-main)

Challenge:	a recent study examines the ability of large language models (LLMs) to paraphrase slang within climate-related tweets . slanted tweets from non-anglocentric countries may contain cultural references and idioms based on sociocultural identities .
Approach:	They investigate the ability of large language models to paraphrase slang within climate-related tweets from Nigeria and the UK.
Outcome:	The proposed model can paraphrase slang within climate-related tweets from Nigeria and the UK . the model can only parse sexist and sex-related slurs, the study shows .

Copied to clipboard

Lifu Tu, Semih Yavuz, Jin Qu, Jiacheng Xu, Rui Meng, Caiming Xiong, Yingbo Zhou

Re-Reading Improves Reasoning in Large Language Models (2024.emnlp-main)

Challenge:	Large language models have shown a powerful ability for text generation, but undesired behaviors such as toxicity and hallucinations can manifest.
Approach:	They propose to formalize text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions.
Outcome:	The proposed approach is effective across three tasks, including keyword-constrained generation, toxicity reduction, and factual correctness in question-answering.

Copied to clipboard

Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, Shuai Ma

Adaptive Axes: A Pipeline for In-domain Social Stereotype Analysis (2024.emnlp-main)

Challenge:	Unlike thought-eliciting prompting methods, RE2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process.
Approach:	They introduce a simple, yet general and effective prompting method, RE2, which rereads the question as input.
Outcome:	The proposed method demonstrates strong generality and compatibility with most thought-eliciting prompting methods, including CoT.

Copied to clipboard

Qingcheng Zeng, Mingyu Jin, Rob Voigt

ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments (2024.emnlp-main)

Challenge:	Existing methods to quantify social stereotypes have struggled to capture the variability in stereotypes across conceptual domains for the same social group.
Approach:	They propose to use text embedding models and adaptive semantic axes to recover stereotypes from contextual representations by using large language models.
Outcome:	The proposed pipeline surpasses token-based methods in capturing in-domain framing and tracks stereotypes along domain-specific semantic axes for in- domain texts.

Copied to clipboard

Sourjyadip Ray, Kushal Gupta, Soumi Kundu, Dr Kasat, Somak Aditya, Pawan Goyal

Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations (2024.emnlp-main)

Challenge:	a global shortage of healthcare workers has demanded the development of smart healthcare assistants.
Approach:	They analyze the healthcare knowledge of existing Large Vision Language Models (LVLMs) using an annotated open-ended task.
Outcome:	The study analyzes the knowledge of large vision language models using open-ended questions . the results highlight the need for specialized, domain-specific solutions .

Copied to clipboard

Jiyi Li

Challenge:	Existing studies on crowd text answer aggregation focus on individual crowd workers' average performance, but the role of LLMs as aggregators is not well-studied.
Approach:	They propose a human-LLM hybrid text answer aggregation method with a Creator-Aggregator Multi-Stage crowdsourcing framework.
Outcome:	The proposed method is based on a Creator-Aggregator Multi-Stage crowdsourcing framework.

Improve Student’s Reasoning Generalizability through Cascading Decomposed CoTs Distillation (2024.emnlp-main)

Copied to clipboard

Chengwei Dai, Kun Li, Wei Zhou, Songlin Hu

Revisiting Supervised Contrastive Learning for Microblog Classification (2024.emnlp-main)

Challenge:	Existing studies have tried to distill these capabilities into smaller language models (SLMs) however, these capabilities are often associated with more parameters, which is not practical to emergent in smaller models.
Approach:	They propose to decompose the traditional single-step learning process into two cascaded learning steps by restructuring the training objectives and concatenating the question with the rationale as input.
Outcome:	Extensive experiments show that the proposed method improves reasoning generalizability and diversity of the model.

Copied to clipboard

Junbo Huang, Ricardo Usbeck

BaitAttack: Alleviating Intention Shift in Jailbreak Attacks via Adaptive Bait Crafting (2024.emnlp-main)

Challenge:	Existing models for microblog classification use pre-training language models (LMs) however, pre-trained LMs are resource-intensive and not suitable for small labs.
Approach:	They propose to fine-tune transformer-based language models with a SCL loss for English microblog classification by comparing two benchmarks.
Outcome:	The proposed method has a performance gain of up to 11.9 percentage points across all subtasks.

Copied to clipboard

Rui Pu, Chaozhuo Li, Rui Ha, Litian Zhang, Lirong Qiu, Xi Zhang

Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective (2024.emnlp-main)

Challenge:	Existing attacks focus on meticulously constructing prompts to disguise harmful intentions . however, incorporation of disguising prompts may incur the challenge of "intention shift"
Approach:	They propose a jailbreak attack component, BaitAttack, to alleviate the effects of intention shift . Bait provides a response to the query, prompting LLMs to rectify or supplement the knowledge within the bait .
Outcome:	The proposed component, BaitAttack, reduces the effects of intention shift within jailbreak attacks.

Copied to clipboard

Zhaotian Weng, Zijun Gao, Jerone Andrews, Jieyu Zhao

Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing (2024.emnlp-main)

Challenge:	Current methods to learn biases from the perspective of model components are limited by their complexity and performance.
Approach:	They propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation within vision-language and multimodal tasks.
Outcome:	The proposed framework is applicable to a wide range of vision-language and multimodal tasks and reduces bias by 22.03% and 9.04% in the MSCOCO and PASCAL-SENTENCE datasets.

Copied to clipboard

Weichuan Wang, Zhaoyi Li, Defu Lian, Chen Ma, Linqi Song, Ying Wei

SciAgent: Tool-augmented Language Models for Scientific Reasoning (2024.emnlp-main)

Challenge:	Existing studies have focused on using Large Language Models to improve translation quality . language mismatch and repetition are two of the main problems with LLMs .
Approach:	They propose to leverage model editing methods to reduce language mismatch and repetition . they propose to fetch intersections of locating results under different language settings .
Outcome:	The proposed methods reduce language mismatch and repetition ratios and enhance translation quality in most cases.

Copied to clipboard

Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun

Global Reward to Local Rewards: Multimodal-Guided Decomposition for Improving Dialogue Agents (2024.emnlp-main)

Challenge:	SciAgent surpasses other LLMs with the comparable size by more than 8.0% in absolute accuracy.
Approach:	They propose a tool-augmented scientific reasoning setting that supplements LLMs with scalable toolsets and builds a benchmark to evaluate LLM’s abilities with tool assistance.
Outcome:	The proposed setting augments LLMs with scalable toolsets and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user.

Copied to clipboard

Dong Won Lee, Hae Park, Yoon Kim, Cynthia Breazeal, Louis-Philippe Morency

Towards Measuring and Modeling “Culture” in LLMs: A Survey (2024.emnlp-main)

Challenge:	Existing methods for asynchronous dialogue agents only use a single global score at the end of the session.
Approach:	They propose a method for aligning an LLM-based dialogue agent for long-term social dialogue . they use local implicit feedback to decompose a human-provided global Explicit reward .
Outcome:	The proposed approach improves the turn-level utterance generation across conversational metrics compared to baseline methods.

Copied to clipboard

Muhammad Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Aji, Jacki O’Neill, Ashutosh Modi, Monojit Choudhury

Challenge:	Existing models are biased towards Western, Anglocentric or American cultures, a problem that is arguably detrimental to the performance of LLMs.
Approach:	They analyze more than 90 recent papers that aim to study cultural representation and inclusion in large language models.
Outcome:	The proposed models are biased towards Western, Anglocentric or American cultures, despite their diversity and their robustness.

ESC-Eval: Evaluating Emotion Support Conversations in Large Language Models (2024.emnlp-main)

Copied to clipboard

Haiquan Zhao, Lingyu Li, Shisong Chen, Shuqi Kong, Jiaan Wang, Kexin Huang, Tianle Gu, Yixu Wang, Jian Wang, Liang Dandan, Zhixu Li, Yan Teng, Yanghua Xiao, Yingchun Wang

Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting (2024.emnlp-main)

Challenge:	Emotion Support Conversation (ESC) is a crucial application for reducing stress and providing emotional guidance.
Approach:	They re-organize 2,801 role-playing cards to define roles of role-players . they train a specific role- playing model called ESC-Role which behaves more like a confused person than GPT-4 .
Outcome:	The proposed model behaves more like a confused person than GPT-4, and the model performs better than GPLs.

Copied to clipboard

Sagnik Mukherjee, Muhammad Adilazuarda, Sunayana Sitaram, Kalika Bali, Alham Aji, Monojit Choudhury

Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features (2024.emnlp-main)

Challenge:	Socio-demographic prompting is a commonly employed approach to study cultural biases in LLMs as well as for aligning models to certain cultures.
Approach:	They propose to use socio-demographic prompting to probe four LLMs with culturally sensitive and non-sensitive cues on datasets that are supposed to be culturally neutral or sensitive.
Outcome:	The proposed model shows significant differences in responses on both kinds of datasets, casting doubt on its robustness.

Copied to clipboard

Xiao Yu, Kejiang Chen, Qi Yang, Weiming Zhang, Nenghai Yu

Hate Personified: Investigating the role of LLMs in content moderation (2024.emnlp-main)

Challenge:	Large language models (LLMs) have revolutionized the field of natural language processing because of their excellent performance on various tasks.
Approach:	They propose a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected.
Outcome:	The proposed method achieves 7.36% and 2.84% improvement in detection performance compared to baselines in detecting texts from different domains generated by GPT-4 and Claude3 respectively.

Copied to clipboard

Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexander Fraser, Tanmoy Chakraborty

Temporally Consistent Factuality Probing for Large Language Models (2024.emnlp-main)

Challenge:	Our work provides preliminary guidelines and highlights the nuances of applying Large Language models in culturally sensitive cases.
Approach:	They propose to use large language models to help with content moderation to assess how well the needs of diverse groups are reflected in annotated posts.
Outcome:	The proposed model is able to leverage community-based flagging efforts and exposure to adversaries.

Copied to clipboard

Ashutosh Bajpai, Aaryan Goyal, Atif Anwer, Tanmoy Chakraborty

A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are used as an alternative knowledge base for many tasks.
Approach:	They propose a temporally consistent factuality probe task that extends the consistency probe in the temporal dimension.
Outcome:	The proposed task extends the definitions of existing metrics to represent consistent factuality across temporal dimension.

Copied to clipboard

Zihao Li, Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann

Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators (2024.emnlp-main)

Challenge:	Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community.
Approach:	They propose to compare multilingual pretraining objectives in a controlled methodological environment with multilingual models.
Outcome:	The proposed model outperforms existing models in 6 languages and demonstrates that multilingual translation is an effective pretraining objective under the right conditions.

Copied to clipboard

Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, Tanmoy Chakraborty

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) and AI assistants are experiencing exponential growth in usage among expert and amateur users.
Approach:	They propose to assess the reliability of current Large Language Models as science communicators . they use a dataset comprising 742 Yes/No queries embedded in complex scientific concepts .
Outcome:	The proposed model outperforms open-access models in scientific question-answering tasks . the model outpersforms GPT-4 Turbo models in many evaluation aspects .

Copied to clipboard

Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, Yu Cheng

Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability (2024.emnlp-main)

Challenge:	Mixture-of-Experts (MoE) has gained increasing popularity as a framework for scaling up large language models.
Approach:	They investigate how to build Mixture-of-Experts (MoE) models from existing large language models . they use expert construction, Continual pre-training and data sampling strategies .
Outcome:	The proposed model outperforms existing models with similar parameters on a wide range of tasks.

Copied to clipboard

Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, Xiaojun Wan

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging (2024.emnlp-main)

Challenge:	Existing methods for evaluation of natural language generation tasks lack reliable data.
Approach:	They propose to use annotations from human and GPT-4 to construct a corpus for NLG evaluation.
Outcome:	The proposed corpus can perform flexible and interpretable evaluations without references and surpasses existing models.

Copied to clipboard

Yiming Ju, Ziyi Ni, Xingrun Xing, Zhixiong Zeng, Hanyu Zhao, Siqi Fan, Zheng Zhang

Generating Demonstrations for In-Context Compositional Generalization in Grounded Language Learning (2024.emnlp-main)

Challenge:	Existing studies suggest that the order of training samples can affect model performance, but this is not the case.
Approach:	They propose to merge supervised fine-tuning models with different data orders to mitigate this imbalance by parameter merging.
Outcome:	The proposed method outperforms the weighted-average method on five datasets.

Copied to clipboard

Sam Spilsbury, Pekka Marttinen, Alexander Ilin

FAME: Towards Factual Multi-Task Model Editing (2024.emnlp-main)

Challenge:	In-Context-learning and few-shot prompting are viable methods for compositional output generation but they are sensitive to the choice of support examples.
Approach:	They propose a method which generates supports and targets current state of the world and then uses them in-context-learning to solve a query.
Outcome:	The proposed agent improves performance on a previously unsolved compositional generalization test without loss of performance in other areas.

Copied to clipboard

Li Zeng, Yingyu Shan, Zeming Liu, Jiashu Yao, Yuhang Guo

MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)

Challenge:	Large language models embed extensive knowledge and perform exceptionally well across tasks. outdated knowledge or factual errors within LLMs can lead to misleading or incorrect responses.
Approach:	They propose to use a dataset to enhance the practicality of model editing to correct inaccurate information within LLMs.
Outcome:	The proposed method performs excellently across tasks and scenarios, confirming its practicality.

Copied to clipboard

Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)

Challenge:	MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning.
Approach:	They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones.
Outcome:	The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs.

Copied to clipboard

Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, Shuai Ma

InfiniPot: Infinite Context Processing on Memory-Constrained LLMs (2024.emnlp-main)

Challenge:	introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance.
Approach:	They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them.
Outcome:	The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods.

Copied to clipboard

Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang

Challenge:	InfiniPot is a KV cache control framework that can handle long input contexts without additional training.
Approach:	They propose a KV cache control framework that can handle long input contexts efficiently without additional training.
Outcome:	The proposed framework outperforms models trained for long contexts in various NLP tasks and is highly efficient and versatile.

VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models (2024.emnlp-main)

Copied to clipboard

Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, Lianwen Jin

CorrSynth - A Correlated Sampling Method for Diverse Dataset Generation from LLMs (2024.emnlp-main)

Challenge:	Existing studies have shown that CLIP models with short summary texts cannot process extensive textual descriptions due to its text encoder's reliance on positional embeddings with length 77.
Approach:	They propose a Contrastive Language-Image Pre-training (CLIP) model which aims to unleash the long-description understanding capability of video CLIP models.
Outcome:	The proposed model can learn the distribution of feature space while expanding the long description capability.

Copied to clipboard

Suhas Kowshik, Abhishek Divekar, Vijit Malik

Defining Knowledge: Bridging Epistemology and Large Language Models (2024.emnlp-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable performance in diverse tasks using zero-shot and few-shot prompting.
Approach:	They propose a method which generates data that is more diverse and faithful to the input prompt using a correlated sampling strategy.
Outcome:	The proposed method overcomes the complexity drawbacks of other guidance-based techniques and improves student metrics and intrinsic metrics upon competitive baselines across four datasets.

Copied to clipboard

Constanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau, Anders Søgaard

TKGT: Redefinition and A New Way of Text-to-Table Tasks Based on Real World Demands and Knowledge Graphs Augmented LLMs (2024.emnlp-main)

Challenge:	Existing literature on large language models (LLMs) define knowledge as a fact if it correctly completes a cloze sentence . but the predictions of semantically equivalent clozing sentences are inconsistent .
Approach:	They review standard definitions of knowledge in epistemology and formalize interpretations applicable to LLMs.
Outcome:	The authors compare the preferences of philosophers and computer scientists in terms of knowledge definitions and evaluation protocols for testing knowledge in accordance with the most relevant definitions.

Copied to clipboard

Peiwen Jiang, Xinbo Lin, Zibo Zhao, Ruhui Ma, Yvonne Chen, Jinhua Cheng

Free your mouse! Command Large Language Models to Generate Code to Format Word Documents (2024.emnlp-main)

Challenge:	Existing studies focus on text-to-table tasks that ignore domain structures and use simple datasets to extract structured information from unstructured text.
Approach:	They propose a new text-to-table task that generates domain knowledge graphs from raw text using a mixed-IE method and a hybrid retrieval augmented generation method.
Outcome:	The proposed dataset improves compatibility with long text-processing tasks by incorporating domain knowledge graphs (KGs) classes into tables.

Copied to clipboard

Shihao Rao, Liang Li, Jiapeng Liu, Guan Weixin, Xiyan Gao, Bing Lim, Can Ma

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models (2024.emnlp-main)

Challenge:	Recent LLMs have significantly improved code generation, making it increasingly accessible to users.
Approach:	They propose an automatic document formatting method, Text-to-Format, driven by various prompting strategies and a high-quality dataset DocFormEval data.
Outcome:	The proposed method improves the efficiency and experience of users in formatting the document and improves document formatting task.

Copied to clipboard

Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) excel in diverse tasks but often underperform in specialized fields due to limited domain-specific or proprietary corpus.
Approach:	They propose a power-law relationship between loss, mixture ratio, and training tokens scale and formalize the trade-off between general and domain-specific capabilities.
Outcome:	The proposed model achieves the desired domain transfer while maintaining general ability and highest utilization of available resources.

Copied to clipboard

Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, Tong Zhang

Rationale-Aware Answer Verification by Pairwise Self-Evaluation (2024.emnlp-main)

Challenge:	Existing multi-modal large language models (MLLMs) are able to process visual inputs by converting them into visual tokens that share the same latent space as language tokens in LLMs.
Approach:	They propose a benchmark that assesses the visual illusion level given spurious images and a pipeline that converts visual inputs into visual tokens.
Outcome:	The proposed benchmark shows that MLLMs suffer from an instinctive bias to varying degrees when presented with spurious images.

Copied to clipboard

Akira Kawabata, Saku Sugawara

On the Robustness of Editing Large Language Models (2024.emnlp-main)

Challenge:	Current approaches to train verifier models neglecting flawed rationales, resulting in an unreliable verifier.
Approach:	They propose a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions.
Outcome:	The proposed method outperforms training methods on three reasoning benchmarks.

Copied to clipboard

Xinbei Ma, Tianjie Ju, Jiyang Qiu, Zhuosheng Zhang, Hai Zhao, Lifeng Liu, Yulong Wang

IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method (2024.emnlp-main)

Challenge:	Existing studies have exhibited impressive success and significant potential.
Approach:	They propose to modify the knowledge memory with minimum computational cost while preserving the performance on the retained knowledge.
Outcome:	The proposed methods avoid retraining to update the model parameters and have demonstrated promising performance and efficiency.

Copied to clipboard

MiHyeon Kim, Juhyoung Park, YoungBin Kim

Distract Large Language Models for Automatic Jailbreak Attack (2024.emnlp-main)

Challenge:	Pre-trained Language Models (PLMs) have shown remarkable performance on diverse NLP tasks through pre-training and fine-tuning.
Approach:	They propose a numerically robust IM-connection incorporating a layer of BERT as a solution of Ordinary Differential Equations (ODEs) . Experimental results validate the robustness of IM BERT under various conditions.
Outcome:	The proposed model outperforms the existing model on the adversarial GLUE dataset by 5.9%p on low-resource scenarios.

Copied to clipboard

Zeguan Xiao, Yan Yang, Guanhua Chen, Yun Chen

Exploring Space Efficiency in a Tree-based Linear Model for Extreme Multi-label Classification (2024.emnlp-main)

Challenge:	Commercial large language models (LLMs) have made great progress in various NLP tasks.
Approach:	They propose a black-box jailbreak framework for automated red teaming of Large language models using an iterative optimization algorithm to conceal malicious content and memory reframing.
Outcome:	The proposed framework outperforms existing jailbreak defense methods and highlights the need to develop more effective and practical defense strategies.

Copied to clipboard

He-Zhe Lin, Cheng-Hung Liu, Chih-Jen Lin

WorryWords: Norms of Anxiety Association for over 44k English Words (2024.emnlp-main)

Challenge:	Extreme multi-label classification (XMC) aims to identify relevant subsets from numerous labels.
Approach:	They propose to store a tree model under the assumption of sparse data under the condition that some features may be unused when training binary classifiers in a trees method.
Outcome:	The proposed method can save 10% of the size of the standard one-vs-rest method for multi-label classification.

Copied to clipboard

Saif Mohammad

Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024.emnlp-main)

Challenge:	Anxiety is a common and beneficial human emotion, but there is still much that is not known about it .
Approach:	They propose a repository of manually derived word–anxiety associations for over 44,450 English words.
Outcome:	The proposed system can track anxiety in streams of text using WorryWords alone.

Copied to clipboard

Sumanth Doddapaneni, Mohammed Khan, Sshubam Verma, Mitesh Khapra

LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration (2024.emnlp-main)

Challenge:	Large Language Models are increasingly relied upon to evaluate text outputs of other LLMs . however, concerns persist over the accuracy of these assessments and the potential for misleading conclusions.
Approach:	They propose a framework to assess the reliability of Large Language Models (LLMs) they propose ' FBI' framework to examine the proficiency of Evaluator LLMs in assessing four critical abilities .
Outcome:	The proposed framework assesses the performance of LLMs in text generation tasks.

Copied to clipboard

Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, Xuanjing Huang

AutoPersuade: A Framework for Evaluating and Explaining Persuasive Arguments (2024.emnlp-main)

Challenge:	Large language models (LLMs) have been successful in understanding language and processing text, but their cost prohibits their practical applications.
Approach:	They propose a multi-agent collaboration method that breaks down lengthy documents into smaller, more manageable chunks and organizes the member agents to read their assigned chunks.
Outcome:	The proposed method achieves 16.42% and 1.63% accuracy gains over existing models on single-hop and multi-hop QA settings.

Copied to clipboard

Till Saenger, Musashi Hinck, Justin Grimmer, Brandon Stewart

Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs (2024.emnlp-main)

Challenge:	Existing tools for persuasion are well-equipped to identify which of a pre-existing set of messages is most persuasive, but they do not offer causal evidence on whether or how they have succeeded.
Approach:	They propose a framework for identifying topical components of persuasive arguments that are autopersuade.
Outcome:	The proposed framework validates the results through human studies and out-of-sample predictions.

Copied to clipboard

Simone Conia, Daniel Lee, Min Li, Umar Farooq Minhas, Saloni Potdar, Yunyao Li

Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning Through Trap Problems (2024.emnlp-main)

Challenge:	XC-Translate is a large-scale, manually-created benchmark for machine translation . current systems struggle to translate texts containing entity names, but KG-MT outperforms state-of-the-art approaches .
Approach:	They propose a method to integrate multilingual knowledge into a neural machine translation model . XC-Translate is the first large-scale, manually-created benchmark for machine translation . they propose KG-MT to integrate cultural-related references into MT models .
Outcome:	The proposed method outperforms state-of-the-art approaches by a large margin compared to NLLB-200 and GPT-4 . the proposed method is based on a multilingual knowledge graph and dense retrieval mechanism .

Copied to clipboard

Jun Zhao, Jingqi Tong, Yurong Mou, Ming Zhang, Qi Zhang, Xuanjing Huang

Scaling Laws for Linear Complexity Language Models (2024.emnlp-main)

Challenge:	Current LLMs lack systematic compositionality, and therefore cannot serve as reliable cognitive models.
Approach:	They propose to introduce logical traps into the original problems of MATH and GSM8K to investigate the compositionality of large language models in mathematical reasoning.
Outcome:	The proposed model can generate infinite combinations from finite learned components.

Copied to clipboard

Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, Yiran Zhong

Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards (2024.emnlp-main)

Challenge:	Existing scaling laws for large language models are unclear, but they are useful for scalability.
Approach:	They propose scaling laws for linear complexity language models to establish a foundation for their scalability.
Outcome:	The proposed models demonstrate superior linguistic proficiency and knowledge retention.

Copied to clipboard

Heejin Do, Sangwon Ryu, Gary Lee

Challenge:	Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation.
Approach:	They propose to integrate actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES.
Outcome:	The proposed scoring-aware multi-reward reinforcement learning integrates actual evaluation schemes into the training process.

Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis (2024.emnlp-main)

Copied to clipboard

Guangliang Liu, Haitao Mao, Jiliang Tang, Kristen Johnson

ATAP: Automatic Template-Augmented Commonsense Knowledge Graph Completion via Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Existing studies on the effectiveness of moral self-correction in large language models have not been conducted.
Approach:	They propose that moral self-correction is a computationally efficient method for reducing harmful content in LLMs.
Outcome:	The proposed method reduces harmful content in LLMs, but it remains under-explored . it can help LLM find shortcut to more morally correct output, the authors argue .

Copied to clipboard

Fu Zhang, Yifan Ding, Jingwei Cheng

LM2: A Simple Society of Language Models Solves Complex Reasoning (2024.emnlp-main)

Challenge:	Commosense knowledge graphs (CKGC) are powerful representations of real-world commonsense knowledge.
Approach:	They propose a framework that uses automatically generated prompt templates combined with pre-trained language models to improve CKGC performance.
Outcome:	The proposed framework mitigates the long-tail problem and improves CKGC performance on a large dataset.

Copied to clipboard

Gurusha Juneja, Subhabrata Dutta, Tanmoy Chakraborty

Clara Meister, Mario Giulianelli, Tiago Pimentel

Challenge:	Existing studies show that providing guidance via decomposing the original question into multiple subproblems elicits more robustness in LLM reasoning.
Approach:	They propose a language-based decomposition, solution and verification framework that modularizes the decomposer, solution, and verification into three different language models.
Outcome:	The proposed model outperforms existing methods on in- and out-domain reasoning problems, outperforming the best baselines by 8.1% on MATH, 7.71% on JEEBench, and 9.7% on MedQA problems.

Copied to clipboard

Linguistics & Cognitive NLP

Challenge:	Existing studies have shown that surprisal theory ignores the possibility of similarity between words and treats them as distinct entities.
Approach:	They propose a new measure of comprehension effort called information value that accounts for communicative equivalences between possible continuations.
Outcome:	The proposed measure of comprehension effort is based on the diversity index of the diversity of communicative units.

Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering (2024.emnlp-main)

Copied to clipboard

Adjali Omar, Olivier Ferret, Sahar Ghannay, Hervé Le Borgne

Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)

Challenge:	Knowledge-Aware Visual Question Answering about Entity tasks require two separate steps to generate accurate answers.
Approach:	They propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion.
Outcome:	The proposed approach improves answer generation through entity retrieval and query expansion.

Copied to clipboard

Jianfeng He, Runing Yang, Linlin Yu, Changbin Li, Ruoxi Jia, Feng Chen, Ming Jin, Chang-Tien Lu

Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP (2024.emnlp-main)

Challenge:	Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods.
Approach:	They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets.
Outcome:	The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable.

Copied to clipboard

Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, Reut Tsarfaty

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training (2024.emnlp-main)

Challenge:	Improvements in language models’ capabilities have pushed their applications towards longer contexts, making long-context evaluation and development an active research area.
Approach:	They propose to unpack the taxonomy of long-context based on the properties that make them more difficult with longer contexts.
Outcome:	The proposed taxonomy is based on the properties that make them more difficult with longer contexts.

Copied to clipboard

Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, Ivan Yamshchikov

SEGMENT+: Long Text Processing with Short-Context Language Models (2024.emnlp-main)

Challenge:	Tokenization is a relatively understudied area, but it can greatly impact model performance and efficiency.
Approach:	They propose a modified BPE tokenizer that removes merges that leave intermediate "junk" tokens from the vocabulary.
Outcome:	The proposed method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression.

Copied to clipboard

Wei Shi, Shuang Li, Kerun Yu, Jinglei Chen, Zujie Liang, Xinhui Wu, Yuxi Qian, Feng Wei, Bo Zheng, Jiaqing Liang, Jiangjie Chen, Yanghua Xiao

Explicit Memory Learning with Expectation Maximization (2024.emnlp-main)

Challenge:	Existing frameworks that increase context window do not guarantee robust performance across long input tasks.
Approach:	They propose a framework that enables language models to handle extended inputs within limited context windows efficiently.
Outcome:	The framework improves performance on long-document question-answering and Needle-in-a-Haystack tasks.

Copied to clipboard

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang

Challenge:	Large Language Models lack reliable learning mechanisms for updating information across interactions.
Approach:	They propose a framework that enhances explicit memory updates via the Expectation-Maximization algorithm.
Outcome:	The proposed framework outperforms existing methods without memory or with static external memory on streaming inference tasks.

Closing the Loop: Learning to Generate Writing Feedback via Language Model Simulated Student Revisions (2024.emnlp-main)

Copied to clipboard

Inderjeet Nair, Jiaye Tan, Xiaotian Su, Anne Gere, Xu Wang, Lu Wang

Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024.emnlp-main)

Challenge:	Recent advances in language models (LMs) have made it possible to automatically generate feedback that is actionable and well-aligned with human-specified attributes.
Approach:	They propose a tool that PROduces Feedback via learning from LM simulated student revisions and propose to iteratively optimize the feedback generator by directly maximizing the effectiveness of students’ overall revising performance.
Outcome:	The proposed approach surpasses baseline methods in effectiveness of improving students’ writing and demonstrates enhanced pedagogical values, even though it was not explicitly trained for this aspect.

Copied to clipboard

Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, Fei Huang

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have revolutionized natural language processing with impressive capabilities, but they lack domain specificity, real-time information and face challenges in solving specialized problems.
Approach:	They propose a multi-LLM approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer.
Outcome:	The proposed model outperforms existing models by demonstrating its effectiveness and advantages in tool learning.

Copied to clipboard

Clement Neo, Shay Cohen, Fazl Barez

Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis (2024.emnlp-main)

Challenge:	Using a method to identify next-token neurons, we find that some attention heads recognize contexts relevant to predicting a token and activate a downstream token-predicting neuron accordingly.
Approach:	They propose a method to identify next-token neurons and determine the upstream attention heads responsible for their activity in LLMs.
Outcome:	The proposed method identifies next-token neurons, finds prompts that highly activate them, and determines the upstream attention heads responsible.

Copied to clipboard

Amey Hengle, Atharva Kulkarni, Shantanu Patankar, Madhumitha Chandrasekaran, Sneha D’silva, Jemima Jacob, Rashmi Gupta

The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoning (2024.emnlp-main)

Challenge:	ANGST is a benchmark for depression-anxiety comorbidity classification from social media posts.
Approach:	They propose a social media-based benchmark for depression-anxiety comorbidity classification . ANGST enables multi-label classification, allowing each post to be simultaneously identified as indicating depression and/or anxiety.
Outcome:	The proposed dataset enables multi-label classification of depression and anxiety . it outperforms existing models but none achieves an F1 score exceeding 72% .

Copied to clipboard

Shaobo Cui, Zhijing Jin, Bernhard Schölkopf, Boi Faltings

Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups (2024.emnlp-main)

Challenge:	Despite its significance, a systematic exploration of commonsense causality is lacking.
Approach:	They focus on taxonomies, benchmarks, acquisition methods, qualitative reasoning, and quantitative measurements in commonsense causality.
Outcome:	The proposed method synthesizes insights from over 200 representative articles and provides a practical guide for beginners.

Copied to clipboard

Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel

Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue (2024.emnlp-main)

Challenge:	Large language models (LLMs) are popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings.
Approach:	They investigate the use of large language models in CWI, LCP, and MWE settings by evaluating their use in zero-shot, few-shot and fine-tuning settings.
Outcome:	The proposed models struggle in certain conditions or achieve comparable results against existing methods.

Copied to clipboard

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, Nanyun Peng

Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done! (2024.emnlp-main)

Challenge:	Existing methods that edit large language models with updated knowledge can cause side effects on the general abilities of LLMs such as reasoning, natural language inference, and question answering.
Approach:	They propose to regularize the edit update weights by imposing constraints on their complexity based on the RElative Change in weighT.
Outcome:	The proposed method can significantly mitigate the side effects while maintaining over 94% editing performance.

Copied to clipboard

Divya Patel, Pathik Patel, Ankush Chander, Sourish Dasgupta, Tanmoy Chakraborty

MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have succeeded in summarizing information in contexts but saliency is subject to user preferences.
Approach:	They propose a framework that measures saliency using user reading histories and contrast in user profiles.
Outcome:	The proposed framework evaluates state-of-the-art LLMs on their ICL performance and shows that they lack true ICPL.

Copied to clipboard

Vishal Saley, Goonjan Saha, Rocktim Das, Dinesh Raghu, Mausam .

***YesBut***: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models (2024.emnlp-main)

Challenge:	Existing datasets lacking comprehensive annotations for medical history-taking are non-English . existing datasets lack comprehensive annotation for medical slots and their attributes .
Approach:	They propose a dataset of doctor-patient dialogues in English for medical history-taking task.
Outcome:	The proposed datasets are available in English and are compared with existing datasets.

Copied to clipboard

Abhilash Nandy, Yash Agarwal, Ashish Patwa, Millon Das, Aman Bansal, Ankit Raj, Pawan Goyal, Niloy Ganguly

Working Memory Identifies Reasoning Limits in Language Models (2024.emnlp-main)

Challenge:	Existing Vision-Language models perform poorly on satirical image detecting tasks . satire and humor are powerful tools to highlight issues, provoke thought, and encourage critical perspective .
Approach:	They propose to use a dataset to evaluate satirical images and satire images to detect satiric images . they also propose to generate the reason behind the image being satiral by generating one half of the image to be satisfying .
Outcome:	The proposed dataset contains 2547 images, 1084 satirical and 1463 non-satirically, with different artistic styles.

Copied to clipboard

Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, Soroush Vosoughi

RAFT: Realistic Attacks to Fool Text Detectors (2024.emnlp-main)

Challenge:	Using large language models, we examine the limitations of their cognitive capabilities and their working memory.
Approach:	They examine the limitations of large language models from a scaling perspective . they also assess various prompting strategies, revealing their diverse impacts on LLM performance.
Outcome:	The proposed models perform poorly on n-back tasks and on prompting strategies.

Copied to clipboard

James Wang, Ran Li, Junfeng Yang, Chengzhi Mao

LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks (2024.emnlp-main)

Challenge:	Large language models (LLMs) have exhibited remarkable fluency across tasks, but their unethical applications are unclear.
Approach:	They propose a grammar error-free black-box attack that exploits LLM embeddings at the word-level while preserving original text quality.
Outcome:	The proposed attack compromises all detectors across domains and is transferable across source models.

Copied to clipboard

Jiaxuan You, Mingjie Liu, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

Challenge:	Existing benchmarks for large language models evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences.
Approach:	They propose a framework which extends established benchmarks to sequential problem-solving settings and provides feedback after each round to build a demonstration memory that the models can query in future tasks.
Outcome:	The proposed framework can improve performance of LLMs by learning from past interactions and improve models' performance over time.

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)

Copied to clipboard

Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella

Challenge:	Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size.
Approach:	They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs .
Outcome:	The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks.

LLM-based Code-Switched Text Generation for Grammatical Error Correction (2024.emnlp-main)

Copied to clipboard

Tom Potter, Zheng Yuan

Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models (2024.emnlp-main)

Challenge:	Code-switching (CSW) is a part of multilingual conversation and is gaining popularity in social and professional settings.
Approach:	They propose to use synthetic data to generate a model capable of correcting grammatical errors in CSW texts.
Outcome:	The proposed model improves on existing systems on an authentic dataset from English as a second language learners.

Copied to clipboard

Mehrdad Farahani, Richard Johansson

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024.emnlp-main)

Challenge:	Generative language models struggle with specialized knowledge that is discussed less frequently on the web.
Approach:	They propose to use a model which decides between parametric and non-parametric knowledge to investigate how it uses the information from the context.
Outcome:	The proposed model can choose between parametric and non-parametric information, but relies more on context than parametric knowledge.

Copied to clipboard

Geewook Kim, Minjoon Seo

Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities (2024.emnlp-main)

Challenge:	Recent advances in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility.
Approach:	They propose to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs.
Outcome:	The proposed models achieve significant improvements in inference throughput while maintaining high performance.

Copied to clipboard

Zihao He, Minh Chu, Rebecca Dorn, Siyi Guo, Kristina Lerman

Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models (2024.emnlp-main)

Challenge:	Social scientists use surveys to learn opinions and beliefs of populations, but these methods are slow, costly, and prone to biases.
Approach:	They propose a framework for aligning large language models to online communities by finetuning instruction-output pairs by an advanced LLM to elicit their beliefs.
Outcome:	The proposed framework enables cost-effective and automated surveying of diverse online communities.

Copied to clipboard

Eldar Kurtic, Amir Moeini, Dan Alistarh

Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models (2024.emnlp-main)

Challenge:	a new benchmark for evaluating the mathematical reasoning on large language models is being developed . popularity of reasoning benchmarks is leading to performance saturation and training set contamination.
Approach:	They introduce a benchmark for evaluating the mathematical reasoning on large language models . they find that models struggle with Mathador-LM, scoring lower than average 3rd graders .
Outcome:	The proposed benchmark improves performance on large language models . it also reduces test-set leakage into training data, a new study shows .

Copied to clipboard

Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, David Acuna

One Thousand and One Pairs: A “novel” challenge for long-context language models (2024.emnlp-main)

Challenge:	Despite recent advances in visual language models, their ability to quantitatively reason about object sizes and distances remains underexplored.
Approach:	They propose a manually annotated benchmark of 241 questions designed for quantitative spatial reasoning and a zero-shot prompting technique that encourages VLMs to use reference objects as visual cues.
Outcome:	The proposed technique improves the performance of the top-performing VLMs by 19 points when a reasoning path using a reference object emerges naturally in the response.

Copied to clipboard

Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, Mohit Iyyer

Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation (2024.emnlp-main)

Challenge:	Existing long-context evaluation methods measure surface-level retrieval capabilities, but do not assess performance on the more challenging task of synthesizing distant and underlying information.
Approach:	They propose a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books.
Outcome:	The proposed model performs better on pairs that require only sentence-level retrieval vs. global reasoning . the proposed model also performs worse on speculative fiction books with extensive world-building .

Copied to clipboard

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, Yun-Hsuan Sung

Do LLMs learn a true syntactic universal? (2024.emnlp-main)

Challenge:	evaluating large language models' output is difficult due to the high cost of human evaluation.
Approach:	They propose a family of foundational large autorater models that train on over 100 quality assessment tasks.
Outcome:	The proposed model outperforms models on 8 of 12 autorater benchmarks on 53 quality assessment tasks.

Copied to clipboard

John Hale, Miloš Stanojević

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets (2024.emnlp-main)

Challenge:	linguistics literature has debated whether large multilingual language models learn language universals . Typological generalizations are a key battleground in such debates - e.g. van der Hulst, 2023, chapter 7).
Approach:	They consider a candidate universal for language universals, the Final-over-Final Condition . they suggest that modern language models may need additional sources of bias to become truly human-like .
Outcome:	The proposed model only seems to recognize the Final-over-Final Condition in German, Russian, Hungarian and Serbian .

Copied to clipboard

Oh Joon Kwon, Daiki Matsunaga, Kee-Eung Kim

How Susceptible are Large Language Models to Ideological Manipulation? (2024.emnlp-main)

Challenge:	Reinforcement learning with human feedback (RLHF) and its offline variant Direct Preference Optimization (DPO) are two of the most important methods for language model (LM) alignment.
Approach:	They propose to use a diversity-seeking RL algorithm called GFlowNet-DPO in an offline preference alignment setting to optimize a model's behavior.
Outcome:	Empirical results show that the proposed algorithm generates far more diverse responses than the baseline methods and is still relatively aligned with human values in dialog generation and summarization tasks.

Copied to clipboard

Kai Chen, Zihao He, Jun Yan, Taiwei Shi, Kristina Lerman

Measuring Psychological Depth in Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have the potential to exert substantial influence on public perceptions and interactions with information.
Approach:	They examine how LLMs can learn and generalize ideological biases from their instruction-tuning data.
Outcome:	The LLMs show a startling ability to absorb ideology from one topic and generalize it to even unrelated ones.

Copied to clipboard

Fabrice Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Yildiz, Miryung Kim, Amit Sahai, Nanyun Peng

Media Attitude Detection via Framing Analysis with Events and their Relations (2024.emnlp-main)

Challenge:	Current evaluations of creative stories focus on objective properties of the text, such as its style, coherence, diversity, and creativity.
Approach:	They propose a framework that measures an LLM's ability to produce authentic and narratively complex stories that provoke emotion, empathy, and engagement.
Outcome:	The proposed framework shows that humans can consistently evaluate stories based on the PDS (0.72 Krippendorff’s alpha).

Copied to clipboard

Jin Zhao, Jingxuan Tu, Han Du, Nianwen Xue

Challenge:	a recent study examined the effects of media framing on public perception and understanding of news articles.
Approach:	They propose to extract framing devices employed by media to assess their role in framating the narrative.
Outcome:	The proposed method surpasses baseline models and offers a more detailed and explainable analysis of media framing effects.

Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)

Copied to clipboard

Yang Ba, Michelle Mancenido, Rong Pan

Challenge:	Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data.
Approach:	They propose a calibration method that incorporates synthetic data without compromising accuracy.
Outcome:	The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks.

Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations (2024.emnlp-main)

Copied to clipboard

Sagi Shaier, Ari Kobren, Philip Ogren

Granular Privacy Control for Geolocation with Vision Language Models (2024.emnlp-main)

Challenge:	Existing work on citation generation has focused on unambiguous settings with single answers, failing to address the complexity of real-world scenarios.
Approach:	They propose a task of QA with source citation in ambiguous settings where multiple valid answers exist, where multiple sources exist.
Outcome:	The proposed framework generates multiple answers and cites their sources, allowing users to verify the factuality of each answer and make informed decisions.

Copied to clipboard

Ethan Mendes, Yang Chen, James Hays, Sauvik Das, Wei Xu, Alan Ritter

MedReadMe: A Systematic Study for Fine-grained Sentence Readability in Medical Domain (2024.emnlp-main)

Challenge:	Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions.
Approach:	They develop a benchmark to evaluate the ability of VLMs to moderate geolocation dialogues with users.
Outcome:	a new benchmark evaluates the ability of VLMs to moderate geolocation conversations with users.

Copied to clipboard

Chao Jiang, Wei Xu

Challenge:	Using fine-grained readability measures is the first step towards making medical texts more accessible.
Approach:	They propose a dataset MedReadMe which measures sentences and complex spans with an annotation tool.
Outcome:	The proposed dataset covers 650 linguistic features and additional complex span features, and is compared against state-of-the-art methods using large language models.

MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification (2024.emnlp-main)

Copied to clipboard

Siddhant Bikram Shah, Shuvam Shiwakoti, Maheep Chaudhary, Haohan Wang

FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization (2024.emnlp-main)

Challenge:	a novel dataset of text-embedded images associated with the LGBTQ+ Pride movement is presented in this paper . a new framework for analyzing text-based images is proposed to address this challenge .
Approach:	They propose a new dataset for machine learning that includes hate, targets of hate, stance, humor and a framework for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model.
Outcome:	The proposed framework achieves superior performance on two real-world datasets.

Copied to clipboard

Mingye Zhu, Yi Liu, Quan Wang, Junbo Guo, Zhendong Mao

Challenge:	Recent advances in preference alignment have significantly improved Large Language Models' ability to generate texts that align with human preferences and values.
Approach:	They propose a constrained optimization approach to detect and mitigate update regression with focal attention.
Outcome:	The proposed approach detects and mitigates update regression with focal attention while maintaining excellent overall performance.

StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children’s Story-Based Learning (2024.emnlp-main)

Copied to clipboard

Jiaju Chen, Yuxuan Lu, Shao Zhang, Bingsheng Yao, Yuanzhe Dong, Ying Xu, Yunyao Li, Qianwen Wang, Dakuo Wang, Yuling Sun

MedCoT: Medical Chain of Thought via Hierarchical Expert (2024.emnlp-main)

Challenge:	Existing story reading systems fail to capture the nuances of how education experts think when conducting interactive story reading activities.
Approach:	They propose to use existing question-answering (QA) datasets to capture experts' annotations and thinking process to construct a story-based annotation framework.
Outcome:	The proposed framework captures experts’ annotations and thinking process and can be used to generate 5, 868 expert-annotated QA pairs with real-world knowledge.

Copied to clipboard

Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Zhou, Zuozhu Liu

Varying Sentence Representations via Condition-Specified Routers (2024.emnlp-main)

Challenge:	Existing methods for medical visual question answering lack robustness and reasoning paths for real-world medical diagnostics.
Approach:	They propose a hierarchical expert verification reasoning chain method to enhance interpretability and accuracy in medical visual question answering.
Outcome:	The proposed method outperforms existing methods on four standard Med-VQA datasets.

Copied to clipboard

Ziyong Lin, Quansen Wang, Zixia Jia, Zilong Zheng

Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialogues (2024.emnlp-main)

Challenge:	Existing sentences cannot account for different aspects of semantic similarity between two sentences.
Approach:	They propose a transformer-style framework that generates conditioned sentences . they propose 'conditional' STS, which measures similarity between two sentences based on condition sentences - a task that requires a sentence embedding model capable of generating distinct representations for the same sentence under different conditions.
Outcome:	The proposed framework is superior to existing models on two condition sentences . it can generate conditioned sentences while maintaining model parameters and computational efficiency .

Copied to clipboard

Jiao Ou, Jiayu Wu, Che Liu, Fuzheng Zhang, Di Zhang, Kun Gai

Information Flow Routes: Automatically Interpreting Language Models at Scale (2024.emnlp-main)

Challenge:	Existing methods target instruction dialogues as learning goal and fine-tune user simulators to pose instructions.
Approach:	They propose to use real instruction dialogues to model complex dialogue flows and pose high-quality instructions.
Outcome:	The proposed method generates diverse, in-depth, and insightful instructions for a given dialogue history.

Copied to clipboard

Javier Ferrando, Elena Voita

A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models (2024.emnlp-main)

Challenge:	Current state-of-the-art language models (LMs) are built on top of the Transformer architecture.
Approach:	They propose to build graphs where nodes correspond to token representations and edges to computations . they show that attention heads and subword merging heads are important .
Outcome:	The proposed model can analyze behavior for specific types of predictions, or different domains.

Copied to clipboard

Houquan Zhou, Zhenghua Li, Bo Zhang, Chen Li, Shaopeng Lai, Ji Zhang, Fei Huang, Min Zhang

Representational Analysis of Binding in Language Models (2024.emnlp-main)

Challenge:	Using an LLM for Chinese spelling correction tasks is completely different from previous approaches . given a Chinese character, there may exist many others with the same or similar pronunciations, or with similar shapes.
Approach:	They propose a training-free prompt-free approach to leverage large language models for Chinese spelling correction task.
Outcome:	The proposed model significantly improves performance on five public datasets, enabling them to compete with state-of-the-art domain-general CSC models.

Copied to clipboard

Qin Dai, Benjamin Heinzerling, Kentaro Inui

CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference (2024.emnlp-main)

Challenge:	Existing research has shown that LMs use a concept called Binding ID (BI) to mark entity-attribute pairs, but have not captured the information from entity activations.
Approach:	They propose to localize the Binding ID mechanism by localizing BI information in LMs by encoding it in a low-rank subspace.
Outcome:	The proposed model can infer attributes for a given entity from a container .

Copied to clipboard

Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Gao Zuchen, Fei Mi, Lanqing Hong

ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures (2024.emnlp-main)

Challenge:	Existing studies have not noticed the safety risks of large language models . authors evaluated 1,400 questions in multi-turn dialogue coreference .
Approach:	They are the first to evaluate LLM safety in multi-turn dialogue coreference . they created a dataset of 1,400 questions and tested five open-source models .
Outcome:	The study shows that model safety decreases in multi-turn dialogue coreference scenarios . the highest success rate was with the LLaMA2-Chat-7b model, while the lowest was with mistral-7B-Instruct model .

Copied to clipboard

Tobias Schimanski, Jingwei Ni, Roberto Martín, Nicola Ranger, Markus Leippold

Context-Aware Adapter Tuning for Few-Shot Relation Learning in Knowledge Graphs (2024.emnlp-main)

Challenge:	Qualitative disclosures typically include narrative descriptions of climate-related risks, opportunities, strategies, and governance.
Approach:	They simulate typical tasks of a sustainability analyst by examining 30 sustainability reports with 16 detailed climate-related questions.
Outcome:	The proposed model combines expert knowledge with embeddings in a dataset with over 8.5K unique question-source-answer pairs labeled by different levels of relevance.

Copied to clipboard

Liu Ran, Zhongzhou Liu, Xiaoli Li, Yuan Fang

Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness (2024.emnlp-main)

Challenge:	Existing methods to predict instances for missing relations on knowledge graphs are limited by their limited training examples.
Approach:	They propose a context-aware adapter for few-shot relation learning in KGs . they propose tunable relation adaptation and contextual information for each relation .
Outcome:	Experiments on three benchmark KGs validate the superiority of RelAdapter over state-of-the-art methods.

Copied to clipboard

Shixuan Ma, Quan Wang

Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection (2024.emnlp-main)

Challenge:	Existing zero-shot detection paradigms that use token cohesiveness are not available for large language models.
Approach:	They propose a generic dual-channel detection paradigm that uses token cohesiveness as a plug-and-play module to improve existing zero-shot detectors.
Outcome:	The proposed model is able to detect human-like text in black-box environments.

Copied to clipboard

Zhanpeng Chen, Zhihong Zhu, Xianwei Zhuang, Zhiqi Huang, Yuexian Zou

From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking (2024.emnlp-main)

Challenge:	Existing methods for multimodal intent detection have two limitations: (i) close entanglement of multimodal semantics with modal structures; (ii) insufficient learning of causal effects of semantic and modality-specific information on the final predictions.
Approach:	They propose a Dual-oriented Disentangled Network with Counterfactual Intervention model that decouples semantics-oriented and modality-oriented representations and a Counterfective Intervention Module that applies causal inference to understand causal effects by injecting confounders.
Outcome:	The proposed model overcomes key limitations in existing systems by effectively disentangling and utilizing modality-specific and multimodal semantic information.

Copied to clipboard

Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei

Symbolic Working Memory Enhances Language Models for Complex Rule Application (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated remarkable performance across various tasks, effectively following instructions to meet diverse user needs.
Approach:	They propose a framework for evaluation benchmarks and attack techniques for LLMs and MLLMs to enhance their security.
Outcome:	The proposed frameworks have been exploited to exploit the weaknesses of LLMs and MLLMs.

Copied to clipboard

Siyuan Wang, Zhongyu Wei, Yejin Choi, Xiang Ren

LLoCO: Learning Long Contexts Offline (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) excel in single-step rule application but struggle with multi-step deductive reasoning when rules are presented non-sequentially.
Approach:	They propose to augment LLMs with external working memory and introduce a neurosymbolic framework for rule application that stores facts and rules in both natural language and symbolic forms, enabling precise tracking.
Outcome:	The proposed framework iteratively performs symbolic rule grounding and LLM-based rule implementation.

Copied to clipboard

Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa

Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024.emnlp-main)

Challenge:	Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation.
Approach:	They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA.
Outcome:	The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering.

Copied to clipboard

Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Wang Chen, Anh Tuan Luu

Mentor-KD: Making Small Language Models Better Multi-step Reasoners (2024.emnlp-main)

Challenge:	Existing methods for generating large language models have been criticized for their complexity and instability.
Approach:	They propose a value-based calibration method to better align Large Language Models with human preferences.
Outcome:	The proposed method surpasses existing methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings.

Copied to clipboard

Hojae Lee, Junho Kim, SangKeun Lee

Are Large Language Models Capable of Generating Human-Level Narratives? (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown impressive emergent capabilities by leveraging Chain-of-Thought (CoT) prompting.
Approach:	They propose a Knowledge Distillation approach which transfers multi-step reasoning ability of Large Language Models (LLMs) to smaller LMs by fine-tuning language models of multi- step rationales generated by LLM teachers.
Outcome:	The proposed method is able to transfer multi-step reasoning ability of LLMs to smaller LMs while addressing data quality and soft label provision.

Copied to clipboard

Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng

MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs (2024.emnlp-main)

Challenge:	a recent HCI study has pointed to gaps in machine storytelling ability at the global level . authors show that LLMs have less suspense and less tension than human stories .
Approach:	They propose a computational framework to analyze narratives through three discourse-level aspects.
Outcome:	The proposed framework analyzes narratives through three discourse-level aspects . it shows that LLMs fall short of human abilities in discourse understanding .

Copied to clipboard

Yerin Hwang, Yongil Kim, Yunah Jang, Jeesoo Bang, Hyunkyung Bae, Kyomin Jung

Can Large Language Models Enhance Predictions of Disease Progression? Investigating Through Disease Network Link Prediction (2024.emnlp-main)

Challenge:	Existing methods to manage topic shifts within on-topic dialogues are limited in their ability to generate training datasets.
Approach:	They propose a data generation framework that automatically generates conversational question-answering datasets with natural topic transitions by leveraging relationships between entities in a knowledge graph.
Outcome:	The proposed framework generates conversational question-answering datasets with natural topic transitions and proves its effectiveness in generating dialogues with topic shifts.

Copied to clipboard

Haohui Lu, Usman Naseem

Searching for Best Practices in Retrieval-Augmented Generation (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have made significant strides in various tasks, yet their effectiveness in predicting disease progression remains relatively unexplored.
Approach:	They propose a large language model with graph prompting and Retrieval-augmented generation to enhance the prediction performance of disease comorbidity within disease networks.
Outcome:	The proposed model outperforms Graph Neural Networks and Graph Prompts and Retrieval-Augmented Generation models in disease progression prediction tasks.

Copied to clipboard

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, Xuanjing Huang

Moral Foundations of Large Language Models (2024.emnlp-main)

Challenge:	Retrieval-augmented generation (RAG) techniques have proven to be effective in integrating up-to-date information, mitigating hallucinations, and enhancing response quality, especially in specialized domains.
Approach:	They propose several strategies for deploying RAG that balance performance and efficiency.
Outcome:	The proposed approaches can significantly enhance question-answering capabilities and accelerate the generation of multimodal content using a “retrieval as generation” strategy.

Copied to clipboard

Marwa Abdulhai, Gregory Serapio-García, Clement Crepy, Daria Valter, John Canny, Natasha Jaques

The Zeno’s Paradox of ‘Low-Resource’ Languages (2024.emnlp-main)

Challenge:	Moral foundations theory (MFT) is a psychological assessment tool that decomposes human moral reasoning into five factors, including care/harm, liberty/oppression, and sanctity/degradation.
Approach:	They propose to use moral foundations theory to analyze whether popular LLMs have acquired a bias towards a particular set of moral values.
Outcome:	The proposed model can be adversarially selected to exhibit a particular moral foundations and can affect downstream tasks.

Copied to clipboard

Hellina Hailu Nigatu, Atnafu Tonja, Benjamin Rosman, Thamar Solorio, Monojit Choudhury

Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization (2024.emnlp-main)

Challenge:	'low resource' languages are understudied by the NLP community, while 'high resource' is referred to as 'achieved', while high-resource languages are referred .
Approach:	They qualitatively analyzed 150 papers from the ACL Anthology and popular speech-processing conferences that mention the keyword ‘low-resource.
Outcome:	The proposed analysis reveals that several interacting axes contribute to ‘low-resourceness’ of a language and why that makes it difficult to track progress for each individual language.

Copied to clipboard

Aseem Srivastava, Smriti Joshi, Tanmoy Chakraborty, Md Shad Akhtar

Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks, but their adaptation to domain-specific intricacies remains challenging.
Approach:	They propose to use a planning engine to orchestrate structuring knowledge alignment to achieve high-order planning by encapsulating domain knowledge and leveraging sheaf convolution learning to enhance its understanding of the dialogue’s structural nuances.
Outcome:	The proposed framework improves on existing LLMs and shows that it can generate better summaries with better quality and better execution.

Copied to clipboard

Pritika Ramu, Koustava Goswami, Apoorv Saxena, Balaji Vasan Srinivasan

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment (2024.emnlp-main)

Challenge:	Prior research addresses generating attributions alongside responses in open domains, either per sentence or per paragraph.
Approach:	They propose a method to decompose generated answers for attribution using template-based in-context learning.
Outcome:	The proposed approach enhances the semantic understanding of abstractive and extractive answers.

Copied to clipboard

Yusuke Hirota, Ryo Hachiuma, Chao-Han Yang, Yuta Nakashima

Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging (2024.emnlp-main)

Challenge:	Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text.
Approach:	They compare standard-format captions and recent GCE processes from the perspectives of gender bias and hallucination.
Outcome:	The proposed methods amplify gender bias by 30.9% and increase hallucination by 59.5%.

Copied to clipboard

Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, Cunhang Fan, Zhao Lv, Dianhui Chu, Zhiying Tu, Dianbo Sui

Embedded Named Entity Recognition using Probing Classifiers (2024.emnlp-main)

Challenge:	Existing methods for parameter pruning fail to utilize the knowledge from pruned parameters.
Approach:	They propose a method that uses manifold learning and the Information Bottleneck measure to merge similar layers to preserve model performance.
Outcome:	The proposed method outperforms pruning methods on multiple datasets and LLMs with quantization and achieves substantial compression ratios.

Copied to clipboard

Nicholas Popovic, Michael Färber

Unleashing the Power of Emojis in Texts via Self-supervised Graph Pre-Training (2024.emnlp-main)

Challenge:	Streaming text generation requires separate models during inference, which increases computational cost, or destructive fine-tuning of the language model.
Approach:	They propose an approach which enables streaming named entity recognition in decoder-only language models without fine-tuning them.
Outcome:	The proposed approach maintains high token generation rates with only a negligible decrease in speed of around 1% compared to a baseline of 43.64%.

Copied to clipboard

Zhou Zhang, Dongzeng Tan, Jiaan Wang, Yilong Chen, Jiarong Xu

Data Contamination Can Cross Language Barriers (2024.emnlp-main)

Challenge:	Emojis have gained immense popularity on social media platforms, serving as a common means to supplement or replace text.
Approach:	They propose a graph pre-train framework for text and emoji co-modeling that incorporates two tasks: node-level graph contrastive learning and edge-level link reconstruction learning.
Outcome:	The proposed framework improves on the Xiaohongshu and Twitter datasets with two types of downstream tasks.

Copied to clipboard

Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, Jingbo Shang

Cross-Lingual Transfer

Challenge:	Existing methods to detect contamination of public benchmarks are too superficial to reflect deeper forms of contamination.
Approach:	They propose generalization-based approaches to unmask a cross-lingual form of contamination that inflates LLMs’ performance while evading current detection methods.
Outcome:	The proposed model outperforms existing detection methods while avoiding contamination of public benchmarks in the pre-training data.

Automated Essay Scoring: A Reflection on the State of the Art (2024.emnlp-main)

Copied to clipboard

Shengjie Li, Vincent Ng

Challenge:	Automated essay scoring (AES) is a key application of natural language processing . it is based on a holistic score that summarizes the essay's overall quality .
Approach:	aaron carroll: automated essay scoring is one of the most important applications in NLP . carroll says the task is still far from being solved, but it's still progressing steadily . he says it'll be interesting to see how researchers can improve performance numbers .
Outcome:	a new neural model can beat existing models on a standard evaluation dataset, authors say . authors: the current model is not enough to improve performance numbers . they say it could spark discussion among researchers on how to move forward .

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024.emnlp-main)

Copied to clipboard

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, Zhaopeng Tu

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs (2024.emnlp-main)

Challenge:	Modern large language models (LLMs) have shown remarkable performance on general language tasks but struggle on complex reasoning tasks.
Approach:	They propose a multi-agent debate framework that encourages divergent thinking in LLMs . they propose to break debate and use a judge to obtain a final solution .
Outcome:	The proposed framework encourages divergent thinking in large language models . it is able to generate novel thoughts even if initial stance is incorrect .

Copied to clipboard

Xin Zhou, Ping Nie, Yiwen Guo, Haojie Wei, Zhanqiu Zhang, Pasquale Minervini, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang

CURE: Context- and Uncertainty-Aware Mental Disorder Detection (2024.emnlp-main)

Challenge:	Existing research seeks to enhance RAG performance by retrieving higher-quality documents or designing RAG-specific LLMs, but internal mechanisms that contribute to RAG’s effectiveness remain underexplored.
Approach:	They propose to examine the internal mechanisms within the popular Mixture-of-Expert (MoE)-based LLMs and examine their ability to improve RAG by examining expert activations.
Outcome:	The proposed method significantly improved the ability of Large Language Models (LLMs) to solve knowledge-intensive tasks.

Copied to clipboard

Migyeong Kang, Goun Choi, Hyolim Jeon, Ji Hyun An, Daejin Choi, Jinyoung Han

Challenge:	Existing methods to detect mental disorders focus on the presence of symptoms, but the context of symptoms is often ignored, leading to errors in symptom identification.
Approach:	They propose to use large language models to extract contextual information while introducing an uncertainty-aware decision fusion network that combines predictions of multiple models based on quantified uncertainty values.
Outcome:	The proposed model detects mental disorders even in situations where symptom information is incomplete.

PepRec: Progressive Enhancement of Prompting for Recommendation (2024.emnlp-main)

Copied to clipboard

Yakun Yu, Shi-ang Qi, Baochun Li, Di Niu

In-Context Compositional Generalization for Large Vision-Language Models (2024.emnlp-main)

Challenge:	Large language models (LLMs) have been gaining in-depth performance in natural language processing domains.
Approach:	They propose a training-free prompting framework that captures knowledge from content-based filtering and collaborative filtering to boost recommendation performance with LLMs.
Outcome:	The proposed framework outperforms traditional deep learning recommendation models and prompt-based recommendation systems on two real-world datasets.

Copied to clipboard

Chuanhao Li, Chenchen Jing, Zhen Li, Mingliang Zhai, Yuwei Wu, Yunde Jia

Improving Zero-shot LLM Re-Ranker with Risk Minimization (2024.emnlp-main)

Challenge:	Recent work shows that in-context learning for large language models exhibits compositional generalization capacity.
Approach:	They propose a method to exhibit in-context compositional generalization in large vision-language models by combining visual and linguistic modalities.
Outcome:	The proposed method reduces redundancy and complexity in in-context learning with LVLMs.

Copied to clipboard

Xiaowei Yuan, Zhao Yang, Yequan Wang, Jun Zhao, Kang Liu

Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are effective Query Likelihood Models, but their estimation is biased and the model's accuracy is poor.
Approach:	They propose a framework which leverages Bayesian decision theory to quantify and mitigate this bias.
Outcome:	The proposed framework improves re-ranking, especially in improving the Top-1 accuracy.

Copied to clipboard

Xianwei Zhuang, Zhihong Zhu, Zhanpeng Chen, Yuxin Xie, Liming Liang, Yuexian Zou

Label Confidence Weighted Learning for Target-level Sentence Simplification (2024.emnlp-main)

Challenge:	Large vision-language models produce unfaithful visual hallucinations, also known as visual halluinations, which hinders their application in multimodal understanding and decision-making.
Approach:	They propose a plug-and-play train-free decoding algorithm for mitigating visual hallucinations . they leverage visual information to construct a coarse-to-fine visual view tree .
Outcome:	The proposed algorithm reduces visual hallucinations (VH) by leveraging visual information to construct a coarse-to-fine visual view tree (CFTree)

Copied to clipboard

Xin Ying Qiu, Jingshen Zhang

Quantum Recurrent Architectures for Text Classification (2024.emnlp-main)

Challenge:	Existing methods for sentence simplification use label confidence weighting to generate pseudo-labeled sentences with varying proficiency levels.
Approach:	They propose a label confidence weighting scheme for multi-level sentence simplification that incorporates a weighting system into the training loss of the encoder-decoder model.
Outcome:	The proposed approach outperforms state-of-the-art confidence weighting methods on English grade-level simplification datasets.

Copied to clipboard

Wenduan Xu, Stephen Clark, Douglas Brown, Gabriel Matos, Konstantinos Meichanetzidis

Tree of Problems: Improving structured problem solving with compositionality (2024.emnlp-main)

Challenge:	Recurrent neural networks (RNNs) were transformative in the early stages of neural NLP, but are now becoming a potentially transformative technology.
Approach:	They develop quantum RNNs with cells based on Parametrised Quantum Circuits (PQCs) they use an angle encoder to define a (non-linear) mapping from a classical word embedding into the quantum Hilbert space.
Outcome:	The proposed models are competitive with RNN baselines on the Rotten Tomatoes dataset and emulator results show they perform better than classical models.

Copied to clipboard

Armel Zebaze, Benoît Sagot, Rachel Bawden

What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable performance across multipletasks through in-context learning.
Approach:	They propose a Tree of Problems (ToP) that is a simpler version of Tree of Thoughts (toT) they propose 'in-context learning' is the ability of Large Language Models (LLMs) to perform a task with the help of a few demonstrations within their context.
Outcome:	The proposed approach outperforms ToT and GoT and performs better on complex reasoning tasks.

Copied to clipboard

Beatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof-Arenas, Luisa Bentivogli

Seg2Act: Global Context-aware Action Generation for Document Logical Structuring (2024.emnlp-main)

Challenge:	Existing bias measurements do not reflect the gender disparities found in machine translation.
Approach:	They conduct a human-centered study to examine if and to what extent bias in machine translation brings harms with tangible costs, such as quality of service gaps between women and men.
Outcome:	The findings advocate for human-centered approaches that can inform the societal impact of bias.

Copied to clipboard

Zichao Li, Shaojie He, Meng Liao, Xuanang Chen, Yaojie Lu, Hongyu Lin, Yanxiong Lu, Xianpei Han, Le Sun

Challenge:	Document logical structuring is crucial for document intelligence due to the complexity of text segment dependencies in the document.
Approach:	They propose an end-to-end, generation-based method for document logical structuring that generates the action sequence via a global context-aware generative model and updates its global context and current logical structure based on the generated actions.
Outcome:	Experiments on ChCatExt and HierDoc datasets show that Seg2Act performs better than previous methods in both supervised and transfer learning settings.

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning (2024.emnlp-main)

Copied to clipboard

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu

Revisiting the Robustness of Watermarking to Paraphrasing Attacks (2024.emnlp-main)

Challenge:	Existing approaches to prune LLMs rely on the C4 dataset as calibration data . arithmetic datasets perform better than pre-training datasets for pruning, whereas chain-of-thought is only useful on certain tasks.
Approach:	They evaluate the selection of calibration data for LLM pruning across a wide range of datasets . they find that C4 is not the optimal calibration data, and that CoT is only useful on certain tasks.
Outcome:	The chosen calibration data significantly impacts the performance of pruned LLMs, the authors found . their results shed light on the importance of carefully selecting calibration data for LLM pruning .

Copied to clipboard

Saksham Rastogi, Danish Pruthi

A Survey of Ontology Expansion for Conversational Understanding (2024.emnlp-main)

Challenge:	Recent watermarking techniques slightly modify the output probabilities of LMs to embed a signal in the generated output that can later be detected.
Approach:	They propose to use a model to produce a watermarking signal that is invariant to semantically-similar inputs to undo the effects of watermarks.
Outcome:	The proposed method undoes the effects of watermarking and dramatically improves the effectiveness of paraphrasing attacks with limited access to model generations.

Copied to clipboard

Jinggui Liang, Yuxia Wu, Yuan Fang, Hao Fei, Lizi Liao

Calibrating Language Models with Adaptive Temperature Scaling (2024.emnlp-main)

Challenge:	Current methods for conversational understanding rely on static ontologies, limiting their ability to handle new and unforeseen user needs.
Approach:	They propose to review the state-of-the-art techniques in OnExp for conversational understanding and highlight emerging frontiers . they categorize existing literature into three main areas: (1) New Intent Discovery, (2) New Slot-Value Discovery, and (3) Joint OnExp.
Outcome:	The proposed methods highlight several emerging frontiers in OnExp to improve agent performance in real-world scenarios and discuss their corresponding challenges.

Copied to clipboard

Johnathan Xie, Annie Chen, Yoonho Lee, Eric Mitchell, Chelsea Finn

Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance? (2024.emnlp-main)

Challenge:	Large language models' confidence scores are degraded after fine-tuning with reinforcement learning from human feedback.
Approach:	They propose a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction.
Outcome:	Adaptive temperature scaling improves calibration by over 10% compared to prior methods . RLHF fine-tuning improves model accuracy, but degradation is not significant .

Copied to clipboard

Fumiya Uchiyama, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo

Why do objects have many names? A study on word informativeness in language use and lexical systems (2024.emnlp-main)

Challenge:	Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and reasoning tasks.
Approach:	They pre-trained decoder-based language models from scratch using ten programming languages and three natural language datasets.
Outcome:	The proposed models outperform natural languages on logical reasoning tasks.

Copied to clipboard

Eleonora Gualdoni, Gemma Boleda

Challenge:	lexical systems contain many different words that can be assigned to the same object . studies on language use have explored how speakers adapt their referring expressions to communicate in context without tackling in-context communication.
Approach:	They propose a simple measure of informativeness for words and lexical systems, grounded in a visual space, and analyze color naming data for English and Mandarin Chinese.
Outcome:	The proposed system allows for a soft mapping between referents and words, taking into account both in-context communication and the structure of the lexical system.

Dual-Space Knowledge Distillation for Large Language Models (2024.emnlp-main)

Copied to clipboard

Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, Jinan Xu

NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition (2024.emnlp-main)

Challenge:	Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities.
Approach:	They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD.
Outcome:	The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies.

Copied to clipboard

Elena Merdjanovska, Ansar Aynetdinov, Alan Akbik

Named Entity Recognition

Challenge:	Existing approaches to named entity recognition often contain a significant percentage of incorrect labels for entity types and boundary boundaries.
Approach:	They propose a noise-robust learning approach that learns from data with partially incorrect labels.
Outcome:	The proposed methods are based on simulated noise and are easier to handle than simulated real noise caused by human error or semi-automatic annotation.

On the Universal Truthfulness Hyperplane Inside LLMs (2024.emnlp-main)

Copied to clipboard

Junteng Liu, Shiqi Chen, Yu Cheng, Junxian He

PairDistill: Pairwise Relevance Distillation for Dense Retrieval (2024.emnlp-main)

Challenge:	Recent studies have explored hallucinations through the lens of internal representations, proposing mechanisms to decipher LLMs’ adherence to facts.
Approach:	They propose to train a universal truthfulness hyperplane that distinguishes the model’s factually correct and incorrect outputs on a diverse collection of over 40 datasets and examine its cross-task, cross-domain, and in-domain generalization.
Outcome:	The proposed model is able to distinguish factual outputs from incorrect outputs on a diverse collection of over 40 datasets.

Copied to clipboard

Chao-Wei Huang, Yun-Nung Chen

User Inference Attacks on Large Language Models (2024.emnlp-main)

Challenge:	Recent advances in dense retrieval have demonstrated remarkable efficacy compared to traditional sparse retrieval methods.
Approach:	They propose to use pairwise relevance distillation to leverage pairwise reranking to enrich the training of dense retrieval models.
Outcome:	The proposed method outperforms existing methods and achieves state-of-the-art results on multiple benchmarks.

Copied to clipboard

Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher Choquette-Choo, Zheng Xu

HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy (2024.emnlp-main)

Challenge:	a large amount of data written by humans is used to train and fine-tune large language models.
Approach:	They propose to infer if a user's data was used to train an LLM by using example-level differential privacy.
Outcome:	The proposed attacks are easy to employ and only require black-box access to an LLM and a few samples from the user.

Copied to clipboard

YongKang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schuetze

Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models (2024.emnlp-main)

Challenge:	Existing approaches to fine-tuning language models use zeroth-order optimizers to conserve GPU memory.
Approach:	They propose a full-parameter fine-tuning strategy which updates a subset of parameters at each training step.
Outcome:	The proposed approach reduces the amount of gradients and optimizer state parameters residing in GPU memory at the same time, thereby reducing GPU memory usage.

Copied to clipboard

Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, Aimin Zhou

Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation (2024.emnlp-main)

Challenge:	Existing studies have revealed that Large Vision-Language Models suffer from hallucinations in practice, including object hallucines, spatial hallucinos, attribute hallucinications, etc.
Approach:	They propose to use CLIP model to mitigate object hallucinations by using a data augmentation method to create negative samples with a variety of hallucinian issues.
Outcome:	The proposed method mitigates object hallucinations and can be used as a visual encoder, effectively alleviating the object halluination issue in LVLMs.

Copied to clipboard

Matthew Raffel, Victor Agostinelli, Lizhong Chen

ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedback (2024.emnlp-main)

Challenge:	Current fine-tuning methods to adapt LLMs for simultaneous translation suffer from several issues, such as unnecessarily expanded training sets, increased prompt sizes, or restriction to a single decision policy.
Approach:	They propose a new paradigm for fine-tuning large language models for simultaneous translation using an attention mask approach.
Outcome:	The proposed model improves translation quality compared to state-of-the-art models on five language pairs while reducing the computational cost.

Copied to clipboard

Qinzhuo Wu, Wei Liu, Jian Luan, Bin Wang

Challenge:	Recent studies have shown that tool-augmented large language models can interact with external tools in multiple rounds and provide a final answer.
Approach:	They propose a tool-augmented large language model that can interact with external tools in multiple rounds and provide a final answer to an instruction.
Outcome:	The proposed framework significantly improves Match Rate, Pass Rate and Win Rate by 26.8%, 20.2%, and 5.6% compared to the SOTA model.

Please note that I’m just an AI: Analysis of Behavior Patterns of LLMs in (Non-)offensive Speech Identification (2024.emnlp-main)

Copied to clipboard

Esra Dönmez, Thang Vu, Agnieszka Falenska

How to Compute the Probability of a Word (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are becoming a part of our everyday lives by being used as tools for information search, content creation, writing assistance, and many more.
Approach:	They propose to use Large Language Models to detect offensive online language in applications with social risk, such as late-life companions and online content moderators.
Outcome:	The proposed models fail to detect offensive language and are therefore unsuitable for use in social applications such as late-life companions and online content moderators.

Copied to clipboard

Tiago Pimentel, Clara Meister

A linguistically-motivated evaluation methodology for unraveling model’s abilities in reading comprehension tasks (2024.emnlp-main)

Challenge:	Language models estimate a probability distribution over strings in a natural language . many recent linguistic studies have been incorrectly computing word probabilities .
Approach:	They propose to use the correct method to compute word probabilities . they highlight issues when relying on models that use end-of-word tokenisers .
Outcome:	Empirically, correcting the widespread bug affects measured outcomes in sentences and lexical optimisation analyses.

Copied to clipboard

Elie Antoine, Frederic Bechet, Géraldine Damnati, Philippe Langlais

Machine Reading Comprehension

Challenge:	Existing models fail for linguistic characteristics of input examples, despite the impressive quantity of scientific studies dedicated to them, the capabilities, limitations, and risks of these models remain largely unknown.
Approach:	They propose to use semantic frame annotation to characterize examples by a small number of complexity factors to account for model’s difficulty.
Outcome:	The proposed evaluation methodology is based on the intuition that certain examples consistently yield lower scores regardless of model size or architecture.

GuardBench: A Large-Scale Benchmark for Guardrail Models (2024.emnlp-main)

Copied to clipboard

Elias Bassani, Ignacio Sanchez

Challenge:	Lack of a standard benchmark for guardrail models poses significant evaluation issues . lack of standardized benchmark makes it hard to compare results across scientific publications.
Approach:	They propose a large-scale benchmark for guardrail models comprising 40 safety evaluation datasets.
Outcome:	The proposed model achieves competitive results without specific fine-tuning without the need for specific fine tuning.

Generate-on-Graph: Treat LLM as both Agent and KG for Incomplete Knowledge Graph Question Answering (2024.emnlp-main)

Copied to clipboard

Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Jun Zhao, Kang Liu

Language models and brains align due to more than next-word prediction and word-level information (2024.emnlp-main)

Challenge:	Existing methods to integrate LLMs with Knowledge Graphs (KGs) however, these methods are often incomplete to cover all the knowledge required to answer questions.
Approach:	They propose to integrate LLMs with Knowledge Graphs (KGs) to address insufficient knowledge and hallucination issues in Large Language Models.
Outcome:	The proposed method outperforms existing methods on two datasets.

Copied to clipboard

Gabriele Merlin, Mariya Toneva

LLMEdgeRefine: Enhancing Text Clustering with LLM-Based Boundary Point Refinement (2024.emnlp-main)

Challenge:	Pretrained language models have been shown to significantly predict brain recordings of people comprehending language.
Approach:	They propose to use two perturbations to design contrasts that control for different types of information.
Outcome:	The proposed model is largely agnostic about the exact linguistic information contained in the conceptual quantities "word-level information" and "multi-word information".

Copied to clipboard

Zijin Feng, Luyang Lin, Lingzhi Wang, Hong Cheng, Kam-Fai Wong

Text Clustering

Challenge:	LLMEdgeRefine is an iterative clustering method enhanced by large language models . existing clustering methods struggle with domain-specific fine-tuning and outliers .
Approach:	They propose an iterative clustering method enhanced by large language models focusing on edge points refinement . authors propose to use LLMs to iterate clusters and iterating to improve semantic coherence .
Outcome:	The proposed method outperforms state-of-the-art methods and offers robustness, adaptability, and cost-efficiency for diverse text clustering applications.

CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures (2024.emnlp-main)

Copied to clipboard

Ekaterina Sviridova, Anar Yeginbergen, Ainara Estarrona, Elena Cabrio, Serena Villata, Rodrigo Agerri

A Simple and Effective L_2 Norm-Based Strategy for KV Cache Compression (2024.emnlp-main)

Challenge:	Existing tools to aid residents in teaching medical doctors to explain decisions are a key objective of AI in education.
Approach:	They present a multilingual dataset for Medical Question Answering where doctors can annotate correct and incorrect diagnoses with argument components and argument relations.
Outcome:	The proposed dataset consists of 558 clinical cases with explanations in English, Spanish, French, Italian and annotated with argument components and argument relations.

Copied to clipboard

Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini

GOME: Grounding-based Metaphor Binding With Conceptual Elaboration For Figurative Language Illustration (2024.emnlp-main)

Challenge:	Existing approaches to reduce the KV cache size involve fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce sequence length.
Approach:	They find a correlation between the L2 norm and attention scores over cached KV pairs . they compress the KV cache based on the L1 norm of key embeddings .
Outcome:	The proposed approach reduces the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy.

Copied to clipboard

Linhao Zhang, Jintao Liu, Li Jin, Hao Wang, Kaiwen Wei, Guangluan Xu

D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation (2024.emnlp-main)

Challenge:	Existing Large Language Models (LLMs) and multimodal models are unable to illustrate figurative language based on literal objects, ignoring the underlying groundings and associations across disparate metaphorical domains.
Approach:	They propose a grounding-based method for metaphor illustration that integrates metaphorical knowledge into systematic instructions for existing large language models.
Outcome:	The proposed method is superior to existing LLMs, diffusion models, or their direct collaboration.

Copied to clipboard

Aida Mostafazadeh Davani, Mark Diaz, Dylan Baker, Vinodkumar Prabhakaran

PALM: Few-Shot Prompt Learning for Audio Language Models (2024.emnlp-main)

Challenge:	Recent studies on annotator subjectivity focus on Western contexts and only document differences across age, gender, or racial groups.
Approach:	They propose a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K English sentences annotated by a pool of more than 4k annotators from 21 countries.
Outcome:	The proposed dataset captures annotators’ moral values along six moral foundations: care, equality, proportionality, authority, loyalty, and purity.

Copied to clipboard

Asif Hanif, Maha Agro, Mohammad Qazi, Hanan Aldarmaki

Annotator-Centric Active Learning for Subjective NLP Tasks (2024.emnlp-main)

Challenge:	Audio-Language Models (ALMs) have recently achieved remarkable success in zero-shot audio recognition tasks, which match features of audio waveforms with class-specific text prompt features.
Approach:	They propose a method which optimizes the feature space of the text encoder branch and optimizes audio waveform features with text prompt features.
Outcome:	The proposed method outperforms existing methods while being less demanding.

Copied to clipboard

Michiel Van Der Meer, Neele Falk, Pradeep Murukannaiah, Enrico Liscio

On the Proper Treatment of Tokenization in Psycholinguistics (2024.emnlp-main)

Challenge:	Annotator-centric active learning addresses the high costs of collecting human annotations by strategically annotating the most informative samples.
Approach:	They propose annotator-centric active learning which incorporates an annotation strategy following data sampling to approximate the full diversity of human judgments.
Outcome:	The proposed approach improves data efficiency and performs well in annotator-centric evaluations.

Copied to clipboard

Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, Ryan Cotterell

Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation (2024.emnlp-main)

Challenge:	Language models are used in computational psycholinguistics to test theories that relate the surprisal of a region of interest to its cognitive cost experienced by readers.
Approach:	They propose to marginalize token-level language models into character-level ones before they are used in psycholinguistic studies.
Outcome:	The proposed model over token strings is better than character-level model, the authors show . the proposed model marginalizes token-level models into character-based models before they are used in psycholinguistic studies.

Copied to clipboard

Anas Himmi, Guillaume Staerman, Marine Picot, Pierre Colombo, Nuno Guerreiro

Jailbreaking LLMs with Arabic Transliteration and Arabizi (2024.emnlp-main)

Challenge:	Neural Machine Translation (NMT) systems suffer from various pathologies, including the generation of translations that are detached from the source content, typically known as hallucinations.
Approach:	They propose to combine detectors and introduce a method for aggregating detectors to detect hallucinations.
Outcome:	The proposed method provides a promising step towards evermore reliable machine translation systems.

Copied to clipboard

Mansour Ghanim, Saleh Almohaimeed, Mengxin Zheng, Yan Solihin, Qian Lou

Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are vulnerable to ‘jailbreak’ attacks, focusing on the Arabic language and its various forms.
Approach:	They propose to use Arabic transliteration and chatspeak to generate unsafe content on platforms like OpenAI GPT-4 and Anthropic Claude 3 Sonnet.
Outcome:	The proposed model could generate unsafe content on platforms like OpenAI GPT-4 and Anthropic Claude 3 Sonnet, highlighting the need for more comprehensive safety training across all language forms.

Copied to clipboard

Zara Siddique, Liam Turner, Luis Espinosa-Anke

Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks (2024.emnlp-main)

Challenge:	Existing research on stereotypes in large language models is limited and focuses on African Ameri- F.
Approach:	They propose to use global bias to probe a set of large language models via perplexity to determine how certain stereotypes are represented in the model's internal representations.
Outcome:	The proposed model amplifys harmful stereotypes and shows that the demographic groups associated with stereotypes remain consistent across model likelihoods and outputs.

Copied to clipboard

Changho Lee, Janghoon Han, Seonghyeon Ye, Stanley Jungkyu Choi, Honglak Lee, Kyunghoon Bae

Recurrent Alignment with Hard Attention for Hierarchical Text Rating (2024.emnlp-main)

Challenge:	Experimental results show that instruction tuning improves zero-shot generalization across various tasks and improves performance of specific tasks.
Approach:	They propose a task selection method that leverages instruction information alone to identify relevant tasks and optimize instruction tuning for specific tasks.
Outcome:	The proposed method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task.

Copied to clipboard

Chenxi Lin, Ren Jiayu, Guoxiu He, Zhuoren Jiang, Haiyan Yu, Xiaomin Zhu

CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification (2024.emnlp-main)

Challenge:	Large language models excel at understanding and generating plain text, but they are not tailored to handle hierarchical text structures or directly predict task-specific properties such as text rating.
Approach:	They propose a framework that integrates Recurrent Alignment with Hard Attention to analyze hierarchically structured text.
Outcome:	The proposed framework outperforms existing state-of-the-art methods on three hierarchical text rating datasets.

Copied to clipboard

Junhui He, Shangyu Wu, Weidong Wen, Chun Jason Xue, Qingan Li

Semformer: Transformer Language Models with Semantic Planning (2024.emnlp-main)

Challenge:	Existing methods for activation sparsification do not capture the relationship between activation and model performance.
Approach:	They propose a general activation sparsification approach using channel-wise thresholding and selective sparsifying to capture the relationship between activation and model performance.
Outcome:	The proposed approach reduces the number of activated neurons during inference by 1.27x over eight downstream tasks while activating fewer parameters than existing methods.

Copied to clipboard

Yongjing Yin, Junran Ding, Kai Song, Yue Zhang

DocCGen: Document-based Controlled Code Generation (2024.emnlp-main)

Challenge:	Neural language models (LLMs) employ teacher forcing to predict tokens based on preceding ground truth tokens.
Approach:	They propose a method for training a Transformer language model that explicitly models the semantic planning of response.
Outcome:	The proposed method exhibits near-perfect performance and mitigates shortcut learning.

Copied to clipboard

Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Kumar, Pushpak Bhattacharyya

Semantics and Sentiment: Cross-lingual Variations in Emoji Use (2024.emnlp-main)

Challenge:	Large language models (LLMs) produce state-of-the-art performance on natural language to code generation for resource-rich general-purpose languages like C++, Java, and Python.
Approach:	They propose a framework that breaks the NL-to-Code generation task into two steps . they use library documentation to detect the correct libraries and schema rules extracted from the documentation to constrain the decoding .
Outcome:	The proposed framework improves different sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code.

Copied to clipboard

Giulio Zhou, Sydelle De Souza, Ella Markham, Oghenetekevwe Kwakpovwe, Sumin Zhao

The Emergence of Compositional Languages in Multi-entity Referential Games: from Image to Graph Representations (2024.emnlp-main)

Challenge:	emojis have been used in social media for a decade but have been inconsistently used in contexts and in isolation.
Approach:	They develop a corpus containing literal meanings for emojis defined by L1 speakers in three languages to assess their e-mail sentiments.
Outcome:	The proposed method shows that emoji semantics differ across languages and how it interacts with sentiment in e-mails.

Copied to clipboard

Daniel Akkerman, Phong Le, Raquel Alhama

Transformers are Multi-State RNNs (2024.emnlp-main)

Challenge:	Language Emergence research uses jointly trained artificial agents to solve a task.
Approach:	They propose a multi-entity game in which targets include multiple entities that are spatially related.
Outcome:	The proposed multi-entity game shows that the emergent languages exhibit a considerable degree of compositionality, but not over all features.

Copied to clipboard

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, Roy Schwartz

Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization (2024.emnlp-main)

Challenge:	Schwartz et al., 2017) have been using transformers for long-range tasks for NLP since the 1990s.
Approach:	They propose a transformer-only transformer with unlimited hidden state size that can be converted into bounded multistate RNNs by fixing the size of their hidden state.
Outcome:	The proposed compression policy outperforms baseline compression policies on long range tasks and LLMs.

Copied to clipboard

Niyati Bafna, Kenton Murray, David Yarowsky

Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion (2024.emnlp-main)

Challenge:	Xue et al., 2021) show that large language models suffer from performance degradation on unseen closely-related languages and dialects relative to their high-resource language neighbour (HRLN).
Approach:	They propose to model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN.
Outcome:	The proposed model offers insights on model robustness to isolated and composed linguistic phenomena and the impact of task and HRL characteristics on PD.

Copied to clipboard

Kerem Zaman, Leshem Choshen, Shashank Srivastava

Collective Critics for Creative Story Generation (2024.emnlp-main)

Challenge:	a recent study examined the effects of model fusion on learning of shortcuts and social biases in fine-tuned language models.
Approach:	They investigate whether model fusion can be used to reduce unwanted knowledge . they examine classification tasks with artificially fusioned models .
Outcome:	The proposed model fusion can reduce unshared knowledge, the authors show . their results show that the model merged models can improve performance and generalize .

Copied to clipboard

Minwook Bae, Hyounghun Kim

Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have enabled the automatic generation of long-form stories containing several thousand words.
Approach:	They propose a framework that creates a story plan and generates based on it, and integrates 'creative' revision mechanism into long-form story generation process.
Outcome:	The proposed framework can significantly enhance story creativity and reader engagement while maintaining narrative coherence.

Copied to clipboard

Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, Alex Warstadt

Challenge:	Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication.
Approach:	They propose a functional pressure that speakers modulate information rate based on location within a hierarchically-structured model of discourse.
Outcome:	The proposed hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication.

Model-based Preference Optimization in Abstractive Summarization without Human Feedback (2024.emnlp-main)

Copied to clipboard

Jaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo, Taesup Kim

Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation? (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) can generate fluent summaries but often introduce inaccuracies by hallucinating content not found in the source document.
Approach:	They propose a method to fine-tune Large Language Models for improved summarization abilities without any human feedback.
Outcome:	The proposed method significantly improves the quality of generated summaries without any human feedback.

Copied to clipboard

Wataru Hashimoto, Hidetaka Kamigaito, Taro Watanabe

NeuroTrialNER: An Annotated Corpus for Neurological Diseases and Therapies in Clinical Trial Registries (2024.emnlp-main)

Challenge:	Named Entity Recognition (NER) is a key task in NLP to find mentions of named entities and classify them into predefined categories.
Approach:	They investigated the impact of data augmentation on confidence calibration and uncertainty estimation in Named Entity Recognition (NER) tasks.
Outcome:	The data augmentation improves calibration and uncertainty in cross-genre and cross-lingual setting, especially in-domain setting.

Copied to clipboard

Simona Doneva, Tilia Ellendorff, Beate Sick, Jean-Philippe Goldman, Amelia Cannon, Gerold Schneider, Benjamin Ineichen

Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting (2024.emnlp-main)

Challenge:	Despite substantial investment, developing new treatments for neurological conditions is a challenging and often unsuccessful endeavour.
Approach:	They propose a corpus for named entity recognition that is annotated clinical trial summaries from ClinicalTrials.gov.
Outcome:	The proposed corpus is annotated for neurological diseases, therapeutic interventions, and control treatments and achieves a close-to-human performance.

Copied to clipboard

Maxime Kayser, Bayar Menzat, Cornelius Emde, Bogdan Bercean, Alex Novak, Abdalá Morgado, Bartlomiej Papiez, Susanne Gaube, Thomas Lukasiewicz, Oana-Maria Camburu

Towards Faithful Knowledge Graph Explanation Through Deep Alignment in Commonsense Question Answering (2024.emnlp-main)

Challenge:	XAI models are being used in safety-critical domains, but their use is limited due to their limited transparency and insufficient model robustness.
Approach:	They evaluated visual, natural language and a combination of both modalities to examine how users use them.
Outcome:	The proposed model is more robust and transparent than previous models.

Copied to clipboard

Weihe Zhai, Arkaitz Zubiaga, Bingquan Liu, Chengjie Sun, Yalong Zhao

Generation with Dynamic Vocabulary (2024.emnlp-main)

Challenge:	Current methods for generating faithful explanations overlook path decoding faithfulness, leading to divergence between graph encoder outputs and model predictions.
Approach:	They propose an algorithm to assess KG representation reliability and an LM-KG distribution-aware Alignment algorithm to improve explanation faithfulness without ground truth.
Outcome:	The proposed algorithm improves explanation faithfulness without ground truth and significantly improves fidelity and model performance.

Copied to clipboard

Yanting Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, Xiaoling Wang

Argument Relation Classification through Discourse Markers and Adversarial Training (2024.emnlp-main)

Challenge:	Using static vocabulary, vocabulary is ignored in advanced generation tasks.
Approach:	They propose a dynamic vocabulary that can involve arbitrary text spans during generation.
Outcome:	The proposed vocabulary can be deployed in a plug-and-play way, thus is attractive for various downstream applications.

Copied to clipboard

Michele Contalbo, Francesco Guerra, Matteo Paganelli

Getting The Most Out of Your Training Data: Exploring Unsupervised Tasks for Morphological Inflection (2024.emnlp-main)

Challenge:	Argument relation classification (ARC) identifies supportive, contrasting and neutral relations between argumentative units.
Approach:	They propose an argument relation classifier that integrates knowledge of discourse markers into a pre-trained RoBERTa model.
Outcome:	The proposed model outperforms existing methods and learns discriminative sentence embeddings supporting the task.

Copied to clipboard

Abhishek Purushothama, Adam Wiemerslage, Katharina Wense

Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval (2024.emnlp-main)

Challenge:	Pre-trained transformers have been shown to be effective in many natural language tasks, but are under-explored for character-level sequence to sequence tasks.
Approach:	They propose to use pre-trained transformers for character-level morphological inflection in several languages to train models for unsupervised tasks.
Outcome:	The proposed model outperforms the best two shared tasks on morphological inflection and graphemeto-phoneme conversion benchmarks.

Copied to clipboard

Dae Yon Hwang, Bilal Taha, Harshit Pande, Yaroslav Nechaev

Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Existing methods for zero-shot learning are sparse, but have been used for dense retrieval (DR) .
Approach:	They propose a novel Universal Document Linking algorithm which links similar documents to enhance synthetic query generation across multiple datasets with different characteristics.
Outcome:	The proposed algorithm surpasses state-of-the-art methods in zero-shot cases.

Copied to clipboard

Po-Heng Chen, Yun-Nung Chen

Prove Your Point!: Bringing Proof-Enhancement Principles to Argumentative Essay Generation (2024.emnlp-main)

Challenge:	Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks.
Approach:	They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks .
Outcome:	The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters.

Copied to clipboard

Ruiyu Xiao, Lei Wu, Yuhang Gou, Weinan Zhang, Ting Liu

Challenge:	Existing arguments generation methods often overlook connections between opinions . Existing methods struggle with providing compelling proof .
Approach:	They propose a two-stage framework for argumentative essay generation with a focus on logical enhancement.
Outcome:	The proposed framework generates argumentative essays with better logical validity and persuasiveness than baseline models.

TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning (2024.emnlp-main)

Copied to clipboard

Kate Sanders, Nathaniel Weir, Benjamin Van Durme

Unsupervised Extraction of Dialogue Policies from Conversations (2024.emnlp-main)

Challenge:	TV-TREES is the first multimodal entailment tree generator for video understanding . it searches for trees of enanglement relationships between text-video evidence and higher-level conclusions that prove question-answer pairs.
Approach:	They propose a multimodal entailment tree generator that promotes interpretable joint-modality reasoning by searching for trees of enanglement relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs.
Outcome:	The proposed approach performs on the TVQA benchmark and shows that it is state-of-the-art on full clips.

Copied to clipboard

Makesh Narsimhan Sreedhar, Traian Rebedea, Christopher Parisien

GRIZAL: Generative Prior-guided Zero-Shot Temporal Action Localization (2024.emnlp-main)

Challenge:	Large language models (LLMs) are used to extract dialogue policies from conversational data.
Approach:	They propose a method for extracting dialogue policies from conversational data using canonical forms and graph traversal algorithms.
Outcome:	The proposed method gives conversation designers greater control and improves the process of developing dialogue policies.

Copied to clipboard

Onkar Susladkar, Gayatri Deshmukh, Vandan Gorade, Sparsh Mittal

Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)

Challenge:	Existing methods to temporally localize videos without prior training examples are lacking due to the complexity of annotated videos.
Approach:	They propose a model that uses multimodal embeddings and dynamic motion cues to localize actions effectively.
Outcome:	GRIZAL outperforms state-of-the-art zero-shot temporal action localization models on ActivityNet, Thumos14 and Charades-STA datasets.

Copied to clipboard

Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, Junmo Kim

FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture (2024.emnlp-main)

Challenge:	Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks.
Approach:	They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks.
Outcome:	The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities.

Copied to clipboard

Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, Desmond Elliott

A Two-Step Approach for Data-Efficient French Pronunciation Learning (2024.emnlp-main)

Challenge:	FoodieQA is a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China.
Approach:	They evaluate vision–language Models and large language models on unseen food images and corresponding questions.
Outcome:	The proposed dataset evaluates vision–language Models and large language models on unseen food images and corresponding questions.

Copied to clipboard

Hoyeon Lee, Hyeeun Jang, Jonghwan Kim, Jaemin Kim

Exploring Intra and Inter-language Consistency in Embeddings with ICA (2024.emnlp-main)

Challenge:	Recent studies have addressed intricate phonological phenomena in French, relying on extensive linguistic knowledge or a significant amount of sentence-level pronunciation data.
Approach:	They propose a grapheme-to-phoneme and post-lexical processing approach to address French phonological phenomena using sentence-level pronunciation data.
Outcome:	The proposed approach mitigates the lack of extensive labeled data and serves as a feasible solution for addressing French phonological phenomena even under resource-constrained environments.

Copied to clipboard

Rongzhi Li, Takeru Matsuda, Hitomi Yanaka

DetoxLLM: A Framework for Detoxification with Explanations (2024.emnlp-main)

Challenge:	Existing studies have shown that ICA can reveal universal semantic axes across languages but lack verification of consistency of independent components within and across languages.
Approach:	They propose to use independent component analysis to identify independent components that are more interpretable than PCA to find universal semantic axes.
Outcome:	The proposed framework ensures the reliability and universality of semantic axes.

Copied to clipboard

Md Tawkat Islam Khondaker, Muhammad Abdul-Mageed, Laks Lakshmanan

Comparing a BERT Classifier and a GPT classifier for Detecting Connective Language Across Multiple Social Media (2024.emnlp-main)

Challenge:	DetoxLLM is a comprehensive end-to-end detoxification framework for toxic language.
Approach:	They propose a comprehensive end-to-end detoxification framework that tackles toxic language across platforms.
Outcome:	The proposed detoxification framework outperforms the SoTA model on human-annotated parallel corpus and offers explanation to promote transparency and trustworthiness.

Copied to clipboard

Josephine Lukito, Bin Chen, Gina Masullo, Natalie Stroud

ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (2024.emnlp-main)

Challenge:	Using social media, researchers have built a variety of text classifiers to understand short-form text . however, there is little discussion regarding what desired language on social media would look like .
Approach:	They propose an approach for detecting connective language from social media discussions using BERT and GPT-3.5 turbo.
Outcome:	The proposed classifier outperforms the existing classifiers in detecting connective language from social media discussions.

Copied to clipboard

Yash Akhauri, Ahmed AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander Rush, Safeen Huda, Mohamed Abdelfattah

Emotion Granularity from Text: An Aggregate-Level Indicator of Mental Health (2024.emnlp-main)

Challenge:	Prior work has focused on contextual sparsity, but it has not been successful.
Approach:	They propose a novel pruning predictor that can shadow the LLM behavior and enforce better sparsity patterns.
Outcome:	The proposed model can shadow the LLM behavior and enforce better sparsity patterns, resulting in 15% improvement in end-to-end accuracy compared to prior methods.

Copied to clipboard

Krishnapriya Vishnubhotla, Daniela Teodorescu, Mallory Feldman, Kristen Lindquist, Saif Mohammad

BLSP-Emo: Towards Empathetic Large Speech-Language Models (2024.emnlp-main)

Challenge:	Emotions play a central role in how we construct meaning and communicate with others.
Approach:	They propose to use temporally-ordered speaker utterances to measure emotion granularity in social media to determine whether they are effective as mental health markers.
Outcome:	The proposed measures of emotion granularity function as markers of mental health conditions.

Copied to clipboard

Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, Jiajun Zhang

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation (2024.emnlp-main)

Challenge:	BLSP-Emo model understands both semantics and emotions in speech and generates empathetic responses.
Approach:	They propose a language-speech pretraining with emotion support that utilizes existing speech and emotion recognition datasets to create an end-to-end speech-language model.
Outcome:	The proposed model can understand both semantics and emotions in speech and generate empathetic responses.

Copied to clipboard

Abhishek Divekar, Greg Durrett

Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024.emnlp-main)

Challenge:	Prior approaches to synthesis use few-shot prompting, which relies on the LLM’s parametric knowledge to generate usable examples.
Approach:	They propose to use a dataset to generate examples of each label from the LLM.
Outcome:	The proposed model significantly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.

Copied to clipboard

Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, Yueting Zhuang

DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts (2024.emnlp-main)

Challenge:	Using large language models, large multimodal models struggle with basic tasks like reading time from a clock and planning a route using a road map.
Approach:	They propose a multimodal self-instruct that synthesizes massive abstract images and visual reasoning instructions.
Outcome:	The proposed model synthesizes 11,193 abstract images and reasoning instructions across eight visual scenarios.

Copied to clipboard

Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty

DEM: Distribution Edited Model for Training with Mixed Data Distributions (2024.emnlp-main)

Challenge:	Data-driven storytelling uses visual aids and visualizations to convey insights.
Approach:	They propose a task for data story generation using large language models and a benchmark containing 1,449 stories from diverse sources.
Outcome:	The proposed framework outperforms non-agentic counterparts in both model-based and human evaluations, but also reveals unique challenges in data story generation.

Copied to clipboard

Dhananjay Ram, Aditya Rawal, Momchil Hardalov, Nikolaos Pappas, Sheng Zha

Challenge:	Recent fine-tuning approaches for large language models require supervised finetun on diverse datasets and follow different distributions.
Approach:	They propose a distribution edited model that integrates models individually trained on each data source with the base model using basic element-wise vector operations.
Outcome:	The proposed model outperforms baseline models on a variety of benchmarks and is cheaper than standard data mixing methods.

Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)

Copied to clipboard

Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer

VerifyMatch: A Semi-Supervised Learning Paradigm for Natural Language Inference with Confidence-Aware MixUp (2024.emnlp-main)

Challenge:	Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown.
Approach:	They propose an approach to edit and re-align alt-texts associated with images using human annotation.
Outcome:	The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks.

Copied to clipboard

Seo Yeon Park, Cornelia Caragea

CaT-Bench: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans (2024.emnlp-main)

Challenge:	Natural language inference (NLI) is a key task for evaluating a model's ability to perform natural language understanding and reasoning.
Approach:	They propose to construct pseudo-generated samples using class-specific fine-tuned large language models (LLMs) . they retain all pseudo-labeled samples, but use MixUp to ensure unlabele .
Outcome:	The proposed approach achieves competitive accuracy compared to strong baselines for NLI datasets in low-resource settings.

Copied to clipboard

Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, Ray Mooney

Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics (2024.emnlp-main)

Challenge:	Existing studies on reasoning in plans focus on classical problems, simulated environments, or restricted language such as PDDL, but real-world plans cannot be tested to test for correctness and reliability.
Approach:	They propose a benchmark question that tests whether a step must necessarily occur before or after another in cooking recipe plans.
Outcome:	The proposed question-driven evaluation shows that SOTA LLMs are underwhelming and biased towards predicting dependence more often, but the best F1 result is 0.73.

Copied to clipboard

Théo Gigant, Camille Guinaudeau, Marc Decombas, Frederic Dufaux

An Empirical Analysis of the Writing Styles of Persona-Assigned LLMs (2024.emnlp-main)

Challenge:	Existing metrics for summarization are reference-based and correlate poorly with relevance . fluency, faithfulness, coherence and relevance are all measures of human evaluation .
Approach:	They propose a reference-free metric that correlates well with human evaluated relevance . n-gram importance weighting is used to weight a summary's importance .
Outcome:	The proposed metric can be used along reference-based metrics to improve their robustness in low quality reference settings.

Copied to clipboard

Manuj Malik, Jing Jiang, Kian Ming Chai

Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks (2024.emnlp-main)

Challenge:	Recent efforts to "personalize" large language models by assigning them specific personas are limited by current knowledge of how well they perform.
Approach:	They use a style embedding model to analyze writing styles of persona-assigned LLMs . they find significant style differences between personas using Kullback-Leibler divergence .
Outcome:	The proposed model shows significant differences in writing styles among personas across socio-demographic groups.

Copied to clipboard

Amit Parekh, Nikolas Vitsakis, Alessandro Suglia, Ioannis Konstas

GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning (2024.emnlp-main)

Challenge:	Evaluating generalisation capabilities of multimodal models based solely on performance on out-of-distribution data fails to capture their true robustness . proposed framework examines the role of instructions and inputs in generalisation abilities of such models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity.
Approach:	They propose a framework that examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity.
Outcome:	The proposed framework examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity.

Copied to clipboard

Aleksander Ficek, Jiaqi Zeng, Oleksii Kuchaiev

CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing (2024.emnlp-main)

Challenge:	Pre-trained large language models can be used for specific tasks and unique information but lack the resources for extensive retraining.
Approach:	They propose to use PEFT methods to adapt large language models while minimizing compute requirements.
Outcome:	The proposed methods outperform GPT models in zero-shot settings but lag behind PEFT.

Copied to clipboard

Xinyi He, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, Dongmei Zhang

Sequential API Function Calling Using GraphQL Schema (2024.emnlp-main)

Challenge:	Existing methods to improve code generation from natural language descriptions are difficult due to complex structure, subtle bugs, and lack of supplementary contents.
Approach:	They propose a framework that enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement.
Outcome:	The proposed framework improves the quality of complex code generation on the DS-1000 and ClassEval datasets.

Copied to clipboard

Avirup Saha, Lakshmi Mandal, Balaji Ganesan, Sambit Ghosh, Renuka Sindhgatta, Carlos Eberhardt, Dan Debrunner, Sameep Mehta

The Illusion of Competence: Evaluating the Effect of Explanations on Users’ Mental Models of Visual Question Answering Systems (2024.emnlp-main)

Challenge:	Function calling using Large Language Models (LLMs) is an active research area . however, sequential function calling using LLMs with interdependence between functions is still under-explored .
Approach:	They propose a dataset representing real-world REST API calls with variable mapping between functions using natural language utterances paired with function call sequences.
Outcome:	The proposed dataset represents real-world REST API calls with variable mapping between functions.

Copied to clipboard

Judith Sieker, Simeon Junker, Ronja Utescher, Nazia Attari, Heiko Wersing, Hendrik Buschmeier, Sina Zarrieß

Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)

Challenge:	Using visual inputs, we hypothesize that explanations will make limited AI capabilities more transparent to users, but our results show that explanation increases users’ perceptions of the system’s competence regardless of its actual performance.
Approach:	They employ a visual question answer and explanation task where participants control the AI system’s limitations by manipulating visual inputs.
Outcome:	The proposed explanations do not increase users’ perceptions of the system’s competence regardless of its actual performance.

Copied to clipboard

Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick

Video-Text Prompting for Weakly Supervised Spatio-Temporal Video Grounding (2024.emnlp-main)

Challenge:	Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages.
Approach:	They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries.
Outcome:	The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian.

Copied to clipboard

Heng Zhao, Zhao Yinjie, Bihan Wen, Yew-Soon Ong, Joey Zhou

A Fast and Sound Tagging Method for Discontinuous Named-Entity Recognition (2024.emnlp-main)

Challenge:	Existing methods extract each candidate tube feature independently by cropping objects from video frame feature, discarding all contextual information such as position change and inter-entity relationship.
Approach:	They propose to use video-text prompts to construct candidate feature instead of cropping tube region from feature map . they also propose negative contrastive samples whose candidate object is erased instead of being highlighted .
Outcome:	The proposed methods surpass existing weakly-supervised methods by a great margin . they draw visual markers over objects tubes as video prompts .

Copied to clipboard

Caio Corro

Factuality of Large Language Models: A Survey (2024.emnlp-main)

Challenge:	Named-entity recognition (NER) is a fundamental natural language processing task . a tagging scheme for discontinuous named entities is proposed .
Approach:	They propose a tagging scheme for discontinuous named entity recognition based on an explicit description of the inner structure of discontinuous mentions.
Outcome:	The proposed method is comparable to state-of-the-art models on three English datasets in the biomedical domain.

Copied to clipboard

Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov

Discovering Biases in Information Retrieval Models Using Relevance Thesaurus as Global Explanation (2024.emnlp-main)

Challenge:	Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios.
Approach:	They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors .
Outcome:	The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors.

Copied to clipboard

Youngwoo Kim, Razieh Rahimi, James Allan

Adaptable Moral Stances of Large Language Models on Sexist Content: Implications for Society and Gender Discourse (2024.emnlp-main)

Challenge:	Currently, local explanations are not effective in predicting the model’s behavior on unseen texts.
Approach:	They propose a method to build a relevance thesaurus containing semantically relevant query term and document term pairs which can augment BM25 scoring functions to better approximate the neural model’s predictions.
Outcome:	The proposed method can augment BM25 scoring functions to better approximate the neural relevance model’s predictions.

Copied to clipboard

Rongchen Guo, Isar Nejadgholi, Hillary Dawkins, Kathleen Fraser, Svetlana Kiritchenko

DISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers (2024.emnlp-main)

Challenge:	Using large language models, large language model learning has become more integrated into our daily lives, making it increasingly important to ensure they reflect ethical and equitable values.
Approach:	They assess how LLMs can apply moral reasoning to both criticize and defend sexist language by evaluating their models and evaluating the moral foundations cited by them.
Outcome:	The models show they can provide comprehensible and contextually relevant text for understanding diverse views on how sexism is perceived.

Copied to clipboard

Rakesh R Menon, Shashank Srivastava

IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning (2024.emnlp-main)

Challenge:	Recent work proposes automatic methods for identifying and explaining systematic biases using keywords.
Approach:	They propose automatic methods for identifying and explaining systematic biases using keywords.
Outcome:	The proposed framework improves classifiers by augmenting training sets with synthetically generated instances or annotated examples via active learning.

Copied to clipboard

Soumya Suvra Ghosal, Samyadeep Basu, Soheil Feizi, Dinesh Manocha

Scope-enhanced Compositional Semantic Parsing for DRT (2024.emnlp-main)

Challenge:	Existing prompt-tuning frameworks lack interpretability, limiting their ability to understand compositional nature of images.
Approach:	They propose a prompt-tuning method that integrates compositional attributes into manual prompts to enhance image-text alignment scores.
Outcome:	The proposed method improves CoOp performance by 7.35% across 10 diverse datasets.

Copied to clipboard

Xiulin Yang, Jonas Groschwitz, Alexander Koller, Johan Bos

The Generation Gap: Exploring Age Bias in the Value Systems of Large Language Models (2024.emnlp-main)

Challenge:	Existing compositional semantic parsers for DRT struggle to produce well-formed representations due to the complexity of the sentence.
Approach:	They propose a compositional, neurosymbolic semantic parser for DRT that uses a novel mechanism for predicting quantifier scope.
Outcome:	The proposed model produces well-formed outputs and performs well on complex sentences.

Copied to clipboard

Siyang Liu, Trisha Maturi, Bowen Yi, Siqi Shen, Rada Mihalcea

TempoFormer: A Transformer for Temporally-aware Representations in Change Detection (2024.emnlp-main)

Challenge:	Using the World Value Survey, we find a general inclination of LLM values towards younger demographics, especially when compared to the US population.
Approach:	They use data from the World Value Survey to examine the alignment of LLM values with specific age groups.
Outcome:	The proposed model can be used to predict the value of a large language model and to assess its performance on 13 categories.

Copied to clipboard

Talia Tseriotou, Adam Tsakalidis, Maria Liakata

Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing? (2024.emnlp-main)

Challenge:	Current approaches to model context and time dynamics are slow and prone to overfitting.
Approach:	They propose a transformer-based and temporally-aware model for dynamic representation learning that is task-agnostic and trained on inter and intra context dynamics.
Outcome:	The proposed model is task-agnostic and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures.

Copied to clipboard

Guillermo Marco, Julio Gonzalo, M.Teresa Mateo-Girona, Ramón Santos

Evaluating Diversity in Automatic Poetry Generation (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have recently shown strong competences generating human-like text, and in particular in short creative writing tasks.
Approach:	They conducted a contest between a novelist and a top performing LLM to determine whether they are ready to compete in creative writing skills with a human creative writer.
Outcome:	The results show that LLMs are far from challenging a top human creative writer.

Copied to clipboard

Yanran Chen, Hannes Gröner, Sina Zarrieß, Steffen Eger

Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models (2024.emnlp-main)

Challenge:	Existing models for creative text generation are not evaluated regarding how different generated poems are from existing training sets.
Approach:	They evaluate the diversity of automatically generated poetry by comparing distributions of generated poetry to distributions in human poetry along structural, lexical, semantic and stylistic dimensions.
Outcome:	The proposed model types show that style-conditioning and character-level modeling increases diversity across virtually all dimensions.

Copied to clipboard

Yi Zhou, Danushka Bollegala, Jose Camacho-Collados

Delving into Qualitative Implications of Synthetic Data for Hate Speech Detection (2024.emnlp-main)

Challenge:	Social biases such as gender or racial biase are reported in language models . a recent study has shown that MLMs encode discriminatory social biase .
Approach:	They analyse temporal corpora of MLMs trained on chronologically ordered temporal snapshots . they find that gender and racial biases are encoded in MLM models .
Outcome:	The proposed model identifies gender biases in MLMs but most remain stable over time . gender bias is associated with higher likelihood scores in some demographic groups .

Copied to clipboard

Camilla Casula, Sebastiano Salto, Alan Ramponi, Sara Tonelli

Grounding Language in Multi-Perspective Referential Communication (2024.emnlp-main)

Challenge:	Recent work on synthetic data for training models for NLP tasks reports mixed results on subjective tasks such as hate speech detection.
Approach:	They propose to use synthetic data to train models for highly subjective tasks such as hate speech detection to investigate the potential and specific pitfalls of using it.
Outcome:	The proposed model outperforms models trained with real data on hate speech detection tasks, but it fails to accurately reflect real-world data on linguistic dimensions and results in different class distributions.

Copied to clipboard

Zineng Tang, Lingjun Mao, Alane Suhr

Threshold-driven Pruning with Segmented Maximum Term Weights for Approximate Cluster-based Sparse Retrieval (2024.emnlp-main)

Challenge:	Using a dataset of 2,970 human-written referring expressions, we find that the performance of automated models in both reference generation and comprehension lags behind that of pairs of human agents.
Approach:	They propose a task and dataset for referring expression generation and comprehension in multi-agent embodied environments where two agents must take into account one another's visual perspective to produce and understand references to objects in a scene.
Outcome:	The proposed model outperforms the strongest proprietary model and improves communicative success from 58.9 to 69.3% when trained with a listener.

Copied to clipboard

Yifan Qiao, Parker Carlson, Shanxiu He, Yingrui Yang, Tao Yang

Text Clustering

Challenge:	Using rank score thresholding, sparse retrieval skips the index at cluster and document levels.
Approach:	They propose a pruning control scheme with a probabilistic guarantee on rank-safeness competitiveness.
Outcome:	The proposed pruning control scheme improves accuracy and safeness while delivering low latency on single-threaded CPU.

Error Analysis of Multilingual Language Models in Machine Translation: A Case Study of English-Amharic Translation (2024.emnlp-main)

Copied to clipboard

Hizkiel Alemayehu, Hamada Zahera, Axel-Cyrille Ngonga Ngomo

MIPD: Exploring Manipulation and Intention In a Novel Corpus of Polish Disinformation (2024.emnlp-main)

Challenge:	Multilingual large language models have significantly advanced machine translation, yet challenges remain for low-resource languages like Amharic.
Approach:	They evaluated the performance of NLLB-200 and M2M in English-Amharic bidirectional translation using the Lesan AI dataset.
Outcome:	The proposed models outperformed the existing models in English-Amharic bidirectional translation using the Lesan AI dataset.

Copied to clipboard

Arkadiusz Modzelewski, Giovanni Da San Martino, Pavel Savov, Magdalena Wilczyńska, Adam Wierzbicki

Unsupervised Discrete Representations of American Sign Language (2024.emnlp-main)

Challenge:	Using a unique methodology, we annotated disinformation in Polish with multiple labels indicating both intents and manipulation techniques employed.
Approach:	They present a novel corpus of 15,356 Polish web articles annotated with multiple labels indicating both disinformation creators’ intents and manipulation techniques employed.
Outcome:	The proposed dataset sheds light on the authors' intention and manipulation techniques in disinformation.

Copied to clipboard

Artem Abzaliev, Rada Mihalcea

Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models (2024.emnlp-main)

Challenge:	Modern NLP models use discrete tokens to represent continuous signals, such as videos, audio, or gestures . modalities that are continuous are difficult to use with discrete models, such a LLM .
Approach:	They propose a method that discretizes sequences of fingerspelling signs into tokens . they also propose 'loss function' to improve interpretability of the tokens.
Outcome:	The proposed method improves the performance of the tokenizer on downstream tasks.

Copied to clipboard

Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, Hyunwoo Kim

Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs (2024.emnlp-main)

Challenge:	While theory of mind (ToM) is naturally developed for humans in childhood, large language models (LLMs) exhibit inconsistency in ToM tasks, despite early reports of successful cases.
Approach:	They propose to evaluate human ToM precursors-perception inference and perception-to-belief inference-in large language models (LLMs) by annotating characters’ perceptions on ToMi and FANToM.
Outcome:	The proposed method significantly improves LLMs’ performance in false belief scenarios.

Copied to clipboard

Mihir Parmar, Hanieh Deilamsalehy, Franck Dernoncourt, Seunghyun Yoon, Ryan Rossi, Trung Bui

Jump Starting Bandits with LLM-Generated Prior Knowledge (2024.emnlp-main)

Challenge:	Existing methods for extractive summarization lack coherence, despite improvements . a human-annotated dataset is used to improve coherency of extractive summary .
Approach:	They propose to use human-annotated datasets to create coherent extractive summaries . they use supervised fine-tuning and natural language user feedback to enhance coherence .
Outcome:	The proposed dataset shows that LLMs can produce coherent summaries with human feedback.

Copied to clipboard

Parand Alamdari, Yanshuai Cao, Kevin Wilson

Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? (2024.emnlp-main)

Challenge:	Contextual multi-armed bandits generate personalized recommendations based on user-specific contexts.
Approach:	They propose an initialization algorithm for contextual bandits by prompting LLMs to produce a pre-training dataset of approximate human preferences for the bandit.
Outcome:	The proposed approach significantly reduces online learning regret and data-gathering costs for training such models.

Copied to clipboard

Fırat Öncel, Matthias Bethge, Beyza Ermis, Mirco Ravanelli, Cem Subakan, Çağatay Yıldız

Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation (2024.emnlp-main)

Challenge:	In the last decade, the generalization and adaptation abilities of deep learning models were evaluated on fixed training and test distributions.
Approach:	They propose to train large language models on unlabeled text corpora and train them online.
Outcome:	The proposed model training on a text domain could degrade its perplexity on the test portion of the same domain.

Copied to clipboard

Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, Le Sun

Virtual Personas for Language Models via an Anthology of Backstories (2024.emnlp-main)

Challenge:	Existing RAG paradigms suffer from the impact of flawed information introduced during the retrieval phrase, thereby diminishing the reliability and correctness of the generated output.
Approach:	They propose a framework that empowers models to discern and process information based on its credibility.
Outcome:	The proposed framework outperforms existing models with retrieval augmentation and exhibits robustness despite increasing noise in the context.

Copied to clipboard

Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Behar, David Chan

Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)

Challenge:	Large language models (LLMs) are trained from vast repositories of text authored by millions of distinct authors, reflecting an enormous diversity of human traits.
Approach:	They propose a method for conditioning LLMs to particular virtual personas by harnessing open-ended life narratives, which they refer to as backstories, and demonstrate that it improves consistency and reliability of experimental outcomes.
Outcome:	The proposed method improves consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations.

Copied to clipboard

Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral

Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies (2024.emnlp-main)

Challenge:	Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them.
Approach:	They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains.
Outcome:	The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans.

Copied to clipboard

Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, Ben Athiwaratkun

The Empirical Variability of Narrative Perceptions of Social Media Texts (2024.emnlp-main)

Challenge:	Existing evaluations that focus on performance metrics miss a key factor: increased effectiveness due to additional compute.
Approach:	They propose to incorporate the compute budget into evaluations to provide a more informative comparison that takes into account both performance metrics and computational cost.
Outcome:	The proposed framework outperforms reasoning strategies when they use comparable compute resources.

Copied to clipboard

Joel Mire, Maria Antoniak, Elliott Ash, Andrew Piper, Maarten Sap

Challenge:	Identifying stories in social media texts provides a lens through which we can study how individuals and communities process and communicate experiences.
Approach:	They construct a taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales.
Outcome:	The proposed model shows that crowd workers disagree on categorical labels, free-text storytelling rationales, authorial intent, and more.

Which questions should I answer? Salience Prediction of Inquisitive Questions (2024.emnlp-main)

Copied to clipboard

Yating Wu, Ritika Mangla, Alex Dimakis, Greg Durrett, Junyi Jessy Li

Revealing Personality Traits: A New Benchmark Dataset for Explainable Personality Recognition on Dialogues (2024.emnlp-main)

Challenge:	Recent work in NLP has taken advantage of question generation capabilities of LLMs to enhance a wide range of applications.
Approach:	They propose a salience predictor for inquisitive questions that is instruction-tuned . they show that highly salient questions are empirically more likely to be answered in the same article .
Outcome:	The proposed model is based on linguist-annotated salience scores of 1,766 questions . it shows that answering salient questions improves comprehension of the text .

Copied to clipboard

Lei Sun, Jinming Zhao, Qin Jin

Continual Test-time Adaptation for End-to-end Speech Recognition on Noisy Speech (2024.emnlp-main)

Challenge:	Current research treats personality recognition as a classification task, failing to reveal the supporting evidence for the recognized personality.
Approach:	They propose a task that aims to reveal the reasoning process as supporting evidence of the personality trait.
Outcome:	The proposed task reveals the reasoning process as supporting evidence of the personality trait.

Copied to clipboard

Guan-Ting Lin, Wei Huang, Hung-yi Lee

Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities (2024.emnlp-main)

Challenge:	Current ASR TTA methods focus on non-continual TTA, which limits cross-sample knowledge learning compared to continual TTA.
Approach:	They propose a Fast-slow TTA framework that leverages the advantage of continual and non-continual TTA and a Dynamic SUTA method that automatically detects domain shifts and resets the model.
Outcome:	The proposed method outperforms non-continual and continual TTA methods while maintaining robustness to domain shifts without requiring domain boundary information.

Copied to clipboard

Sachit Menon, Richard Zemel, Carl Vondrick

CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)

Challenge:	Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning.
Approach:	They propose a method to unlock the visual reasoning capabilities of multimodal large language models by using a metaphorical ‘whiteboard’ to draw out reasoning steps as images and return these images back to the model for further processing.
Outcome:	The proposed method shows that it can be used on four difficult tasks that involve visual and spatial reasoning with no demonstrations or specialized modules.

Copied to clipboard

Weixi Tong, Tianyi Zhang

Self-Training Large Language and Vision Assistant for Medical Question Answering (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem.
Approach:	They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases.
Outcome:	The proposed framework outperforms existing methods on four code generation datasets and five programming languages.

Copied to clipboard

Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, Zhiqiang Tao

SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization (2024.emnlp-main)

Challenge:	Existing methods for collecting medical data are expensive and time-consuming.
Approach:	They propose a method to train a large-scale LVLM capable of auto-generating medical visual instruction data to improve data efficiency.
Outcome:	The proposed method shows that it performs well across three major visual question answering (VQA) benchmarks.

Copied to clipboard

Prakamya Mishra, Zonghai Yao, Parth Vashisht, null Feiyun, Beining Wang, Vidhi Mody, Hong Yu

Defending Jailbreak Prompts via In-Context Adversarial Game (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) struggle with factual inaccuracies, a critical issue in clinical NLP applications where errors could lead to serious consequences.
Approach:	They propose a pipeline that leverages >100B parameter GPT variants to act as synthetic experts to generate edit feedback without additional human annotations.
Outcome:	The proposed pipeline aims to improve the quality of clinical note summarizations by generating edit feedback without human annotations.

Copied to clipboard

Yujun Zhou, Yufei Han, Haomin Zhuang, Kehan Guo, Zhenwen Liang, Hongyan Bao, Xiangliang Zhang

Detecting Online Community Practices with Large Language Models: A Case Study of Pro-Ukrainian Publics on Twitter (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications, but concerns regarding their security persist.
Approach:	They propose an adversarial game that leverages agent learning to extend knowledge to defend against jailbreaks.
Outcome:	The proposed game shows that LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates across various attack scenarios.

Copied to clipboard

Kateryna Kasianenko, Shima Khanehzar, Stephen Wan, Ehsan Dehghan, Axel Bruns

Multilingual Topic Classification in X: Dataset and Analysis (2024.emnlp-main)

Challenge:	Existing methods for identifying practices within social media are not yet available.
Approach:	They propose a methodological workflow for computational identification of such practices within social media texts by using open-source models and OpenAI’s large language models.
Outcome:	The proposed method improves accuracy and supports context-sensitive moderation and advancing the understanding of online community dynamics.

Copied to clipboard

Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Jose Camacho-Collados

MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)

Challenge:	Social media platforms such as X (Twitter), Snapchat and Instagram provide an environment for content creation and information sharing.
Approach:	They propose a multilingual dataset featuring tweet topic classification in four languages . they leverage X-Topic to perform cross-linguistic and multilingual analysis .
Outcome:	The proposed dataset includes topics in four languages and is useful for cross-linguistic analysis and the development of robust multilingual models.

Copied to clipboard

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

Updating CLIP to Prefer Descriptions Over Captions (2024.emnlp-main)

Challenge:	Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions.
Approach:	They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4.
Outcome:	The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information.

Copied to clipboard

Amir Zur, Elisa Kreiss, Karel D’Oosterlinck, Christopher Potts, Atticus Geiger

CmdCaliper: A Semantic-Aware Command-Line Embedding Model and Dataset for Security Research (2024.emnlp-main)

Challenge:	Current metrics for imagetext similarity tend to be insensitive to the text's purpose.
Approach:	They propose to use a model that assigns higher scores to descriptions than captions . they use parameter efficient fine-tuning and a loss objective to shed light on the distinction .
Outcome:	The proposed model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and sheds light on the caption–description distinction.

Copied to clipboard

Sian-Yao Huang, Cheng-Lin Yang, Che-Yu Lin, Chun-Ying Huang

Back to School: Translation Using Grammar Books (2024.emnlp-main)

Challenge:	Currently, command-line embeddings are limited due to the lack of comprehensive datasets for the field due to privacy and regulation concerns.
Approach:	They propose a command-line embedding model called CmdCaliper for training and unbiased evaluation using a set of large language models comprising 28,520 similar command- line pairs.
Outcome:	The proposed model suppresses state-of-the-art sentences with ten times more parameters across various tasks.

Copied to clipboard

Jonathan Hus, Antonios Anastasopoulos

VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)

Challenge:	Current large language models require massive amounts of parallel sentences to perform machine translations for high resource languages.
Approach:	They propose to incorporate grammar books into the prompt of GPT-4 to improve machine translation and evaluate the performance on 16 topologically diverse low-resource languages.
Outcome:	The proposed method improves on 16 low-resource languages on 16 topologically diverse languages.

Copied to clipboard

Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang

Towards Aligning Language Models with Textual Feedback (2024.emnlp-main)

Challenge:	Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations.
Approach:	They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models .
Outcome:	The proposed approach is effective across three video captioning models.

Copied to clipboard

Saüc Lloret, Shehzaad Dhuliawala, Keerthiram Murugesan, Mrinmaya Sachan

AMPO: Automatic Multi-Branched Prompt Optimization (2024.emnlp-main)

Challenge:	Using textual feedback, language models can be trained to learn from textual inputs.
Approach:	They propose an approach that aligns language models with user preferences expressed in text.
Outcome:	The proposed approach outperforms PPO on toxicity reduction, summarization, and dialog response tasks while achieving the same performance with only 20% of the samples.

Copied to clipboard

Sheng Yang, Yurong Wu, Yan Gao, Zineng Zhou, Bin Zhu, Xiaodi Sun, Jian-Guang Lou, Zhiming Ding, Anbang Hu, Yuan Fang, Yunsong Li, Junyan Chen, Linjun Yang

DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translators (2024.emnlp-main)

Challenge:	Existing prompt engineering techniques are limited to producing single flow instructions, struggling with handling diverse patterns.
Approach:	They propose an automatic prompt optimization method that iteratively develops a multi-branched prompt using failure cases as feedback.
Outcome:	The proposed method achieves the best results across five tasks and demonstrates significant optimization efficiency due to adoption of a minimal search strategy.

Copied to clipboard

Xinglin Lyu, Junhui Li, Yanqing Zhao, Min Zhang, Daimeng Wei, Shimin Tao, Hao Yang, Min Zhang

DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-Editing (2024.emnlp-main)

Challenge:	Concatenating large language models are adapted to context-aware neural machine translation in a concatenated way . a recent paradigm shift has been witnessed in discourse-related challenges such as zero pronoun translation .
Approach:	They propose an alternative adaptation approach to make large language models discriminately model and utilize inter- and intra-sentence contexts.
Outcome:	The proposed approach outperforms concatenation mode and improves performance in discourse modeling.

Copied to clipboard

Devleena Das, Vivek Khetan

Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models (2024.emnlp-main)

Challenge:	Recent advances in language modelling have led to the availability of many pre-trained language models (PLMs); however, how much data is needed to fine-tune PLMs for downstream tasks?
Approach:	They propose a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset to fine- tune PLMs for text-editing tasks.
Outcome:	The proposed framework is as accurate as CoEDIT across eight different datasets consisting of six different editing tasks, while finetuning on 70% less data.

Copied to clipboard

Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Yamaguchi, Rieko Kubo, Shinji Nishimoto, Yu Takagi

“They are uncultured”: Unveiling Covert Harms and Social Threats in LLM Generated Conversations (2024.emnlp-main)

Challenge:	Recent studies have assessed different levels of semantic content, such as speech, objects, and stories, separately.
Approach:	They used functional magnetic resonance imaging to record brain activity while watching 8.3 hours of dramas and movies.
Outcome:	The findings show that LLMs predict human brain activity more accurately than traditional language models, particularly for complex background stories.

Copied to clipboard

Preetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury, Tanu Mitra

Multi-expert Prompting Improves Reliability, Safety and Usefulness of Large Language Models (2024.emnlp-main)

Challenge:	Prior studies on LLM harms focus on Western concepts like race and gender, overlooking cultural concepts from other parts of the world.
Approach:	They propose a set of seven metrics to examine the presence of covert harms in LLM-generated conversations.
Outcome:	The proposed model detects that seven out of eight LLMs generated conversations riddled with CHAST, characterized by malign views expressed in seemingly neutral language, compared to Western ones such as race.

Copied to clipboard

Do Long, Duong Yen, Anh Tuan Luu, Kenji Kawaguchi, Min-Yen Kan, Nancy Chen

Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification? (2024.emnlp-main)

Challenge:	Existing enhancements of ExpertPrompting improve the large language model generation process.
Approach:	They propose a novel enhancement of ExpertPrompting to improve LLM generation by simulating multiple experts, aggregating their responses and selecting the best among individual and aggregated responses.
Outcome:	The proposed enhancement outperforms ExpertPrompting and comparable baselines in truthfulness, factuality, informativeness, usefulness and harmfulness.

Copied to clipboard

Gabriel Roccabruna, Massimo Rizzoli, Giuseppe Riccardi

Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown promising performance in temporal reasoning tasks such as temporal question answering.
Approach:	They propose to use large language models to detect temporal relations between events with in-context learning and lightweight fine-tuning approaches to assess their performance.
Outcome:	The proposed models significantly underperform smaller encoder-only models based on RoBERTa in the Temporal Relation Classification task.

Copied to clipboard

Keunwoo Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, Joyce Chai

Waterfall: Scalable Framework for Robust Text Watermarking and Provenance for LLMs (2024.emnlp-main)

Challenge:	Emergent In-context Learning on Videos induces in-contact learning over video and text . eILeV-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions.
Approach:	They implement Emergent In-context Learning on Videos (EILeV) that induces in-contact learning over video and text by capturing key properties of pre-training data.
Outcome:	The proposed training paradigm outperforms off-the-shelf VLMs in few-shot video narration for novel, rare actions.

Copied to clipboard

Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, Bryan Kian Hsiang Low

MASIVE: Open-Ended Affective State Identification in English and Spanish (2024.emnlp-main)

Challenge:	Existing text watermarking methods are not robust enough against paraphrasing attacks . existing methods lack robustness to paraphrases and are not scalable to millions of users .
Approach:	They propose a training-free framework for robust and scalable text watermarking . they propose to use large language models as paraphrasers and a combination of techniques .
Outcome:	The proposed framework improves scalability, verifiability and computational efficiency compared to existing methods.

Copied to clipboard

Nicholas Deas, Elsbeth Turcan, Ivan Mejia, Kathleen McKeown

You Make me Feel like a Natural Question: Training QA Systems on Transformed Trivia Questions (2024.emnlp-main)

Challenge:	Existing models that fail to understand cultural and language influences the meaning of emotional terms like "love" a new study shows that smaller finetuned models outperform much larger LLMs on region-specific span prediction tasks.
Approach:	They propose to use a reddit reddits dataset to identify a set of affective states . they find that smaller finetuned multilingual models outperform larger LLMs .
Outcome:	The proposed model outperforms larger models on span prediction task even on region-specific Spanish affective states.

Copied to clipboard

Tasnim Kabir, Yoo Yeon Sung, Saptarashmi Bandyopadhyay, Hao Zou, Abhranil Chandra, Jordan Boyd-Graber

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality (2024.emnlp-main)

Challenge:	Existing question-answering datasets are expensive and difficult to annotate and time-consuming to gather.
Approach:	They propose to transform Manchester questions into web queries using the same question datasets.
Outcome:	The proposed questions can be trained on a Manchester QA dataset using the Quiz Bowl (QB) sample.

Copied to clipboard

Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, Soroush Vosoughi

Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling (2024.emnlp-main)

Challenge:	Recent studies combine LoRA with Mixture-of-Experts (MoE) to improve performance in Large Language Models.
Approach:	They propose a method to combine LoRA and Mixture-of-Experts (MoE) to improve performance in Large Language Models.
Outcome:	The proposed method reduces redundancy in LoRA experts within the MoE architecture, and improves training quality across layers.

Copied to clipboard

Irfan Robbani, Paul Reisert, Surawat Pothong, Naoya Inoue, Camélia Guerraoui, Wenzhi Wang, Shoichi Naito, Jungmin Choi, Kentaro Inui

Challenge:	Prior work on quality assessment has focused on numerical scoring and fallacy type-labeling tasks, without aiming to analyze fallacy logic structures.
Approach:	They propose four sets of explainable templates for common informal logical fallacies designed to explicate a fallacy’s implicit logic.
Outcome:	The proposed models achieve a high agreement score and reasonable coverage 83% on 400 fallacious arguments and state-of-the-art language models struggle with detecting fallacy templates (0.47 accuracy).

Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions (2024.emnlp-main)

Copied to clipboard

Leena Mathur, Paul Pu Liang, Louis-Philippe Morency

Challenge:	Building socially-intelligent AI agents involves creating agents that can sense, perceive, reason about, learn from, and respond to affect, behavior, and cognition of other agents.
Approach:	They propose a set of technical challenges and open questions for researchers to advance Social-AI.
Outcome:	The proposed frameworks are based on the social intelligence competencies that evolved over thousands of years in Homo sapiens and are expected to be the foundations for the development of social-intelligent AI agents.

RAt: Injecting Implicit Bias for Text-To-Image Prompt Refinement Models (2024.emnlp-main)

Copied to clipboard

Ziyi Kou, Shichao Pei, Meng Jiang, Xiangliang Zhang

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese (2024.emnlp-main)

Challenge:	Text-to-image prompt refinement (T2I-Refine) aims to rephrase or extend an input prompt with more descriptive details that can be leveraged to generate images with higher quality.
Approach:	They develop an adversarial prompt attacking framework that implicitly attacks input prompts with intentional adversarials to generate images with higher quality.
Outcome:	The proposed framework can implicitly attack input prompts with implicit concept biases to generate images with higher quality and explicit visual bias towards the target group.

Copied to clipboard

Rifki Putri, Faiz Haznitrama, Dea Adhista, Alice Oh

Can Language Models Induce Grammatical Knowledge from Indirect Evidence? (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models.
Approach:	They investigate the effectiveness of using Large Language Models to generate culturally relevant commonsense QA datasets for Indonesian and Sundanese languages using both LLMs and human annotators.
Outcome:	The proposed model generates 4.5K questions per language, compared with 4.5k for Indonesian and 4.5km for Sundanese.

Copied to clipboard

Miyu Oba, Yohei Oseki, Akiyo Fukatsu, Akari Haga, Hiroki Ouchi, Taro Watanabe, Saku Sugawara

Jialiang Xu, Shenglan Li, Zhaozhuo Xu, Denghui Zhang

Challenge:	Recent advances in language models have shown remarkable progress in various tasks.
Approach:	They introduce a dataset that incorporates wug words and inject them into pretraining data and evaluate them on evaluation data.
Outcome:	The proposed model does not induce grammatical knowledge even after repeated exposure to instances with the same structure but differing only in lexical items from evaluation instances in certain language phenomena.

Copied to clipboard

SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding (2024.emnlp-main)

Challenge:	Existing studies have focused on the occurrence of copyright violations in LLM output, but a negative answer would suggest that LLMs will become the primary facilitator and accelerator of copy right infringement behavior.
Approach:	They propose to examine whether LLMs respect copyright information in user input . they use a set of language models, user prompts, and copyrighted materials .
Outcome:	The proposed model will be the primary facilitator and accelerator of copyright infringement behavior, the study finds . the study also provides a benchmark dataset serving as a test bed for evaluating infringement behaviors by LLMs .

Copied to clipboard

Ryan Sun, Tianyi Zhou, Xun Chen, Lichao Sun

Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have limited inference speed due to sequential token generation . Spechub is a novel, efficient sampling-verification method for MDSD that improves acceptance rates with only linear computational overhead.
Approach:	They propose a method that uses a smaller draft model to generate multiple token sequences . Spechub generates 0.05-0.27 and 0.02-0.16 more tokens per step than RRS and RRS without replacement .
Outcome:	The proposed method improves acceptance rates with only linear computational overhead.

Copied to clipboard

YeonJoon Jung, Jaeseong Lee, Seungtaek Choi, Dohyeon Lee, Minsoo Kim, Seung-won Hwang

Rethinking the Role of Proxy Rewards in Language Model Alignment (2024.emnlp-main)

Challenge:	Existing methods to increase the robustness of pre-trained language models (PLMs) against unseen ASR systems produce noisy inputs for SLU models, which can significantly degrade their performance.
Approach:	They propose to introduce ASR-plausible noises into pre-trained language models by cutting off the non-causal effect of noises.
Outcome:	The proposed method improves the robustness and generalizability of SLU models against unseen ASR systems by cutting off the non-causal effect of noises.

Copied to clipboard

Sungdong Kim, Minjoon Seo

Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant (2024.emnlp-main)

Challenge:	Typically, the human feedback is used to train a proxy reward model (RM), and a policy model is optimized over the reward signal from the RM using RL.
Approach:	They aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL).
Outcome:	The proposed model shows competitive performances with strong open-source RMs in alignment benchmarks.

Copied to clipboard

Abhirama Penamakuri, Anand Mishra

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)

Challenge:	Existing knowledge-aware text-based visual question answering methods are based on textual entities in images.
Approach:	They propose a visual text entity linking module that harnesses a state-of-the-art visual text recognition engine and the power of a large multimodal model to perform visual text-entity linking.
Outcome:	The proposed approach surpasses the previous best approach by 23.3% on an absolute scale and establishes a new state of the art.

Copied to clipboard

Stefano Perrella, Lorenzo Proietti, Pere-Lluís Huguet Cabot, Edoardo Barba, Roberto Navigli

IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning (2024.emnlp-main)

Challenge:	Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices.
Approach:	They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases.
Outcome:	The proposed framework offers clearer insights than correlation with human judgments.

Copied to clipboard

Soeun Lee, Si-Woo Kim, Taewhan Kim, Dong-Jin Kim

Encoding Spreadsheets for Large Language Models (2024.emnlp-main)

Challenge:	Existing text-only training methods overlook the modality gap between using text data during training and employing images during inference.
Approach:	They propose a novel approach that aligns text features with visually relevant features to mitigate the modality gap between using text data during training and employing images during inference.
Outcome:	The proposed method outperforms the state-of-the-art methods in image captioning and video captioning by a significant margin compared to training with text data.

Copied to clipboard

Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, Dongmei Zhang

Let’s discuss! Quality Dimensions and Annotated Datasets for Computational Argument Quality Assessment (2024.emnlp-main)

Challenge:	Spreadsheets are characterized by their extensive two-dimensional grids, flexible layouts, and varied formatting options, which pose significant challenges for large language models (LLMs).
Approach:	They propose a structural-anchor-based compression, inverse index translation, and data-format-aware aggregation module to compress spreadsheets effectively.
Outcome:	The proposed method outperforms the existing model in GPT4 and achieves a state-of-the-art 78.9% F1 score.

Copied to clipboard

Rositsa Ivanova, Thomas Huber, Christina Niklaus

Automatic sentence segmentation of clinical record narratives in real-world data (2024.emnlp-main)

Challenge:	Argumentation is a key competence and an important cultural technique in democratic societies.
Approach:	They propose to create domain-specific datasets and methods to assess argument quality.
Outcome:	The proposed methods address gaps in the literature and aid future research in the domain.

Copied to clipboard

Dongfang Xu, Davy Weissenbacher, Karen O’Connor, Siddharth Rawal, Graciela Hernandez

One-to-Many Communication and Compositionality in Emergent Communication (2024.emnlp-main)

Challenge:	Sentence segmentation is a linguistic task used as a pre-processing step in many NLP applications.
Approach:	They propose a sequence labeling classifier that predicts sentence spans using a dynamic sliding window based on the prediction of each input sequence.
Outcome:	The proposed method outperforms state-of-the-art systems on clinical notes and on five other datasets to assess its generalizability and performance.

Copied to clipboard

Heeyoung Lee

Challenge:	Compositional languages leverage rules that derive meaning from combinations of simpler constituents.
Approach:	They investigate the effects of one-to-many communication environment on emergent languages where a single speaker broadcasts its message to multiple listeners to cooperatively solve a task.
Outcome:	The proposed model shows that broadcasting the speaker’s message to multiple listeners does not induce more compositional languages.

Bayesian Example Selection Improves In-Context Learning for Speech, Text and Visual Modalities (2024.emnlp-main)

Copied to clipboard

Siyin Wang, Chao-Han Yang, Ji Wu, Chao Zhang

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? (2024.emnlp-main)

Challenge:	Large language models (LLMs) can adapt to new tasks easily and efficiently in a training-free manner.
Approach:	They propose to use eBayesian in-context example selection method to extend the inference probability conditioned on in-constitut examples based on Bayes’ theorem to select in-strategy examples . Experimental results show the efficacy and robustness of their method on various models, tasks and modalities.
Outcome:	The proposed method is based on the eBayesian in-context example selection approach.

Copied to clipboard

Alexander Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, Mehdi Ali

Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models (2024.emnlp-main)

Challenge:	a study of multilingual pre-trained LLMs on parallel instruction-tuning benchmarks shows that instruction-following models can be used across languages by up to 9.9%.
Approach:	They conduct an extensive study of the performance of multilingual pre-trained LLMs instruction-tuned on parallel instruction-uning datasets.
Outcome:	The proposed model improves cross-lingual instruction following capabilities by 9.9% on a large and mid-sized LLM on parallel instruction-tuning datasets.

Copied to clipboard

Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral

Linear Layer Extrapolation for Fine-Grained Emotion Classification (2024.emnlp-main)

Challenge:	Existing logical reasoning evaluation benchmarks focus on simplistic single-step or multi-step reasoning with limited set of inference rules.
Approach:	They propose to use a multi-step logical reasoning evaluation dataset to measure their ability for human-like multi- step logical thinking.
Outcome:	The proposed dataset covers three logic types including propositional, first-order, and non-monotonic logic with various inference rules and depths.

Copied to clipboard

Mayukh Sharma, Sean O’Brien, Julian McAuley

Task Oriented In-Domain Data Augmentation (2024.emnlp-main)

Challenge:	Existing studies show that Transformer-based language models are more factual accurate in later layers .
Approach:	They propose a method that optimizes contrast based on the selected intermediate layer . they observe a similar pattern for fine-grained emotion classification in text .
Outcome:	Experiments show that the proposed method outperforms standard methods in fine-grained emotion classification tasks.

Copied to clipboard

Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, Jian Jiao

Numerical Reasoning

Challenge:	Existing methods for large language models suffer from two major issues: in-domain data are scarce compared with general domain-agnostic data.
Approach:	They propose a task-oriented in-domain data augmentation framework that uses in- domain data selection and task-orientated synthetic passage generation to adapt LLMs to two domains: advertisement and math.
Outcome:	The proposed framework improves LLM performance by 8% in the advertisement domain and 7.5% in the math domain.

SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)

Copied to clipboard

Shruti Singh, Nandan Sarkar, Arman Cohan

Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules (2024.emnlp-main)

Challenge:	SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles.
Approach:	They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs.
Outcome:	The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering.

Copied to clipboard

Zhuocheng Gong, Ang Lv, Jian Guan, Wei Wu, Huishuai Zhang, Minlie Huang, Dongyan Zhao, Rui Yan

No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages (2024.emnlp-main)

Challenge:	Empirical results show that MoMs consistently outperform vanilla transformers .
Approach:	They propose an architecture that allows for a mixture-of-modules computation that uses a finite set of modules defined by multi-head attention and feed-forward networks.
Outcome:	The proposed architecture outperforms vanilla Transformers and their variants in multiple ways.

Copied to clipboard

Youssef Mohamed, Runjia Li, Ibrahim Ahmad, Kilichbek Haydarov, Philip Torr, Kenneth Church, Mohamed Elhoseiny

PREDICT: Multi-Agent-based Debate Simulation for Generalized Hate Speech Detection (2024.emnlp-main)

Challenge:	Traditionally, vision research focused on unambiguous class labels, whereas ArtELingo emphasizes diversity of opinions over languages and cultures.
Approach:	They propose a vision-language benchmark that spans 28 languages and encompasses approximately 200,000 annotations.
Outcome:	The proposed benchmark spans 28 languages and encompasses approximately 200,000 annotations . the challenge is to build machine learning systems that assign emotional captions to images .

Copied to clipboard

Someen Park, Jaehoon Kim, Seungwan Jin, Sohyun Park, Kyungsik Han

TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR (2024.emnlp-main)

Challenge:	Existing methods to generalize hate speech detection models have been limited by the labeling criteria between datasets.
Approach:	They propose a framework that uses the concept of multi-agent for hate speech detection that uses a set of labeling criteria to create multiple agents based on the induced labeling of given datasets.
Outcome:	The proposed framework achieves superior cross-evaluation performance compared to methods that focus on specific labeling criteria or majority voting methods.

Copied to clipboard

Shashi Kumar, Srikanth Madikeri, Juan Pablo Zuluaga Gomez, Iuliia Thorbecke, Esaú Villatoro-tello, Sergio Burdisso, Petr Motlicek, Karthik S, Aravind Ganapathiraju

ApiQ: Finetuning of 2-Bit Quantized Large Language Model (2024.emnlp-main)

Challenge:	Existing approaches to automatic speech recognition use cascaded pipelines for tasks like voice activity detection, diarization, transcription and subsequent processing.
Approach:	They propose a single Transducer-based model that integrates task-specific tokens into the reference text during ASR model training, streamlining inference and eliminating the need for separate NLP models.
Outcome:	The proposed model outperforms the existing pipeline on speaker change detection, endpointing, and NER tasks while outperforming the existing model in individual task performance.

Copied to clipboard

Baohao Liao, Christian Herold, Shahram Khadivi, Christof Monz

Memorize Step by Step: Efficient Long-Context Prefilling with Incremental Memory and Decremental Chunk (2024.emnlp-main)

Challenge:	Memory-efficient finetuning of large language models (LLMs) has attracted huge attention with the increasing size of LLMs due to the constraints posed by GPU memory limitations and the effectiveness of these methods compared to full finetune.
Approach:	They propose a memory-efficient finetuning framework called ApiQ to restore lost information from quantization by initializing LoRA components and quantizing weights of LLMs.
Outcome:	The proposed framework maintains the original LLM’s activation precision while mitigating error propagation from shallower into deeper layers.

Copied to clipboard

Zhiyuan Zeng, Qipeng Guo, Xiaoran Liu, Zhangyue Yin, Wentao Shu, Mianqiu Huang, Bo Wang, Yunhua Zhou, Linlin Li, Qun Liu, Xipeng Qiu

A Morphology-Based Investigation of Positional Encodings (2024.emnlp-main)

Challenge:	Existing methods to optimize LLM for long sequences for long documents are slow and consume memory.
Approach:	They propose a method that starts with a small memory size and gradually increases it . they propose Decremental Chunk based on Incremental Memory (IMDC) which reduces chunk size while increasing memory size .
Outcome:	The proposed method is faster (1.45x) and reduces GPU memory consumption by 23.3% compared to fixed-size memory.

Copied to clipboard

Poulami Ghosh, Shikhar Vashishth, Raj Dabre, Pushpak Bhattacharyya

I love pineapple on pizza != I hate pineapple on pizza: Stance-Aware Sentence Transformers for Opinion Mining (2024.emnlp-main)

Challenge:	Contemporary deep learning models handle languages with diverse morphology . morphological complexity of languages is closely linked with positional encodings .
Approach:	They propose to use positional encodings to integrate morphological complexity into deep learning models.
Outcome:	The proposed model improves on 22 languages and 5 downstream tasks.

Copied to clipboard

Vahid Ghafouri, Jose Such, Guillermo Suarez-Tangil

BiasWipe: Mitigating Unintended Bias in Text Classifiers through Model Interpretability (2024.emnlp-main)

Challenge:	Sentence transformers excel at grouping topically similar texts, but struggle to differentiate opposing viewpoints on the same topic.
Approach:	They propose to fine-tune sentence transformers with arguments for and against controversial claims to enhance their utility for social computing tasks.
Outcome:	The proposed model improves opinion mining and stance detection tasks by combining human-generated controversial claims with stance-aware sentences.

Copied to clipboard

Mamta Mamta, Rishikant Chigrupaatii, Asif Ekbal

ArMeme: Propagandistic Content in Arabic Memes (2024.emnlp-main)

Challenge:	Existing methods to mitigate unintended bias in social media platforms are re-training and adding extra parameters to the model.
Approach:	They propose a technique to mitigate unintended bias in language models by pruning the neuron weights responsible for univ bias.
Outcome:	The proposed technique achieves fairness by pruning the neuron weights responsible for unintended bias without loss of original performance.

Copied to clipboard

Firoj Alam, Abul Hasnat, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain

Language is Scary when Over-Analyzed: Unpacking Implied Misogynistic Reasoning with Argumentation Theory-Driven Prompts (2024.emnlp-main)

Challenge:	a lack of media literacy is a major factor contributing to the spread of misleading information on social media.
Approach:	They analyze a dataset of 6K Arabic memes with manual annotations . they propose to develop computational tools for their detection .
Outcome:	The proposed dataset is a first resource for Arabic multimodal research.

Copied to clipboard

Arianna Muti, Federico Ruggeri, Khalid Khatib, Alberto Barrón-Cedeño, Tommaso Caselli

Thoughts to Target: Enhance Planning for Target-driven Conversation (2024.emnlp-main)

Challenge:	a new study aims to understand the implicit reasoning used to convey misogynistic comments in Italian and English.
Approach:	They propose misogyny detection as an Argumentative Reasoning task and use argumentation theory to build large language models to understand the implicit reasoning used to convey misogany in Italian and English.
Outcome:	The proposed task is an argumentative reasoning task in Italian and English.

Copied to clipboard

Zhonghua Zheng, Lizi Liao, Yang Deng, Ee-Peng Lim, Minlie Huang, Liqiang Nie

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging (2024.emnlp-main)

Challenge:	Empirical results demonstrate that our method significantly improves the planning ability of LLMs, especially in target-driven conversations.
Approach:	They propose a two-stage framework to improve the LLMs’ capability in planning conversations towards designated targets by distilling natural language plans from a target-driven conversation corpus and generating new plans with demonstration-guided in-context learning.
Outcome:	The proposed framework improves the ability of conversational models to plan towards designated targets and can be used to build extensive conversational AI.

Copied to clipboard

Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, Pradeep Dasigi

Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation (2024.emnlp-main)

Challenge:	Training data compositions for Large Language Models (LLMs) can significantly affect their downstream performance.
Approach:	They propose a method which trains individual models on subsets of a training corpus and reuses them across evaluations of combinations of subset.
Outcome:	The proposed method improves training efficiency by scaling only linearly with respect to new data.

Copied to clipboard

Zhe Cao, Zhi Qu, Hidetaka Kamigaito, Taro Watanabe

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters (2024.emnlp-main)

Challenge:	Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously.
Approach:	They propose to fine-tune a language in its intrinsic subspace with a tiny fraction of entire parameters.
Outcome:	The proposed methods outperform full-parameter fine-tuning up to 2.25 spBLEU scores and reduce trainable parameters to 0.4% for high and medium-resource languages and 1.6% for low-resourced ones.

Copied to clipboard

Zhiyu Guo, Hidetaka Kamigaito, Taro Watanabe

Challenge:	Recent studies have focused on scaling the context size of large language models (LLMs) however, the enormous inference costs of LLMs limit their applications.
Approach:	They propose a method which uses attention scores and the l 1 norm to evaluate token importance.
Outcome:	Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B show that the proposed method outperforms attention-score-only baselines in over 12 tasks.

Generative Subgraph Retrieval for Knowledge Graph–Grounded Dialog Generation (2024.emnlp-main)

Copied to clipboard

Jinyoung Park, Minseok Joo, Joo-Kyung Kim, Hyunwoo Kim

Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers (2024.emnlp-main)

Challenge:	Existing methods for knowledge graph–grounded dialog generation fail to leverage the rich knowledge of pretrained language models.
Approach:	They propose a method for dialog generation that integrates dialog history with a knowledge graph.
Outcome:	The proposed method achieves state-of-the-art in knowledge graph–grounded dialog generation on OpenDialKG and KOMODIS datasets.

Copied to clipboard

Tuc Nguyen, Thai Le

Generalizing Clinical De-identification Models by Privacy-safe Data Augmentation using GPT-4 (2024.emnlp-main)

Challenge:	Existing studies show that augmenting the training data of pre-trained language models with parametric fine-tuning methods can enhance their robustness under adversarial attacks.
Approach:	They propose an approach that fine-tunes PLMs with adapters and adversarial augmentation via mixup to leverage existing knowledge from a set of pre-known attacks.
Outcome:	The proposed approach achieves best trade-off between training efficiency and robustness under adversarial attacks compared to baselines on five downstream tasks across six varied black-box attacks and 2 PLMs.

Copied to clipboard

Woojin Kim, Sungeun Hahm, Jaejin Lee

Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game (2024.emnlp-main)

Challenge:	De-identification (de-ID) is critical for patient confidentiality in clinical data management due to the difficulty of retaining training corpora and labeling standards vary across institutions.
Approach:	They propose to exploit GPT-4 for data augmentation through one-shot and zero-shot prompts to exploit the problem of PHI leakage by redacting PHI before processing.
Outcome:	The proposed approach significantly improves on three types of F1 scores in cross-dataset testing.

Copied to clipboard

Prisha Samdarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan

GottBERT: a pure German Language Model (2024.emnlp-main)

Challenge:	We evaluate the performance of large language models (LLMs) against expert and novice human players.
Approach:	They propose to use the New York Times Connections game as a test bed to evaluate the abstract reasoning capabilities of large language models (LLMs) they propose to test the ability of large-language models to be able to cluster and categorize words using semantic relations.
Outcome:	The proposed game is a test bed for evaluating abstract reasoning capabilities in humans and AI systems.

Copied to clipboard

Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, Martin Boeker

Computational Meme Understanding: A Survey (2024.emnlp-main)

Challenge:	Pre-trained language models have advanced natural language processing (NLP) despite the introduction of BERT, single-language models are still relevant.
Approach:	They present a German singlelanguage RoBERT model pre-trained exclusively on the German portion of the OSCAR dataset.
Outcome:	The GottBERT model outperforms the existing models on Named Entity Recognition and text classification tasks.

Copied to clipboard

Khoi Nguyen, Vincent Ng

CoverICL: Selective Annotation for In-Context Learning via Active Graph Coverage (2024.emnlp-main)

Challenge:	Computational Meme Understanding (CMU) is a collection of tasks involving the automated comprehension of memes.
Approach:	They propose a comprehensive taxonomy for memes along three dimensions – forms, functions, and topics and introduce three key tasks for Computational Meme Understanding, namely classification, interpretation, and explanation.
Outcome:	The proposed model is based on a taxonomy of memes along three dimensions and is compared to existing models and datasets.

Copied to clipboard

Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefa Rangwala, Christos Faloutsos, George Karypis

Retrieval-enriched zero-shot image classification in low-resource domains (2024.emnlp-main)

Challenge:	In-context learning (ICL) uses few-shot labeled examples to perform selective annotation.
Approach:	They propose an algorithm that incorporates uncertainty sampling into selective annotation for ICL . CoverICL builds a nearest-neighbor graph based on the semantic similarity between candidate ICL examples .
Outcome:	The proposed algorithm outperforms existing methods for low-budget active learning (AL) it is up to 2x more budget-efficient than SOTA methods for high-budge AL.

Copied to clipboard

Nicola Dall’Asen, Yiming Wang, Enrico Fini, Elisa Ricci

I-AM-G: Interest Augmented Multimodal Generator for Item Personalization (2024.emnlp-main)

Challenge:	Low-resource domains are those where data or annotations are scarce.
Approach:	They propose a retrieval-based method for low-resource domains that trains without training . they use web-crawled databases to retrieve relevant textual information from query images .
Outcome:	The proposed method outperforms existing training-based methods in low-resource domains . it retrieves relevant textual information from large web-crawled databases .

Copied to clipboard

Xianquan Wang, Likang Wu, Shukang Yin, Zhi Li, Yanjiang Chen, Hufeng Hufeng, Yu Su, Qi Liu

Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps (2024.emnlp-main)

Challenge:	e-commerce and recommender systems lack a framework for personalized generation . a new framework extracts tags from multimodal information of items that the user has interacted with .
Approach:	They propose a framework that extracts tags from multimodal information and rewrites item description . they then use a decoupled text-to-text and image-to image retriever to search for similar item text .
Outcome:	The proposed framework can generate results aligned with user preferences . it can be used in e-commerce and recommender systems to win over diverse user base .

Copied to clipboard

Giuseppe Attanasio, Beatrice Savoldi, Dennis Fucci, Dirk Hovy

Enhancing Language Model Alignment: A Confidence-Based Approach to Label Smoothing (2024.emnlp-main)

Challenge:	a new class of multitasks, multilingual neural networks, has recently pushed the boundaries of speech-related tasks.
Approach:	They evaluate performance of two widely used multilingual automatic speech recognition models . they find clear gender disparities, with the advantaged group varying across languages .
Outcome:	The proposed models are compared on 19 languages from eight language families and two speaking conditions.

Copied to clipboard

Baihe Huang, Hiteshi Sharma, Yi Mao

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have remarkable capabilities across various domains . Reinforcement Learning with Human Feedback (RLHF) phase is crucial for training . label smoothing is a technique that replaces hard labels with soft labels .
Approach:	They propose a method that iteratively updates the label smoothing parameter based on preference labels and model forecasts.
Outcome:	The proposed method improves the performance of large language models on state-of-the-art alignment tasks.

Copied to clipboard

Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist

Show and Guide: Instructional-Plan Grounded Vision and Language Model (2024.emnlp-main)

Challenge:	Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment.
Approach:	They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data.
Outcome:	The proposed algorithm can estimate the optimal policy even from off-policy data.

Copied to clipboard

Diogo Glória-Silva, David Semedo, Joao Magalhaes

Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents (2024.emnlp-main)

Challenge:	Existing plans-following language models (LLMs) are not capable of multimodal input and output, resulting in inconsistent performance on multimodal tasks.
Approach:	They propose a multimodal plan-following language model that integrates both textual plans and visual information to bring cross-modality to instructional tasks.
Outcome:	The proposed model performs well on multimodal and textual dialogue in a plan-grounded setting.

Copied to clipboard

Bandhav Veluri, Benjamin Peloquin, Bokai Yu, Hongyu Gong, Shyamnath Gollakota

QuBE: Question-based Belief Enhancement for Agentic LLM Reasoning (2024.emnlp-main)

Challenge:	Existing spoken dialogue models are half-duplex in nature and require explicit prompting by the user or implicit tracking of interruption or silence events.
Approach:	They propose to integrate time information into Llama3-8b so that they run synchronously with the real-world clock.
Outcome:	The proposed model outperforms state-of-the-art in dialogue meaningfulness while maintaining naturalness.

Copied to clipboard

Minsoo Kim, Jongyoon Kim, Jihyuk Kim, Seung-won Hwang

CompAct: Compressing Retrieved Documents Actively for Question Answering (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models (LLMs) have led to an explosion of interest in their deployment as agents.
Approach:	They propose a method that enhances agents’ focus on task-relevant contexts by constructing a belief state via question answering.
Outcome:	The proposed method outperforms established baselines and achieves marked improvements on the BeIR zero-shot retrieval benchmark.

Copied to clipboard

Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, Jaewoo Kang

An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models (2024.emnlp-main)

Challenge:	Existing methods to condense extensive documents with no loss of information are difficult to implement in real-world scenarios.
Approach:	They propose a framework that employs an active strategy to condense extensive documents without losing key information.
Outcome:	The proposed framework improves performance and compression rate on multi-hop question-answering benchmarks.

Copied to clipboard

Fatemeh Shiri, Xiao-Yu Guo, Mona Far, Xin Yu, Reza Haf, Yuan-Fang Li

Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models (2024.emnlp-main)

Challenge:	Large Multimodal Models (LMMs) have shown impressive generalization ability on vision and language tasks, but their spatial understanding is under-explored.
Approach:	They construct a VQA dataset to analyze LMMs' spatial reasoning capabilities.
Outcome:	The proposed model is stronger at basic object detection than complex spatial reasoning.

Copied to clipboard

Jiaxin Zhang, Wendi Cui, Yiran Huang, Kamalika Das, Sricharan Kumar

Local Contrastive Editing of Gender Stereotypes (2024.emnlp-main)

Challenge:	Large language models capture factual knowledge across a wide range of domains, but refining their capabilities on previously seen knowledge remains a challenge.
Approach:	They propose a synthetic knowledge ingestion method that leverages fine-grained synthesis and interleaved generation to construct high-quality data representations from raw knowledge sources.
Outcome:	The proposed method outperforms baseline methods on question-answering tasks spanning finance, biomedicine, and open-generation domains.

Copied to clipboard

Marlene Lutz, Rochelle Choenni, Markus Strohmaier, Anne Lauscher

De-Identification of Sensitive Personal Data in Datasets Derived from IIT-CDIP (2024.emnlp-main)

Challenge:	Stereotypical bias encoded in language models (LMs) poses a threat to safe language technology . current research lacks a thorough understanding of manifestations of biases in specific model weights.
Approach:	They propose a method that localizes and edits weights associated with gender bias . they use local contrastive editing to localize and control a small subset of weights .
Outcome:	The proposed method localizes and controls a small subset of weights that encode gender bias.

Copied to clipboard

Stefan Larson, Nicole Lima, Santiago Diaz, Amogh Joshi, Siddharth Betala, Jamiu Suleiman, Yash Mathur, Kaushal Prajapati, Ramla Alakraa, Junjie Shen, Temi Okotore, Kevin Leach

RAR: Retrieval-augmented retrieval for code generation in low resource languages (2024.emnlp-main)

Challenge:	Large volumes of data are becoming increasingly important for training machine learning models for document understanding tasks like classification, information extraction, and visual question answering.
Approach:	They propose a data de-identification pipeline that replaces sensitive data with synthetic, but realistic, data that preserves the utility of de-identified documents.
Outcome:	The proposed method preserves the utility of the de-identified documents so that they can continue to be used in various document understanding applications.

Copied to clipboard

Avik Dutta, Mukul Singh, Gust Verbruggen, Sumit Gulwani, Vu Le

STAR: SocioTechnical Approach to Red Teaming Language Models (2024.emnlp-main)

Challenge:	Either examples or documentation are commonly used for improved code generation.
Approach:	They propose retrieval augmented retrieval as a two-step method for selecting relevant examples and documentation.
Outcome:	The proposed method outperforms example and grammar retrieval on low-resource languages . it also outperformed two-step retrieval when used independently .

Copied to clipboard

Laura Weidinger, John Mellor, Bernat Pegueroles, Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, A. Bergman, Mikel Rodriguez, Verena Rieser, William Isaac

Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA (2024.emnlp-main)

Challenge:	STAR is a sociotechnical framework that improves on current best practices for red teaming safety of large language models.
Approach:	They propose a sociotechnical framework that improves on current best practices for red teaming safety of large language models.
Outcome:	The proposed framework improves on current best practices for red teaming safety of large language models.

Copied to clipboard

Maharshi Gor, Hal Daumé Iii, Tianyi Zhou, Jordan Boyd-Graber

Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)

Challenge:	Recent advances in large language models have led to claims of AI surpassing humans in QA tasks . authors: models are purportedly acing tests that many humans find challenging .
Approach:	They propose a framework that enables quantitative assessment and comparison of problem-solving abilities in QA agents.
Outcome:	The proposed framework uncovers distinctficiency patterns in knowledge domains and reasoning skills.

Copied to clipboard

Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang

Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass.
Approach:	They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens.
Outcome:	The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost .

Copied to clipboard

Tarun Tater, Sabine Schulte Im Walde, Diego Frassinelli

Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark (2024.emnlp-main)

Challenge:	a recent study examines the visual representation of concrete concepts using images from Bing and YFCC.
Approach:	They examine the variability in visual representations by using images of concrete and abstract concepts from Bing and YFCC.
Outcome:	The proposed model can distinguish between concrete and abstract concepts using basic visual features, the authors show . their model outperforms other models in the nearest neighbor analysis, but it is more complex and requires more visual features .

Copied to clipboard

Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Tucker Balch, Manuela Veloso, Svitlana Vyetrenko

Can LLMs Learn Uncertainty on Their Own? Expressing Uncertainty Effectively in A Self-Training Manner (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) are a critical tool for time series analysis and reporting in many fields, including healthcare, finance, climate, and many more.
Approach:	They propose a framework for rigorously evaluating the capabilities of Large Language Models (LLMs) on time series understanding, encompassing both univariate and multivariate forms.
Outcome:	The proposed framework delineates various characteristics inherent in time series data.

Copied to clipboard

Shudong Liu, Zhaocong Li, Xuebo Liu, Runzhe Zhan, Derek Wong, Lidia Chao, Min Zhang

Challenge:	Large language models (LLMs) exhibit excessive, random, and uninformative uncertainty rendering them unsuitable for decision-making in human-computer interactions.
Approach:	They propose an uncertainty-aware instruction tuning method that aligns LLMs’ perception with the probabilistic uncertainty of the generation.
Outcome:	The proposed method improves LLMs' performance by 45.2%, with reasonably good out-of-domain generalization capabilities.

Preference-Guided Reflective Sampling for Aligning Language Models (2024.emnlp-main)

Copied to clipboard

Hai Ye, Hwee Tou Ng

Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP (2024.emnlp-main)

Challenge:	Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs.
Approach:	They propose a more efficient method for iterative data generation and model re-training that leverages tree-based tree-derived generation framework to enable more efficient sampling.
Outcome:	The proposed method significantly outperforms repeated random sampling in best-of-N sampling on AlpacaEval and Arena-Hard.

Copied to clipboard

Pieter Delobelle, Giuseppe Attanasio, Debora Nozza, Su Lin Blodgett, Zeerak Talat

Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs (2024.emnlp-main)

Challenge:	a measure’s intended use and reliability assessment are often unclear or entirely absent from the literature examining bias measures in natural language processing.
Approach:	They propose a set of desiderata to assess the degree to which a measure’s results enable informed action and a review of 146 papers proposing bias measures in NLP.
Outcome:	The proposed desiderata are based on 146 papers proposing bias measures in natural language processing (NLP) . they show that key elements of actionability, including a measure’s intended use and reliability assessment, are often unclear or entirely absent.

Copied to clipboard

Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap

Challenge:	Recent advances in large language models have enabled richer social simulations . however, the role of information asymmetry in these simulations has been overlooked .
Approach:	They develop an evaluation framework to simulate social interactions with LLMs in different settings.
Outcome:	The proposed framework performs better in unrealistic, omniscient simulation settings but struggles in those with information asymmetry.

A Simple LLM Framework for Long-Range Video Question-Answering (2024.emnlp-main)

Copied to clipboard

Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, Gedas Bertasius

Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing (2024.emnlp-main)

Challenge:	a recent study has shown that short video understanding is not trivial due to the need for long-range temporal reasoning capabilities.
Approach:	They propose a language-based short- and long-range question-answering framework LLoVi . they propose 'multi-round summarization prompt' that asks the LLM to summarize the captions .
Outcome:	The proposed framework outperforms the state-of-the-art on the EgoSchema dataset and to grounded VideoQA.

Copied to clipboard

Akshat Gupta, Sidharth Baskaran, Gopala Anumanchipalli

Challenge:	Recent work using Rank-One Model Editing (ROME) has shown that there are certain facts that the algorithm is unable to edit without breaking the model.
Approach:	They propose to use a model editing method called Rank-One Model Editing to make multiple edits to a single model without breaking it.
Outcome:	The proposed method improves generalization and locality of model editing and improves model collapse compared to the original implementation of ROME.

Casablanca: Data and Models for Multidialectal Arabic Speech Recognition (2024.emnlp-main)

Copied to clipboard

Bashar Talafha, Karima Kadaoui, Samar Magdy, Mariem Habiboullah, Chafei Chafei, Ahmed El-Shangiti, Hiba Zayed, Mohamedou Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah El Mekki, El Moatez Billah Nagoudi, Benelhadj Saadia, Hamzah Alsayadi, Walid Al-Dhabyani, Sara Shatnawi, Yasir Ech-chammakhy, Amal Makouar, Yousra Berrachedi, Mustafa Jarrar, Shady Shehata, Ismail Berrada, Muhammad Abdul-Mageed

Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations (2024.emnlp-main)

Challenge:	despite recent advances in speech processing, the majority of world languages and dialects remain uncovered.
Approach:	They propose to collect and transcribe a new Arabic dataset for eight dialects . they also develop strong baselines exploiting the new dataset .
Outcome:	The proposed dataset covers eight Arabic dialects, including Algerian, Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni.

Copied to clipboard

Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria

Communicating with Speakers and Listeners of Different Pragmatic Levels (2024.emnlp-main)

Challenge:	Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to harmful content.
Approach:	They propose a training-free framework that enhances LLM safety across different scenarios.
Outcome:	The proposed framework significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.

Copied to clipboard

Kata Naszadi, Frans Oliehoek, Christof Monz

RECANTFormer: Referring Expression Comprehension with Varying Numbers of Targets (2024.emnlp-main)

Challenge:	Using a model of the speaker's intentions, people can achieve pragmatic interpretations using a variety of reasoning abilities.
Approach:	They propose to model the interaction between speakers and listeners with different levels of pragmatic competence and to model their level of reasoning abilities.
Outcome:	The proposed model is based on a simulating language learning and conversing between speakers and listeners with different levels of reasoning abilities.

Copied to clipboard

Bhathiya Hemanthage, Hakan Bilen, Phil Bartie, Christian Dondrup, Oliver Lemon

Challenge:	Existing methods for REC assume that a single referring expression always refers to a one instance in the image.
Approach:	They propose a one-stage method that generates bounding boxes for objects referred to in natural language expressions.
Outcome:	The proposed method outperforms baselines in three GREC datasets.

Sprout: Green Generative AI with Carbon-Efficient LLM Inference (2024.emnlp-main)

Copied to clipboard

Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari

Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs (2024.emnlp-main)

Challenge:	Sprout reduces the carbon footprint of inference in large language models by over 40% in real-world evaluations.
Approach:	Sprout introduces "generation directives" to guide autoregressive generation process . et al. cites Llama model and global electricity grid data as examples .
Outcome:	Sprout reduces the carbon footprint of generative AI models by over 40% in real-world evaluations using the Llama model and global electricity grid data.

Copied to clipboard

Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, Mark Dredze

T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings (2024.emnlp-main)

Challenge:	Journalists engage in multiple steps in news writing that depend on human creativity, such as exploring different “angles” and selecting sources.
Approach:	They propose to use large language models to help journalists plan their news coverage . they find that LLMs recommend more creative angles and more informational sources .
Outcome:	The proposed models align better with humans when recommending angles, compared with informational sources.

Copied to clipboard

Björn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, Samuel Weinbach

SpeechQE: Estimating the Quality of Direct Speech Translation (2024.emnlp-main)

Challenge:	Tokenizers are crucial for encoding information in Large Language Models, but their development has stagnated.
Approach:	They propose a tokenizer that embeds words through sparse activation patterns over character triplets . they show competitive downstream performance with a parameter reduction of more than 85% .
Outcome:	The proposed approach achieves competitive downstream performance with a parameter reduction of more than 85% on embedding layers.

Copied to clipboard

HyoJung Han, Kevin Duh, Marine Carpuat

Assessing and Verifying Task Utility in LLM-Powered Applications (2024.emnlp-main)

Challenge:	Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored.
Approach:	They propose a new quality estimation system based on cascaded and end-to-end architectures.
Outcome:	The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation.

Copied to clipboard

Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qingyun Wu, Chi Wang, Ahmed Awadallah, Charles Clarke, Julia Kiseleva

Challenge:	Rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks.
Approach:	They propose a framework to propose criteria tailored to the unique purpose of any given application and propose corresponding criteria for the application.
Outcome:	The proposed framework provides a comprehensive assessment of the effectiveness and robustness of two open source datasets including Math Problem solving and ALFWorld House-hold related tasks.

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models (2024.emnlp-main)

Copied to clipboard

Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric Xing

Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree (2024.emnlp-main)

Challenge:	Empirical evaluations on eight recent LLMs reveal that DRPO significantly enhances alignment performance, enabling base models to outperform their SFT/RLHF-tuned counterparts.
Approach:	They propose a tuning-free approach to self-alignment called Dynamic Rewarding with Prompt Optimization (DRPO) it leverages a dynamic rewarding mechanism to identify and rectify alignment weaknesses .
Outcome:	The proposed approach outperforms existing methods and is highly adaptable to various alignment challenges.

Copied to clipboard

Harbani Jaggi, Kashyap Coimbatore Murali, Eve Fleisig, Erdem Biyik

Adversarial Text Generation using Large Language Models for Dementia Detection (2024.emnlp-main)

Challenge:	Disagreement among annotators can reveal nuances in subjective tasks that lack a simple ground truth .
Approach:	They propose three approaches to predict annotator ratings on the toxicity of text . they integrate annotators' history, demographics, survey information into their models .
Outcome:	The proposed approach outperforms other methods in toxicity rating prediction.

Copied to clipboard

Youxiang Zhu, Nana Lin, Kiran Balivada, Daniel Haehn, Xiaohui Liang

xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics (2024.emnlp-main)

Challenge:	Large language models excel in text classification tasks, but they do not perform well with picture description.
Approach:	They propose an interpretable classification approach by Adversarial Text Generation (ATG) that could relate dementia detection with other tasks.
Outcome:	The proposed approach achieves 85% accuracy, >10% improvement over the previous methods.

Copied to clipboard

Daniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko, Steffen Eger

The Greatest Good Benchmark: Measuring LLMs’ Alignment with Utilitarian Moral Dilemmas (2024.emnlp-main)

Challenge:	State-of-the-art trainable machine translation evaluation metrics rely on large encoders . this makes them computationally expensive and inaccessible to researchers with limited resources.
Approach:	They propose a method to extract knowledge stored in large encoders and a pipeline for efficient black-box distillation.
Outcome:	The proposed model surpasses COMET-22 and BLEURT-20 on the WMT22 dataset by 6.4%.

Copied to clipboard

Giovanni Marraffini, Andrés Cotton, Noe Hsueh, Axel Fridman, Juan Wisznia, Luciano Corro

Challenge:	Our analysis across 15 diverse LLMs reveals consistently encoded moral preferences that diverge from established moral theories and lay population moral standards.
Approach:	They propose to evaluate the moral judgments of large language models using utilitarian dilemmas to determine their moral alignment.
Outcome:	The findings highlight the ‘artificial moral compass’ of Large Language Models, offering insights into their moral alignment.

FairFlow: Mitigating Dataset Biases through Undecided Learning for Natural Language Understanding (2024.emnlp-main)

Copied to clipboard

Jiali Cheng, Hadi Amiri

Style-Shifting Behaviour of the Manosphere on Reddit (2024.emnlp-main)

Challenge:	Existing debiasing frameworks can detect known dataset biases and spurious correlations in data.
Approach:	They propose a framework that learns to be undecided in its predictions for data samples . they propose 'contrary' objective that learn debiased and robust representations from biased views .
Outcome:	The proposed framework outperforms existing methods against out-of-domain and hard test samples without compromising performance.

Copied to clipboard

Jai Aggarwal, Suzanne Stevenson

The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective (2024.emnlp-main)

Challenge:	Hate speech groups (HSGs) may negatively influence online platforms through their distinctive language, which may affect the tone and topic of discussion in other spaces if spread beyond the HSGs.
Approach:	They explore the linguistic style of the Manosphere on reddit and how it reflects their linguistic styles across communities.
Outcome:	The linguistic style of the Manosphere on Reddit is studied to determine whether it is harmful to health and community health.

Copied to clipboard

Yihan Ma, Xinyue Shen, Yixin Wu, Boyang Zhang, Michael Backes, Yang Zhang

Holistic Evaluation for Interleaved Text-and-Image Generation (2024.emnlp-main)

Challenge:	Recent research has shown that high-quality prompts are essential for LLMs to produce accurate and relevant responses.
Approach:	They analyze 10,538 in-the-wild prompts collected from various platforms and develop a framework that decomposes the prompts into eight key components.
Outcome:	The proposed framework decomposes 10,538 in-the-wild prompts into eight components.

Copied to clipboard

Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, Lifu Huang

FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)

Challenge:	Existing evaluation benchmarks do not support arbitrarily interleaved images and text for both inputs and outputs.
Approach:	They propose to use a benchmark to evaluate interleaved text-and-image generation . they define five evaluation aspects for InterleavatedEval, a reference-free metric .
Outcome:	The proposed benchmarks cover a limited number of domains and use cases and lack comparableity-based metrics.

Copied to clipboard

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev

The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Copied to clipboard

Alexander Choi, Syeda Sabrina Akter, J.p. Singh, Antonios Anastasopoulos

Is Child-Directed Speech Effective Training Data for Language Models? (2024.emnlp-main)

Challenge:	Large language models have shown capabilities close to human performance in various analytical tasks.
Approach:	They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions .
Outcome:	The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks.

Copied to clipboard

Steven Feng, Noah Goodman, Michael Frank

RevMUX: Data Multiplexing with Reversible Adapters for Efficient LLM Batch Inference (2024.emnlp-main)

Challenge:	High-performing language models are typically trained on hundreds of billions of words, but human learners use language fluently after far less training data.
Approach:	They train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset.
Outcome:	The proposed models show that child language input is not valuable for training language models.

Copied to clipboard

Yige Xu, Xu Guo, Zhiwei Zeng, Chunyan Miao

Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024.emnlp-main)

Challenge:	Large language models (LLMs) have brought a great breakthrough to the natural language processing community, but their high throughput demands make them difficult to handle concurrent queries.
Approach:	They propose a parameter-efficient data multiplexing framework that integrates a reversible design in the multiplexer and can be reused to perform reverse operations and restore individual samples for classification.
Outcome:	The proposed framework improves inference efficiency while maintaining satisfactory classification performance.

Copied to clipboard

Juncai Li, Ru Li, Xiaoli Li, Qinghua Chai, Jeff Pan

M3Hop-CoT: Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thought (2024.emnlp-main)

Challenge:	Existing approaches to abstract inference ignore the polysemy and hierarchical nature of concepts . prevailing approaches disregard how arguments might entail differently across various concept levels, thereby missing potential enlargement connections.
Approach:	They propose a framework that organizes arguments hierarchically and delves into entailment relations at diverse concept levels.
Outcome:	The proposed framework improves the model's generalization and reasoning prowess in natural language inference.

Copied to clipboard

Gitanjali Kumari, Kirtan Jain, Asif Ekbal

GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation (2024.emnlp-main)

Challenge:	Recent studies have shown that Large Language Models (LLMs) neglect cultural diversity and key aspects like emotion and contextual knowledge hidden in the visual modalities.
Approach:	They propose a framework for misogynous meme identification using a multimodal multimodal prompting principle and a CLIP-based classifier.
Outcome:	The proposed framework performs well on the SemEval-2022 task 5 dataset, and is generalizable across different datasets.

Copied to clipboard

Govind Ramesh, Yao Dou, Wei Xu

RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation (2024.emnlp-main)

Challenge:	Using large language models to jailbreak is important for testing safety and security issues.
Approach:	They propose an approach that leverages the reflective capabilities of large language models for jailbreaking with only black-box access.
Outcome:	The proposed method achieves jailbreak success rates of 98% on GPT-4, 92% on GTP-4 Turbo, and 94% on Llama-3.1-70B in under 7 queries.

Copied to clipboard

Kiseung Kim, Jay-Yoon Lee

Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (2024.emnlp-main)

Challenge:	Existing approaches to retrieval augmented generation (RAG) are based on parametric knowledge and external knowledge.
Approach:	They propose a weakly supervised method for training a relevance estimator (RE) that provides relative relevance between contexts as previous rerankers did, and provides confidence, which can be used to classify whether given context is useful for answering the given question.
Outcome:	The proposed framework improves previously unreferenced large language models and can be trained with a small generator without labels for correct contexts.

Copied to clipboard

Vatsal Gupta, Pranshu Pandya, Tushar Kataria, Vivek Gupta, Dan Roth

Simul-MuST-C: Simultaneous Multilingual Speech Translation Corpus Using Large Language Model (2024.emnlp-main)

Challenge:	Language models display sensitivity to input perturbations, causing concerns about trust among users.
Approach:	They propose a methodology to examine how input perturbations affect language models across various scales, including pre-trained models and large language models.
Outcome:	The proposed methods enhance the model’s robustness to input perturbations and if exposure to one perturbation enhances or diminishes its performance with respect to other perturbations.

Copied to clipboard

Mana Makinae, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text (2024.emnlp-main)

Challenge:	Simultaneous speech translation (SiST) begins translating before the entire source input is received.
Approach:	They propose a dataset that rearranges sentences into segmented monotonic data for simultaneous speech translation using the Large Language Model.
Outcome:	The proposed dataset improves quality and latency in siST translations by rearranging sentences into segmented monotonic data.

Copied to clipboard

Pritika Ramu, Aparna Garimella, Sambaran Bandyopadhyay

On the Fragility of Active Learners for Text Classification (2024.emnlp-main)

Challenge:	Existing measures for table quality evaluation fail to capture the overall semantics of tables . failure to accurately assess table quality can result in including subpar content or overlooking valuable tables in documents.
Approach:	They propose a method that captures table semantics by breaking down a table into atomic statements and comparing them with ground truth statements.
Outcome:	The proposed method shows stronger correlation with human judgments of table quality across four datasets.

Copied to clipboard

Abhishek Ghose, Emma Nguyen

BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers (2024.emnlp-main)

Challenge:	Active learning (AL) techniques optimally utilize a labeling budget by iteratively selecting instances that are most valuable for learning.
Approach:	They propose to use active learning techniques to iteratively select instances that are most valuable for learning.
Outcome:	The proposed framework is used to benchmark active learning techniques for text classification using pre-trained representations.

Copied to clipboard

Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May Dongmei Wang, Joyce Ho, Chao Zhang, Carl Yang

Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval (2024.emnlp-main)

Challenge:	Developing effective biomedical retrieval models is important for excelling at knowledge-intensive biomedically tasks but still challenging due to the lack of sufficient publicly annotated biomedic data and computational resources.
Approach:	They propose a series of dense retrievers for enhancing biomedical retrieval via unsupervised pre-training on large biomedically corpora, followed by instruction fine-tuning on a combination of labeled datasets and synthetic pairs.
Outcome:	Experiments on 5 biomedical tasks across 11 datasets confirm the performance of the retrieval model on various biomedically demanding tasks.

Copied to clipboard

Jonghyun Song, Cheyon Jin, Wenlong Zhao, Andrew McCallum, Jay-Yoon Lee

M3D: MultiModal MultiDocument Fine-Grained Inconsistency Detection (2024.emnlp-main)

Challenge:	Experimental results show that using only bi-encoders as an intermediate reranker can improve top-1 accuracy with negligible slowdown (7%).
Approach:	They propose a framework that compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other.
Outcome:	The proposed framework compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other.

Copied to clipboard

Chia-Wei Tang, Ting-Chih Chen, Kiet Nguyen, Kazi Sajeed Mehrab, Alvi Ishmam, Chris Thomas

MedAdapter: Efficient Test-Time Adaptation of Large Language Models Towards Medical Reasoning (2024.emnlp-main)

Challenge:	Existing methods for fact-checking claim are limited by ambiguous information and lack sample-level predictions.
Approach:	They propose a method that predicts the logical relationship of each aspect of a claim from a set of multimodal documents.
Outcome:	The proposed method outperforms existing models on two benchmarks while providing finer-grained predictions, explanations, and evidence.

Copied to clipboard

Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Haotian Sun, Hang Wu, Carl Yang, May Dongmei Wang

EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records (2024.emnlp-main)

Challenge:	Large language models (LLMs) have improved generation and reasoning capabilities compared to traditional BERT-sized models due to massive number of parameters and extensive pre-training on vast textual corpora.
Approach:	They propose a unified post-hoc adapter for test-time adaptation of large language models . they propose to fine-tune only a small BERT-sized adapter to rank candidate LLMs .
Outcome:	The proposed adapter improves performance on four biomedical tasks without requiring computational resources or sharing data with third parties.

Copied to clipboard

Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, May Dongmei Wang

SimLLM: Detecting Sentences Generated by Large Language Models Using Similarity between the Generation and its Re-generation (2024.emnlp-main)

Challenge:	EHRAgent enables clinicians to interact with EHRs using natural language . reliance on rule-based conversion systems often necessitates additional training or effort from data engineers.
Approach:	They propose a large language model agent that generates and executes code in natural language to facilitate clinicians in directly interacting with EHRs.
Outcome:	The proposed agent outperforms the strongest baseline by up to 29.6% in success rate on three real-world EHR datasets.

Copied to clipboard

Hoang-Quoc Nguyen-Son, Minh-Son Dao, Koji Zettsu

CELLO: Causal Evaluation of Large Vision-Language Models (2024.emnlp-main)

Challenge:	Prior studies have detected the generation of non-analogous text with substantial differences between original and generated content.
Approach:	They propose a method to detect analogous machine-generated sentences that closely mimic human-written ones by estimating the similarity between an input sentence and its generated counterpart.
Outcome:	The proposed approach outperforms existing methods in academic dishonesty, spam dissemination, and misinformation propagation.

Copied to clipboard

Meiqi Chen, Bo Peng, Yan Zhang, Chaochao Lu

Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair (2024.emnlp-main)

Challenge:	Recent advances in large vision-language models have improved causal reasoning abilities . however, current models struggle with tasks like causal reasoning .
Approach:	They propose a fine-grained and unified definition of causality involving interactions between humans and objects.
Outcome:	The proposed model surpasses traditional commonsense causality by including explicit causal graphs . it also shows that current LVLMs can benefit from a causally inspired prompting strategy .

Copied to clipboard

Yusuke Sakai, Mana Makinae, Hidetaka Kamigaito, Taro Watanabe

Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)

Challenge:	Existing siMT corpora are limited due to high costs and limited annotator capabilities.
Approach:	They propose a method to convert ST corpora into interpretation-style corpors by fine-tuning models with Large Language Models.
Outcome:	The proposed method reduces latency while achieving better quality compared to other models.

Copied to clipboard

Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang

MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)

Challenge:	Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining.
Approach:	They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts.
Outcome:	The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings .

Copied to clipboard

Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu

ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering (2024.emnlp-main)

Challenge:	Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored.
Approach:	They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios.
Outcome:	The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC.

Copied to clipboard

Francesco Molfese, Simone Conia, Riccardo Orlando, Roberto Navigli

ABLE: Personalized Disability Support with Politeness and Empathy Integration (2024.emnlp-main)

Challenge:	Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely opaque.
Approach:	They propose a zero-shot question answering framework that combines retrieval, case-based reasoning and introspection to improve the model's performance and interpretability.
Outcome:	The proposed framework outperforms existing LLMs and previous knowledge integration approaches in commonsense reasoning benchmarks and achieves an average accuracy improvement of 4.5 points.

Copied to clipboard

Kshitij Mishra, Manisha Burja, Asif Ekbal

Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models (2024.emnlp-main)

Challenge:	Adaptive, Bespoke, Listen and Empathetic is a conversational support system for physical disabilities that tracks user personas and provides personalized support according to user person preferences.
Approach:	They propose a conversational support system that tracks user personas and integrates politeness and empathy levels into responses to ensure that support interactions are tailored to each user's characteristics and preferences.
Outcome:	The proposed system is based on a conversational dataset enriched with user profile annotations and tested on 84 users with physical disabilities.

Copied to clipboard

Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, Jinyoung Yeo

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code (2024.emnlp-main)

Challenge:	Prior work has used LLMs to generate programming language and applied external compilers for such tasks.
Approach:	They propose a framework that expresses task-level logic with pseudocode and tailors it to each instance and simulates execution of it.
Outcome:	The proposed framework outperforms baselines in diverse reasoning tasks.

Copied to clipboard

Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang, Jinyoung Yeo

Improving Minimum Bayes Risk Decoding with Multi-Prompt (2024.emnlp-main)

Challenge:	Large language models (LLMs) have made great progress in code generation, however, they still produce errors.
Approach:	They propose a RL environment that provides feedback on code editing by analyzing the performance of the revised code in unit tests.
Outcome:	The proposed model outperforms baselines in enhancing open-source code LLMs’ code editing, making them comparable with closed-source LLM.

Copied to clipboard

David Heineman, Yao Dou, Wei Xu

Deciphering Cognitive Distortions in Patient-Doctor Mental Health Conversations: A Multimodal LLM-Based Detection and Reasoning Framework (2024.emnlp-main)

Challenge:	Existing methods to generate LLMs with a single ‘best’ prompt are unstable and sub-optimal in practice.
Approach:	They propose to decode multiple candidate generations from a prompt bank at inference-time and use Minimum Bayes Risk (MBR) to select a final output.
Outcome:	The proposed method improves MBR across a set of conditional generation tasks and models.

Copied to clipboard

Gopendra Singh, Sai Vemulapalli, Mauajama Firdaus, Asif Ekbal

Nearest Neighbor Normalization Improves Multimodal Retrieval (2024.emnlp-main)

Challenge:	Cognitive distortion research sheds light on pervasive errors in thinking patterns . authors present method for detecting and reasoning about cognitive distortions .
Approach:	They propose a method for detecting and reasoning about cognitive distortions using Large Language Models.
Outcome:	The proposed method improves accuracy and depth of detection and reasoning tasks in a zero-shot manner.

Copied to clipboard

Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, Tristan Thrush

Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning (2024.emnlp-main)

Challenge:	Recent training-free methods suggest that accuracy can be improved without fine-tuning.
Approach:	They propose a method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization.
Outcome:	The proposed method improves retrieval metrics for all contrastive models and datasets and does not require training on the reference database.

Copied to clipboard

Shengguang Wu, Shusheng Yang, Zhenglun Chen, Qi Su

LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering (2024.emnlp-main)

Challenge:	Existing methods to assess social-pragmatic inference in large language models are inadequacy, and preferential tuning is the best approach.
Approach:	They propose to use free-form models' responses as a measure to assess social-pragmatic reasoning and advocate for preference optimization over supervised finetuning (SFT).
Outcome:	The proposed model outperforms supervised finetuning (SFT) and offers a near-free launch in pragmatic abilities without compromising general capabilities.

Copied to clipboard

Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang

Context-aware Watermark with Semantic Balanced Green-red Lists for Large Language Models (2024.emnlp-main)

Challenge:	Existing long-context Large Language Models (LLMs) struggle with the “lost in the middle” issue.
Approach:	They propose a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG’s understanding of complex long-context knowledge.
Outcome:	The proposed system outperforms long-context LLMs, advanced RAG, and vanilla RAG on three multi-hop datasets.

Copied to clipboard

Yuxuan Guo, Zhiliang Tian, Yiping Song, Tianlun Liu, Liang Ding, Dongsheng Li

Knowledge Graph Enhanced Large Language Model Editing (2024.emnlp-main)

Challenge:	Recent research suggests that watermarking methods cause degradation of text quality due to semantic disparities between the watermarked text and the unwatermarked text.
Approach:	They propose a semantic-aware watermark method that generates a watermark key considering contexts to split a green/red list for watermark injection.
Outcome:	The proposed method reduces performance drop due to adding bias on green lists . it also allows green lists to cover almost all semantics .

Copied to clipboard

Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, Zhumin Chen

‘Quis custodiet ipsos custodes?’ Who will watch the watchmen? On Detecting AI-generated peer-reviews (2024.emnlp-main)

Challenge:	Existing methods for editing large language models struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of post-edit LLMs in processing edited knowledge.
Approach:	They propose a model editing method that leverages knowledge graphs to enhance LLM editing by capturing changes in associated knowledge by constructing an external graph.
Outcome:	The proposed method improves the generalization ability of LLMs in processing edited knowledge.

Copied to clipboard

Sandeep Kumar, Mohit Sahu, Vardhan Gacche, Tirthankar Ghosal, Asif Ekbal

Mitigating Open-Vocabulary Caption Hallucinations (2024.emnlp-main)

Challenge:	Recent studies have focused on generic AI-generated text detection or estimating fraction of peer-reviews that can be AI-generated.
Approach:	They propose a model that detects whether a peer-review is written by ChatGPT and a reviewer-generated model that generates similar outputs upon re-prompting.
Outcome:	The proposed model is more robust, but paraphrasing is more effective.

Copied to clipboard

Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, Hadar Averbuch-Elor

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes (2024.emnlp-main)

Challenge:	Existing methods for image captioning ignore the long-tailed nature of hallucinations . a new framework is proposed to address hallucines in image captions in the open-vocabulary setting .
Approach:	They propose a framework to address hallucinations in image captioning in the open-vocabulary setting.
Outcome:	The proposed framework surpasses the CHAIR benchmark in diversity and accuracy in open-vocabulary captioning.

Copied to clipboard

Kosuke Nishida, Kyosuke Nishida, Kuniko Saito

ALVIN: Active Learning Via INterpolation (2024.emnlp-main)

Challenge:	Existing methods to train large language models that require a non-uniform model norm are not effective.
Approach:	They propose a technique that allows for uniformity of the norm of the model parameters . they propose 'weight scaling as reparameterization' to adjust the norm to the parameter .
Outcome:	The proposed technique outperforms existing methods and stabilizes training with the transformer decoders.

Copied to clipboard

Michalis Korakakis, Andreas Vlachos, Adrian Weller

Filtered Direct Preference Optimization (2024.emnlp-main)

Challenge:	Experimental results show that Active Learning methods ignore example groups whose prevalence may vary . supervised fine-tuning remains a critical component of model development, authors say .
Approach:	They propose an approach that uses interpolations to create anchors between examples . they propose to use the model to identify informative examples that counteract shortcuts .
Outcome:	The proposed model outperforms state-of-the-art active learning methods on six datasets . it prioritizes high-certainty instances that integrate representations from different example groups .

Copied to clipboard

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

Challenge:	Existing studies on the impact of RLHF on text quality have focused on reward-model-free RL.
Approach:	They propose an extension of direct preference optimization to improve model performance by analyzing the quality of the preference dataset.
Outcome:	The proposed method improves the performance of models optimized with DPO over those optimized with reward-model-based RLHF.

Instruction Fine-Tuning: Does Prompt Loss Matter? (2024.emnlp-main)

Copied to clipboard

Mathew Huerta-Enochian, Seung Ko

Entity Insertion in Multilingual Linked Corpora: The Case of Wikipedia (2024.emnlp-main)

Challenge:	Recent research in language modeling has made huge advances in training instruction-following agents.
Approach:	They analyze the effects of various prompt loss token weights for supervised instruction fine-tuning.
Outcome:	The proposed model outperforms models fine-tuned on short-completion data on multiple-choice and short-generation benchmarks.

Copied to clipboard

Tomás Feith, Akhil Arora, Martin Gerlach, Debjit Paul, Robert West