Findings of the Association for Computational Linguistics: EACL 2024
Chem-FINESE: Validating Fine-Grained Few-shot Entity Extraction through Text Reconstruction (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for fine-grained few-shot entity extraction are difficult to implement in the chemical domain due to the information overload of scientific papers. |
| Approach: | They propose a sequence-to-sequence based few-shot entity extraction approach . it uses a seq2seq entity extractor and a self-validation module to reconstruct original input sentence . |
| Outcome: | The proposed framework achieves 8.26% and 6.84% performance gains on two datasets. |
GPTs Are Multilingual Annotators for Sequence Generation Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods of data annotation are time-consuming and expensive . complexity of crowdsourcing increases when dealing with low-resource languages . |
| Approach: | They propose an autonomous method to gather unlabeled data and label them using large language models. |
| Outcome: | The proposed method is cost-efficient and applicable for low-resource language annotation. |
Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have not addressed the heterogeneous and hierarchical properties inherent in EHR data. |
| Approach: | They propose a medical code-centric multimodal contrastive EHR learning framework with hierarchical regularisation that integrates multifaceted information encompassing medical codes, demographics, and clinical notes. |
| Outcome: | The proposed framework integrates multifaceted information encompassing medical codes, demographics, and clinical notes using a tailored network design and bimodal contrastive losses. |
FlexiQA: Leveraging LLM’s Evaluation Capabilities for Flexible Knowledge Selection in Open-domain Question Answering (2024.findings-eacl)
Copied to clipboard
| Challenge: | Current methods for open-domain question answering lacks the hallucination and relevance of acquired knowledge to the given question. |
| Approach: | They propose a new pipeline that utilizes the diverse evaluation capabilities of large language models to select knowledge effectively and flexibly. |
| Outcome: | The proposed pipeline combines the strengths of both paradigms and overcomes their shortcomings. |
Hyper-BTS Dataset: Scalability and Enhanced Analysis of Back TranScription (BTS) for ASR Post-Processing (2024.findings-eacl)
Copied to clipboard
Chanjun Park, Jaehyung Seo, Seolhwa Lee, Junyoung Son, Hyeonseok Moon, Sugyeong Eo, Chanhee Lee, Heuiseok Lim
| Challenge: | Automatic Speech Recognition (ASR) post-processing requires substantial amounts of data, requiring expensive phonetic transcription experts. |
| Approach: | They propose a "Hyper-BTS" dataset that is five times larger than prior studies . they propose criteria for categorizing error types within ASR post-processing . |
| Outcome: | The proposed method can generate ASR inputs from clean text using a text-to-speech system. |
ParrotTTS: Text-to-speech synthesis exploiting disentangled self-supervised representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | ParrotTTS can train a multi-speaker variant using transcripts from a single speaker in low resource setup and generalizes to languages not seen while training the self-supervised backbone. |
| Approach: | They propose a modular text-to-speech synthesis model that can train a multi-speaker variant using transcripts from a single speaker. |
| Outcome: | The proposed model outperforms state-of-the-art multi-lingual text-to-speech models using only a fraction of paired data as latter. |
NavHint: Vision and Language Navigation Agent with a Hint Generator (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work on vision and language navigation relies on navigation-related losses to establish the connection between vision and modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment. |
| Approach: | They propose to provide indirect supervision to the navigation agent through a hint generator that generates visual descriptions during navigation. |
| Outcome: | The proposed method improves the navigation performance and interpretability of the R2R and R4R datasets. |
Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies show that only the textual component of hateful memes enables the multimodal classifier to generalize across domains while the image component proves highly sensitive to a specific training dataset. |
| Approach: | They propose to use only the textual component of hateful memes to generalize across different domains while the image component is highly sensitive to a specific training dataset. |
| Outcome: | The proposed model performs similarly to hate-meme classifiers in a zero-shot setting, while the introduction of meme’s image captions worsens performance by an average F1 of 0.02. |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages (2024.findings-eacl)
Copied to clipboard
Nikita Martynov, Mark Baushenko, Anastasia Kozlova, Katerina Kolomeytseva, Aleksandr Abramov, Alena Fenogenova
| Challenge: | Recent advances in large language models have shown impressive text generation and language understanding capabilities, evident in benchmarks like SuperGLUE, GEM, BigBench etc. |
| Approach: | They propose a method for generative spelling correction that can be extended to any language with minor changes. |
| Outcome: | The proposed method can be extended to any language with minor changes, and is based on a set of generative models with a single-domain and multi-domain test sets. |
How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)
Copied to clipboard
| Challenge: | a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models. |
| Approach: | They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings. |
| Outcome: | The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks. |
Large Language Models for Psycholinguistic Plausibility Pretesting (2024.findings-eacl)
Copied to clipboard
| Challenge: | Psycholinguists typically use language models to create controlled materials . plausibility judgments are often based on coarse-grained judgements, but fine-grounded ones do not . |
| Approach: | They investigate whether Language Models can be used to generate plausibility judgments . they find that plausible judgements from LMs are highly related to human judgements - whereas other LM models are not . |
| Outcome: | The proposed language models can generate plausibility judgments from human evaluators . the proposed models do not provide satisfactory discriminative power . |
Modeling Aspect Sentiment Coherency via Local Sentiment Aggregation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have not explored aspect sentiment coherency, including its implications in adversarial defense. |
| Approach: | They propose a local sentiment aggregation paradigm that models aspect sentiment coherency . they demonstrate the capability of LSA in adversarial defense . |
| Outcome: | The proposed model outperforms existing models and achieves state-of-the-art sentiment classification performance. |
An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have proposed reference-free evaluations of image captions . however, these approaches are restrictive and favor captions with similar vocabulary but different meanings. |
| Approach: | They propose to use reference-free metrics to evaluate image captions . they propose to combine lexical overlap and semantics to identify fine-grained errors . |
| Outcome: | The proposed metrics struggle to identify fine-grained errors, the authors show . CLIPScore, UMIC, and PAC-S are sensitive to variations in image-relevant objects mentioned in the caption . |
Barriers to Effective Evaluation of Simultaneous Interpretation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have relied on out-of-the-box machine translation metrics to evaluate interpretation data, but they do not account for human judgments of interpretation quality. |
| Approach: | They propose to use machine translation metrics to evaluate human interpretations to address potential barriers to disfluency, summarization, paraphrasing and segmentation. |
| Outcome: | The proposed model achieves better correlation with human judgments than state-of-the-art metrics. |
Inconsistent dialogue responses and how to recover from them (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to assess and bolster utterance consistency of chat systems have been shown difficult to detect. |
| Approach: | They propose to use annotators to write dialogue responses and recovery utterances to assess and bolster utteration consistency of chat systems. |
| Outcome: | The proposed dataset significantly improves the detection and resolution of inconsistencies in chat conversations. |
MUG: Interactive Multimodal Grounding on User Interfaces (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior studies modeled multimodal UI grounding in one round, but such an interaction is inherently iterative. |
| Approach: | They propose a task where a user and an agent collaborate on an interface screen . they use a dataset of 77,820 sequences of human user-agent interaction on mobile interfaces . |
| Outcome: | The proposed task improves the absolute task completion by 18% over the entire test set and 31% over the challenging split. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Current multilingual agreement (MA) methods require parallel data between multiple language pairs, which is not always realistic and optimize the agreement in an ambiguous direction, which hampers the translation performance. |
| Approach: | They propose a novel multilingual agreement framework that optimizes agreement bidirectionally with the Kullback-Leibler Divergence loss. |
| Outcome: | The proposed method improves strong baselines on the task of multilingual neural machine translation with three benchmarks: TED Talks, News, and Europarl. |
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining. |
| Approach: | They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora. |
| Outcome: | The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages. |
OYXOY: A Modern NLP Test Suite for Modern Greek (2024.findings-eacl)
Copied to clipboard
Konstantinos Kogkalidis, Stergios Chatzikyriakidis, Eirini Giannikouri, Vasiliki Katsouli, Christina Klironomou, Christina Koula, Dimitris Papadakis, Thelka Pasparaki, Erofili Psaltaki, Efthymia Sakellariou, Charikleia Soupiona
| Challenge: | a select few languages claim the lion's share in the literature, boasting a plethora of models and a constant stream of results. |
| Approach: | They propose a linguistically motivated evaluation suite for Greek NLP . they propose linguistic-motivated tasks for natural language inference, word sense disambiguation and metaphor detection . |
| Outcome: | The proposed evaluation suite for Greek NLP is linguistically motivated and technically relevant. |
A Comprehensive Evaluation of Inductive Reasoning Capabilities and Problem Solving in Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Inductive reasoning is fundamental to both human and artificial intelligence. |
| Approach: | They evaluated the inductive reasoning abilities of current Large Language Models (LLMs) and their performance on symbolic tasks. |
| Outcome: | The proposed models fail on symbolic tasks and show that chain-of-thought prompts help them by decomposing the problem-solving process, but the LLMs learn limitedly. |
Towards efficient self-supervised representation learning in speech processing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models require several GPUs for days to pretrain, generating environmental concerns because of their high energy consumption. |
| Approach: | They propose an efficient self-supervised model that uses a single GPU during 24 to 48 hours of pretraining to address high computational costs. |
| Outcome: | The proposed model represents two orders of magnitude better than existing models. |
Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large pre-trained language models such as GPT-3.5 and GPT-4 have gained significant attention in natural language research due to limited computational resources or inaccessible parameters. |
| Approach: | They propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs while editing their output. |
| Outcome: | The proposed framework significantly improves GPT-3.5’s performance in logical form-to-text conversion and low-resource machine translation, surpassing other state-of-the-art (SOTA) LLM post-editing methods in cross-domain settings. |
Noise Contrastive Estimation-based Matching Framework for Low-Resource Security Attack Pattern Recognition (2024.findings-eacl)
Copied to clipboard
| Challenge: | Techniques, Tactics and Procedures (TTPs) mapping is a difficult task for CTI extraction . conventional learning approaches target the problem in the classical multiclass/label classification setting . |
| Approach: | They propose a neural matching architecture that incorporates a sampling-based learn-to-compare mechanism to facilitate the learning process. |
| Outcome: | The proposed model reduces the complexity of competing over large label space. |
Large Language Models for Scientific Information Extraction: An Empirical Study for Virology (2024.findings-eacl)
Copied to clipboard
| Challenge: | Scholarly communication in the digital age is facing significant challenges due to the overwhelming volume of publications. |
| Approach: | They propose to use Wikipedia infoboxes and structured Amazon product descriptions to create structured scholarly contribution summaries using text generation capabilities of LLMs. |
| Outcome: | The proposed model can be applied to complex IE tasks within terse domains like Science with 1000x fewer parameters than the state-of-the-art GPT-davinci. |
Re3val: Reinforced and Reranked Generative Retrieval (2024.findings-eacl)
Copied to clipboard
| Challenge: | generative retrieval models encode pointers to information in a corpus as an index within the model’s parameters. |
| Approach: | They propose a generative retrieval model that leverages contextual information to rerank retrieved page titles and utilizes REINFORCE to maximize rewards generated by constrained decoding. |
| Outcome: | The proposed model can't be tuned for the downstream readers as decoding the page title is a non-differentiable operation. |
Entity Linking in the Job Market Domain (2024.findings-eacl)
Copied to clipboard
| Challenge: | In Natural Language Processing, entity linking (EL) has centered around Wikipedia, but yet remains underexplored for the job market domain. |
| Approach: | They propose to use a bi-encoder and an autoregressive model to link fine-grained span-level skill mentions to a specific taxonomy entry to quantify labor market demands. |
| Outcome: | The proposed model outperforms GENRE in strict evaluation, but performs better in loose evaluation. |
(Chat)GPT v BERT Dawn of Justice for Semantic Change Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. |
| Approach: | They propose to use (Chat)GPT to solve two diachronic extensions of the Word-in-Context task: TempoWiC and HistoWic. |
| Outcome: | The proposed technology performs significantly worse than the foundational GPT version of (Chat)GPT for studying semantic change. |
Towards Unified Uni- and Multi-modal News Headline Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Current approaches to multimodal summarization and headline generation are limited by hierarchical cross-modal encoders and modality-specific decoders. |
| Approach: | They propose a task formulation that utilizes a simple encoder-decoder model to generate headlines from uni- and multimodal news articles. |
| Outcome: | The proposed model is trained on data of several modalities and extends the decoder to handle the multimodal output. |
On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models (2024.findings-eacl)
Copied to clipboard
Thilini Wijesiriwardene, Ruwan Wickramarachchi, Aishwarya Naresh Reganti, Vinija Jain, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | Analogies facilitate the transfer of meaning and knowledge from one domain to another. |
| Approach: | They propose to use large language models to encode syntactic and semantic structures of sentences to identify sentence analogies. |
| Outcome: | The LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies. |
Contextualization Distillation from Large Language Model for Knowledge Graph Completion (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing knowledge graph completion models lack textual information, which limits their performance . a plug-in-and-play approach is needed to train small models in descriptive context . |
| Approach: | They propose a plug-in-and-play approach to knowledge graph completion that prompts LLMs to generate descriptive context. |
| Outcome: | The proposed method improves performance on Wikipedia articles and synset definitions. |
Differentially Private Natural Language Models: Recent Advances and Future Directions (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to great success in various natural language processing tasks. |
| Approach: | They propose a systematic review of recent advances in DP deep learning models . they discuss some differences and additional challenges of DP-NLP . |
| Outcome: | The proposed method can prevent reconstruction attacks and protect against potential side knowledge while maintaining the privacy of sensitive data. |
Learning to Compare Financial Reports for Financial Forecasting (2024.findings-eacl)
Copied to clipboard
| Challenge: | Public companies in the US are required to publish annual reports that contain over 25,000 words across all sections and a high percentage of boilerplate content that does not change much year-to-year. |
| Approach: | They propose to model complex, cross-document relationships between financial reports using paired financial reports. |
| Outcome: | The proposed model can predict company risk and correlation from financial reports . the proposed model is able to recognize complex, nuanced relationships with complex signals . |
Arukikata Travelogue Dataset with Geographic Entity Mention, Coreference, and Link Annotation (2024.findings-eacl)
Copied to clipboard
Shohei Higashiyama, Hiroki Ouchi, Hiroki Teranishi, Hiroyuki Otomo, Yusuke Ide, Aitaro Yamamoto, Hiroyuki Shindo, Yuki Matsuda, Shoko Wakamiya, Naoya Inoue, Ikuya Yamada, Taro Watanabe
| Challenge: | et al., 2006) considers geographic relatedness among geo-entity mentions in document-level geoparsing. |
| Approach: | They present a Japanese travelogue dataset that considers geographic relatedness among geo-entity mentions. |
| Outcome: | The proposed dataset includes 200 travelogue documents with rich geo-entity information . it shows that human activities, mobility, and events are often described with natural language expressions of locations or geographic entities (geo-entities) |
Knowledge Generation for Zero-shot Knowledge-based VQA (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent knowledge-based visual question answering methods do not explicitly show the knowledge needed to answer the questions and therefore lack interpretability. |
| Approach: | They propose a method which generates knowledge from an LLM and incorporates it into a zero-shot manner. |
| Outcome: | The proposed method performs better than previous zero-shot K-VQA methods on two benchmarks and is generally relevant and helpful. |
Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have focused on the effectiveness of contrastive learning in deep learning. |
| Approach: | They propose a method to improve sentence representation of unsupervised contrastive learning by examining the role of temperature in VRL and SRL. |
| Outcome: | The proposed method improves representation of unsupervised contrastive learning by cooling the temperature of the representation space. |
Bootstrap Your Own PLM: Boosting Semantic Features of PLMs for Unsuperivsed Contrastive Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have shown that SimCSE significantly improves the performance of pretrained language models on the sentence representation benchmark. |
| Approach: | They propose a method called IFM which reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
| Outcome: | The proposed method reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
Personalized Abstractive Summarization by Tri-agent Generation Pipeline (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing research shows that large language models do not consistently satisfy users' preferences or expectations. |
| Approach: | They propose a tri-agent generation pipeline that includes a generator, an instructor, and an editor to enhance output personalization. |
| Outcome: | The proposed pipeline generates outputs that better meet user expectations on two abstractive summarization datasets. |
Revisiting the Markov Property for Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Statistical machine translation (SMT) has employed Markov models, but autoregressive models are less effective. |
| Approach: | They propose to use a Markov Autoregressive Transformer to model neural machine translation using four WMT benchmarks. |
| Outcome: | The proposed model performs better than autoregressive models on four WMT benchmarks. |
Reward Engineering for Generating Semi-structured Explanation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Unstructured natural language explanations lack a comprehensive explanation mechanism to verify a model's true reasoning capabilities. |
| Approach: | They propose a reward engineering method which uses semi-structured explanations to verify a model's true reasoning capabilities. |
| Outcome: | The proposed method achieves new state-of-the-art on two semi-structured explanation generation benchmarks (ExplaGraph and COPA-SSE) . |
Towards Context-Based Violence Detection: A Korean Crime Dialogue Dataset (2024.findings-eacl)
Copied to clipboard
| Challenge: | Currently, there are three main branches of violence detection, including surveillance of potential threats in offline situation and automatic prevention of harmful media. |
| Approach: | They propose to use the Korean Crime Dialogue Dataset to classify violence that occurs in offline scenarios. |
| Outcome: | The proposed dataset shows that understanding varying relationships among interlocutors improves the performance of crime dialogue classification. |
Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in building sentence embedding models have centered on replacing traditional human-generated text datasets with those generated by LLMs. |
| Approach: | They propose a loss function that incorporates Positive-Negative sample Augmentation within the contrastive learning objective to enhance sentence embeddings using both human and LLM-generated datasets. |
| Outcome: | The proposed model mitigates the sentence anisotropy problem in Wikipedia corpus and improves Spearman’s correlation in standard Semantic Textual Similarity (STS) tasks (+1.47% compared to CLHAIF). |
Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues (2024.findings-eacl)
Copied to clipboard
| Challenge: | blending multiple languages within a single conversation presents a formidable challenge, given the wide-ranging variations influenced by individual speaking styles and cultural backgrounds. |
| Approach: | They propose a novel approach to harness the Big Five personality traits acquired in an unsupervised manner from code-mixed conversations to bolster the performance of response generation. |
| Outcome: | The proposed approach enhances contextual relevance and performance of the proposed model by combining personality traits with dialogue context. |
Morality is Non-Binary: Building a Pluralist Moral Sentence Embedding Space using Contrastive Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing NLP methods treat morality as binary, ranging from right to wrong. |
| Approach: | They propose to build a pluralist moral sentence embedding space using contrastive learning methods to examine relationships among moral elements. |
| Outcome: | The proposed method shows that pluralism can be captured in an embedding space. |
Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing direct S2TT systems have been unable to disambiguate utterances where prosody plays a crucial role. |
| Approach: | They propose to use contrastive evaluation to quantitatively measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. |
| Outcome: | The proposed system improves overall accuracy 12.9% and improves intent scores 15.6%. |
Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have demonstrated ChatGPT's remarkable few-shot, even zero-shot learning abilities when compared to other models. |
| Approach: | They quantitatively evaluate the performance of ChatGPT on inter-sentential relations such as temporal relations, causal relations, and discourse relations. |
| Outcome: | The proposed model performs well on temporal relations, causal relations, and discourse relations. |
Backtracing: Retrieving the Cause of the Query (2024.findings-eacl)
Copied to clipboard
| Challenge: | a number of online content portals allow users to ask questions to supplement their understanding. |
| Approach: | They propose a task of backtracing to retrieve the text segment that most likely caused a user query. |
| Outcome: | The proposed method improves on the backtracing task in three domains . the results show that there is room for improvement and new retrieval approaches . |
Unsupervised Multilingual Dense Retrieval via Generative Pseudo Labeling (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods often yield inferior performance in multilingual retrieval, requiring a large amount of paired data, which is costly. |
| Approach: | They propose an Unsupervised Multilingual dense Retriever trained without paired data which iteratively improves performance of multilingual retrievers. |
| Outcome: | The proposed framework outperforms supervised baselines on two benchmark datasets and shows that iterative training improves the performance. |
Investigating grammatical abstraction in language models using few-shot learning of novel noun gender (2024.findings-eacl)
Copied to clipboard
| Challenge: | a new study shows that language models can generalise novel noun gender from one to two learning examples and apply it across agreement contexts. |
| Approach: | They conduct a noun learning experiment to assess whether a transformer and an LSTM can achieve human-like abstraction of grammatical gender in French. |
| Outcome: | The proposed models generalise gender from one to two learning examples and apply gender across agreement contexts, albeit with a bias for the masculine gender category. |
On-the-fly Denoising for Data Augmentation in Natural Language Understanding (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve data augmentation performance may introduce noisy data that impairs training. |
| Approach: | They propose an on-the-fly denoising technique that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original dataset. |
| Outcome: | The proposed method improves on text classification and question-answering tasks on general augmentation techniques and prevents overfitting on noisy labels. |
Style Vectors for Steering Generative Large Language Models (2024.findings-eacl)
Copied to clipboard
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, Tobias Hecking
| Challenge: | Large language models (LLMs) can be trained on vast corpora and can generate text in a nuanced and parameterisable way. |
| Approach: | They propose to add style vectors to the activations of hidden layers during text generation to steer output towards specific styles. |
| Outcome: | The proposed approach differs from prompt engineering in that it can be nuanced and parameterisable. |
Consistent Joint Decision-Making with Heterogeneous Learning Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to handle inconsistencies in correlated decisions are insufficient for tasks like hierarchical image classification and text summa-rization. |
| Approach: | They propose a decision-making framework that promotes consistency among decisions made by diverse models while utilizing external knowledge. |
| Outcome: | The proposed framework is superior to baselines on multiple datasets. |
Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage (2024.findings-eacl)
Copied to clipboard
| Challenge: | a new study examines the association capabilities of large language models . as models scale up, their ability to associate entities/information intensifies . however, there is a performance gap when associating commonsense knowledge versus PII, with the latter showing lower accuracy. |
| Approach: | They examine the association capabilities of large language models and identify factors that influence their proficiency in associating information. |
| Outcome: | The proposed models show a performance gap when associating commonsense knowledge versus PII, with the latter showing lower accuracy. |
Probing Critical Learning Dynamics of PLMs for Hate Speech Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on pretrained language models (PLMs) for hate speech detection have not investigated how their performance is affected by pretraining and finetuning. |
| Approach: | They propose to compare pretrained language models, evaluate their seed robustness, finetuning settings, and the impact of pretraining data collection time. |
| Outcome: | The proposed models show that they are more robust than other models and that they have a better chance of performing better than domain-specific models. |
Embible: Reconstruction of Ancient Hebrew and Aramaic Texts Using Transformers (2024.findings-eacl)
Copied to clipboard
| Challenge: | inscriptions are damaged over time due to earthquakes, fires, political conflicts, and other natural and human-related causes. |
| Approach: | They evaluate masked language modeling approach to reconstruct illegible inscriptions . they use transformer-based models to fine-tune models on Biblical texts . |
| Outcome: | The proposed model achieves the highest text completion accuracy on masked parts of the Biblical texts. |
Stateful Memory-Augmented Transformers for Efficient Dialogue Modeling (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing Transformers models are computationally expensive for long context inputs. |
| Approach: | They propose a transformer that can interchange information between memory states and context . they evaluate the efficiency of their model on three dialogue datasets and two language datasets . |
| Outcome: | The proposed model is compatible with existing transformer models and can preserve dialogue history information. |
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024.findings-eacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | Embeddings in transformers encode vast amounts of linguistic nuances and patterns. |
| Approach: | They investigate the anisotropy dynamics and intrinsic dimension of embeddings in transformers . they found that transformer decoders exhibit a bell-shaped anisotropie profile . |
| Outcome: | The investigated embeddings exhibit a bell-shaped curve with the highest anisotropy concentrations in the middle layers . the intrinsic dimension increases in the initial phases of training, indicating an expansion into higher-dimensional space. |
MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms (2024.findings-eacl)
Copied to clipboard
Patrick Lee, Alain Chirino Trujillo, Diana Cuevas Plancarte, Olumide Ojo, Xinyi Liu, Iyanuoluwa Shode, Yuan Zhao, Anna Feldman, Jing Peng
| Challenge: | Euphemisms are a linguistic device used to soften or neutralize language that may otherwise be harsh or awkward to state directly. |
| Approach: | They train a multilingual transformer model to disambiguate potentially euphemistic terms in multilingual and cross-lingual settings. |
| Outcome: | The proposed model performs better than monolingual models on the disambiguation task compared to monolingual ones in multilingual and cross-lingual settings. |
PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens. |
| Approach: | They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations. |
| Outcome: | Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods. |
Do-Not-Answer: Evaluating Safeguards in LLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | a dataset evaluating harmful capabilities in large language models is available at https://github.com/Libr-AI/do-not-answer. |
| Approach: | They collect an open-source dataset to evaluate the safeguards in large language models . they find that simple BERT-style classifiers can achieve results comparable to GPT-4 . |
| Outcome: | The proposed dataset compares the safety of six popular LLMs to GPT-4 on automatic safety evaluation. |
Do Language Models Know When They’re Hallucinating References? (2024.findings-eacl)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. |
| Approach: | They propose to use hallucinated book and article references as "model organism" of hallucinism research . authors propose queries to the language model to identify hallucinous references . |
| Outcome: | The authors show that language models can identify hallucinated references without external resources . they show that LMs often produce inconsistent author lists for hallucinos, but also accurately recall the authors of real references . |
Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys (2024.findings-eacl)
Copied to clipboard
| Challenge: | integrating cultural dimensions with dialogue encoding features can enhance the predictive accuracy and quality of dialogue agents. |
| Approach: | They propose to incorporate cultural dimensions into dialogue encoding features to enhance the predictive accuracy of dialogue agents. |
| Outcome: | The proposed model improves the accuracy and quality of dialogue predictions by incorporating cultural dimensions with dialogue encoding features. |
CEO: Corpus-based Open-Domain Event Ontology Induction (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing event-centric NLP models restrict their generalization capabilities by limiting the pre-defined ontology. |
| Approach: | They propose a Corpus-based Event Ontology induction model to relax the restriction imposed by pre-defined ontologies. |
| Outcome: | The proposed model can induce a hierarchical event ontology with meaningful names on eleven open-domain corpora, making it more trustworthy and easier to be further curated. |
Rethinking STS and NLI in Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. |
| Approach: | They propose to rethink semantic textual similarity (STS) and natural language inference (NLI) models with task-specific prompts and model overconfidence to capture disagreements between human judgements. |
| Outcome: | The proposed models are able to capture human opinions on individual examples without any parameter modifications. |
Learning High-Quality and General-Purpose Phrase Representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained language models for phrasal embeddings are unnecessarily complex and require to be pre-tuned on a corpus with context sentences. |
| Approach: | They propose a framework to learn phrase representations in a context-free fashion. |
| Outcome: | The proposed framework generates superior phrase embeddings while requiring a smaller model size. |
Explaining Language Model Predictions with High-Impact Concepts (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to explain large language models (LLMs) are mostly correlational and lack causal features due to compositional nature of languages. |
| Approach: | They propose a framework to provide impact-aware explanations for large language models that are robust to feature changes and influential to the model’s predictions. |
| Outcome: | The proposed explanations improve on real and synthetic tasks and are robust to feature changes and influential to the model’s predictions. |
Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has revealed that machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. |
| Approach: | They propose a metric to detect spurious tokens and a family of regularization methods to mitigate spurious correlations in text classification. |
| Outcome: | The proposed method prevents spurious clusters and significantly improves the robustness of classifiers without auxiliary data. |
On the Intractability to Synthesize Factual Inconsistencies in Summarization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for detecting factual inconsistencies in abstractive summarization are lacking in factual consistency detection. |
| Approach: | They propose to use real model-generated summaries with human annotations to detect factual inconsistencies. |
| Outcome: | The proposed model outperforms the SOTA on CoGenSumm, FactCC, Frank, and SummEval datasets. |
IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on social media bias detection focus on fine-tuning models specific to particular datasets and testing them on corresponding test sets. |
| Approach: | They propose a general bias detection framework, IndiVec, built upon large language models and vector databases. |
| Outcome: | The proposed framework outperforms baseline methods on four political bias datasets and provides explicit top-k indicators to interpret bias predictions. |
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)
Copied to clipboard
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. |
| Approach: | They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
| Outcome: | The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
Computational Morphology and Lexicography Modeling of Modern Standard Arabic Nominals (2024.findings-eacl)
Copied to clipboard
| Challenge: | Modern Standard Arabic (MSA) nominals present many morphological and lexical modeling challenges that have not been consistently addressed before. |
| Approach: | They propose to use a morphological framework to model Arabic nominals using a proposed morphology framework. |
| Outcome: | The proposed model improves accuracy and consistency compared to a commonly used morphological analyzer and generator. |
Relabeling Minimal Training Subset to Flip a Prediction (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to identify and relabel training subsets that can flip a prediction are not efficient, argues a new study. |
| Approach: | They propose an algorithm to identify and relabel the smallest training subset St needed to flip a prediction. |
| Outcome: | The proposed algorithm can flip a prediction on a test point xt with 2% of training points . the proposed method can be used for multiple purposes including evaluating model robustness . |
Why Generate When You Can Discriminate? A Novel Technique for Text Classification using Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for text classification using autoregressive language models are limited . authors propose a novel technique for text classification using autoreregressives . |
| Approach: | They propose a two-step technique for text classification using autoregressive language models . they use a set of perplexity and log-likelihood based numeric features to elicit a text instance . |
| Outcome: | The proposed technique eliminates parameter updates in LMs and does not limit training examples . it is evaluated across 5 datasets and compares with multiple competent baselines . |
Autism Detection in Speech – A Survey (2024.findings-eacl)
Copied to clipboard
| Challenge: | a range of studies have been done on autism in voice, speech and language . females are under-researched in the field, and there are few experiments with transformers . |
| Approach: | They analyse studies of how autism is displayed in voice, speech and language . they define autism and which comorbidities might influence the correct detection . |
| Outcome: | The authors show that there is already a lot of research on autism in speech, but there are still some shortcomings. |
Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work on multimodal content classification has not addressed these challenges. |
| Approach: | They propose to use two auxiliary tasks to fine-tune multimodal models to address hidden cross-modal semantics and weak image-text relationships when modeling text and images. |
| Outcome: | The proposed model improves by up to 2.6 F1 score across five diverse social media datasets. |
What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing and new approaches to zero-shot knowledge composition are lacking in NLP. |
| Approach: | They propose a framework for zero-shot module composition that unifies existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. |
| Outcome: | The proposed framework enables a systematic unification of concepts and enables the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. |
IndiFoodVQA: Advancing Visual Question Answering and Reasoning with a Knowledge-Infused Synthetic Data Generation Pipeline (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large Vision Language Models lack domain-specific data for reasoning on complex problems. |
| Approach: | They propose to use explicit knowledge-infused questions, answers, and reasons to answer and reason upon the questions. |
| Outcome: | The proposed model improves by 25% over the baseline model. |
MAPLE: Micro Analysis of Pairwise Language Evolution for Few-Shot Claim Verification (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for verification of claims are limited by the availability of labeled data. |
| Approach: | They propose a method that explores the alignment between a claim and its evidence using a seq2seq model and a novel semantic measure. |
| Outcome: | The proposed method shows significant performance improvements over baselines SEED, PET and LLaMA 2 across three fact-checking datasets. |
Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Event detection is a crucial information extraction task in many domains, such as Wikipedia or news. |
| Approach: | They propose to use subject-object relations obtained from a rule-based open information extraction system to enhance domain transfer of trigger detection (TD) they combine this enhanced transfer with masked language modeling on the target domain, observing further TD transfer gains. |
| Outcome: | The proposed model improves the transfer of triggers between domains and reduces performance drops when using a low-resource source domain to a high-res target domain. |
Exploring efficient zero-shot synthetic dataset generation for Information Retrieval (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in large language models offer a new avenue of generating synthetic training data to train neural retrieval models for unlabelled data collections. |
| Approach: | They propose a method to generate high-quality synthetic datasets using a small language model and a filtering mechanism to ensure the quality of generated questions. |
| Outcome: | The proposed method outperforms unsupervised retrieval methods such as BM25 and pretrained monoT5. |
Clustering-based Sampling for Few-Shot Cross-Domain Keyphrase Extraction (2024.findings-eacl)
Copied to clipboard
| Challenge: | Scientific domain-specific pre-training has led to state-of-the-art keyphrase extraction performance with a majority of benchmarks being within the domain. |
| Approach: | They propose to leverage topic information available in the data to build a clustering-based sampling approach that facilitates selecting a few samples to label from the target domain . |
| Outcome: | The proposed approach leads to 26.35 points in performance when compared to selecting few-shot samples uniformly at random. |
Random Smooth-based Certified Defense against Text Adversarial Attack (2024.findings-eacl)
Copied to clipboard
| Challenge: | Textual adversarial examples train models on the worst-case text generated by substituting words in original texts with synonyms, but due to the discrete word embedding representations, the large search space hinders the robust training efficiency. |
| Approach: | They propose to treat the word substitution as a continuous perturbation on the word embedding representation and apply random smooth techniques to approximate the word replacement operation. |
| Outcome: | The proposed method outperforms conventional methods and improves the robustness in training. |
Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness (2024.findings-eacl)
Copied to clipboard
| Challenge: | Poorly formulated questions can lead to user frustration and dissatisfaction . |
| Approach: | They propose to leverage key features that contribute to the classification of clarifying questions, enhancing user satisfaction and system performance. |
| Outcome: | The proposed model improves with a minimum performance boost of 45% in traditional classifiers, especially in large language models. |
Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Cross-lingual transfer of language models trained on high-resource languages such as English has been limited due to the high cost of obtaining non-English conversational data. |
| Approach: | They introduce a parallel and large-scale multilingual conversation dataset that is used for cross-lingual alignment pretraining by translating the English-only Schema-Guided Dialogue dataset into 105 other languages. |
| Outcome: | The proposed model performs well on slot-filling and intent classification tasks, and is able to perform well in other languages. |
Correcting Language Model Outputs by Editing Salient Layers (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models that use manual layer selection require prior domain knowledge and expensive empirical layer selection methods. |
| Approach: | They propose a model editing approach that selectively edits a small subset of model parameters to update the factual knowledge. |
| Outcome: | The proposed solution matches the accuracy of previous approaches with only 1/3 of their edits, enabling efficient updates to the parametric knowledge in large language models. |
Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback (2024.findings-eacl)
Copied to clipboard
| Challenge: | In many approaches to Natural Language Processing tasks, language is inherently interactive. |
| Approach: | They propose to use human-AI collaboration to improve human-human interaction by providing feedback that the agent can understand and utilize. |
| Outcome: | The proposed task is an interactive grounded language understanding task in a MineCraft-like world. |
Goodhart’s Law Applies to NLP’s Explanation Benchmarks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Popular methods for "explaining" the outputs of natural language processing (NLP) models operate by highlighting a subset of input tokens that ought, in some sense, to be salient. |
| Approach: | They propose to inflate a model’s comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution inputs. |
| Outcome: | The proposed metrics exploit the tendency for extracted explanations and complements to be “out-of-support” relative to each other and in-distribution inputs. |
Syllable-level lyrics generation from melody exploiting character-level language model (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained language models specifically designed at the syllable level are not available. |
| Approach: | They propose to exploit character-level language models for syllable-level lyrics generation from symbolic melody. |
| Outcome: | The proposed system improves coherence and correctness of generated lyrics without training expensive language models. |
Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca (2024.findings-eacl)
Copied to clipboard
| Challenge: | Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. |
| Approach: | They employ a dataset and machine translations of it to form multilingual data and use it to tune LLMs. |
| Outcome: | The proposed model is on par or better than a model for each language, and multilingual tuning with downsampled data is as powerful and robust. |
Prompt Perturbation Consistency Learning for Robust Language Models (2024.findings-eacl)
Copied to clipboard
Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan
| Challenge: | Large language models have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization. |
| Approach: | They propose a method to reduce the performance drop of large language models by regularizing the divergence between losses from clean and perturbed samples. |
| Outcome: | The proposed approach recovers on average 59% and 69% of the performance drop for IC and SF tasks while using ten times fewer augmented data samples. |
Enhancing Society-Undermining Disinformation Detection through Fine-Grained Sentiment Analysis Pre-Finetuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | a new method for disinformation detection is needed to address the issue of disinformation, authors argue . a series of rigorous experiments establishes a notable connection between disinformation and fine-grained sentiment labels . |
| Approach: | They propose a method leveraging pre-finetuning concept for efficient detection and removal of disinformation that may undermine society. |
| Outcome: | The proposed method improves performance across languages and languages, showing promising results. |
Minimal Distillation Schedule for Extreme Language Model Compression (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for teacher assistant-based distillation require multiple trials to find the optimal teacher assistant. |
| Approach: | They propose a method that allows scheduling of an optimal teacher assistant in just one trial . they show that student performance is positively correlated with the scale-performance tradeoff . |
| Outcome: | The proposed method can select the optimal teacher assistant in just one trial . it can be used to compare performance of student and teacher assistants on GLUE benchmarks. |
Event Semantic Classification in Context (2024.findings-eacl)
Copied to clipboard
| Challenge: | In this work, we focus on the semantic classification of events in context to help machines gain a deeper understanding of events. |
| Approach: | They propose to integrate event semantics into downstream tasks to help machines understand events better. |
| Outcome: | The proposed model improves the understanding of events in context. |
Local and Global Contexts for Conversation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing pretrained conversation models lack the correlation and connection between local and global contexts. |
| Approach: | They propose a local-global hierarchical transformer model that combines local and global contexts. |
| Outcome: | The proposed model outperforms existing conversation models on performance metrics with significant margins. |
Aspect-based Key Point Analysis for Quantitative Summarization of Reviews (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on review summarization use only major opinions, but ignore minority opinions and fail to quantify opinion prevalence. |
| Approach: | They propose a framework for quantitative review summarization using aspect-based key point analysis (ABKPA) they use aspect-basic sentiment analysis to automatically annotate silver labels for matching aspect-sentiment pairs . |
| Outcome: | The proposed framework outperforms state-of-the-art baselines on Yelp reviews on five business categories. |
Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent work has struggled to achieve consistent results due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism. |
| Approach: | They propose a model that leverages the controllability of VQVAE to guide the self-attention mechanism in Transformer-based VAEs to improve semantic control and generation. |
| Outcome: | The proposed model outperforms existing state-of-the-art VAE models in terms of control and preservation of semantic information across different tasks. |
High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pretrained large language models (LLMs) can bridge the performance gap for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
| Approach: | They propose to use pretrained large language models to bridge this gap by automating and evaluating data-to-text generation in under-resourced languages. |
| Outcome: | The proposed model can set the state of the art for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
Antonym vs Synonym Distinction using InterlaCed Encoder NETworks (ICE-NET) (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing research on antonym-synonym distinction is limited by the sparsity of the feature space. |
| Approach: | They propose to capture and model relation-specific properties of antonyms and synonyms pairs . ICE-NET outperforms existing research by a relative score of upto 1.8% in F1-measure . |
| Outcome: | The proposed model outperforms existing models by 1.8% in the F1-measure. |
Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similarity (2024.findings-eacl)
Copied to clipboard
Eric Khiu, Hasti Toossi, Jinyu Liu, Jiaxu Li, David Anugraha, Juan Flores, Leandro Roman, A. Seza Doğruöz, En-Shiun Lee
| Challenge: | Existing approaches for predicting the performance of NLP models for low-resource languages (LRLs) focus on high-resourced languages, overlooking LRLs and domain shifts. |
| Approach: | They investigate the impact of domain similarity on predicting performance of machine translation models in low-resource languages. |
| Outcome: | The results show that domain similarity has the most important impact on predicting the performance of Machine Translation models. |
Does CLIP Bind Concepts? Probing Compositionality in Large Image Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large-scale neural network models combining text and images have made incredible progress in recent years, but to what extent they encode compositional representations of the concepts over which they operate remains an open question . |
| Approach: | They compare the performance of a large pretrained vision and language model (CLIP) to a set of three synthetic datasets designed to test concept binding. |
| Outcome: | The proposed model can encode compositional concepts and bind variables in a structure-sensitive way, e.g., differentiating ‘cube behind sphere’ from ‘cub behind cube’. |
Code-Switching and Back-Transliteration Using a Bilingual Model (2024.findings-eacl)
Copied to clipboard
| Challenge: | a wealth of Judeo-Arabic texts is available on the internet, but most speakers of Arabic are unfamiliar with the Hebrew script. |
| Approach: | They propose two machine-learning models for transliterating Judeo-Arabic into Arabic . they also propose a model for identifying non-Arabian words, predominantly Hebrew and Aramaic . |
| Outcome: | The proposed models outperform previous solutions for transliterating Judeo-Arabic into Arabic and identifying non-Arabian words, mainly Hebrew and Aramaic. |
Tsetlin Machine Embedding: Representing Words Using Logical Expressions (2024.findings-eacl)
Copied to clipboard
| Challenge: | Embedding words in vector space is a fundamental first step in state-of-the-art natural language processing. |
| Approach: | They propose to embed words in vector space using propositional logic instead of dense vectors . they evaluate embeddings on intrinsic and extrinsic benchmarks and visualize word clusters based on their results . |
| Outcome: | The proposed model outperforms GLoVe on six classification tasks. |
Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communities (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies treat ideology as a liberal/conservative binary and fail to capture the spectrum of ideologies that may organically emerge in interconnected online communities. |
| Approach: | They propose a method that uses finetuning language models to probe nuanced ideologies of online communities by analyzing discussions of the 2020 election on Twitter. |
| Outcome: | The proposed approach shows higher alignment than baselines for the proposed approach. |
Unified Embeddings for Multimodal Retrieval via Frozen LLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | Unified Embeddings for Multimodal Retrieval (UniMuR) is a simple but effective approach that embeds multimodal inputs and retrieves visual and textual outputs via frozen Large Language Models (LLMs). |
| Approach: | They propose a method that embeds multimodal inputs and retrieves visual and textual outputs via frozen Large Language Models (LLMs). |
| Outcome: | The proposed method significantly reduces LLM’s modality bias towards generating text-only outputs and achieves strong image/text retrieval ability. |
Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods (2024.findings-eacl)
Copied to clipboard
| Challenge: | Transfer learning methods have shown that reusing non-task-specific knowledge can speed up task-specific learning in the latter. |
| Approach: | They propose to port whole functional modules that encode task-specific knowledge from one model to another using parameter-efficient finetuning techniques. |
| Outcome: | The proposed methods outperform the two alternatives, but there are differences between the two. |
Exploiting Class Probabilities for Black-box Sentence-level Attacks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing sentences use either no feedback or only the class labels for sentence-level attacks. |
| Approach: | They propose an algorithm that uses class probabilities for black-box sentence-level attacks and investigate the effectiveness of using class probabilties on the attack’s success. |
| Outcome: | The proposed algorithm is evaluated against baselines and classifiers and compares with the existing models to determine whether it is worthy or practical to use class probabilities for black-box sentence-level attacks. |
Learning Label Hierarchy with Supervised Contrastive Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to supervised contrastive learning treat each class as independent and therefore consider all classes to be equally important. |
| Approach: | They propose a family of Label-Aware SCL methods that incorporate hierarchical information to SCL by leveraging similarities between classes. |
| Outcome: | The proposed method outperforms baseline supervised approaches on three datasets. |
GrounDial: Human-norm Grounded Safe Dialog Response Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent conversational AI systems generate unsafe responses agreeing to offensive user input or including toxic content. |
| Approach: | They propose a method where response safety is achieved by grounding responses to commonsense social rules without fine-tuning. |
| Outcome: | The proposed approach is quantitatively and qualitatively safer even without additional data or tuning. |
Trainable Hard Negative Examples in Contrastive Learning for Unsupervised Abstractive Summarization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for contrastive learning rely on manual negative examples and are poor in quality and adaptability during training. |
| Approach: | They propose a framework that learns trainable negative examples for contrastive learning in unsupervised abstractive summarization by combining a negative example network and a representation network. |
| Outcome: | The proposed approach eliminates the need for manual negative example design and improves on two benchmark datasets. |
Low-Resource Counterspeech Generation for Indic Languages: The Case of Bengali and Hindi (2024.findings-eacl)
Copied to clipboard
| Challenge: | NLP is a technique that generates counterspeech that “counters” the vicious tone of online abuse and dilutes/ameliorates their rippling effect over the social network. |
| Approach: | They propose to use neural architectures to generate counterspeech that can "counter" the vicious tone of online abuse and dilute/ameliorate their rippling effect over the social network. |
| Outcome: | The proposed model can generate counterspeech in monolingual setups and is more transferable when languages belong to the same language family. |
Teaching Probabilistic Logical Reasoning to Transformers (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to reasoning using transformers are limiting, resulting in inconsistent results in arithmetic and QA benchmarks. |
| Approach: | They propose a novel approach that utilizes probabilistic logical rules as constraints in the fine-tuning phase without relying on them in the inference stage. |
| Outcome: | The proposed approach improves the transformer-based language model’s intrinsic reasoning and makes their probabilistic logical reasoning process more explicit and explainable. |
On Measuring Context Utilization in Document-Level MT Systems (2024.findings-eacl)
Copied to clipboard
| Challenge: | Current studies on document-level translation evaluation focus on sentence-level models which are inadequate for capturing improvements in discourse phenomena. |
| Approach: | They propose to complement accuracy-based evaluation with measures of context utilization. |
| Outcome: | The proposed model can be used to handle context-dependent discourse phenomena using an automatic annotation tool. |
Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing systems that make predictions and ask questions are unable to have a mutual exchange of opinions. |
| Approach: | They propose to use a dataset and computational framework to allow systems to have beneficial discussions with humans, improving the accuracy by 25 points on a natural language inference task. |
| Outcome: | The proposed system improves accuracy by 25 points on a natural language inference task. |
Autoregressive Score Generation for Multi-trait Essay Scoring (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing holistic approaches to score essays using pre-trained BERT-based models are inefficient, leading to inferior qualities in data-scarce traits. |
| Approach: | They propose an autoregressive prediction of multi-trait scores using pre-trained T5 models. |
| Outcome: | The proposed model shows over 5% improvement in prompts and traits compared to previous models . |
CMA-R: Causal Mediation Analysis for Explaining Rumour Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on explainable fake news or rumour detection by and large use attention weights as explanation, but the use of attention weighted explanations is problematic. |
| Approach: | They propose a causal mediation analysis approach to explain the decision-making process of neural models for rumour detection on Twitter by identifying salient tweets that explain model predictions and highlighting causally impactful words in the tweets. |
| Outcome: | The proposed approach shows strong agreement with human judgements for critical tweets determining the truthfulness of stories. |
Morphology Aware Source Term Masking for Terminology-Constrained NMT (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research in terminology-constrained NMT systems focuses on data-driven approaches to generating translations. |
| Approach: | They propose a method that appends target term lemmas to their corresponding source terms in the input sentence while retaining essential grammatical information. |
| Outcome: | The proposed method improves on the “copy-and-inflect” method in two translation directions with different levels of source morphological complexity. |
Improving Backchannel Prediction Leveraging Sequential and Attentive Context Awareness (2024.findings-eacl)
Copied to clipboard
| Challenge: | Backchannels are short and often affirmative or empathetic responses from a listener during a conversation . et al. (2010) showed that timely backchanneling can enhance storytelling ability . |
| Approach: | They propose a context-aware backchannel prediction approach that leverages a pretrained wav2vec model to enhance backchannel performance. |
| Outcome: | The proposed approach improves performance in Korean and English datasets . it leverages the pretrained wav2vec model for encoding audio signal . |
SENSE-LM : A Synergy between a Language Model and Sensorimotor Representations for Auditory and Olfactory Information Extraction (2024.findings-eacl)
Copied to clipboard
| Challenge: | SENSE-LM is a system for the extraction of sensory references in textual corpus . it is based on large language models and linguistic resources such as sensorimotor norms . |
| Approach: | They propose a system that extracts sensory references from large corpus of textual documents using large language models and linguistic resources such as sensorimotor norms. |
| Outcome: | The proposed system is evaluated on two sensory functions, Olfaction and Audition, and compared with state-of-the-art methods. |
Analyzing the Role of Part-of-Speech in Code-Switching: A Corpus-Based Study (2024.findings-eacl)
Copied to clipboard
| Challenge: | Code-switching (CS) is a common linguistic phenomenon wherein speakers fluidly transition between languages in conversation. |
| Approach: | They propose to use a part-of-speech (POS)-based analysis of Spanish-English and Mandarin-English corpora to examine the propensity of bilinguals to engage in CS. |
| Outcome: | The findings confirm the existence of a statistically significant connection between POS and the likelihood of CS across language pairs, but show that it diminishes as tokens distance themselves from CS instances. |
In-Contextual Gender Bias Suppression for Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally expensive. |
| Approach: | They propose to suppress gender biases by providing textual preambles from manually designed templates and real-world statistics without accessing model parameters. |
| Outcome: | The proposed methods suppress gender biases in English LLMs using a CrowsPairs dataset without accessing model parameters. |
Parameter-Efficient Fine-Tuning: Is There An Optimal Subset of Parameters to Tune? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has illuminated the possibility of selective parameter-efficient fine-tuning, which retains the inference speed of the original model and comes at no additional computational cost. |
| Approach: | They propose to selectively update only a small subset of parameters during the fine-tuning process, keeping the remaining parameters frozen during training. |
| Outcome: | The proposed methods retain the inference speed of the original model and come at no additional computational cost. |
Contextualized Topic Coherence Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing topic models that estimate the interpretability of topics are difficult to compare due to their nature as unsupervised models. |
| Approach: | They propose to use contextualized topic coherence metrics to simulate human-centered coherency evaluation while maintaining the efficiency of other automated methods. |
| Outcome: | The proposed metrics better reflect human judgment on topics extracted from short text collections by avoiding highly scored topics that are meaningless to humans. |
ProMISe: A Proactive Multi-turn Dialogue Dataset for Information-seeking Intent Resolution (2024.findings-eacl)
Copied to clipboard
| Challenge: | Work done during internship at Amazon Alexa AI. |
| Approach: | They propose to use iterative suggested question-answering conversation to improve the trade-off between satisfaction of the user’s intent and keeping the information exchange natural. |
| Outcome: | The proposed proposed question-answering conversation improves the satisfaction of the user’s intent while keeping the information exchange natural and cognitive load of the interaction minimal on the users. |
CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Neural machine translation systems exhibit limited robustness in handling source-side linguistic variations. |
| Approach: | They propose a dialectal benchmark to quantify the robustness of MT systems to handle source-side linguistic variations. |
| Outcome: | The proposed benchmark demonstrates that large MT models face challenges translating dialectal variants. |
QAEVENT: Event Extraction as Question-Answer Pairs Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | a novel approach to document-level event extraction requires linguistic expertise. |
| Approach: | They propose a question-and-answer-pair representation of document-level events as question and answer pairs (QAEVENT) this allows for more scalable and faster annotations from crowdworkers without linguistic expertise. |
| Outcome: | The proposed method produces high-quality data for document-level event extraction compared to pre-defined schemas . it is scalable and faster than existing methods, with no linguistic expertise. |
Sequence Shortening for Context-Aware Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Context-aware Machine Translation aims to improve translations of sentences by incorporating surrounding sentences as context. |
| Approach: | They propose to use latent representation of source sentence as context in a multi-encoder architecture to achieve higher accuracy on contrastive datasets. |
| Outcome: | The proposed architectures achieve comparable BLEU and COMET scores on contrastive datasets and comparable accuracies on the single- and multi-encoder approaches. |
Jigsaw Pieces of Meaning: Modeling Discourse Coherence with Informed Negative Sample Synthesis (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work on creating “informed” incoherent samples for coherence modeling has focused on permutations of a coherent document . |
| Approach: | They propose to use Constituency trees, Part-of-speech, semantic overlap to create “informed” negative samples that better represent or mimic incoherence. |
| Outcome: | The proposed methods improve the quality of the negative sample. |
Non-Exchangeable Conformal Language Generation with Nearest Neighbors (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate reliability of generated text are lacking in natural language generation. |
| Approach: | They propose a non-exchangeable conformal prediction method that provides bounds on coverage . they validated their method with k-NN retrieval and show that it produces encouraging results . |
| Outcome: | The proposed method produces encouraging results in machine translation and language modeling tasks. |
Evidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to ODQA use a simple yet effective retriever-reader framework, but this approach is not always effective in abstractive tasks. |
| Approach: | They propose a method that leverages synthetic distractor samples to learn to discriminate evidence passages from distractors. |
| Outcome: | The proposed method is validated on multiple abstractive open-domain question answering tasks. |
Self-training Strategies for Sentiment Analysis: An Empirical Study (2024.findings-eacl)
Copied to clipboard
| Challenge: | Sentiment analysis is a crucial task in natural language processing. |
| Approach: | They propose to leverage a small amount of labeled and unlabeled data to train models with self-training. |
| Outcome: | The proposed method improves the performance of small language models in several few-shot settings while reducing the cost of annotations. |
Language is All a Graph Needs (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work on integrating graph problems into generative language modeling framework remains limited. |
| Approach: | They propose an LLM with instructions based on natural language to perform graph tasks. |
| Outcome: | The proposed model surpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets and sheds light on generative LLMs as new foundation model for graph machine learning. |
Unraveling the Dynamics of Semi-Supervised Hate Speech Detection: The Impact of Unlabeled Data Characteristics and Pseudo-Labeling Strategies (2024.findings-eacl)
Copied to clipboard
| Challenge: | Semi-supervised learning addresses the need for large amounts of labeled training data for state-of-the-art approaches. |
| Approach: | They propose to leverage unlabeled data to reduce the amount of annotated data required for machine learning based hate speech detection by using a semi-supervised approach. |
| Outcome: | The proposed approach reduces the amount of annotated data required by state-of-the-art models by leveraging unlabeled data. |
When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (2024.findings-eacl)
Copied to clipboard
| Challenge: | Using large language models (LMs) for query or document expansion can improve generalization in information retrieval. |
| Approach: | They conduct the first comprehensive analysis of large language models (LMs) for query or document expansion. |
| Outcome: | The proposed expansions improve retrieval performance for weaker models but harm stronger models. |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
Let’s Negotiate! A Survey of Negotiation Dialogue Systems (2024.findings-eacl)
Copied to clipboard
Haolan Zhan, Yufei Wang, Zhuang Li, Tao Feng, Yuncheng Hua, Suraj Sharma, Lizhen Qu, Zhaleh Semnani Azad, Ingrid Zukerman, Reza Haf
| Challenge: | Recent research has focused on negotiation dialogue systems, but no systematic review of this task has been conducted. |
| Approach: | They propose to provide a systematic review of negotiation dialogue systems and to provide an overview of current research. |
| Outcome: | The proposed systems are based on the literature and are compared against existing systems. |
Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing language models such as Transformer-based models fail to predict the conversation outcome. |
| Approach: | They propose to integrate human-annotated domain knowledge and LLM-generated features to provide richer context to counseling conversations. |
| Outcome: | The proposed model improves by 15% when combined with human-annotated domain knowledge and LLM-generated features. |
Better Explain Transformers by Illuminating Important Information (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing explanations focus on the input and output of the Transformers, resulting in confusing results. |
| Approach: | They propose to highlight important information and eliminate irrelevant information by a refined information flow on top of the layer-wise relevance propagation method. |
| Outcome: | The proposed method outperforms baseline models on classification and question-answering datasets with over 3% to 33% improvement on explanation metrics. |
Testing the Depth of ChatGPT’s Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5’s Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking (2024.findings-eacl)
Copied to clipboard
| Challenge: | In the months since its release, ChatGPT and its underlying model, GPT3.5, have garnered massive attention due to their potent mix of capability and accessibility. |
| Approach: | They examine GPT3.5's aptitude for visual tasks using ASCII-art without overt distillation into a lingual summary. |
| Outcome: | The proposed model performs well on image recognition and generation tasks. |
Cross-lingual Editing in Multilingual Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models editing techniques (METs) can efficiently update outdated LLMs without retraining. |
| Approach: | They propose a cross-lingual model editing paradigm where a fact is edited in one language and the subsequent update propagation is observed across other languages. |
| Outcome: | The proposed techniques perform well in multilingual models with knowledge stored in multiple languages. |
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive. |
| Approach: | They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner. |
| Outcome: | The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning. |
AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents (2024.findings-eacl)
Copied to clipboard
| Challenge: | AccentFold uses spatial relationships to improve speech recognition for accented speech . existing methods for accent recognition have been limited due to data scarcity and budget constraints . |
| Approach: | They propose a method that exploits spatial relationships between learned accent embeddings to improve downstream automatic speech recognition. |
| Outcome: | The proposed method outperforms baseline methods in accented speech training. |
Hierarchical and Dynamic Prompt Compression for Efficient Zero-shot API Usage (2024.findings-eacl)
Copied to clipboard
| Challenge: | Long prompts present a significant challenge for LLM-based systems that learn to use unseen APIs directly in-context from their documentation. |
| Approach: | They propose a method that learns to compress a prompt into a few "gist token" activations during finetuning but is ineffective in compressing API documentation. |
| Outcome: | The proposed model keeps the simplicity, efficiency, and large compression factor (20x on SGD) of the gist token approach while achieving significantly better accuracy. |
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Contrastive language-image pre-training models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval. |
| Approach: | They propose a fine-tuning approach to enhance the representations of CLIP models for paraphrases by leveraging large language models. |
| Outcome: | The proposed model improves on baseline models across paraphrased retrieval, visual genome relation and attribution, and seven semantic textual similarity tasks. |
Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods often fail to produce objective results and favor high similarity to the ground-truth question-answer pairs. |
| Approach: | They propose an alternative approach to evaluate question-answer generation using Generative Interpretation (GI) GI outperforms existing evaluation methods in terms of human alignment . |
| Outcome: | The proposed approach outperforms existing evaluation methods in human alignment and shows comparable performance with GPT3.5, only with BART-large. |
Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained language models perform well in In-Topic setups, but face challenges in Cross-Topical setups where testing data is derived from distinct topics. |
| Approach: | They propose a probing-based approach to analyze pre-trained language models in a Cross-Topic setup to better understand the reasons behind generalization gaps. |
| Outcome: | The proposed approach improves on pre-trained language models in In-Topic setups and Cross-Topical scenarios. |
LLM-GEm: Large Language Model-Guided Prediction of People’s Empathy Levels towards Newspaper Article (2024.findings-eacl)
Copied to clipboard
| Challenge: | Empathy is a key component of human-to-human interactions, and is often overlooked due to the inherent noise in crowdsourced annotations. |
| Approach: | They propose a large language model-guided empathy prediction system that rectifies annotation errors based on defined annotation selection threshold and makes annotations reliable for conventional empathy prediction models. |
| Outcome: | The proposed system rectifies annotation errors based on defined selection threshold and makes the annotations reliable for conventional empathy prediction models, e.g., BERT-based pre-trained language models. |
ICE-Score: Instructing Large Language Models to Evaluate Code (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text. |
| Approach: | They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages. |
| Outcome: | The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks. |
CReSE: Benchmark Data and Automatic Evaluation Framework for Recommending Eligibility Criteria from Clinical Trial Information (2024.findings-eacl)
Copied to clipboard
| Challenge: | Eligibility criteria (EC) are defined as a set of conditions an individual must meet to participate in a clinical trial. |
| Approach: | They propose to recommend EC based on clinical trial information, including trial titles, and introduce an automatic evaluation framework to assess clinical validity of the EC recommendation model. |
| Outcome: | The proposed model outperforms existing language models pre-trained on the biomedical domain in EC clustering. |
BMX: Boosting Natural Language Generation Metrics with Explainability (2024.findings-eacl)
Copied to clipboard
| Challenge: | Modern language model (LM) based natural language generation evaluation metrics achieve astonishing results in grading machine generated sentences like humans would. |
| Approach: | They use word-level explanations to convert a word- level score into a segment-level score and combine this segment- level with the original metric to obtain a better metric. |
| Outcome: | The proposed metrics improve across MT and summarization datasets while achieving small improvements on machine translation. |
Joint Inference of Retrieval and Generation for Passage Re-ranking (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for re-ranking documents are sparse and do not require training. |
| Approach: | They propose a method that optimizes mutual information between query and passage distributions by integrating cross-encoders and generative models in the re-ranking process. |
| Outcome: | The proposed method outperforms conventional re-rankers and language model scorers in open-domain QA retrieval settings and diverse retrieval benchmarks under zero-shot settings. |
DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI (2024.findings-eacl)
Copied to clipboard
Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | DialogStudio is the largest and most diverse collection of dialogue datasets . existing datasets lack diversity and comprehensiveness, authors say . |
| Approach: | They introduce DialogStudio: the largest and most diverse collection of dialogue datasets . DialogStuio aggregates more than 80 diverse dialogue dataset . |
| Outcome: | a new dataset is created to improve the quality and diversity of dialogue datasets . DialogStudio is the largest and most diverse collection of dialogue data . |
Exploring hybrid approaches to readability: experiments on the complementarity between linguistic features and transformers (2024.findings-eacl)
Copied to clipboard
| Challenge: | Linguistic features have been a key component of the automatic assessment of text readability (ARA) with the development in the ARA field, the research moved to Deep Learning (DL) |
| Approach: | They compare 6 hybrid approaches to Machine Learning and DL on 4 corpora and found they are the most robust on smaller datasets and across languages. |
| Outcome: | The proposed approaches perform better on smaller datasets and across languages and tasks. |
Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to analyze speech representations using pretraining data are difficult to achieve for endangered languages. |
| Approach: | They propose an unsupervised method to examine the level of abstraction in vector representations of speech from a pretrained model to determine their level of abstractness. |
| Outcome: | The proposed method is fully unsupervised and could be used in comparative studies on under-documented languages. |
The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work evaluated how often PLMs can correctly predict a subject and a relation . previous work focused on evaluating how much PLM know, but this study focused on the internal state of knowledge inside them. |
| Approach: | They examine how often PLMs can correctly predict a subject and a relation . they also examine how knowledge inside PLM is embodied in the internal state . |
| Outcome: | The proposed model improves on the accuracy of the evidence paragraphs and manually written prompts. |
HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing code summarization approaches ignore the interplay of dependencies among program elements and code hierarchy. |
| Approach: | They propose a code summarization approach utilizing Heterogeneous Code Representations (HCRs) and HierarchyNet. |
| Outcome: | The proposed method improves on existing models and pre-trained models. |
Understanding the effects of language-specific class imbalance in multilingual fine-tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models have been developed to reduce the amount of resources needed to perform classification tasks. |
| Approach: | They modify traditional class weighing approach to reduce imbalance by calculating class weights separately for each language. |
| Outcome: | The proposed model improves performance and reduces the promotion of uninformative features. |
NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries (2024.findings-eacl)
Copied to clipboard
Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, Haidong Zhang
| Challenge: | Creating spreadsheet formulas remains a tedious and error-prone task for many end-users . a novel task is proposed to generate spreadsheet formulae from a user's NL query . |
| Approach: | They propose a task to generate formulas that are grounded on a spreadsheet table given a Natural Language query as input. |
| Outcome: | The proposed task generates formulas that are grounded on a spreadsheet table, given a natural language query as input. |