Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the application and evaluation of Large Language Models (LLMs) but LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains. |
| Approach: | They propose an abductive-deductive framework that integrates Large Language Models with an external backward-chaining solver to refine step-wise natural language explanations. |
| Outcome: | The proposed framework improves explanations generated via in-context learning methods and Chain-of-Thought (CoT) on ethical NLI tasks while producing formal proofs describing and supporting models’ reasoning. |
Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions (2024.eacl-long)
Copied to clipboard
| Challenge: | a fundamental characteristic of natural language definitions is that they are widely abundant, pos-1. |
| Approach: | They propose a multi-relational model that explicitly leverages definitions' semantic structure to derive word embeddings. |
| Outcome: | The proposed model can preserve the semantic mapping required for interpretable traversal while imposing constraints on definitions while maintaining the recursive semantic structure. |
Anisotropy Is Inherent to Self-Attention in Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | despite their success, Transformers models suffer from a representation degeneration problem . anisotropy is a property of hidden representations that makes them unexpectedly close to each other . |
| Approach: | They investigate the representation degeneration problem in a self-supervised learning model based on Transformers . anisotropy is a property of hidden representations which makes them unexpectedly close to each other . |
| Outcome: | The representation degeneration problem is a phenomenon widely observed among self-supervised learning methods based on Transformers. |
Generating Benchmarks for Factuality Evaluation of Language Models (2024.eacl-long)
Copied to clipboard
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham
| Challenge: | Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself and might under-represent domain specific or rare facts. |
| Approach: | They propose a method that transforms a factual corpus into a benchmark evaluating an LM's propensity to generate true facts from the corpus . |
| Outcome: | The proposed framework transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. |
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs (2024.eacl-long)
Copied to clipboard
| Challenge: | Lack of access to model details has raised concerns about data contamination among researchers. |
| Approach: | They conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. |
| Outcome: | The proposed models have been exposed to 4.7M samples from 263 benchmarks during the first year after their release. |
Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing text-to-SQL datasets that capture complex reasoning are limited by their execution accuracy. |
| Approach: | They present a bilingual text-to-SQL dataset specific to complex reasoning . their evaluation shows that Archer challenges the capabilities of current models . |
| Outcome: | The proposed dataset challenges state-of-the-art models with 6.73% execution accuracy . the dataset contains 1,042 English and 1,042, Chinese questions and 521 unique SQL queries . |
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Augmented Language Models (LLMs) over-rely on task-specific demonstrations that limits their generalizability and computational cost. |
| Approach: | They propose a query-tool grounding algorithm that is generalizable to various tasks . they delegate tool grounding and execution to small language models and LLMs . |
| Outcome: | The proposed algorithm outperforms baselines on 14 datasets and shows it can be generalized to different tasks. |
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled impressive zero-shot capabilities across various natural language tasks. |
| Approach: | They propose two ways to exploit the emergent abilities of large language models for NLG assessment. |
| Outcome: | The proposed methods improve performance and positional biases in comparisons between candidates. |
Parameter-Efficient Conversational Recommender System as a Language Processing Task (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to recommend items are categorized into attribute-based and generation-based methods. |
| Approach: | They propose to represent items in natural language and formulate a conversational recommender system that can be optimized in a single stage without relying on non-textual metadata. |
| Outcome: | The proposed model can be optimized in a single stage, without relying on non-textual metadata such as a knowledge graph. |
OpenPI2.0: An Improved Dataset for Entity Tracking in Texts (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work shows that explicit modeling entity states benefits LMs in procedural tasks. |
| Approach: | They propose a dataset where entities and attributes are fully canonicalized and additional entity salience annotations are added. |
| Outcome: | The proposed dataset outperforms existing models on question answering and classical planning tasks. |
A Comparative Multidimensional Analysis of Empathetic Systems (2024.eacl-long)
Copied to clipboard
| Challenge: | Empathetic dialogue systems have received significant attention, but no systematic review has verified these limitations. |
| Approach: | They analyze 21 empathetic dialogue systems using automated methods to examine their progress. |
| Outcome: | The results show that empathetic dialogue systems lack specificity, reflection levels, diversity . the results also offer guidance for developing future systems . |
Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent approaches to multi-hop question answering rely on in-context learning . however, these models contain billions of parameters making them inefficient at inference time. |
| Approach: | They propose a framework that allows for improving smaller language models with less than 10 human-annotated QA pairs by synthesizing millions of multi-hop questions and claims to fine tune language models. |
| Outcome: | The proposed framework improves model performance on multi-hop question answering and fact verification benchmarks while being almost one-third the size in parameter count. |
Language Models as Inductive Reasoners (2024.eacl-long)
Copied to clipboard
| Challenge: | Inductive reasoning is a core component of human intelligence. |
| Approach: | They propose a task to induce natural language rules from natural language facts using natural language as representation for knowledge instead of formal language. |
| Outcome: | The proposed task surpasses baselines in both automatic and human evaluations. |
SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024.eacl-long)
Copied to clipboard
David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, En-Shiun Lee
| Challenge: | despite progress in building multilingual language models evaluation is limited to a few languages with available datasets . despite this, we create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Approach: | They create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Outcome: | The proposed dataset addresses the lack of evaluation dataset for Natural Language Understanding (NLU) for many languages, it is the first publicly available evaluation dataset. |
FinBPM: A Framework for Portfolio Management-based Financial Investor Behavior Perception Model (2024.eacl-long)
Copied to clipboard
| Challenge: | a portfolio management framework based on reinforcement learning is needed to optimize stock price movements. |
| Approach: | They propose a framework that takes irrational investment into account when calculating portfolio weights . they use financial text to analyze intrinsic value information of companies and time series data . |
| Outcome: | The proposed framework gains 13.26% returns over state-of-the-art models while controlling for risk. |
Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions (2024.eacl-long)
Copied to clipboard
| Challenge: | Using model uncertainty as supervision for deciding when to ask may not be the most effective way to resolve model uncertainty. |
| Approach: | They propose to generate clarification questions based on model uncertainty estimation and compare it to several alternatives to generate questions . |
| Outcome: | The proposed approach improves the model uncertainty of a collaborative dialogue task and shows that it is more effective than other alternatives. |
Like a Good Nearest Neighbor: Practical Content Moderation and Text Classification (2024.eacl-long)
Copied to clipboard
| Challenge: | Few-shot text classification systems are infeasible to deploy and use reliably due to their dependence on prompting and billion-parameter language models. |
| Approach: | They propose a modification to SetFit that fine-tunes a Sentence Transformer under a contrastive learning paradigm and achieves similar results to more unwieldy systems. |
| Outcome: | The proposed model fine-tunes a Sentence Transformer under a contrastive learning paradigm and achieves similar results to more unwieldy systems. |
Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon (2024.eacl-long)
Copied to clipboard
| Challenge: | Prior work extended multilingual models to other languages due to the unavailability of labeled and unlabeled training data. |
| Approach: | They use multilingual lexicons to enhance multilingual models capabilities in low-resource languages . they focus on zero-shot sentiment analysis tasks across 34 languages based on a single sentence . |
| Outcome: | The proposed model improves zero-shot performance across 34 languages without using any sentence-level sentiment data. |
CEAN: Contrastive Event Aggregation Network with LLM-based Augmentation for Event Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | Event Extraction is a crucial yet arduous task in natural language processing (NLP), as its performance is hindered by laborious data annotation. |
| Approach: | They propose a Contrastive Event Aggregation Network with LLM-based Augmentation to promote low-resource learning and reduce data noise for event extraction. |
| Outcome: | The proposed approach achieves new state-of-the-art results on the ACE2005 and ERE-EN datasets. |
How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models? (2024.eacl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual translation models with massive coverage are becoming of the backbone of many translation systems. |
| Approach: | They propose to use a gradient-based inference-time controller to control a pretrained multilingual model by using a model with attribute annotations. |
| Outcome: | The proposed model performs well on pretrained multilingual models and is attribute- rather than language-specific. |
MultiMUC: Multilingual Template Filling on MUC-4 (2024.eacl-long)
Copied to clipboard
William Gantt, Shabnam Behzad, Hannah An, Yunmo Chen, Aaron White, Benjamin Van Durme, Mahsa Yarmohammadi
| Challenge: | We present multilingual parallel template filling datasets for MUCs . systems were required to extract one template per incident, containing details about perpetrators, victims, weapons used . |
| Approach: | They introduce MultiMUC, the first multilingual parallel corpus for template filling . they obtain automatic translations from a strong multilingual machine translation system . |
| Outcome: | The proposed dataset includes translations of the classic MUC-4 template filling benchmark into Arabic, Chinese, Farsi, Korean, and Russian. |
Align and Augment: Generative Data Augmentation for Compositional Generalization (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on semantic parsing has shown that seq2seq models find compositional generalization challenging. |
| Approach: | They propose a data-augmentation strategy that exploits alignment annotations between sentences and their corresponding meaning representations to improve compositional generalization. |
| Outcome: | The proposed model improves compositional generalization performance by exploiting alignment annotations between sentences and their corresponding meaning representations. |
UNSEE: Unsupervised Non-contrastive Sentence Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Unsupervised Non-Contrastive Sentence Embeddings demonstrates better performance compared to SimCSE in the Massive Text Embing (MTEB) benchmark. |
| Approach: | They introduce UNSEE, which stands for Unsupervised Non-Contrastive Sentence Embeddings, which demonstrates better performance compared to SimCSE in the Massive Text Embing benchmark. |
| Outcome: | The proposed solution achieves better performance than contrastive objectives on the Massive Text Embedding (MTEB) benchmark. |
EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning (2024.eacl-long)
Copied to clipboard
Kinjal Basu, Keerthiram Murugesan, Subhajit Chaudhury, Murray Campbell, Kartik Talamadupula, Tim Klinger
| Challenge: | Text-based games (TBGs) combine natural language understanding with reasoning. |
| Approach: | They propose an exploration-guided reasoning agent for textual reinforcement learning that integrates natural language with reasoning. |
| Outcome: | The proposed agent outperforms baseline agents on TWG and TWC games. |
From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for text segmentation are small in scale, synthesized, or only contain well-structured documents. |
| Approach: | They propose a benchmark YTSeg focusing on spoken content that is unstructured and unstructures . they also introduce an efficient hierarchical segmentation model MiniSeg that outperforms state-of-the-art benchmarks. |
| Outcome: | The proposed model outperforms state-of-the-art models on unstructured spoken content . the proposed model could be used for "smart chaptering" tasks . |
Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels (2024.eacl-long)
Copied to clipboard
| Challenge: | Obtaining high-quality labeled data that accurately represents complexity of real-world scenarios can be expensive, time-consuming, or even impractical. |
| Approach: | They propose to use Fréchet Inception Distance to measure distance between judged items and retrieved results. |
| Outcome: | The proposed method improves on a MS MARCO dataset and TREC Deep Learning Tracks query sets. |
Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have indicated that NLI models have an understanding of lexical and compositional semantics. |
| Approach: | They propose a framework to assess the extent of semantic sensitivity in NLI models . they use adversarially generated examples with minor semantics-preserving surface-form variations . |
| Outcome: | The proposed framework shows that NLI models struggle with minor variations requiring knowledge of compositional semantics . |
Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties (2024.eacl-long)
Copied to clipboard
| Challenge: | Mainstream cross-lingual task-oriented dialogue systems often overlook the transfer to lower-resource colloquial varieties due to limited test data. |
| Approach: | They propose to train a model for intent recognition and slot-filling in English and apply it to other languages. |
| Outcome: | The proposed model performs better than existing models on English and other languages. |
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents (2024.eacl-long)
Copied to clipboard
| Challenge: | Using chain-of-thought prompting, large language models perform better on complex reasoning tasks. |
| Approach: | They propose a prompting framework that decomposes a question into a sequence of actions and executes them over the document to obtain the answer. |
| Outcome: | The proposed framework outperforms zero-shot and chain-of-thought prompting on a QuALITY dataset . it proposes a plan based on actions mined from a training set and executes it step by step . |
LAraBench: Benchmarking Arabic AI with Large Language Models (2024.eacl-long)
Copied to clipboard
Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, Firoj Alam
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly influenced the landscape of language and speech research. |
| Approach: | They used GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM to tackle 33 distinct tasks across 61 datasets. |
| Outcome: | The proposed model outperforms SOTA models in zero-shot learning, with a few exceptions. |
SentenceLDA: Discriminative and Robust Document Representation with Sentence Level Topic Model (2024.eacl-long)
Copied to clipboard
| Challenge: | SentenceLDA is a sentence-level topic model that can be used to discriminate between different contexts. |
| Approach: | They propose a sentence-level topic model that extends the semantic unit from word to sentence and a corpus-level key opinion mining model that uses a lexical property to discriminate between different contexts. |
| Outcome: | The proposed model returns more discriminative document representation than other topic models while maintaining LDA’s elegant probabilistic interpretability. |
Towards Hierarchical Spoken Language Disfluency Modeling (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing solutions to speech dysfluency modeling are limited and expensive for low-income families. |
| Approach: | They propose a hierarchical unconstrained dysfluency modeling approach that addresses both dysfluencies transcription and detection to eliminate the need for extensive manual annotation. |
| Outcome: | The proposed approach eliminates the need for extensive manual annotation and improves the accuracy of the proposed model in phonetic transcription. |
Finding a Needle in the Adversarial Haystack: A Targeted Paraphrasing Approach For Uncovering Edge Cases with Minimal Distribution Distortion (2024.eacl-long)
Copied to clipboard
| Challenge: | Adversarial attacks against Language models (LMs) are a significant concern. |
| Approach: | They propose an approach to automatically learn a policy to generate challenging examples that improve the model’s performance. |
| Outcome: | The proposed approach outperforms baselines and exhibits generalizability across classifiers and datasets. |
FAIR: Filtering of Automatically Induced Rules (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce the human annotation efforts require a diverse set of rules to assign labels to unlabeled data. |
| Approach: | They propose an automatic rule-filtering algorithm to filter out a large set of automatically created rules from a small set of labeled features. |
| Outcome: | The proposed approach achieves statistically significant results over existing methods. |
NNOSE: Nearest Neighbor Occupational Skill Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | a new method for extracting occupational skills from text is needed to overcome the scarcity of skills across datasets. |
| Approach: | They propose a method that leverages multiple datasets to extract occupational skills from text . they propose NNOSE to extract neighboring skills from other datasets . |
| Outcome: | The proposed method improves infrequent skill extraction without additional fine-tuning. |
GAINER: Graph Machine Learning with Node-specific Radius for Classification of Short Texts and Documents (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in Graph Machine Learning (GML) have led to the development of numerous models tailored for processing text for various natural language applications. |
| Approach: | They propose a framework called Graph mAchine learnIng with Node-spEcific Radius that is aimed at graph-based NLP. |
| Outcome: | The proposed framework is non-neural and novel for graph-based NLP. |
MAFIA: Multi-Adapter Fused Inclusive Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) are widely used in NLP for various tasks. |
| Approach: | They propose to modularly debias a pre-trained language model across multiple bias dimensions using structured knowledge and a large generative model. |
| Outcome: | The proposed model is able to debias a pre-trained language model across multiple bias dimensions in a semi-automated way. |
Code-Switched Language Identification is Harder Than You Think (2024.eacl-long)
Copied to clipboard
| Challenge: | Code switching (CS) is a common phenomenon in written and spoken communication, but is handled poorly by many NLP applications. |
| Approach: | They propose to use CS language identification for corpus building to make it more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. |
| Outcome: | The proposed system is based on a sentence-level multi-label tagging problem and provides recommendations for future work. |
Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM (2024.eacl-long)
Copied to clipboard
| Challenge: | a novel method to train a smaller model with LLMs for zero-shot text classification requires immense computational resources due to their substantial model size. |
| Approach: | They propose a method which leverages the generative power of large language models to train a smaller model. |
| Outcome: | The proposed method outperforms state-of-the-art methods when limited data is available. |
Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Neural networks are notoriously hard to interpret and slightly mysterious to researchers and practitioners alike. |
| Approach: | They formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based-sensitivity. |
| Outcome: | The transformer is more sensitive to hyperparameters according to both metrics, but not batch size . large models, multilinguality of NLP models and tasks make hyperparametric tuning more expensive . |
Examining Gender and Racial Bias in Large Vision–Language Models Using a Novel Dataset of Parallel Images (2024.eacl-long)
Copied to clipboard
| Challenge: | a new wave of large vision–language models (LVLMs) incorporate images as input in addition to text . a recent study examined potential gender and racial biases in such systems based on the perceived characteristics of the people in the input images. |
| Approach: | They examine potential gender and racial biases in large vision–language models . they query a dataset of AI-generated images of people to see whether they differ . |
| Outcome: | The proposed dataset shows that the images differ in gender and race according to the perceived characteristics of the person depicted. |
ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases (2024.eacl-long)
Copied to clipboard
| Challenge: | Reasoning over Commonsense Knowledge Bases (CSKBs) is a way to acquire new commonsense knowledge based on reference knowledge in original CSKB and external prior knowledge. |
| Approach: | They propose a plugin to provide and check explicit relational constraints over prompting techniques. |
| Outcome: | The proposed method improves on existing prompting techniques and CSKB reasoning. |
A* shortest string decoding for non-idempotent semirings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing single shortest path algorithm is undefined for weighted finite-state automata over non-idempotent semirings because such semiring does not guarantee existence of a shortest pathway. |
| Approach: | They propose an algorithm which finds the shortest string for a weighted non-deterministic automaton over non-idempotent semirings using the backwards shortest distance of an equivalent deterministic algorithm. |
| Outcome: | The proposed algorithm only visits a small fraction of the states in the companion semiring, but only visits one state if determinization is performed "on the fly". |
Importance-Aware Data Augmentation for Document-Level Neural Machine Translation (2024.eacl-long)
Copied to clipboard
| Challenge: | Document-level neural machine translation (DocNMT) models can be difficult and expensive to train due to data sparsity. |
| Approach: | They propose an Importance-Aware Data Augmentation algorithm that augments training data based on token importance information estimated by the norm of hidden states and training gradients. |
| Outcome: | The proposed algorithm outperforms strong DocNMT baselines and several data augmentation approaches on three widely-used benchmarks. |
Lost in Translationese? Reducing Translation Effect Using Abstract Meaning Representation (2024.eacl-long)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) can be used to reduce translationese in text . if translationeses are not addressed in training or test sets, evaluation scores can be overinflated . |
| Approach: | They propose to use Abstract Meaning Representation (AMR) to reduce translationese in translated texts. |
| Outcome: | The proposed approach outperforms other methods based on machine translation and paraphrase generation. |
Comparing Template-based and Template-free Language Model Probing (2024.eacl-long)
Copied to clipboard
| Challenge: | Template-based and template-based approaches rank models differently except for the top domain-specific models. |
| Approach: | They evaluate 16 different cloze-task language model probing approaches on 10 probing English datasets to answer questions about model rankings and absolute scores. |
| Outcome: | The results show that the template-based and template-free approaches rank models differently except for the top domain-specific models. |
Desiderata For The Context Use Of Question Answering Systems (2024.eacl-long)
Copied to clipboard
| Challenge: | Prior work has uncovered a set of common problems in state-of-the-art context-based question answering systems, such as a lack of attention to the context when it conflicts with a model’s parametric knowledge and a loss of consistency with their answers. |
| Approach: | They propose to examine the desiderata for context-based question answering systems and then compare them to a set of prior work. |
| Outcome: | The proposed models are based on 15 datasets and evaluated on 5 datasets. |
Scaling up Discovery of Latent Concepts in Deep NLP Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models focus on clustering latent spaces within pre-training models . authors propose metrics for assessing the quality of discovered latent concepts . |
| Approach: | They propose metrics to assess the quality of discovered latent concepts . they propose to scale latent concept discovery to larger datasets and models . |
| Outcome: | The proposed clustering algorithms improve performance while maintaining quality of the obtained concepts. |
AnthroScore: A Computational Linguistic Measure of Anthropomorphism (2024.eacl-long)
Copied to clipboard
| Challenge: | Anthropomorphism is commonplace in people's interactions with technology . anthropomorphizing language can suggest undue accountability and agency in technologies . |
| Approach: | They propose an automatic metric of implicit anthropomorphism in language . they use a masked language model to quantify how non-human entities are implicitly framed as human by the surrounding context. |
| Outcome: | The proposed metric measures how non-human entities are implicitly framed as human by the surrounding context. |
Centering the Speech Community (2024.eacl-long)
Copied to clipboard
| Challenge: | In remote speech communities, people interact with the outside world using a variety of an institutional language. |
| Approach: | They propose to use local languages to support their collaboration in a remote community in the far north of australia to explore the functional differences between oral and institutional languages. |
| Outcome: | The proposed language technologies are better aligned with local interests and aspirations than the first author's western framing of language as data for exploitation by machines. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing web-mined corpora for low-resource languages have serious quality issues, especially for lowresource language pairs. |
| Approach: | They ranked each corpus according to a similarity measure and evaluated different portions of this ranked corpus. |
| Outcome: | The results show that the quality of web-mined corpora for low-resource languages is significantly different from human-curated corporats. |
VOLTAGE: A Versatile Contrastive Learning based OCR Methodology for ultra low-resource scripts through Auto Glyph Feature Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | UNESCO has classified 2500 out of 7000 languages spoken worldwide as endangered . low resource languages are at a greater risk of extinction due to lack of unsupervised OCR methodologies for low resource . |
| Approach: | They propose a contrastive learning based OCR methodology that leverages auto-glyph feature recommendation for cluster-based labelling. |
| Outcome: | The proposed method achieves 95% accuracy for machine printed and 87% for handwritten samples on Takri script. |
Unsupervised Contrast-Consistent Ranking with Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Language models contain ranking-based knowledge and are powerful solvers of in-context ranking tasks. |
| Approach: | They propose to use a model to elicit language models' ranking knowledge without supervision by using a pairwise, pointwise and listwise prompting method. |
| Outcome: | The proposed method is inspired by an unsupervised probing method called Contrast-Consistent Search (CCS). |
Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent content when parametric knowledge conflicts with knowledge in the input document. |
| Approach: | They propose a method to enhance factual adaptiveness while achieving factual consistency on original datasets. |
| Outcome: | The proposed method improves factual adaptiveness while achieving factual consistency on original datasets. |
Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent laws like “right to explanations” have spurred research in developing interpretable models . a recent study has shown that multimodal explanations improve performance in generating textual justifications . |
| Approach: | They propose to use visual and textual modalities to explain why a given meme is cyberbullying . they use a Contrastive Language-Image Pretraining approach to generate textual justifications . |
| Outcome: | The proposed model improves performance in visual and textual explanations and identifies the visual evidence supporting a decision. |
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models with instruction tuning are resource-intensive . a recent study suggests that the performance of LLMs scales proportionally with the size of the model. |
| Approach: | They propose to distill knowledge from instruction-tuned LLMs into much smaller ones . they develop a large set of 2.58M instructions based on existing and newly-generated instructions . |
| Outcome: | The proposed models are comparable to strong baselines while being much smaller in size. |
Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformer (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for cognate identification are based on distributions of phonemes and make little use of cognacy labels. |
| Approach: | They propose a transformer-based architecture inspired by computational biology for automated cognate detection. |
| Outcome: | The proposed architecture performs better than existing methods with increased supervision. |
Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent trends in learning monolingual and multilingual sentence embeddings are based on contrastive learning (CL) among an anchor, one positive and multiple negative instances. |
| Approach: | They propose to leverage multiple positives to improve learning of multilingual sentence embeddings by using an anchor, one positive, and multiple negative instances. |
| Outcome: | The proposed approach improves retrieval, semantic similarity, and classification performance on unseen languages. |
Moderation in the Wild: Investigating User-Driven Moderation in Online Discussions (2024.eacl-long)
Copied to clipboard
| Challenge: | Effective content moderation is imperative for fostering healthy and productive discussions in online domains. |
| Approach: | They propose to document and release a dataset of comments in which users act as moderators. |
| Outcome: | The proposed dataset contains 1000 comment-reply pairs with crowdsourced annotations from a large annotator pool and fine-grained annotation schema targeting the functions of moderation, stylistic properties(aggressiveness, subjectivity, sentiment), constructiveness, and individual perspectives of the annotators on the task. |
Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching (2024.eacl-long)
Copied to clipboard
| Challenge: | Multilingual models exhibit impressive cross-lingual transfer capabilities on unseen languages, but performance is impacted when there is a script disparity with the languages used in the model’s pre-training data. |
| Approach: | They propose a novel method to align a resource-rich language's script with a target language and train a classifier that can make informed decisions regarding the appropriate processing of each token. |
| Outcome: | The proposed model can be used to transfer a language's scripts across multiple languages, but it is suboptimal for mixed languages, where only a subset benefits while the rest is impeded. |
Where Do We Go From Here? Multi-scale Allocentric Relational Inferencefrom Natural Spatial Descriptions (2024.eacl-long)
Copied to clipboard
| Challenge: | Current NLP navigation studies focus on egocentric local descriptions that require reasoning over the agent’s local perception. |
| Approach: | They propose to use a dataset to analyse English geospatial instructions to find locations and paths from natural language descriptions. |
| Outcome: | The proposed task and dataset includes 10,404 examples of English geospatial instructions for reaching a target location using map-knowledge. |
Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on extractive summarisation but limited attention has been paid to abstractive summaries. |
| Approach: | They propose to trace bias in abstractive summarisation models to social media opinions using different models and adaptation methods. |
| Outcome: | The proposed model is compared with other models and adaptation methods to summarise social media opinions using different models and adaption methods. |
Document Structure in Long Document Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing long-document Transformers do not learn representations of document structure during pretraining. |
| Approach: | They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference. |
| Outcome: | The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance. |
The Role of Data Curation in Image Captioning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing image captioning models treat all samples equally, neglecting mismatched data . Several other techniques have relied on curriculum learning strategies to adapt learning to the difficulty of the task. |
| Approach: | They propose to actively curate difficult samples in datasets using curriculum learning strategies to improve captioning models. |
| Outcome: | The proposed methods outperform existing models on the Flickr30K and COCO datasets. |
Large-Scale Bitext Corpora Provide New Evidence for Cognitive Representations of Spatial Terms (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent evidence suggests that there exist two classes of cognitive representations within the spatial terms of a language. |
| Approach: | They propose a pipeline for extracting, isolating, and aligning spatial terms from parallel text . they find evidence that variability in functional terms differs significantly from that of geometric terms . |
| Outcome: | The proposed pipeline extracts, isolates, and aligns spatial terms in basic locative constructions from parallel text. |
REFINER: Reasoning Feedback on Intermediate Representations (2024.eacl-long)
Copied to clipboard
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, Boi Faltings
| Challenge: | Language models (LLMs) have shown remarkable performance by explicitly generating intermediate inferences,e.g., chain-of-thought prompting. |
| Approach: | They propose a framework for finetuning LMs to generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. |
| Outcome: | Empirical evaluations of REFINER on three diverse reasoning tasks show that it significantly improves over baseline models. |
HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations (2024.eacl-long)
Copied to clipboard
| Challenge: | Using demographic factors, pre-trained language models can adapt to demographic changes. |
| Approach: | They propose a framework to measure demographic alignment of language models with a target demographic for the first time. |
| Outcome: | The proposed framework outperforms human-machine language models in age-related tasks and outperformed a typical 21-year-old at memorization. |
LOCOST: State-Space Models for Long Document Abstractive Summarization (2024.eacl-long)
Copied to clipboard
Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari
| Challenge: | State-space models are a low-complexity alternative to transformers for text generation . however, the quadratic complexity of the input length restricts the application of large pretrained models to long texts. |
| Approach: | They propose an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. |
| Outcome: | The proposed model saves memory and memory during training and inference time while saving 50% and 87% of memory. |
A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation (2024.eacl-long)
Copied to clipboard
| Challenge: | Extensive research has been devoted to adversarial attacks against NMT models . perturbations of inputs can mislead the target model, resulting in incorrect outputs . |
| Approach: | They propose an adversarial attack framework that alters the class of output translations of an NMT model and a classifier to craft adversarials whose translations belong to a different class . |
| Outcome: | The proposed approach has a more substantial effect on the translation by altering the overall meaning, which leads to a different class determined by an oracle classifier. |
Improving Generalization in Semantic Parsing by Increasing Natural Language Variation (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches generate question reformulations via models trained on Spider or only introduce local changes. Existing methods generate question form reformulation but lack robustness. |
| Approach: | They use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations by generating more realistic and diverse questions. |
| Outcome: | The proposed model improves on the new spider dataset by using a few prompts. |
Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
No Error Left Behind: Multilingual Grammatical Error Correction with Pre-trained Translation Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Grammatical Error Correction (GEC) research has primarily focused on English with little coverage for other languages. |
| Approach: | They propose a multilingual machine translation model that can be fine-tuned to improve error correction out-of-the-box. |
| Outcome: | The proposed model outperforms similar-sized MT5 models and competes favourably with larger models. |
Quantifying Stereotypes in Language (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies define a sentence as stereotypical and anti-stereotypical, but they lack a fine-grained quantification of stereotypes. |
| Approach: | They quantify stereotypes in language by annotating a dataset to quantify stereotype of sentences. |
| Outcome: | The proposed models validate the findings of the current studies. |
Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Model (2024.eacl-long)
Copied to clipboard
| Challenge: | Motivational Interviewing (MI) is a counselling technique used to guide people towards behaviour change. |
| Approach: | They propose a method for distilling reflections from a foundational language model into smaller models that can be owned and controlled. |
| Outcome: | The proposed method achieves 100% success rate on hold-out test set and 90% on the GPT-2 XL. |
Multi-Reference Benchmarks for Russian Grammatical Error Correction (2024.eacl-long)
Copied to clipboard
| Challenge: | Using the union of the references increases system scores by more than 10 points, but not across error types. |
| Approach: | They propose multi-reference benchmarks for the Grammatical Error Correction of Russian . they use two existing single-refer datasets for a total of 7,444 learner sentences . |
| Outcome: | The proposed benchmarks show that new raters tend to make more changes, especially at the lexical level, compared to the original rater. |
Plan-Grounded Large Language Models for Dual Goal Conversational Settings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs can follow user instructions, but it is unclear how they can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation. |
| Approach: | They propose a dual-purpose mixed-initiative conversational setting where the LLM grounds the conversation on an arbitrary plan and seeks to satisfy both a procedural plan and user instructions. |
| Outcome: | The proposed model achieves 2.1x improvement over a strong baseline and good generalization to unseen domains. |
“Define Your Terms” : Enhancing Efficient Offensive Speech Classification with Definition (2024.eacl-long)
Copied to clipboard
| Challenge: | Multiple studies have proposed various semantically related yet subtle distinct categories of offensive speech. |
| Approach: | They propose a meta-learning architecture that incorporates the input’s label and definition for classification via Prototypical Network. |
| Outcome: | The proposed model achieves 75% of the maximal F1-score while using less than 10% of the available training data across 4 datasets. |
VlogQA: Task, Dataset, and Baseline Models for Vietnamese Spoken-Based Machine Reading Comprehension (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing datasets for machine reading comprehension tasks in Vietnamese focus on written documents, such as Wikipedia articles, online newspapers, or textbooks. |
| Approach: | They propose to capture Vietnamese spoken language in natural settings and use it to create a machine-learning corpus for machine reading comprehension tasks. |
| Outcome: | The proposed corpus consists of 10,076 question-answer pairs based on 1,230 transcript documents sourced from YouTube . |
CEV-LM: Controlled Edit Vector Language Model for Shaping Natural Language Generations (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing control approaches primarily adjust the semantic (e.g., emotion, topics), structural (e-speech, parts-of-seech), and lexical (el-s-sp-s) properties of text, but are insufficient to accomplish complex objectives such as pacing which control the complexity and readability of the text. |
| Approach: | They propose a lightweight semi-autoregressive language model that uses edit vectors to control three complementary metrics that quantify the shape of text. |
| Outcome: | The proposed model provides significantly more targeted and precise control of speed, volume, and circuitousness while using less training data, and containing fewer parameters. |
It’s All Relative: Learning Interpretable Models for Scoring Subjective Bias in Documents from Pairwise Comparisons (2024.eacl-long)
Copied to clipboard
| Challenge: | a new model to score subjective bias in documents is developed to perform pairwise comparisons . a recent study shows that the model can be explained and validated for other domains based on the training data. |
| Approach: | They propose an interpretable model to score subjective bias in Wikipedia articles . they train the model on pairs of revisions of the same Wikipedia article . |
| Outcome: | The proposed model can interpret parameters to discover words most indicative of bias . it compares legal texts, news media and law amendments in three settings . |
HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification (2024.eacl-long)
Copied to clipboard
| Challenge: | Hierarchical text classification is a complex subtask under multi-label text classification . the relevance of document sections can vary based on the hierarchy level, necessitating a dynamic document representation. |
| Approach: | They propose a text-generation-based framework that uses language models to encode dynamic text representations. |
| Outcome: | The proposed framework surpasses existing methods while handling data and mitigating class imbalance. |
M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection (2024.eacl-long)
Copied to clipboard
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate fluent responses to user queries, but they are also susceptible to misuse in journalism, education, and academia. |
| Approach: | They propose a large-scale benchmark for machine-generated text detection that is a multi-generator, multi-domain, and multi-lingual corpus. |
| Outcome: | The proposed system can detect machine-generated text and pinpoint misuse . the proposed system is based on a large-scale benchmark dataset . |
A Truly Joint Neural Architecture for Segmentation and Parsing (2024.eacl-long)
Copied to clipboard
| Challenge: | Contemporary multilingual dependency parsers can parse a diverse set of languages, but performance is lower for Morphologically Rich Languages. |
| Approach: | They propose a joint neural architecture where a lattice-based representation is provided to an arc-factored model and solves the morphological segmentation and syntactic parsing tasks at once. |
| Outcome: | The proposed architecture is language-agnostic and language-based to improve on Hebrew . it shows that the proposed model can parse morphological segmentation and syntactic parsing tasks at once. |
ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text (2024.eacl-long)
Copied to clipboard
| Challenge: | Lexical normalization is a fundamental task in Natural Language Processing (NLP) it involves the transformation of words into their canonical forms. |
| Approach: | They present a corpus of Vietnamese words annotated by human annotators for the Vietnamese lexical normalization task. |
| Outcome: | The best-performing system achieved 57.74% using the Error Reduction Rate (ERR) metric with the Leave-As-Is (LAI) baseline. |
Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) text-to-text generation methods are unable to generate coherent and fluent texts due to discrete nature of text. |
| Approach: | They propose to integrate discrete diffusion models (DDM) into NAR text-to-text generation and integrate BART to improve the performance. |
| Outcome: | The proposed method outperforms competing methods and surpasses autoregressive methods on 7 datasets. |
Unleashing the Power of Discourse-Enhanced Transformers for Propaganda Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing systems focused on the surface words, ignoring the linguistic structure of the texts. |
| Approach: | They propose to use discourse analysis to analyze paragraph-level and token-level classifications and propose a Transformer architecture that can be used to detect propaganda. |
| Outcome: | The proposed system improves on English and Russian texts and shows strong correlations between propaganda instances and discourse spans. |
Predicting Client Emotions and Therapist Interventions in Psychotherapy Dialogues (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the role of client emotions and therapist interventions in predicting treatment outcomes from psychotherapy dialogues. |
| Approach: | They propose to model the therapist-intervention-prediction-based dialogue acts at the utterance-level using a pan-theoretical schema and fine-tuned language models. |
| Outcome: | The proposed model predicts the coherence between client self-reports on emotion and utterance-level emotions. |
Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies (2024.eacl-long)
Copied to clipboard
| Challenge: | Autoregressive decoding is expensive for many sequence-to-sequence tasks, but for some downstream tasks, the actual decoding output is not needed, just attributes of the sequence. |
| Approach: | They propose non-autoregressive proxy models that can efficiently predict scalar-valued sequence-level attributes from the encodings, avoiding the expensive decoding stage. |
| Outcome: | The proposed models outperform ensembles in machine translation (MT) and automatic speech recognition (ASR) while being significantly faster. |
3D Rotation and Translation for Hyperbolic Knowledge Graph Embedding (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing knowledge graph embeddings do not capture relation patterns, but they capture symmetry, antisymmetry, inversion, commutative composition, non-commutable composition, hierarchy, and multiplicity. |
| Approach: | They propose a 3D Rotation and Translation in Hyperbolic space model that captures relation patterns simultaneously. |
| Outcome: | The proposed model outperforms state-of-the-art models in terms of accuracy, hierarchy property, and other relation patterns in low-dimensional space, while performing similarly in high-dimensional spaces. |
Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Ranking (2024.eacl-long)
Copied to clipboard
| Challenge: | Chinese geographic re-ranking task aims to find the most relevant addresses among retrieved candidates. |
| Approach: | They propose a framework to integrate Chinese geographic semantics into re-ranking pipelines. |
| Outcome: | The proposed framework improves on two Chinese benchmark datasets. |
Style-News: Incorporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Using generative models, the issues of producing hallucinatory contents have been raised in various domains, e.g., law, writing. |
| Approach: | They propose a style-aware neural news generator that mimics the style of real news to deceive people by identifying which publisher the style corresponds to and training a model to detect fake news. |
| Outcome: | The proposed framework outperforms state-of-the-art models in terms of fluency, content preservation, and style adherence. |
Graph-based Clustering for Detecting Semantic Change Across Time and Languages (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to detect semantic change using contextualized embeddings are underperforming . a graph-based clustering approach captures nuanced changes in word senses across time and languages . |
| Approach: | They propose a graph-based clustering approach to capture nuanced changes in word senses across time and languages. |
| Outcome: | The proposed approach surpasses previous approaches in the SemEval2020 binary classification task across four languages. |
Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Pretrained language models learn cross-lingual knowledge and perform well on diverse tasks when finetuned. |
| Approach: | They propose a zero-shot prompting approach that captures cross-lingual word sense with a contextual prompt. |
| Outcome: | The proposed approach outperforms baselines on recall in many evaluation languages without additional training or finetuning. |
Anchor Points: Benchmarking Models with Much Fewer Examples (2024.eacl-long)
Copied to clipboard
| Challenge: | Modern language models exhibit powerful but brittle behavior, leading to larger and more diverse benchmarks. |
| Approach: | They propose to use anchor points to select small subsets of a language model-prompt dataset to capture model behavior across the entire dataset. |
| Outcome: | The proposed technique outperforms standard benchmarks in language models with 1-30 anchor points . the proposed technique can be used to compare models on different regions of the dataset . |
SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling (2024.eacl-long)
Copied to clipboard
| Challenge: | Visual storytelling aims to automatically generate a coherent story based on a given image sequence. |
| Approach: | They propose a framework that represents the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge. |
| Outcome: | The proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations. |
Discovering and Articulating Frames of Communication from Social Media Using Chain-of-Thought Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | a method for Discovering and Articulating FoCs is proposed . 86.2% of the FoC encoded by communication experts were also uncovered . |
| Approach: | They propose a method for Discovering and Articulating FoCs that uses Chain-of-Thought prompting and In-Context Active Curriculum Learning to uncover FoC. |
| Outcome: | The proposed method uncovered 86.72% of the FoCs encoded by communication experts on the same reference dataset. |
VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to “vet” labels from noisy captions for weakly-supervised object detection are limited for object detection. |
| Approach: | They propose a technique to “vet” labels extracted from noisy captions and use them for weakly-supervised object detection without any bounding boxes. |
| Outcome: | The proposed method improves WSOD without label vetting by 30% on five datasets. |
WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at answering WSC questions, but their ability to generate such questions remains less explored. |
| Approach: | They propose a tree-of-experts prompting method which enhances the generation of WSC instances by incorporating new 'ambiguous' and 'offensive' categories. |
| Outcome: | The proposed method enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods) it extends the framework by incorporating new ‘ambiguous’ and ‘offensive’ categories, providing a deeper insight into model overconfidence and bias. |
Kardeş-NLU: Transfer to Low-Resource Languages with Big Brother’s Help – A Benchmark and Evaluation for Turkic Languages (2024.eacl-long)
Copied to clipboard
| Challenge: | Cross-lingual transfer (XLT) driven by massively multilingual language models (mmLMs) has been shown to be ineffective for low-resource (LR) target languages with little (or no) representation in mmLM’s pretraining . |
| Approach: | They propose a benchmark to evaluate cross-lingual transfer (XLT) to LR languages that do have a close HR relative and a framework to integrate Turkish into XLT. |
| Outcome: | The proposed configuration is of practical relevance for more of the world’s languages: XLT to LR languages that do have a close HR relative. |
Chaining Event Spans for Temporal Relation Grounding (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to understanding temporal relations between events have relied on answer overlaps as a proxy label to distinguish similar and dissimilar questions. |
| Approach: | They propose a timeline reasoning network that elicits proper reasoning behaviors through a module for predicting time spans of events. |
| Outcome: | The proposed approach outperforms existing methods by resolving spurious overlaps using the predicted timeline. |
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses. |
| Approach: | They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses. |
| Outcome: | The proposed model performs better on diverse summarisation tasks than existing models. |
AnaDE1.0: A Novel Data Set for Benchmarking Analogy Detection and Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | Textual analogies that make comparisons between two concepts are often used for explaining complex ideas, creative writing, and scientific discovery. |
| Approach: | They propose a task that includes three synergistic tasks: detecting documents containing analogies, extracting text segments that make up the analogy, and identifying the (source and target) concepts being compared. |
| Outcome: | The proposed task performs well on all sub-tasks and smaller models perform better than non-finetuned ChatGPT, suggesting high task difficulty. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
Learning to Retrieve In-Context Examples for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve in-context learning performance are highly sensitive to the quality of the incontext examples provided. |
| Approach: | They propose a framework to iteratively train dense retrievers that can identify high-quality in-context examples for large language models. |
| Outcome: | The proposed model improves performance by retrieving examples with similar patterns, and the gains are consistent across LLMs of varying sizes. |
EnCore: Fine-Grained Entity Typing by Pre-Training Entity Encoders on Coreference Chains (2024.eacl-long)
Copied to clipboard
| Challenge: | Entity typing is the task of assigning semantic types to entities mentioned in text. |
| Approach: | They propose to pre-train an entity encoder such that embeddings of coreferring entities are more similar to each other. |
| Outcome: | The proposed method improves state-of-the-art on fine-grained entity typing and entity extraction. |
Unsupervised stance detection for social media discussions: A generic baseline (2024.eacl-long)
Copied to clipboard
| Challenge: | stance detection methods are designed for specific network types, either homophilic or heterophilic, and fail to generalize to both. |
| Approach: | They propose to generalize a graph neural network based on text embeddings to homophilic and homophilic networks. |
| Outcome: | The proposed model outperforms state-of-the-art methods across heterophilic and homophilic networks. |
Putting Context in Context: the Impact of Discussion Structure on Text Classification (2024.eacl-long)
Copied to clipboard
| Challenge: | Current text classification approaches focus on the content to be classified, but contextual information is neglected in many cases. |
| Approach: | They propose to integrate contextual information into a transformer-based model by feeding it as natural language input into . they also experiment with different amounts of training data and analyse local discussion networks in a privacy-compliant way. |
| Outcome: | The proposed model can be generalized to other datasets and is privacy-compliant. |
Aligning Large and Small Language Models via Chain-of-Thought Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting empowers Large Language Models to solve complex reasoning tasks in a step-wise manner. |
| Approach: | They propose a method for aligning and transferring reasoning abilities between larger and smaller Language Models by using CoT-Demonstrations. |
| Outcome: | The proposed method outperforms baselines on question-answering and mathematical reasoning benchmarks. |
Disentangling the Roles of Target-side Transfer and Regularization in Multilingual Machine Translation (2024.eacl-long)
Copied to clipboard
| Challenge: | Multilingual Machine Translation (MMT) benefits from knowledge transfer across different language pairs, but performance differences between one-to-many and many-to-1 translation are negligible. |
| Approach: | They conduct a large-scale study that varies the auxiliary target-side languages along two dimensions to show the dynamic impact of knowledge transfer on the main language pairs. |
| Outcome: | The proposed model can translate between multiple languages with minimal positive transfer ability. |
Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have unlocked unprecedented applications of AI. |
| Approach: | They propose to use a social benchmark to evaluate the bias protection provided by Large Language Models (LLMs) with a variety of tasks with varying complexities to assess their effectiveness. |
| Outcome: | The proposed benchmark shows that both ChatGPT and GPT-4 have strong biases with respect to nationality, gender, race, and religion. |
Rainbow - A Benchmark for Systematic Testing of How Sensitive Visio-Linguistic Models are to Color Naming (2024.eacl-long)
Copied to clipboard
| Challenge: | Visio-linguistic models have been gaining popularity for tasks that require a deeper understanding of multimodalities. |
| Approach: | They compile a probing dataset to test multi-modal alignment around color . they show that models have trouble with prepositions and verbs . |
| Outcome: | The proposed model is superior to models that do not rely on pre-extracted image features and is able to perform well with noisy pre-training data. |
CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable generalization capabilities, performing well on various tasks such as question answering (QA), complex reasoning, and code generation. |
| Approach: | They propose to augment training data of smaller language models with automatically generated counterfactuals (CF) instances to improve out-of-domain (OOD) performance of SLMs in extractive question answering setup. |
| Outcome: | The proposed approach improves out-of-domain (OOD) performance of small language models in extractive question answering setup. |
UP5: Unbiased Foundation Model for Fairness-aware Recommendation (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are gaining a foothold in Recommender Systems (RS) but there is growing concern that LLMs perpetuate stereotypes and may result in unfair recommendations. |
| Approach: | They propose a counterfactually-fair-prompt method for LLM-based recommendation that is based on unbiased foundation mOdels. |
| Outcome: | The proposed method achieves better recommendation performance with a high level of fairness on two real-world datasets. |
Human Temporal Inferences Go Beyond Aspectual Class (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on aspectual classification in English has been motivated as a pre-requisite for Natural Language Understanding (NLU) in cases where temporal reasoning is required. |
| Approach: | They propose to classify English verb phrases into situation aspect categories by gathering crowd-sourced judgements from non-expert, native English participants. |
| Outcome: | The proposed approach uses a crowd-sourced dataset from non-expert, native English participants to examine aspectual entailments in English. |
It is not True that Transformers are Inductive Learners: Probing NLI Models with External Negation (2024.eacl-long)
Copied to clipboard
| Challenge: | et al., 2017) show that NLI models learn to treat external negation as a distractor . e-learning models fail to inductively learn the role of negation for NLI tasks . |
| Approach: | They propose that models fine-tuned on NLI datasets learn to treat external negation as a distractor, effectively ignoring its presence in hypothesis sentences. |
| Outcome: | The proposed models learn to treat external negation as a distractor, the authors show . they also fail to inductively learn the law of the excluded middle for a single prefix . |
Polarized Opinion Detection Improves the Detection of Toxic Language (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating polarized annotations are un-normalized and difficult to exploit in machine learning. |
| Approach: | They propose a method for K-class text classification that exploits polarized texts in the dataset. |
| Outcome: | The proposed method exploits polarized texts in a dataset and can improve classification performance. |
Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations (2024.eacl-long)
Copied to clipboard
| Challenge: | Acoustic word embeddings are vector representations of spoken words . self-supervised learning (SSL)-based speech models are popular for speech recognition . |
| Approach: | They explore the effectiveness of the Correspondence Auto-Encoder to obtain improved AWEs . they use self-supervised learning on large amounts of unlabelled speech data . |
| Outcome: | The Correspondence Auto-Encoder (CAE) model outperforms MFCC models on language discrimination . the model achieves best results in Polish, Portuguese, Spanish, French, and English . |
Investigating Agency of LLMs in Human-AI Collaboration Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | We examine how LLMs can be measured and managed for Agency . a model that manifests high Intentionality, Motivation, Self-Efficacy, and Self-Regulation is more likely to be perceived as strongly agentive. |
| Approach: | They collect a dataset of 83 human-human collaborative interior design conversations containing 908 conversational snippets annotated for Agency features. |
| Outcome: | The proposed models show that they manifest high Intentionality, Motivation, Self-Efficacy, and Self-Regulation, and are more likely to be perceived as agentive. |
SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking (2024.eacl-long)
Copied to clipboard
| Challenge: | In-context learning with Large Language Models (LLMs) is a promising avenue of research in Dialog State Tracking (DST). |
| Approach: | They propose a data generation framework tailored for Dialog State Tracking that uses large language models to synthesize natural, coherent, and free-flowing dialogues with DST annotations. |
| Outcome: | The proposed framework improves joint goal accuracy by 4-5% over the zero-shot baseline on MultiWOZ 2.1 and 2.4. |
Argument Mining as a Text-to-Text Generation Task (2024.eacl-long)
Copied to clipboard
| Challenge: | Argument Mining (AM) aims to uncover the argumentative structures within a text. |
| Approach: | They propose a method that generates argumentatively annotated text using a pretrained encoder-decoder language model and a pre-trained decoder. |
| Outcome: | The proposed method achieves state-of-the-art performance on three types of benchmark datasets. |
Answering legal questions from laymen in German civil law system (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on questions asked by experts, such as lawyers or legal scholars. |
| Approach: | They use a dataset to analyze laymen's legal questions paired with answers from lawyers and grounded to concrete law book paragraphs to find out what limitations exist. |
| Outcome: | The proposed system could help laymen in real situations without understanding law . the proposed system is based on 21k laymen’s legal questions paired with answers from lawyers and grounded to concrete law book paragraphs. |
An Empirical Analysis of Diversity in Argument Summarization (2024.eacl-long)
Copied to clipboard
| Challenge: | Current methods for summarizing arguments miss an important aspect of diversity . authors examine three aspects of diversity in argument summarization . |
| Approach: | They propose three aspects of diversity that are important for accommodating multiple perspectives. |
| Outcome: | The proposed models lack the diversity of opinions, sources, and annotators. |
What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies show that identifying verifiable claims is difficult, whereas identifying unverifiably claims is more challenging. |
| Approach: | They hypothesize that breaking down claims into smaller units increases our understanding which properties impact verifiability. |
| Outcome: | The proposed corpus of evidence is based on the first corpus for scientific fact verification annotated with subject–relation–object triplets, evidence documents, and fact-checking verdicts. |
Approximate Attributions for Off-the-Shelf Siamese Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing attribution methods cannot tackle Siamese encoders since they compare two inputs rather than processing a single one. |
| Approach: | They propose an attribution method specifically targeted for Siamese encoders that can be adjusted and fine-tuned to retain original model's predictive performance. |
| Outcome: | The proposed method retains the original model's predictive performance and can be applied to off-the-shelf models. |
Describing Images Fast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing models of visuo-linguistic variation are weak to moderately trained to capture such a variation in visual outputs. |
| Approach: | They use a corpus of Dutch image descriptions with eye-tracking data to investigate the nature of the variation in visuo-linguistic signals. |
| Outcome: | The proposed model lacks biases about what makes a stimulus complex for humans and what leads to variations in human outputs. |
Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge (2024.eacl-long)
Copied to clipboard
| Challenge: | Using low-resource languages, multilingual language models (ML-LMs) have been developed to transfer factual knowledge across languages. |
| Approach: | They ask how ML-LMs acquire and represent factual knowledge . they use a multilingual factual information probing dataset to investigate ML . |
| Outcome: | The findings highlight the challenge of maintaining consistency factual knowledge across languages. |
Comparing Knowledge Sources for Open-Domain Scientific Claim Verification (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing systems for fact-checking scientific claims assume that the documents containing the evidence are already provided and annotated or contained in a limited corpus. |
| Approach: | They perform an array of experiments to test the performance of open-domain claim verification systems on four datasets of biomedical and health claims in different settings. |
| Outcome: | The proposed system performs better with biomedical and health claims, while Wikipedia is more suited for everyday health concerns. |
Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy (2024.eacl-long)
Copied to clipboard
| Challenge: | Uncertainty estimation is an important diagnostic tool for statistical models. |
| Approach: | They propose to adapt similarity-sensitive Shannon entropy (S3E) for NMT by incorporating a concept borrowed from theoretical ecology. |
| Outcome: | The proposed framework improves quality estimation and named entity recall, and improves translation quality. |
LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text (2024.eacl-long)
Copied to clipboard
Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, Kyryl Truskovskyi
| Challenge: | a recent study focused on detecting legal violations within unstructured textual data . a similar study focused only on associating violations with potentially affected individuals . |
| Approach: | They constructed two datasets using Large Language Models (LLMs) they publicize the results to advance legal natural language processing research . |
| Outcome: | The proposed datasets and the code used for the experiments have been released to advance legal natural language processing (NLP) |
𝜇PLAN: Summarizing using a Content Plan as Cross-Lingual Bridge (2024.eacl-long)
Copied to clipboard
Fantine Huot, Joshua Maynez, Chris Alberti, Reinald Kim Amplayo, Priyanka Agrawal, Constanza Fierro, Shashi Narayan, Mirella Lapata
| Challenge: | Recent advances in abstractive summarization have focused on English, but more recently, with the advent of large pre-trained models, the task is becoming more complex. |
| Approach: | They propose an approach to cross-lingual summarization that uses an intermediate planning step as a cross-linguistic bridge. |
| Outcome: | The proposed approach achieves state-of-the-art in terms of informativeness and faithfulness on the XWikis dataset. |
Exploring Data Augmentation in Neural DRS-to-Text Generation (2024.eacl-long)
Copied to clipboard
| Challenge: | Neural networks are notoriously data-hungry, resulting in ungrammatical texts . data augmentation requires a specific design for a structurally rich input format . |
| Approach: | They propose to selectively augment a training set with new data by adding and varying two specific lexical categories, i.e. proper and common nouns. |
| Outcome: | The proposed approach selectively augments a training set with new data by adding and varying two specific lexical categories, i.e. proper and common nouns. |
Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work shows that Large Language Models (LLMs) are not robust to complex language understanding tasks due to reliance on spurious correlations of training datasets. |
| Approach: | They propose a method for measuring model reliance on spurious features by exploiting chosen biases on out-of-distribution (OOD) datasets. |
| Outcome: | The proposed method shows that the reported OOD gains of debiasing methods can't be explained by mitigated reliance on biased features, suggesting that biases are shared among different QA datasets. |
Improving Contrastive Learning in Emotion Recognition in Conversation via Data Augmentation and Decoupled Neutral Emotion (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to model context of utterances and speaker are inadequate . despite the improvements, there are still intrinsic challenges in the ERC dataset . |
| Approach: | They propose a supervised contrastive learning method specifically oriented for ERC task . they employ a data augmentation method emulating the emotion dynamics in a conversation and a method addressing the predominance and ambiguity of neutral emotion. |
| Outcome: | The proposed method emulates the emotion dynamics in a conversation and addresses the predominance and the ambiguity of neutral emotion. |
Neuralign: A Context-Aware, Cross-Lingual and Fully-Neural Sentence Alignment System for Long Texts (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing sentence alignment systems focus on auxiliary information such as document metadata and hyperparameter-sensitive techniques, and neglect the crucial role that context plays in the alignment process. |
| Approach: | They propose a context-aware, end-to-end and fully-neural architecture for sentence alignment that maps source and target sentences in long documents by contextualizing their sentence embeddings with respect to the other sentences in the document. |
| Outcome: | The proposed system maps source and target sentences in long documents by contextualizing their sentence embeddings with respect to the other sentences in the document. |
Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing explanations for speech classification models are difficult to interpret and make mistakes. |
| Approach: | They propose to explain speech classification models by using word-level and paralinguistic attributes to measure the impact of each audio segment aligned with a word on the outcome. |
| Outcome: | The proposed explanations correctly represent the model’s inner workings and are plausible to humans. |
Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training (2024.eacl-long)
Copied to clipboard
| Challenge: | End-to-end (E2E) spoken language understanding models are constrained by the cost of collecting speech-semantics pairs. |
| Approach: | They propose a model that learns E2E SLU without speech-semantics pairs . they propose cross-modal selective self-training (CMSST) to address imbalance and noise issues . |
| Outcome: | The proposed model learns E2E SLU without speech-semantics pairs . the proposed model requires the domains of speech-text and text-sensitization to match . |
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models (2024.eacl-long)
Copied to clipboard
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz
| Challenge: | Recent work suggests that Large Language Models (LLMs) exhibit Neural Theory-of-Mind (N-ToM) however, prior work reached conflicting conclusions regarding those abilities. |
| Approach: | They examine the extent of Large Language Models’ N-ToM abilities through an extensive evaluation of 6 tasks and find that LLMs struggle with adversarial examples . |
| Outcome: | The proposed metrics show that LLMs exhibit certain N-ToM abilities, but this behavior is far from robust. |
NevIR: Negation in Neural Information Retrieval (2024.eacl-long)
Copied to clipboard
| Challenge: | Negation is a common everyday phenomenon and has been a consistent area of weakness for language models. |
| Approach: | They ask IR models to rank two documents that differ only by negation . they find that most current information retrieval models do not consider negation. |
| Outcome: | The proposed benchmarks show that most current models do not consider negation . the results are similar to those found in the literature, but are poorer than random ranking . |
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) may hallucinate and generate false information despite pre-training on factual data. |
| Approach: | They propose a new evaluation metric that measures the extent to which model-produced answers are directly found in underlying text corpora. |
| Outcome: | The proposed evaluation metric measures the extent to which model-produced answers are directly found in underlying text corpora. |
Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to assigning ICD codes to clinical text are time-consuming, labor intensive, and error-prone. |
| Approach: | They propose to adapt a Transformer-based model to a longformer model and use it to encode clinical narratives. |
| Outcome: | The proposed approach outperforms current state-of-the-art models in ICD coding with the label embeddings contributing to the good performance. |
Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue (2024.eacl-long)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogues require a balance between being specific to what the conversation partner has said and being attributable to an underlying source document. |
| Approach: | They propose a framework that allows to experiment with various plan variables supported by prior work . they show that metric-aware planning mechanisms are better at automatic evaluations but underperform in human judgment compared to metric agnostic mechanisms. |
| Outcome: | The proposed framework supports metric-agnostic and metric aware content planning, but it underperforms in human judgment. |
SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models have a tendency to make confidently wrong predictions, highlighting the need for uncertainty quantification (UQ) . previous studies focused on aleatoric uncertainty, but the full spectrum of uncertainties, including epistemic, remains inadequately explored. |
| Approach: | They propose a method to quantify uncertainty in large language models (LLMs) they use a set of perturbations and an aggregation module to generalize the method. |
| Outcome: | The proposed method improves model uncertainty calibration and reduces expected calibration error by 50% on average. |
TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan
| Challenge: | Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text. |
| Approach: | They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space. |
| Outcome: | The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. |
Advancing Precise Outline-Conditioned Text Generation with Task Duality and Explicit Outline Control (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on outline-conditioned text generation focus on generating text using provided outlines as rough sketches, but lack of clarity and rationality of the rough outlines hampers quality of the generated text. |
| Approach: | They propose a novel task that requires generating stories based on specific, sentence-level outlines. |
| Outcome: | The proposed framework improves the quality of precise outline-conditioned text generation. |
Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing vision-and-language models perform better on multimodal tasks, but there is little understanding of how multimodal learning can help visual representations. |
| Approach: | They conduct a probing analysis of visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. |
| Outcome: | The proposed model improves vision-and-language models on label and attribute prediction tasks while vision-only models are stronger on dense prediction tasks. |
Creating Suspenseful Stories: Iterative Planning with Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Automated story generation has been a challenge in NLP for many years. |
| Approach: | They propose an iterative-prompting-based method that is grounded in two theoretical foundations of story suspense from cognitive psychology and narratology. |
| Outcome: | The proposed method works in a fully zero-shot manner and does not rely on any supervised story corpora. |
Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning (2024.eacl-long)
Copied to clipboard
Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl
| Challenge: | Existing prompt transfer techniques lack consideration for dialogue-specific information. |
| Approach: | They propose a method which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task. |
| Outcome: | The proposed method significantly outperforms baselines on two dialogue summarization benchmarks. |
Ask, Assess, and Refine: Rectifying Factual Consistency and Hallucination in LLMs with Metric-Guided Feedback Learning (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have heralded unprecedented capabilities in information seeking and text generation, but challenges remain regarding citation errors and generating information not present in the evidence (hallucination). |
| Approach: | They propose a framework to assess citation errors and hallucination using an explicit evaluation paradigm to formulate actionable natural language feedback. |
| Outcome: | The proposed approach improves correctness, fluency, and citation quality and reduces hallucinations in the results. |
Effective Controllable Bias Mitigation for Classification and Retrieval using Gate Adapters (2024.eacl-long)
Copied to clipboard
| Challenge: | a recent study focused on learning separate modules for on-demand debiasing. |
| Approach: | They propose a modular debiasing module with a controllable gate adapter . they demonstrate that the module can reduce the bias of search results . |
| Outcome: | The proposed module can reduce biases on three classification tasks with four protected attributes while maintaining higher task performance. |
STable: Table Generation Framework for Encoder-Decoder Models (2024.eacl-long)
Copied to clipboard
Michał Pietruszka, Michał Turski, Łukasz Borchmann, Tomasz Dwojak, Gabriela Nowakowska, Karolina Szyndler, Dawid Jurkiewicz, Łukasz Garncarek
| Challenge: | Existing approaches to infer text-to-table neural models are limited to raw text, but the proposed framework is capable of unifying a variety of problems involving natural language. |
| Approach: | They propose a framework for text-to-table neural models that utilizes a generalized sequential method that comprehends information from all cells in the table. |
| Outcome: | The proposed framework outperforms previous approaches on several challenging datasets and outperformed existing models by up to 15%. |
A RelEntLess Benchmark for Modelling Graded Relations between Named Entities (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing Knowledge Graphs do not cover graded relations, yet they are difficult to draw a line between those that satisfy them and those that do not. |
| Approach: | They propose a benchmark in which entity pairs have to be ranked according to how much they satisfy a given graded relation. |
| Outcome: | The proposed model outperforms several publicly available LLMs and closed conversational models. |
A Multimodal Framework to Detect Target Aware Aggression in Memes (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent research on memes’ detrimental facets is skewed towards high-resource languages, such as Bengali. |
| Approach: | They propose a dataset MIMOSA that annotates annotated memes across five aggression target categories in Bengali and propose 'Multimodal Attentive Fusion' to detect aggression targets. |
| Outcome: | The proposed method outperforms state-of-the-art methods in Bengali and in low-resource languages. |
Graph Guided Question Answer Generation for Procedural Question-Answering (2024.eacl-long)
Copied to clipboard
Hai Pham, Isma Hadji, Xinnuo Xu, Ziedune Degutyte, Jay Rainey, Evangelos Kazakos, Afsaneh Fazly, Georgios Tzimiropoulos, Brais Martinez
| Challenge: | a new method for question-answer generation from procedural text is sub-optimal for training QA models. |
| Approach: | They propose a method for generating exhaustive and high-quality training data from procedural text . they use procedural data to represent each step and the overall flow of the procedure as graphs . |
| Outcome: | The proposed method outperforms existing methods on task-specific question answering tasks. |
Contrastive Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Hallucinations occur when the target side sentence is detached from the source side sentence, or in other words, when there is a low contribution of the source sentence to the generation of the target sentence. |
| Approach: | They propose to use Contrastive Decoding to maximise the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
| Outcome: | The proposed algorithm maximises the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
Leveraging fine-tuned Large Language Models with LoRA for Effective Claim, Claimer, and Claim Object Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on identifying claims has focused on sentence level, neglecting supplementary attributes such as the claimer and claim object of the claim. |
| Approach: | They propose a novel approach to detect claims using large language models in natural language understanding and text generation. |
| Outcome: | The proposed approach transforms claim, claimer and claim object detection task into QA setting. |
Should I try multiple optimizers when fine-tuning a pre-trained Transformer for NLP tasks? Should I tune their hyperparameters? (2024.eacl-long)
Copied to clipboard
| Challenge: | NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer-learning techniques. |
| Approach: | They propose to use a variant of Stochastic Gradient Descent (SGD) to select among numerous variants, often with minimal or no tuning of the optimizer’s hyperparameters. |
| Outcome: | Experiments with five GLUE datasets, two models and seven popular optimizers show that tuning just the learning rate is as good as tuning all the hyperparameters. |
GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on salient entity extraction relies on crowdsourcing or user statistics to derive labels for entities. |
| Approach: | They propose a dataset that defines salience using human summaries and shows high agreement between annotations based on whether a source entity is mentioned in the summary. |
| Outcome: | The proposed dataset shows that pre-trained models and zero-shot LLM prompting fail to capture salient entities in generated summaries. |
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on sociodemographic prompting have not explored the effectiveness of this technique. |
| Approach: | They propose to use sociodemographic prompting to steer models towards answers that humans with specific sociodemography would give. |
| Outcome: | The proposed technique can improve zero-shot learning by focusing on human sociodemographic profiles. |
Threat Behavior Textual Search by Attention Graph Isomorphism (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to analyze malware behavior only disclose a subset of behaviors due to inherent difficulties. |
| Approach: | They propose a novel malware behavior search technique that is based on graph isomorphism at the attention layers of Transformer models. |
| Outcome: | The proposed technique outperforms state-of-the-art methods in a case study of 10 real-world malwares by 6-14%. |
Identifying Narrative Content in Podcast Transcripts (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to study narrativity in novels, social media and patient records are limited. |
| Approach: | They propose to process podcast transcripts and extract narrative content from podcasts . they use annotations to enable future research into narrativity within a large corpus of podcast episodes. |
| Outcome: | The proposed methods compare to existing methods and can enable future research into narrativity within a large corpus of approximately 100,000 podcast episodes. |
Frequency Explains the Inverse Correlation of Large Language Models’ Size, Training Data Amount, and Surprisal’s Fit to Reading Times (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. |
| Approach: | They present a series of analyses showing that word frequency is a key explanatory factor underlying these two trends. |
| Outcome: | The results show that word frequency is a key explanatory factor underlying these two trends. |
Presentations by the Humans and For the Humans: Harnessing LLMs for Generating Persona-Aware Slides from Documents (2024.eacl-long)
Copied to clipboard
Ishani Mondal, Shwetha S, Anandhavelu Natarajan, Aparna Garimella, Sambaran Bandyopadhyay, Jordan Boyd-Graber
| Challenge: | Existing efforts to automate document-to-slide generation have failed to adapt to the persona of target audience or duration of presentation. |
| Approach: | They propose a concept of end-user specification-aware document to slides conversion that incorporates end- user specifications into the conversion process. |
| Outcome: | The proposed model can create persona-aware presentations tailored to the persona of target audience and cognitive abilities of target audiences. |
ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. |
| Approach: | They propose a prompt-based method for token-level sequence labeling tasks . they propose to decompose an input sentence into single tokens and apply one prompt template to each token. |
| Outcome: | The proposed method outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer . the method also attains state-of-the-art performance when employed with the mT5 model . |
Small Language Models Improve Giants by Rewriting Their Outputs (2024.eacl-long)
Copied to clipboard
| Challenge: | despite impressive performance of large language models, they lag behind specialized models in various tasks. |
| Approach: | They propose a training model that can be integrated with different LLMs at inference to improve their performance without task-specific training. |
| Outcome: | The proposed model outperforms standard models on four natural language generation tasks. |
Unintended Bias Detection and Mitigation in Misogynous Memes (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing models that detect misogyny are not able to detect unintended biases in memes, perpetuating harmful stereotypes and reinforcing negative attitudes. |
| Approach: | They propose to measure and mitigate unintentional bias in misogynous memes detection models by using a contextualized scene graph-based multimodal network (CTXSGMNet) they also evaluate their generalizability by evaluating their performance on a few benchmark meme datasets. |
| Outcome: | The proposed model achieves state-of-the-art performance on the SemEval-2022 Task 5 (MAMI task) dataset, showcasing its promising performance in terms of Equity of Odds and F1 score. |
A Weak Supervision Approach for Few-Shot Aspect Based Sentiment Analysis (2024.eacl-long)
Copied to clipboard
Robert Vacareanu, Siddharth Varia, Kishaloy Halder, Shuai Wang, Giovanni Paolini, Neha Anna John, Miguel Ballesteros, Smaranda Muresan
| Challenge: | Existing methods to improve few-shot performance in aspect-based sentiment analysis (ABSA) require complex interactions between the target and the polarity of the sentiment. |
| Approach: | They propose a pipeline approach to construct a noisy ABSA dataset and adapt it to the ABSA tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art on the aspect extraction sentiment classification task and is capable of performing the harder aspect sentiment triplet extraction task. |
Counterfactual Reasoning with Knowledge Graph Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Knowledge graph embeddings were originally developed to infer true but missing facts in incomplete knowledge repositories. |
| Approach: | They propose a task that requires models to reason on a counterfactual KG. |
| Outcome: | The proposed task connects knowledge graph completion and counterfactual reasoning. |
System-Level Natural Language Feedback (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on NL feedback focus on instance-level approaches to refine specific examples, but we present a framework for system-level use of NL. |
| Approach: | They propose a framework for system-level use of natural language feedback . they use feedback to formalize system-design decisions in a human-in-the-loop-process . |
| Outcome: | The proposed framework improves search query and dialog response generation and human written instance-level feedback brings further gains over GPT-3.5 written feedback. |
Syntactic Preposing and Discourse Relations (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on discourse has ignored non-canonical syntax . a mask-filling task shows that preposing can affect discourse-relational senses . |
| Approach: | They propose to use preposing to mark information status and structure discourse flow . they use a mask-filling task to predict when a constituent appears in canonical position . |
| Outcome: | The results show that the top-ranked mask-fillers agree more often with "gold" annotations in the Penn Discourse TreeBank than in the latter case. |
Can we obtain significant success in RST discourse parsing by using Large Language Models? (2024.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that LLMs with tens of billion parameters can perform discourse parsing tasks. |
| Approach: | They employ Llama 2 and fine-tune it with QLoRA to achieve similar results . they show that LLMs with tens of billion parameters can perform a wide range of NLP tasks . |
| Outcome: | The proposed model performs better than existing models on three benchmark datasets. |
Ameli: Enhancing Multimodal Entity Linking with Fine-Grained Attributes (2024.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that understanding attributes of mentions from text descriptions and visual images plays a vital role in multimodal entity linking. |
| Approach: | They propose to integrate attributes into multimodal entity linking using a text-image-based knowledge base. |
| Outcome: | The proposed approach integrates attributes into disambiguation. |
Generative Dense Retrieval: Memory Can Be a Burden (2024.eacl-long)
Copied to clipboard
| Challenge: | Empirical results show that Generative Dense Retrieval (GDR) achieves an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability. |
| Approach: | They propose a Generative Dense Retrieval paradigm that auto-decodes document identifiers given a query and uses memory to avoid memory confusion. |
| Outcome: | Empirical results show that the proposed paradigm improves on the small-scale corpora and improves scalability. |
Backward Compatibility During Data Updates by Weight Interpolation (2024.eacl-long)
Copied to clipboard
| Challenge: | Retraining a model with a larger amount of training data introduces negative flips . retraining the model with the updated data introduce negative flipping . |
| Approach: | They propose a backward compatible weight interpolation method to improve model predictions without regression bugs. |
| Outcome: | The proposed method reduces negative flips without sacrificing accuracy . it is straight forward to implement and does not increase inference cost. |
Gradient-Based Language Model Red Teaming (2024.eacl-long)
Copied to clipboard
| Challenge: | generative language models generate unsafe responses by producing adversarial prompts . red teaming is labor-intensive and difficult to scale when done by humans. |
| Approach: | They propose a red teaming method that generates diverse prompts that are likely to cause an LM to generate unsafe responses. |
| Outcome: | The proposed method is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach. |
Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Test (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that moral judgment depends on the language in which the dilemma is presented. |
| Approach: | They extend the work of beyond English, to 5 new languages (Chinese, Hindi, Russian, Spanish and Swahili) and probe three LLMs that show substantial multilingual text processing and generation abilities. |
| Outcome: | The models show substantial multilingual text processing and generation abilities. |
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in training multilingual models on large datasets have shown promising results in knowledge transfer across languages. |
| Approach: | They challenge the assumption that high zero-shot performance reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. |
| Outcome: | The proposed model can achieve high performance on multilingual benchmarks and on low-resource languages. |
Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition (2024.eacl-long)
Copied to clipboard
| Challenge: | Few-shot named entity recognition (NER) uses only a few annotated examples to identify named entities within text. |
| Approach: | They propose to leverage natural language descriptions of each entity type to perform few-shot named entity recognition. |
| Outcome: | The proposed model learns to interpret verbalized descriptions of entities using natural language descriptions of their types and their verbalizations. |
MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to automating ML are time-consuming and difficult to understand for human developers. |
| Approach: | They propose a framework that leverages large language models to develop ML solutions for novel tasks. |
| Outcome: | The proposed framework bridges the gap between machine intelligence and human knowledge by exploiting state-of-the-art large language models. |
Text-Guided Image Clustering (2024.eacl-long)
Copied to clipboard
Andreas Stephan, Lukas Miklautz, Kevin Sidak, Jan Philip Wahle, Bela Gipp, Claudia Plant, Benjamin Roth
| Challenge: | Current image clustering methods neglect the use of generated textual descriptions. |
| Approach: | They propose to use image captioning and visual question-answering to cluster images . they propose a new approach to inject task- or domain knowledge into image clustering . |
| Outcome: | The proposed method outperforms existing methods on eight image clustering datasets. |
CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models for many-class classification suffer from verbalizer ambiguity . a significant disparity exists between the pre-training and fine-tuning stages of the model . |
| Approach: | They propose a method to tune pre-trained language models to a broad spectrum of tasks . they use an instance-dependent soft prefix to complement language verbalizers in many-class classification . |
| Outcome: | The proposed method outperforms baselines on many-class datasets. |