Findings of the Association for Computational Linguistics: EMNLP 2023
Multi Document Summarization Evaluation in the Presence of Damaging Content (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics evaluate a summary based on relevance and consistency with the source documents. |
| Approach: | They propose to measure the ability of MDS systems to handle damaging documents in their input set by lexical similarity and language model likelihood. |
| Outcome: | The proposed metrics show that they can summarize a set of documents without damaging content. |
Guiding AMR Parsing with Reverse Graph Linearization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsing aims to extract an abstract semantic graph from a sentence. |
| Approach: | They propose a new framework that allows for reversed linearization of AMR graphs . they propose to combine sequence-to-sequence approaches with a linearized graph . |
| Outcome: | The proposed framework outperforms the best AMR parser by 0.8 and 0.5 Smatch scores on the AMR 2.0 and AMR 3.0 datasets. |
Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Song translation requires both translation of lyrics and alignment of music notes . human translators of songs need to have a mastery of cultural traditions and the poetic usage of both source and target languages . |
| Approach: | They propose a model that can model lyric translation and lyrics-melody alignment . they use an encoder-decoder framework that can translate lyrics and determine number of aligned notes . |
| Outcome: | The proposed framework can translate lyrics and determine the number of aligned notes at each decoding step. |
Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users (2023.findings-emnlp)
Copied to clipboard
Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, Mitesh Khapra
| Challenge: | Indian subcontinent is home to diverse languages written in multiple scripts . widespread use of romanization and lack of standardization means accurate transliteration models form a critical component in the NLP stack for Indian languages used by over 735 million Internet users. |
| Approach: | They propose to build a transliteration dataset using monolingual and parallel corpora and human annotators. |
| Outcome: | The proposed model improves accuracy by 15% on the Dakshina test set and establishes strong baselines on the Aksharantar test set. |
Pretraining Without Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that state-space models (SSMs) outperform standard and deep learning for long-range sequence modeling. |
| Approach: | They propose a model that combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. |
| Outcome: | The proposed model outperforms standard and standard sequence modeling architectures on speech generation and the long range arena benchmarks. |
Time-Aware Representation Learning for Time-Sensitive Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA datasets do not include sufficient time expressions, and language models have difficulty understanding the relationships between time specifiers and numerical values. |
| Approach: | They propose a Time-Context-dependent Span Extraction task and build a time-context dependent data generation framework for model training. |
| Outcome: | The proposed model outperforms baseline models up to 8.5 of the F1-score in the TimeQA dataset. |
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent surge of interest in developing evaluation metrics based on pretrained large language models (LLMs) can better cope with lexical variation. |
| Approach: | They propose to replace computation-intensive transformers with lighter alternatives and employ linear and quadratic approximations for alignment algorithms on top of LLM representations. |
| Outcome: | The proposed approach replaces computation-intensive transformers with lighter alternatives and employs linear and quadratic approximations for alignment algorithms on top of LLM representations. |
Unsupervised Opinion Summarization Using Approximate Geodesics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for opinion summarization are limited due to the scarcity of data. |
| Approach: | They propose a system to perform unsupervised extractive opinion summarization using a dictionary-based representation learning model that generates topical representations of texts. |
| Outcome: | The proposed system achieves strong performance on three opinion summarization datasets. |
Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has shown that static word embeddings can encode words’ frequencies, but little has been studied about this behavior. |
| Approach: | They propose to use static word embeddings to encode words' frequencies and to assess the impact of this relationship on embeddable bias metrics. |
| Outcome: | The proposed model shows that word embeddings can produce higher similarity between high-frequency words than other embeddables. |
Improving Classifier Robustness through Active Generative Counterfactual Data Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for finding meaningful counterfactuals rely on human annotation or implicit label invariance . a small amount of human-annotated counterf actual data can generate a robust dataset with learned labels. |
| Approach: | They propose a framework that generates counterfactuals by actively sampling from regions of uncertainty and automatically labeling them with a learned auxiliary classifier. |
| Outcome: | The proposed framework generates a large number of diverse counterfactuals and labels them with a learned classifier. |
Data Augmentation Techniques for Machine Translation of Code-Switched Texts: A Comparative Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Code-switching (CSW) text generation is a popular solution to address data scarcity. |
| Approach: | They compare linguistic theories, lexical replacements and back-translation approaches to Egyptian Arabic-English CSW. |
| Outcome: | The proposed methods perform best on machine translation and quality evaluation. |
On the Relation between Sensitivity and Accuracy in In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. |
| Approach: | They propose a few-shot selective prediction method that abstains from sensitive predictions. |
| Outcome: | The proposed method outperforms confidence-based and entropy-based methods on ten classification datasets. |
Self-distilled Transitive Instance Weighting for Denoised Distantly Supervised Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to reducing wrongly labeled instances are based on a bag-level setting . however, sentence-level training is vulnerable to the noise brought by DS, which limits its application. |
| Approach: | They propose a transitive instance weighting mechanism integrated with the self-distilled BERT backbone to generate dynamic instance weights for denoised sentence-level training. |
| Outcome: | The proposed method can tackle wrongly labeled instances and prevent overfitting. |
MWE as WSD: Solving Multiword Expression Identification with Word Sense Disambiguation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to word sense disambiguation use encodings of the sense gloss and context information to improve performance. |
| Approach: | They propose a poly-encoder architecture which uses the sense gloss to improve WSD performance. |
| Outcome: | The proposed approach outperforms the state-of-the-art in word sense disambiguation by 1.9 F1 points and on the PARSEME 1.1 English dataset. |
Dual Contrastive Learning Framework for Incremental Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In incremental learning, large models learn and refresh knowledge continuously . many approaches have been proposed to preserve knowledge from previous tasks while learning new concepts in online NLP applications. |
| Approach: | They propose a dual contrastive learning framework that fosters transferability across different tasks . they use global contrastive and task-specific learning to promote a generalized embedding space . |
| Outcome: | The proposed framework outperforms the current state-of-the-art methods on text datasets. |
Reference Free Domain Adaptation for Translation of Noisy Questions with Question Specific Rewards (2023.findings-emnlp)
Copied to clipboard
Baban Gain, Ramakrishna Appicharla, Soumya Chennabasavaraj, Nikesh Garera, Asif Ekbal, Muthusamy Chelliah
| Challenge: | Creating a synthetic parallel corpus from noisy data is also difficult due to its noisy nature. |
| Approach: | They propose a training methodology that fine-tunes the NMT system only using source-side data to balance adequacy and fluency. |
| Outcome: | The proposed method surpasses the MLE-based fine-tuning approach by achieving a 1.9 BLEU improvement. |
Filtered Semi-Markov CRF (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for sequence labeling tasks such as Named Entity Recognition (NER) suffer from quadratic complexity over sequence length and poor performance compared to CRF. |
| Approach: | They propose a variant of Semi-Markov CRF that incorporates a filtering step to eliminate irrelevant segments, reducing complexity and search space. |
| Outcome: | The proposed method outperforms both CRF and Semi-CRF on several NER benchmarks while being significantly faster. |
Data Pruning for Efficient Model Pruning in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have demonstrated encouraging performance in various NLP tasks at the cost of over-parametrized networks and high memory requirements. |
| Approach: | They combine data pruning with movement pruning for Neural Machine Translation to enable efficient fine-pruning by leveraging cross-entropy scores of individual training instances. |
| Outcome: | The proposed pruning strategy outperforms other pruning methods on a translation task and shows that training cross-entropy scores can reduce the steps required for convergence and training time. |
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. |
| Approach: | They propose a large language model to split long ASR transcripts into segments that can be independently translated to maximize translation quality. |
| Outcome: | The proposed model improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 sets. |
Re-Temp: Relation-Aware Temporal Representation Learning for Temporal Knowledge Graph Completion (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models ignore ability to skip irrelevant snapshots according to entity-related relations in query . TKGC is difficult and even large-scale pre-trained language models such as gist ignore explicit temporal information. |
| Approach: | They propose a model that leverages explicit temporal embedding as input to skip unnecessary information for prediction. |
| Outcome: | The proposed model outperforms all state-of-the-art models on six datasets . it incorporates skip information flow after each timestamp to skip unnecessary information . |
RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that current TMSC systems rely on textual information, and the progress in tackling this task has slowed down. |
| Approach: | They propose to integrate both visual and textual information to improve the performance of TMSC by considering multimodal information. |
| Outcome: | The proposed model integrates both visual and textual information to improve performance. |
Lexical Entrainment for Conversational Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conversational agents are expected to possess human-like features such as lexical entrainment (LE). |
| Approach: | They propose a dataset and a measure for LE for conversational systems to explicitly integrate LE into conversational system. |
| Outcome: | The proposed dataset and a measure for LE for conversational systems address this human-like phenomenon. |
AutoReply: Detecting Nonsense in Dialogue with Discriminative Replies (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for dialogue detection make many errors in their own messages . a dataset of long dialogues richly grounded in the game state contains many errors . |
| Approach: | They propose to use an annotated dialogue dataset to generate automatic responses for dialogue models. |
| Outcome: | The proposed model outperforms handcrafted replies and performs on par with supervised learning approaches. |
Follow-on Question Suggestion via Voice Hints for Voice Assistants (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Query suggestion is a standard feature of screen-based search experiences, but it is not trivial to implement in voice-based settings. |
| Approach: | They propose a task of suggesting questions with compact voice hints to allow users to ask follow-up questions. |
| Outcome: | The proposed approach is based on a dataset of 6681 input questions and human written hints and is highly linguistically motivated. |
Bidirectional Masked Self-attention and N-gram Span Attention for Constituency Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attention mechanisms for constituency parsing lack directional information needed to form sentence spans. |
| Approach: | They propose a bidirectional masked and N-gram span Attention model which captures the explicit dependencies between each word and enhances the representation of the output span vectors. |
| Outcome: | The proposed model achieves state-of-the-art performance on the Penn Treebank and Chinese Penn TreeBank datasets with F1 scores of 96.47 and 94.15 respectively. |
CR-COPEC: Causal Rationale of Corporate Performance Changes to learn from Financial Reports (2023.findings-emnlp)
Copied to clipboard
Ye Chun, Sunjae Kwon, Kyunghwan Sohn, Nakwon Sung, Junyoup Lee, Byoung Seo, Kevin Compher, Seung-won Hwang, Jaesik Choi
| Challenge: | CR-COPEC is a large-scale domain-adaptation causal sentence dataset to detect financial performance changes of corporate. |
| Approach: | They propose a large-scale domain-adaptation causal sentence dataset to detect financial performance changes of corporate. |
| Outcome: | The proposed dataset can be used by investors and analysts without having to read through all the documents. |
Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using attention heads, we can explore how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. |
| Approach: | They propose to investigate how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. |
| Outcome: | The proposed model exhibits a higher degree of similarity with humans in plausibility processing compared to other Transformer language models. |
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models have shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. |
| Approach: | They propose a method to automatically obtain function signatures and associated Unit Tests, suitable for RL training of Code Synthesis models. |
| Outcome: | The proposed model improves on a pre-trained model on various benchmarks and is more efficient than previous models. |
Unlocking the Heterogeneous Landscape of Big Data NLP with DUUI (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Automated analysis of large corpora is a complex task, especially in terms of time efficiency. |
| Approach: | They propose a framework for automatic distributed analysis of text corpora that leverages Big Data experience and virtualization with Docker. |
| Outcome: | The proposed framework is scalable, flexible, lightweight, and feature-rich for automatic distributed analysis of text corpora. |
Towards Agile Text Classifiers for Everyone (2023.findings-emnlp)
Copied to clipboard
Maximilian Mozes, Jessica Hoffmann, Katrin Tomanek, Muhamed Kouate, Nithum Thain, Ann Yuan, Tolga Bolukbasi, Lucas Dixon
| Challenge: | Text-based safety classifiers are widely used for content moderation and increasingly to tune generative language model behavior. |
| Approach: | They propose to use small, targeted datasets to train safety classifiers using small, iterative datasets that can be quickly developed for a particular policy. |
| Outcome: | The proposed method can be quickly developed for a specific policy with a labeled dataset of as few as 80 examples. |
Beyond Good Intentions: Reporting the Research Landscape of NLP for Social Good (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have created a vast number of applications that are aimed at social good applications. |
| Approach: | They propose a dataset with three tasks that can help identify NLP4SG papers and characterize the NLP landscape by: (1) identifying the papers that address a social problem, (2) mapping them to the corresponding UN Sustainable Development Goals, and (3) identifying their methods. |
| Outcome: | The proposed dataset can help identify NLP4SG papers and characterize the NLP landscape by: (1) identifying the papers that address a social problem, (2) mapping them to the corresponding UN Sustainable Development Goals (SDGs), and (3) identifying their methods. |
PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing question answering systems rely on large, high-quality training data. |
| Approach: | They propose a synthetic data generation method which decomposes cross-lingual QA into two stages . they apply a question generation model to the English side and annotation projection to translate both questions and answers. |
| Outcome: | The proposed method outperforms existing methods on cross-lingual QA datasets. |
Sharing, Teaching and Aligning: Knowledgeable Transfer Learning for Cross-Lingual Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | X-STA is a new approach for cross-lingual machine reading comprehension . the variation of answer span positions in different languages makes it difficult to transfer knowledge across languages. |
| Approach: | They propose a method that leverages an attentive teacher to subtly transfer the answer spans of the source language to the answer output space of the target. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on three multi-lingual datasets. |
BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have improved performance of text classification tasks, but they still suffer from spurious domain-specific clues. |
| Approach: | They propose a method to augment pretrained language models by generating texts in any desired genre and on any desired topic. |
| Outcome: | The proposed method improves on genre classification tasks while showing no improvement for other topics. |
Toward Stronger Textual Attack Detectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | despite the high performance of deep learning techniques, trained models remain vulnerable to adversarial attacks . authors present LAROUSSE, STAKEOUT and other approaches to detect adversarials . LARousSE is unsupervised, hyperparameter free and non-differentiable . |
| Approach: | They propose a framework to detect adversarial attacks and an extended benchmark to test them . they demonstrate that LAROUSSE outperforms previous methods and allows to identify interesting factor of detection rate variations. |
| Outcome: | The proposed framework outperforms existing methods and allows to identify interesting factor of detection rate variations. |
MEAL: Stable and Active Learning for Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot classification have high variance across different sets of few shots and finetuning runs. |
| Approach: | They propose novel ensembling methods that significantly reduce run variability and introduce a new active learning criterion for *data selection*. |
| Outcome: | The proposed method significantly reduces run variability and improves performance on five tasks. |
Structure and Label Constrained Data Augmentation for Cross-domain Few-shot NER (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require large datasets with accurate annotations that are labor-intensive and time-consuming. |
| Approach: | They propose a method to leverage domain gaps to model cross-domain few-shot named entity recognition (NER) NER is a natural language processing task to detect entity mentions and classify them into predefined labels . |
| Outcome: | The proposed method achieves state-of-the-art or competitive results on standard datasets. |
Weakly-supervised Deep Cognate Detection Framework for Low-Resourced Languages Using Morphological Knowledge of Closely-Related Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cognate detection focus on orthographic, phonetic or contextual models, which under-perform for most under-resourced languages. |
| Approach: | They propose a language-agnostic weakly-supervised deep cognate detection framework for under-resourced languages using morphological knowledge from closely related languages. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on cognate detection datasets across languages and can be extended to a wide range of languages from any language family. |
SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-SQL aims to automate the process of generating SQL queries on a database from natural language text. |
| Approach: | They propose a method to improve few-shot prompting capabilities of Text-to-SQL for Large Language Models (LLMs) they propose 'SQlPrompt' which aims to diversify the SQL proposals during consistency selection with different prompt designs and foundation models. |
| Outcome: | The proposed method outperforms previous approaches for in-context learning with zero labeled data by a large margin, closing the gap with finetuning state-of-the-art with thousands of labeles. |
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing foundation models can only perform the best in one type of understanding tasks. |
| Approach: | They propose a method for training a general foundation model, X-FM, using text, image, and image-text data. |
| Outcome: | The proposed method outperforms existing foundation models on language, vision, and vision-language understanding tasks. |
Trigger Warnings: Bootstrapping a Violence Detector for Fan Fiction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing guidelines for proactively alerting readers of potentially disturbing content have been proposed. |
| Approach: | They propose to use a labeled corpus of narrative fiction from a popular fan fiction site to determine whether to assign a trigger warning to an English story. |
| Outcome: | The proposed task achieves F1 scores between 0.8 and 0.9 on three datasets . the authors show that assigning trigger warnings for violence is feasible . |
Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Code pre-trained models have been proposed and widely applied in the domain of code intelligence. |
| Approach: | They propose a method that uses a plug-and-play graph neural network module as a tunable prefix to exploit structural information of source code. |
| Outcome: | The proposed method exploits structural information of source code and could replace full fine-tuning. |
Counterfactual Augmentation for Multimodal Learning Under Presentation Bias (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage. |
| Approach: | They propose a method for correcting presentation bias using generated counterfactual labels by augmentation of the labels by the user. |
| Outcome: | The proposed method improves performance in an oracle setting compared to uncorrected models and existing bias-correction methods. |
A Table-to-Text Framework with Heterogeneous Multidominance Attention and Self-Evaluated Multi-Pass Deliberation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Table-to-text works have been widely applied in different domains, such as weather forecast and financial report generation. |
| Approach: | They propose a table-to-text approach on top of Self-evaluated multi-pass Generation and Heterogenous Multidominance Attention to explore the hierarchical structure. |
| Outcome: | The proposed method outperforms several SOTA methods quantitatively and qualitatively on three public datasets. |
Crossing the Aisle: Unveiling Partisan and Counter-Partisan Events in News Reporting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work in NLP has only studied media bias via linguistic style and word usage. |
| Approach: | They annotate a dataset containing 8,511 (counter-)partisan event annotations in 304 news articles from ideologically diverse media outlets. |
| Outcome: | The proposed dataset contains 8,511 (counter-)partisan event annotations in 304 news articles from ideologically diverse media outlets. |
Video-Text Retrieval by Supervised Sparse Multi-Grained Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in video-text retrieval have led to improved representation learning methods. |
| Approach: | They propose a multi-grained sparse learning framework to learn an aligned sparsen space shared between video and text for video-text retrieval. |
| Outcome: | The proposed framework is superior to existing methods on video-text retrieval benchmarks. |
Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Intent discovery remains a crucial task in natural language processing . identifying novel, unseen intents remains one of the biggest challenges in this field . |
| Approach: | They propose a multi-language approach to intent discovery using Adapters and a Transformer architecture. |
| Outcome: | The proposed pipeline outperforms baselines in two zero-shot settings for intent classification and unseen intent discovery. |
ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation (2023.findings-emnlp)
Copied to clipboard
Kun Zhang, Xiexiong Lin, Yuanzhuo Wang, Xin Zhang, Fei Sun, Cen Jianhe, Hexiang Tan, Xuhui Jiang, Huawei Shen
| Challenge: | Existing methods that align natural language with SQL Language underestimate inherent structural characteristics of SQL and lead to structure errors. |
| Approach: | They propose a retrieval-argument framework that aligns natural language with SQL Language and trains one encoder-decoder-based model to fit all questions. |
| Outcome: | The proposed framework improves accuracy and robustness of text-to-SQL generation on five datasets. |
Approximating Two-Layer Feedforward Networks for Efficient Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work uses sparse Mixtures of Experts (MoEs) to build resource-efficient large language models. |
| Approach: | They propose a general framework that unifies various methods to build two-layer NNs . they propose methods to improve both MoEs and PKMs based on their results . |
| Outcome: | The proposed framework improves both MoEs and product-key memories (PKMs) it shows that MoE's are competitive with dense Transformer-XL on two different scales while being much more resource efficient. |
Adapter-TST: A Parameter Efficient Method for Multiple-Attribute Text Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies explore performing text style transfer on attributes like age, gender, formality, politeness, and formality. |
| Approach: | They propose a framework that freezes the pre-trained model’s original parameters and enables the development of a multiple-attribute text style transfer model. |
| Outcome: | The proposed model outperforms state-of-the-art models on sentiment transfer and multiple-attribute transfer tasks with significantly less computational resources. |
Solving the Right Problem is Key for Translational NLP: A Case Study in UMLS Vocabulary Insertion (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a gap exists between research output and real-world task for automated NLP systems . a recent study shows that powerful models alone will not yield translational NLP solutions . |
| Approach: | They propose a formulation for UMLS vocabulary insertion which mirrors the real-world task . they propose measurable qualitative improvements to editors who carry out the UVI task based on strong datasets . |
| Outcome: | The proposed model outperforms existing models and improves the UVI task. |
Improving Cross-lingual Transfer through Subtree-aware Word Reordering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that multilingual language models are not effective when dealing with less-represented languages. |
| Approach: | They propose a powerful reordering method that learns word-order patterns conditioned on the syntactic context from a small amount of annotated data. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks and is effective in both zero-shot and few-shot scenarios. |
Novel Slot Detection With an Incremental Setting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current dialogue systems face diverse user requests and rapid change domains, making quickly adapt to scenarios with previous unseen slot types becomes a major challenge. |
| Approach: | They propose an incremental novel slot detection task which separates the dialogue system to deal with novel types as two major phrases: 1) model discovers unknown slots; 2) training model to possess the capability to handle new classes. |
| Outcome: | The proposed approach overcomes catastrophic forgetting during the process of INSD and is highly effective. |
Self-supervised Post-processing Method to Enrich Pretrained Word Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods that use external resources to make word vectors specialize depend on the lexicon. |
| Approach: | They propose a self-supervised extension of extrofitting by its own word vector distribution. |
| Outcome: | The proposed method improves word similarity embeddings on similarity tasks without external resources. |
Automatic Model Selection with Large Language Models for Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought and Program-Aided Language Models offer different strengths and weaknesses. |
| Approach: | They propose a model selection method that uses a large language model to select between two different reasoning methods. |
| Outcome: | The proposed method shows significant performance improvements across eight reasoning datasets with Codex, ChatGPT, and GPT-4. |
ARKitSceneRefer: Text-based Localization of Small Objects in Diverse Real-World 3D Indoor Scenes (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for 3D referring expression comprehension cover large objects and small objects, such as cooking tools and office supplies. |
| Approach: | They propose a 3D referring expression comprehension dataset that uses 3D scenes to ground text representations onto objects in 3D environments. |
| Outcome: | The proposed dataset covers 15k objects of 1,605 indoor scenes and is significantly larger than existing datasets. |
Improving Question Generation with Multi-level Content Planning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest key phrase selection is essential for question generation, yet it is difficult to connect disjointed phrases into meaningful questions, especially for long context. |
| Approach: | They propose a QG framework that uses multi-level content planning to generate questions from a given context and an answer. |
| Outcome: | The proposed framework outperforms baselines on two popular QG datasets. |
Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized general natural language preprocessing tasks, but their performance in financial domains is not evaluated comprehensively. |
| Approach: | They propose a framework to evaluate financial language models on financial tasks . they compare performance of auto-encoding language models and ChatGPT . |
| Outcome: | The proposed framework compares the performance of auto-encoding language models and the LLM ChatGPT on financial tasks. |
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering (2023.findings-emnlp)
Copied to clipboard
Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang, Rakesh Menon, Md Parvez, Zhe Feng
| Challenge: | Hallucination is a well-known phenomenon in text generated by large language models . state-of-the-art LLMs still have a number of weaknesses, including the tendency to generate hallucinatory statements without considering the factuality . |
| Approach: | They propose a dataset that captures hallucinations made by retrieval-augmented LLMs . they propose to use these methods to help detect hallucinosity in QA tasks . |
| Outcome: | The proposed method captures hallucinations made by retrieval-augmented LLMs for QA tasks. |
InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that multi-modal data representations tend to cluster within a limited convex cone, hindering retrieval performance. |
| Approach: | They propose a method that uses graph convolution and average pooling to separate representations by increasing distances between data points. |
| Outcome: | The proposed method reduces representation degeneration problem by increasing distances between data points. |
Dissecting In-Context Learning of Translations in GPT-3 (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on fewshot prompting in Large Language Models has focused on selecting the few-shot samples for prompting. |
| Approach: | They propose a method to add demonstration attributes to prompting in machine translations by perturbations of high-quality in-domain demonstrations. |
| Outcome: | The proposed method improves upon the zero-shot translation performance of GPT-3, even making it competitive with few-shot prompted translations. |
Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Open-domain dialog generates search queries that help obtain relevant knowledge for holding informative conversations. |
| Approach: | They propose to integrate social commonsense reasoning into internet search queries . they use a commonsensible dialog system to establish connections related to the conversation topic . |
| Outcome: | The proposed framework overcomes limitations of existing query generation techniques based on explicit dialog information and produces more relevant, specific, and compelling queries. |
MixTEA: Semi-supervised Entity Alignment with Mixture Teaching (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn informative entity embeddings are insufficient for semi-supervised entity alignment. |
| Approach: | They propose a semi-supervised method which guides the model learning with an end-to-end mixture teaching of manually labeled mappings and probabilistic pseudo mappings. |
| Outcome: | The proposed method is superior to existing methods on benchmark datasets and further analyses. |
Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to enhance dense retrieval models are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. |
| Approach: | They propose an unsupervised method to enhance passage retrieval in zero-shot settings by iterating a loop that a dense retriever learns from supervision signals provided by a reranker. |
| Outcome: | The proposed method outperforms leading supervised and unsupervised retrievers on the BEIR benchmark while showing strong adaptation abilities to tasks and domains that were unseen during training. |
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that TESTA reduces the number of visual tokens by 75% and thus accelerates video encoding. |
| Approach: | They propose a method to condense video semantics by aggregating similar frames and patches within each frame. |
| Outcome: | The proposed method reduces visual tokens by 75% and accelerates video encoding. |
Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for extracting temporal information from text are not suitable for time-sensitive questions. |
| Approach: | They propose to use existing temporal information extraction systems to construct temporal graphs of events, times, and temporal relations in questions and documents. |
| Outcome: | The proposed method outperforms graph convolution-based approaches on SituatedQA and TimeQA. |
The Internal State of an LLM Knows When It’s Lying (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown exceptional performance in various tasks, but one of their main drawbacks is generating inaccurate or false information with a confident tone. |
| Approach: | They propose to train a classifier that outputs the probability that a statement is truthful based on the hidden layer activations of the LLM as it reads or generates the statement. |
| Outcome: | The proposed classifier achieves an average of 71% to 83% accuracy labeling which sentences are true versus false, depending on the LLM base model. |
Factual Relation Discrimination for Factuality-oriented Abstractive Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing factuality-oriented abstractive summarization models only consider the integration of factual information and ignore the causes of factuual errors. |
| Approach: | They propose a factuality-oriented abstractive summarization model that can identify the causes of factual errors. |
| Outcome: | The proposed model outperforms state-of-the-art models in factual metrics. |
Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-modal entity alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). |
| Approach: | They propose a novel MMEA transformer that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance alignment task. |
| Outcome: | The proposed transformer hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. |
Is a Prestigious Job the same as a Prestigious Country? A Case Study on Multilingual Sentence Embeddings and European Countries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Our analysis reveals that the strongest dimension in all models correlates with the political and economic distinction between Western and Eastern Europe and the country’s GDP. |
| Approach: | They propose to prompt models with templated sentences that they machine-translate into 12 European languages and analyze the most prominent dimension in the embeddings. |
| Outcome: | The political distinction between Eastern and Western Europe and the country’s economic strength in terms of GDP are the most prominent features in the embeddings. |
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent inquiries reveal a lack of robust ToM in large language models . current models focus on different aspects of ToM and are prone to shortcuts and data leakage. |
| Approach: | They propose to taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. |
| Outcome: | The proposed model breaks ToM into individual components and treats LLMs as agents physically and socially situated in interactions with humans. |
Text Augmented Spatial Aware Zero-shot Referring Image Segmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing zero-shot referring image segmentation methods focus on global-level alignment of image-text pairs, neglecting fine-grained matching between referring sentence and local image regions. |
| Approach: | They propose a zero-shot referring image segmentation task that is training-free . they use a mask proposal network and a text-augmented spatial-correction score . |
| Outcome: | The proposed method outperforms state-of-the-art zero-shot referring image segmentation methods. |
IRFL: Image Recognition of Figurative Language (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Figures of speech are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. |
| Approach: | They develop a dataset for multimodal figurative language understanding using human annotation and an automatic pipeline to generate a multimodal dataset. |
| Outcome: | The proposed dataset performs better than human vision and language models compared with a human dataset . |
Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for prompt tuning can overfit to few-shot training samples, causing overfitting . authors propose a new framework for prompt learning with supervised meta-learning . |
| Approach: | They propose a self-supervised meta-prompt learning framework with MEta-gradient Regularization for few-shot generalization that leverages self-recognized meta-learning with a diverse set of meta-tasks to learn a universal prompt initialization using only unlabeled data. |
| Outcome: | The proposed framework learns a universal prompt initialization for efficient adaptation using only unlabeled data. |
An Adaptive Prompt Generation Framework for Task-oriented Dialogue System (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing black-box large language models (LLMs) have excellent performance in task-oriented dialogue (TOD) tasks, but obtaining suitable prompts for specific tasks is challenging. |
| Approach: | They propose a black-box large language model that generates domain and slot information in the belief state, which serves as prior knowledge for subsequent prompt generation. |
| Outcome: | The proposed framework outperforms existing prompting methods on the MultiWOZ 2.0 dataset. |
Temporal Knowledge Graph Reasoning Based on N-tuple Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Temporal Knowledge Graphs (TKGs) only contain their core entities and form them as quadruples. |
| Approach: | They propose to describe a temporal fact more accurately as an n-tuple . they propose to use a neural network to learn evolutional representations of entities . |
| Outcome: | The proposed model oversimplifies and causes information loss on two datasets. |
Make Your Decision Convincing! A Unified Two-Stage Framework: Self-Attribution and Decision-Making (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for explaining black-box model behavior are unreliable . large-scale pre-trained models often rely on superficial clues for predictions . |
| Approach: | They propose a unified two-stage framework that uses subsequences from the input text as a rationale to generate model decision. |
| Outcome: | The proposed framework achieves competitive results on five reasoning datasets and in semi-supervised scenarios. |
Adaptive Structure Induction for Aspect-based Sentiment Analysis with Spectral Perspective (2023.findings-emnlp)
Copied to clipboard
| Challenge: | incorporating structure information can enhance the performance of aspect-based sentiment analysis. |
| Approach: | They propose to use pre-trained language models to induct latent structures from a spectrum perspective. |
| Outcome: | The proposed model shortens Aspects-sentiment Distance and improves structure induction ability. |
NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
Peter West, Ronan Bras, Taylor Sorensen, Bill Lin, Liwei Jiang, Ximing Lu, Khyathi Chandu, Jack Hessel, Ashutosh Baheti, Chandra Bhagavatula, Yejin Choi
| Challenge: | a new commonsense knowledge model, NovaCOMET, combines knowledge and general task models. |
| Approach: | They propose an open commonsense knowledge model that combines knowledge and general task models. |
| Outcome: | The proposed model matches or exceeds existing knowledge models on commonsense reasoning tasks. |
In-Context Demonstration Selection with Cross Entropy Difference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. |
| Approach: | They propose a cross-entropy difference method for selecting in-context demonstrations that uses parameter efficient finetuning to train small models on training data. |
| Outcome: | The proposed method outperforms baseline selection methods on a mix-domain dataset and shows that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example. |
The Past, Present, and Future of Typological Databases in NLP (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Typological information is inconsistent with each other and other sources of typological information, such as linguistic grammars. |
| Approach: | They propose to examine disagreements between typological databases and their uses in NLP by exploring disagreements across databases and resources. |
| Outcome: | The proposed view of typology has significant potential in the future, including in language modeling in low-resource scenarios. |
SoulChat: Improving LLMs’ Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used in psychological counseling to provide universal advice. |
| Approach: | They constructed a multi-turn empathetic conversation dataset with 2 million samples . they found that the model's empathy ability is enhanced when finetuning . |
| Outcome: | Experiments show that large language models can be finetuned to provide empathy . but, when applied to mental health or emotional support conversation, there are three main issues . |
Can ChatGPT Assess Human Personalities? A General Evaluation Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLM. |
| Approach: | They propose to use Myers–Briggs Type Indicator (MBTI) tests to generate unbiased prompts and replace the subject in question statements to enable flexible queries and assessments. |
| Outcome: | The proposed framework enables LLMs to flexibly assess personalities of different groups of people. |
MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) like GPT-4 are not able to handle multi-modal open-domain question answering in a zero-shot manner. |
| Approach: | MoqaGPT uses divide-and-conquer strategy to extract answers from each modality separately. |
| Outcome: | MoqaGPT improves on MMCoQA dataset by +37.91 points and EM by +34.07 points. |
Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for understanding users’ contextual search intent show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. |
| Approach: | They propose to use large language models to generate multiple query rewrites and hypothetical responses and to aggregate them into an integrated representation that can robustly represent the user’s real contextual search intent. |
| Outcome: | The proposed framework can generate multiple query rewrites and hypothetical responses and can be used to represent the user’s real contextual search intent. |
DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-based metrics are limited by their reliance on human input. |
| Approach: | They propose to adapt some reference-based metrics to assess system summary against human-written references. |
| Outcome: | The proposed model outperforms reference-based metrics on two datasets and is comparable to reference-free metrics. |
Toxicity in chatgpt: Analyzing persona-assigned language models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing community. |
| Approach: | They evaluate toxicity in over half a million generations of ChatGPT by assigning it a persona . they find that outputs engage in incorrect stereotypes, harmful dialogue, hurtful opinions . |
| Outcome: | a new study shows that assigning a persona to a chatbot can increase toxicity in half a million generations. |
Execution-Based Evaluation for Open-Domain Code Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ODEX is the first open-domain EXecution-based natural language (NL) to Python code generation dataset. |
| Approach: | They propose to use a dataset to extend the scope of coding queries to more realistic settings by using open-domain EXecution-based natural language (NL) to Python. |
| Outcome: | The proposed dataset has 945 NL-Code pairs and 1,707 human-written test cases. |
Syntax-Aware Retrieval Augmented Code Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural code generation models with token-level retrieval capabilities are often noisy and time-consuming. |
| Approach: | They propose a token-level retrieval augmented code generation method that leverages syntax constraints for the retrieval of datastores. |
| Outcome: | The proposed method reduces the impact of retrieve noise on code generation on two datasets. |
Selecting Key Views for Zero-Shot Entity Linking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Entity linking is a task of assigning ambiguous mentions in textual input to entities in knowledge bases. |
| Approach: | They propose a framework to align mentions in text to entities in knowledge bases . they use unsupervised clustering to select key views from descriptions . |
| Outcome: | The proposed framework achieves state-of-the-art on the zero-shot entity linking dataset. |
Is Explanation the Cure? Misinformation Mitigation in the Short Term and Long Term (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using natural language processing (NLP), there is an ongoing shift towards NLPbased solutions such as fake news detection and generation of fact-checked, counterfactual explanations. |
| Approach: | They compare the effectiveness of a warning label and state-of-the-art counterfactual explanations generated by natural language generation (GPT4) models in debunking misinformation. |
| Outcome: | The proposed explanations significantly decrease participants’ self-reported belief in fake claims for the short-term and long-term. |
Improving the Robustness of Summarization Models by Detecting and Removing Input Noise (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summarization models are typically evaluated using test data that is identically distributed as training data. |
| Approach: | They propose a method to detect and remove input noise from documents to be summarized without extra training or auxiliary models. |
| Outcome: | The proposed method recovers a large fraction of the loss in performance, sometimes as large as 11 ROUGE-1 points, without extra training, auxiliary models, or prior knowledge of the type of noise. |
How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to detect AI-generated text are inadequate, causing misuse of the text. |
| Approach: | They propose a universal evasive prompt framework that can prompt any PLM to generate “human-like” text that can mislead detectors. |
| Outcome: | The proposed approach can prompt any PLM to generate “human-like” text that can mislead detectors. |
Knowledge is a Region in Weight Space for Fine-tuned Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuning on different datasets from the same underlying task form s looser clusters. |
| Approach: | They analyze the weight space and the underlying loss landscape of different models to determine how they are interconnected and how they perform. |
| Outcome: | The proposed method improves accuracy on 11 out of 12 datasets by starting from the center of the region, if not more, than using the pretrained model. |
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-annotator datasets can improve performance when they expand from a single annotation per instance to multiple annotations. |
| Approach: | They propose a multi-annotator simulation process to generate datasets with varying annotation budgets and compare them to a single annotation per instance. |
| Outcome: | The proposed model can generate datasets with varying annotation budgets and show that similar datasets can lead to varying performance gains. |
On the Risk of Misinformation Pollution with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study demonstrates that large language models can be misused for generating credible-sounding misinformation . however, the ability to produce credible text raises concerns regarding their potential misuse . |
| Approach: | They propose three defense strategies to mitigate misinformation generated by Large Language Models . they propose a threat model and simulate potential misuse scenarios . |
| Outcome: | The proposed defense strategies have shown promising results, albeit with costs. |
Dolphin: A Challenging and Diverse Benchmark for Arabic NLG (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for Arabic are limited, but they can be used to measure performance of different languages. |
| Approach: | They propose a benchmark for Arabic that addresses the need for a framework dedicated to Arabic languages and varieties. |
| Outcome: | The proposed benchmark covers 13 different tasks in Arabic and spans 50 test splits. |
Hierarchical Enhancement Framework for Aspect-based Argument Mining (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods have primarily treated ABAM as a nested named entity recognition problem, overlooking the need for tailored strategies to effectively address the specific challenges of ABA M tasks. |
| Approach: | They propose a layer-based Hierarchical Enhancement Framework (HEF) for Aspect-Based Argument Mining and introduce three new components to improve the performance and accuracy. |
| Outcome: | Experiments on multiple datasets and tasks verify the effectiveness of the proposed framework and components. |
MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown nearly saturated performance on many NLP tasks. |
| Approach: | They construct multiple sensitive factors time QA which encompasses three temporal factors . they test current mainstream LLMs with different parameter sizes . |
| Outcome: | The proposed model incorporates three temporal factors with 2,853 samples . the results show that LLMs fall behind smaller models on these factors . |
What Makes Chain-of-Thought Prompting Effective? A Counterfactual Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using a few-shot prompt, we examine the effects of symbols and patterns on in-context learning in large language models. |
| Approach: | They employ a counterfactual prompting approach by manipulating examples and testing the consequences on model behavior. |
| Outcome: | The proposed approach allows us to understand the relative contributions of symbols and patterns on in-context learning. |
Perceptual Structure in the absence of grounding: the impact of abstractedness and subjectivity in color language for LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that color perception and color language are suitable for empirically studying the problem. |
| Approach: | They propose to quantify alignment between a defined color space and a feature space in a language model by learning a mapping between embedding space and color space. |
| Outcome: | The results show that there is considerable alignment between a defined color space and the feature space defined by a language model. |
A Dataset for Investigating the Impact of Context for Offensive Language Detection in Tweets (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Offensive language detection is crucial in natural language processing . we investigated the importance of contextual information for detecting offensive language in tweets . |
| Approach: | They investigated the importance of contextual information for detecting offensive language in tweets . they used a Turkish tweet dataset with over 28,000 tweet-reply pairs . |
| Outcome: | The proposed model performs better with and without contextual information than with and with contextual information. |
Remember what you did so you know what to do next (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown large language models (LLMs) to be poor fit for a simulated robot to achieve 30 classes of goals. |
| Approach: | They use the 6B parameter GPT-J language model to create a plan for a simulated robot to achieve 30 classes of goals in ScienceWorld. |
| Outcome: | The proposed model outperforms the state-of-the-art by a factor of 1.4 when training on as many prior steps as will fit, and the results are 2.3x better than the state of the-art. |
An Empirical Study of Multimodal Model Merging (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that model merging can generate a multi-task solution without synchronous training. |
| Approach: | They propose to merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient architecture. |
| Outcome: | The proposed model merging outperforms naive models on various tasks with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% ADE20k. |
Learning to Abstract with Nonparametric Variational Information Bottleneck (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Learning textual embeddings is costly as they are tokenization specific and require different models to be trained for each level of abstraction. |
| Approach: | They propose a character-level representation model which learns different levels of abstraction in different layers of the same model. |
| Outcome: | The proposed model is more robust to adversarial perturbations and can learn to compress representations to different levels of abstraction. |
Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on manipulating entity features to find pairwise relations, yet neglect the more fundamental structural information that links disparate entity pairs together. |
| Approach: | They propose a Visual Relation Extraction framework that generates relation predictions on entity pairs extracted from scanned images and incorporates global structural knowledge into the representations of the entities. |
| Outcome: | The proposed framework outperforms existing methods in fine-tuning setting and yields stronger data-efficient performance in the low-resource setting. |
Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that sequence-to-sequence (seq2sequ) models struggle with compositional generalization (CG) a crucial property of human language learning is its compositional globalization (GC), the algebraic ability to understand and produce a potentially infinite number of novel combinations from known components. |
| Approach: | They propose a sequence-to-sequence (seq2sequ) extension which learns to compose representations of different encoder layers dynamically for different tasks. |
| Outcome: | The proposed model achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of the proposed model. |
SelectNoise: Unsupervised Noise Injection to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, MT systems for low-resource languages lack parallel data and monolingual data. |
| Approach: | They propose an unsupervised approach to generate noisy HRLs training data by selective candidate extraction and noise injection. |
| Outcome: | The proposed model outperforms strong baselines on 12 ELRLs in a zero-shot setting . |
Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation methods for dense retrieval models use unadapted rerank models, leading to imprecise labels. |
| Approach: | They propose to adapt a rerank model to the target domain before using it for label generation. |
| Outcome: | The proposed model achieves better results across three retrieval datasets. |
Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on LLMs evaluation with exams are lacking in cognitive research on their overall knowledge structure. |
| Approach: | They conduct an evaluation using a human test dataset based on Bloom Taxonomy to reveal the knowledge structures of Large Language Models and gain insights of their cognitive capabilities. |
| Outcome: | The proposed model can pass AP, SAT, and Leetcode exams, but lacks the cognitive power to perform on human exams. |
Simpler neural networks prefer subregular languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Inductive biases of neural networks are still poorly understood, says dr. johansen . subregular languages are thought to form a bound on human phonological patterns . |
| Approach: | They apply a relaxation of L0 regularization which induces sparsity to study inductive biases of LSTMs. |
| Outcome: | The proposed method is based on a relaxation of L0 regularization, which induces sparsity, and a subregular language bias in LSTMs is related to the cognitive bias observed in human phonology. |
Simple Hardware-Efficient PCFGs with Independent Left and Right Productions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scaling dense PCFGs to thousands of nonterminals have shown to be beneficial for unsupervised parsing, but they still perform poorly as a language model and as an unsupervised model. |
| Approach: | They propose a simple PCFG formalism with independent left and right productions that scales more effectively as a language model and as an unsupervised parser. |
| Outcome: | The proposed formalism scales better as a language model and as an unsupervised parser despite imposing a stronger independence assumption compared to low-rank approaches. |
R3 Prompting: Review, Rephrase and Resolve for Chain-of-Thought Reasoning in Large Language Models under Noisy Context (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have evaluated LLMs under noise-free context but the dilemma for LLM to produce inaccurate results under noisy context has not been fully investigated. |
| Approach: | They propose a new method for CoT reasoning using Chain-of-Thought prompting that interacts with LLMs to perform key sentence extraction, variable declaration and answer prediction. |
| Outcome: | The proposed method outperforms existing CoT prompting methods on five reasoning tasks under noisy context. |
Quality Estimation-Assisted Automatic Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing APE and QE combination strategies have not shown significant performance gains in the field of automatic post-editing (APE). |
| Approach: | They propose to train a model on APE and QE tasks to improve the APE performance by using a multi-task learning methodology that treats both tasks as a 'bargaining game' they also investigate various existing combination strategies and show that their approach achieves state-of-the-art performance for a ‘distant’ language pair, viz., English-Marathi. |
| Outcome: | The proposed model improves on two different language pairs, viz., English-Marathi and English-German. |
Adapter Pruning using Tropical Characterization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on adapter pruning have not examined the optimal number of adapter parameters needed for downstream applications. |
| Approach: | They propose an adapter pruning approach that prunes adapter parameters without changing the orientation of underlying tropical hypersurfaces. |
| Outcome: | The proposed approach prunes adapter layers without changing the orientation of underlying tropical hypersurfaces. |
Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge (2023.findings-emnlp)
Copied to clipboard
Shajith Ikbal, Udit Sharma, Hima Karanam, Sumit Neelam, Ronny Luss, Dheeraj Sreedhar, Pavan Kapanipathi, Naweed Khan, Kyle Erwin, Ndivhuwo Makondo, Ibrahim Abdelaziz, Achille Fokoue, Alexander Gray, Maxwell Crouse, Subhajit Chaudhury, Chitra Subramanian
| Challenge: | Various approaches have been tried to map predicate components of a natural language (NL) text segment onto their corresponding predicates within a knowledge base (KB). |
| Approach: | They propose a neuro-symbolic approach to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base question answering systems. |
| Outcome: | The proposed approach achieves an average performance gain of 17% on CLUTRR and relation linking in a KBQA system. |
TaTA: A Multilingual Table-to-Text Dataset for African Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Gehrmann, Sebastian Ruder, Vitaly Nikolaev, Jan Botha, Michael Chavinda, Ankur Parikh, Clara Rivera
| Challenge: | Existing data-to-text generation datasets are limited to English and a small number of other languages. |
| Approach: | They create the first large multilingual table-to-text dataset with a focus on African languages. |
| Outcome: | The proposed dataset includes 8,700 examples in nine languages including four African languages and a zero-shot test language. |
Explain-then-translate: an analysis on improving program translation with self-generated explanations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using self-generated natural language explanations improves zero-shot performance by 12% on average. |
| Approach: | They propose to use self-generated natural language explanations as an intermediate step for code-to-code translation with language models. |
| Outcome: | The proposed approach improves zero-shot performance by 12% on average . the proposed approach is not evaluated on a broader set of languages including low-resource languages. |
Can Brain Signals Reveal Inner Alignment with Human Languages? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Brain Signals, such as Electroencephalography, and human languages have been explored independently for many downstream tasks, however, the connection between them has not been well explored. |
| Approach: | They introduce a multimodal transformer alignment model to observe coordinated representations between EEG and language. |
| Outcome: | The proposed method achieved an F1-score improvement of 1.7% on ZuCo and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCO for relation detection. |
DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current Event Extraction methods focus on high-resource scenarios, which requires large amount of annotated data. |
| Approach: | They propose a demonstration-based learning paradigm for EE to fully use annotated data . they propose EE as a natural language generation task guided by schema-based prompts . |
| Outcome: | The proposed model outperforms current methods in low-resource scenarios. |
GLGR: Question-aware Global-to-Local Graph Reasoning for Multi-party Dialogue Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for multi-hop reasoning are lacking for local graph reasoning . existing approaches neglect local semantic structures in utterances . |
| Approach: | They propose a question-aware global-to-local graph reasoning approach that expands the canonical Interlocutor-Utterance graph by introducing a query node. |
| Outcome: | The proposed approach outperforms existing methods on Molweni and FriendsQA. |
Towards Mitigating LLM Hallucination via Self Reflection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown promise for generative and knowledge-intensive tasks including question-answering (QA) but the practical deployment still faces challenges, notably the issue of “hallucination”, where models generate plausible-sounding but unfaithful or nonsensical information. |
| Approach: | They propose a self-reflection methodology that incorporates knowledge acquisition and answer generation to address the issue of "hallucination" they use a set of LLMs to generate a more accurate and factually accurate answer. |
| Outcome: | The proposed approach improves factuality, consistency, and entailment of the generated answers. |
Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of manual annotation and recognition relied on a predefinition of features and required technical knowledge. |
| Approach: | They propose a 3D normalization method for MediaPipe’s 2D pose and a novel human-readable way of representing the 3D standardized pose data. |
| Outcome: | The proposed method outperforms the common 2D normalization on a small, diverse Japanese Sign Language (JSL) sociolinguistic dataset. |
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, Partha Talukdar
| Challenge: | Existing datasets are often informed by established research directions in the NLP community. |
| Approach: | They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
| Outcome: | The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
DiffuVST: Narrating Fictional Scenes with Global-History-Guided Denoising Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for visual storytelling suffer from low inference speed and are not well-suited for synthetic scenes. |
| Approach: | They propose a diffusion-based system that generates visual descriptions as a single conditional denoising process. |
| Outcome: | The proposed system improves inter-sentence coherence and image-to-text fidelity. |
DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to mitigate gender bias in pre-trained language models are often evaluated on datasets that check the extent to which the model is gender-neutral in its predictions. |
| Approach: | They propose to use a manually curated dataset to measure gender bias and to measure useful gender knowledge. |
| Outcome: | The proposed dataset aims to quantify gender biases and to assess their impact on useful gender knowledge. |
Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times. |
| Approach: | They propose to evaluate surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. |
| Outcome: | The proposed model variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, while smaller models show a ‘tipping point’ at convergence after the decrease in language model perplexity . |
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing explanation datasets for large language models are limited to the English language and general domain, leading to a scarcity of linguistic diversity and a lack of resources in specialized domains, such as medical. |
| Approach: | They propose to use a medical dataset to assess the interpretability of Large Language Models (LLMs) . they propose to analyze medical text and generate rationales for their decisions . |
| Outcome: | The proposed model passes the pharmacist examination with a 75.7% accuracy, while other models like ChatGPT fail. |
CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles (2023.findings-emnlp)
Copied to clipboard
| Challenge: | CLASS empowers ITS with two key capabilities: first, it equips it with essential problem-solving strategies, and second, it facilitates natural language interactions, fostering engaging student-tutor conversations. |
| Approach: | They propose a design framework called Conversational Learning with Analytical Step-by-Step Strategies (CLASS) that empowers ITS with two key capabilities: first, a carefully curated dataset and second, facilitating natural language interactions. |
| Outcome: | The proposed framework empowers ITS with two key capabilities: first, it equips it with essential problem-solving strategies, and second, it facilitates natural language interactions, fostering engaging student-tutor conversations. |
Normal-Abnormal Decoupling Memory for Medical Report Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for capturing nuanced visual information are prone to data bias and noise. |
| Approach: | They propose a normal-abnormal semantic decoupling network that utilizes abnormal pattern memory to optimize visual extraction through the extraction of abnormal semantics from the reports. |
| Outcome: | The proposed approach surpasses the current state-of-the-art methods on the benchmark MIMIC-CXR and shows excellent performance on the same dataset. |
mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models display surprising multilingual capabilities despite being pre-trained on English data. |
| Approach: | They propose a multilingual sequence-to-sequence model that disentangles language-specific information from language-agnostic information. |
| Outcome: | The proposed model outperforms existing models on representative natural language understanding and generation tasks in 40+ languages. |
ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are becoming general-purpose APIs, requiring visual knowledge to be understood. |
| Approach: | They propose to evaluate the visual capability of large-scale large-language models through visual commonsense evaluation using a human-annotated dataset. |
| Outcome: | The proposed dataset compares the visual commonsense knowledge of large-scale models with those of unimodal LLMs and visually augmented models. |
MultiCoNER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a core task in Natural Language Processing. |
| Approach: | They present a dataset for fine-grained Named Entity Recognition covering 33 entity classes across 12 languages in monolingual and multilingual settings. |
| Outcome: | The proposed dataset covers 33 entity classes across 12 languages in monolingual and multilingual settings. |
A Query-Parallel Machine Reading Comprehension Framework for Low-resource NER (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental task in natural language processing. |
| Approach: | They propose a query-parallel MRC-based approach to named entity recognition . the model is trained with parameter-efficient tuning technique, making it more data-efficient . |
| Outcome: | The proposed model performs competitively against strong baseline methods in resource-rich settings and achieves state-of-the-art results in low-resource settings. |
BiSPN: Generating Entity Set and Relation Set Coherently in One Pass (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to extract entities and relation triples from text are limited. |
| Approach: | They propose a bipartite set prediction network to generate entity set and relation set in parallel. |
| Outcome: | The proposed model can generate entity set and relation set in parallel, while maintaining coherence between the predicted entities and relation sets. |
MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for engagingness evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. |
| Approach: | They propose to use large language models to evaluate engagingness in dialogue . they propose to include prompts and translated prompts in the model . |
| Outcome: | The proposed model outperforms existing methods on evaluation of engagingness in dialogue across languages. |
Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing financial PLMs are not pretrained on sufficiently diverse financial data, leading to subpar generalization performance. |
| Approach: | They propose to pretrain financial PLMs on financial corpus and train financial models on financial data. |
| Outcome: | The proposed financial language models outperform existing financial PLMs on financial tasks even for unseen corpus groups. |
LLMDet: A Third Party Large Language Models Generated Text Detection Tool (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing detection tools rely on access to LLMs and can only distinguish between machine-generated and human-authored text. |
| Approach: | They propose a model-specific, secure, efficient, and extendable detection tool that can source text from specific LLMs. |
| Outcome: | The proposed tool can source text from specific LLMs, such as GPT-2, OPT, LLaMA, and others. |
RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have focused on producing concise observations while neglecting the precise attributes that determine the severity of diseases. |
| Approach: | They propose a model that generates precise radiology reports via dynamic disease progression reasoning by combining historical and spatiotemporal information. |
| Outcome: | Experiments on two publicly available datasets show the proposed model can generate precise and accurate radiology reports with dynamic disease progression reasoning. |
Causal Intervention for Abstractive Related Work Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models ignore the inherent causality during related work generation, leading to spurious correlations which downgrade the models’ generation quality and generalizability. |
| Approach: | They propose a Causal Intervention Module for Related Work Generation (CaM) that captures causal relationships in related work generation and implements causal interventions to mitigate the negative impact of spurious correlations. |
| Outcome: | The proposed framework improves the quality and coherence of generated related work by capturing causalities in the generation process. |
G-SPEED: General SParse Efficient Editing MoDel (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated incredible capabilities in understanding, generating, and manipulating languages. |
| Approach: | They propose a general SParse Efficient Editing MoDel which can fulfill diverse editing requirements through a single model while maintaining low computational costs. |
| Outcome: | The proposed model can fulfill diverse editing requirements through a single model while maintaining low computational costs. |
Attack Prompt Generation for Red Teaming and Defending Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies construct attack prompts via manual or automatic methods, but these methods have limitations on cost and quality. |
| Approach: | They propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning and a defense framework that fine-tunes victim LLM's through iterative interactions with the attack framework. |
| Outcome: | The proposed approach is based on experiments on different LLMs to evaluate their effectiveness against red teaming attacks. |
Smart “Chef”: Verifying the Effect of Role-based Paraphrasing for Aspect Term Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect Term Extraction (ATE) is a task of automatically extracting aspect terms from sentences. |
| Approach: | They propose to automatically rewrite sentences from virtual experts with different roles . they leverage ChatGPT to determine virtual experts in the considered domains . |
| Outcome: | The proposed method can be used to expand the predictions obtained on the original sentences without retraining or fine-tuning the baseline extractors. |
Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning (2023.findings-emnlp)
Copied to clipboard
Yougang Lyu, Jitai Hao, Zihan Wang, Kai Zhao, Shen Gao, Pengjie Ren, Zhumin Chen, Fang Wang, Zhaochun Ren
| Challenge: | Existing methods for predicting judgment results for multiple defendants are ineffective. |
| Approach: | They propose a method to predict the judgment results for each defendant in multi-defendant cases . they formalize the multi-diffendant judgment process as hierarchical reasoning chains . |
| Outcome: | The proposed method can predict the judgment results for multiple defendants in multi-defendant cases. |
Interpreting Indirect Answers to Yes-No Questions in Multiple Languages (2023.findings-emnlp)
Copied to clipboard
Zijie Wang, Md Hossain, Shivam Mathur, Terry Melo, Kadir Ozler, Keun Park, Jacob Quintero, MohammadHossein Rezaei, Shreya Shakya, Md Uddin, Eduardo Blanco
| Challenge: | Existing models for Yes-no questions skip polar keywords and instead use long explanations that must be interpreted. |
| Approach: | They propose a distant supervision approach to collect training data and show that direct answers are useful to train models to interpret indirect answers. |
| Outcome: | The proposed model achieves a 68% to 76% F1-score on multilingual Question-Answering benchmarks. |
Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Few-shot named entity recognition methods struggle with out-of-domain (OOD) examples due to their reliance on manual labeling for the target domain. |
| Approach: | They propose a framework to enable generalization to an unseen target domain with only a few labeled examples. |
| Outcome: | The proposed framework achieves significant performance improvements on in-domain and cross-domain datasets. |
Intervention-Based Alignment of Code Search with Execution Feedback (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing code search training datasets approximate text-code co-occurrences as positive execution feedback, but this approximation may misalign models’ retrieval decisions from ground-truth correctness. |
| Approach: | They propose a code intervention-based reinforcement learning approach that perturbs training code to result in misalignment, then tests models’ decisions and corrects them with the execution feedback by reinforcement learning. |
| Outcome: | The proposed method induces the execution feedback from perturbation, without actual execution, and then tests models’ decisions and corrects them with the execution input by reinforcement learning. |
Enhancing Neural Machine Translation with Semantic Units (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to model and leverage semantic units in natural language do not provide a complete understanding of the whole sentence. |
| Approach: | They propose a method which models the integral meanings of semantic units within a sentence . they propose 'word pair encoder' to help identify the boundaries of semantic unit boundaries . |
| Outcome: | The proposed method outperforms baselines and supports the semantic unit representation of subwords and tokens. |
DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot topic classification are limited due to the volume of information pouring in from the Internet . a new framework is proposed to train a classifier for few shot topics . |
| Approach: | They propose a framework to train a classifier for few-shot topic classification using a customized dataset and a dense retriever model. |
| Outcome: | The proposed framework shows superior performance on few-shot topic classification tasks compared to baselines that use in-context learning . |
A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge. |
| Approach: | They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. |
| Outcome: | The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players. |
Generative Calibration for In-context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning is one of the most exciting features of large language models . performance is sensitive to various configurations of the prompt, such as the choice or order of the training examples. |
| Approach: | They propose to calibrate the in-context predictive distribution by adjusting the label marginal . they find that the proposed method outperforms the ICL and state-of-the-art calibration methods . |
| Outcome: | The proposed method outperforms state-of-the-art methods by 27% absolute in macro-F1. |
Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to few-shot relation extraction require training. |
| Approach: | They propose a method for few-shot relation extraction using large language models, called CoT-ER, chain-of-thought with explicit evidence reasoning. |
| Outcome: | The proposed approach achieves competitive performance compared to the fully-supervised state-of-the-art approach on the FewRel1.0 and FewRela2.0 datasets. |
AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end speech translation (ST) models need large amount of training data to perform well. |
| Approach: | They propose a shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. |
| Outcome: | The proposed method achieves better performance on the MUST-C dataset, with higher inference speed and lower memory usage. |
No offence, Bert - I insult only humans! Multilingual sentence-level attack on toxicity detection networks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new sentence-level attack on toxic detection models is shown to work on seven languages . toxicity detection systems are used to silence the voices of criticism, causing echo chambers . |
| Approach: | They propose a sentence-level attack that adds positive words to a hateful message . they show the attack works on seven languages from three different language families . |
| Outcome: | The proposed attack is shown to work on seven languages from three different language families. |
Manipulating the Perceived Personality Traits of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Psychology research has long explored aspects of human personality like extroversion, agreeableness and emotional stability, three of the personality traits that make up the ‘Big Five’. |
| Approach: | They propose to use text generated from large language models to evaluate perceived personality traits and to frame them as tools for controlling personas in dialog systems. |
| Outcome: | The proposed models predict personality traits in different contexts and can be manipulated in a predictable way. |
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new few-shot LLM-based chatbot is able to provide factual and engaging responses . a novel hybrid human-and-LLM evaluation methodology is used to evaluate the system . |
| Approach: | They propose a few-shot LLM-based chatbot that almost never hallucinates . they distill WikiChat into a 7B-parameter LLaMA model with minimal loss of quality . |
| Outcome: | The proposed system outperforms retrieval-based and LLM-based systems on the Wikipedia corpus. |
Automated Few-Shot Classification with Instruction-Finetuned Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing few-shot learning approaches combine language models with prompts, but they often require domain knowledge and substantial guesswork. |
| Approach: | They propose a method to eliminate the need for handcrafted prompts by generating two distinct, semantically meaningful class descriptions and a selection mechanism via cross-validation. |
| Outcome: | The proposed method outperforms state-of-the-art few-shot learning methods over 12 datasets, spanning 8 classification tasks. |
Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language-Model-as-a-Services (LMaaSs) support a variety of user tasks through in-context learning from prompts. |
| Approach: | They propose a lightweight automatic prompt generation method that meta-trains a prompt generation model to enable robust learning from the contexts created by the generated prompts. |
| Outcome: | The proposed method improves performance on unseen tasks by 19.4% compared to the state-of-the-art prompt generation method. |
Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on word analogies, but they neglect structures that underpin analogical reasoning. |
| Approach: | They propose a task to abduct structures that form an analogy between two systems to evaluate their analogical reasoning abilities. |
| Outcome: | The proposed task is based on 400 scientific analogies from 13 different fields and is compared with a standard SCAR benchmark. |
HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods that encode a sequence in its entirety for contrast with others often neglect local representation learning. |
| Approach: | They propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness. |
| Outcome: | The proposed framework improves training efficiency and effectiveness by dividing a sequence into several segments and using local and global contrastive learning to model relationships. |
Density-Aware Prototypical Network for Few-Shot Relation Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies treat NOTA as an extra class and treat it the same as known relations. |
| Approach: | They propose a density-aware prototypical network to treat various instances distinctly . they separate known instances and isolate NOTA instances, respectively . their code will be made public after the paper is accepted . |
| Outcome: | The proposed method outperforms strong baselines with robustness towards different NOTA rates. |
Improved Training of Deep Text Clustering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for deep clustering optimization with shallow models have limited performance due to poor power of feature learning. |
| Approach: | They propose a general deep clustering optimization method that leverages information feedback to construct generalized labels to optimize the deep model. |
| Outcome: | The proposed method reduces the impact of noise on the clustering process by using correlation relationship between the samples. |
RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on retrieval-augmented language models has two main problems: determining what information to retrieve and effectively combining retrieved information during generation. |
| Approach: | They propose a retrieval-augmented language model that captures current and future information from source and target text into a latent space. |
| Outcome: | The proposed model is more efficient than explicit raw text, but limited by context length and noise. |
RefGPT: Dialogue Generation of GPT, by GPT, and for GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have impressive capability to resolve a wide range of NLP tasks by fine-tuning high-quality instruction data. |
| Approach: | They propose a method to generate huge truthful and customized dialogues without worrying about factual errors caused by the model hallucination. |
| Outcome: | The proposed method solves the model hallucination in dialogue generation by restricting the LLMs to leverage the given reference instead of reciting their own knowledge to generate dialogues. |
INA: An Integrative Approach for Enhancing Negotiation Strategies with Reward-Based Dialogue Agent (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a novel negotiation agent is designed for the online marketplace . a dialogue agent can negotiate on price and other factors . |
| Approach: | They propose a novel negotiation agent that is integrative in nature and can negotiate on price and other factors. |
| Outcome: | The proposed agent is integrative in nature and can negotiate on price and other factors. |
Large Language Models are Better Reasoners with Self-Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve complex natural language processing tasks require multiple steps to verify the answers. |
| Approach: | They propose to use chain of thought prompting to solve reasoning tasks with large language models. |
| Outcome: | The proposed method can improve reasoning performance on arithmetic, commonsense, and logical reasoning datasets. |
Multi-Granularity Information Interaction Framework for Incomplete Utterance Rewriting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches in Incomplete Utterance Rewriting (IUR) fail to capture the source of important words, introducing words from irrelevant utterances. |
| Approach: | They propose a framework to capture the multi-granularity of semantic information and fetch the relevant utterance. |
| Outcome: | The proposed framework outperforms state-of-the-art models on two benchmark datasets . it can capture the source of important words and fetch the relevant utterance . |
Accuracy is not enough: Evaluating Personalization in Summarizers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing accuracy measures cannot evaluate the degree of personalization of summarization models. |
| Approach: | They propose to use a PENS dataset to analyze the degree of personalization of ten different summarization models. |
| Outcome: | The proposed measure can evaluate the degree of personalization of summarization models using the PENS dataset. |
For Generated Text, Is NLI-Neutral Text the Best Text? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a perfectly informative agent would eschew utterances that are redundant or contradict that which they have already said. |
| Approach: | They propose to use a pre-trained NLI model to assess whether a sentence entails, contradicts, or is neutral to prompt and preceding text. |
| Outcome: | The proposed model is predictive of generation errors made by GPT-3. |
Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experiments on multiple data sets show that the proposed approach consistently improves performance and outperforms several alternative methods. |
| Approach: | They propose to integrate a text classifier into an existing TAR stopping rule to train it without the need for additional annotations. |
| Outcome: | Experiments on multiple data sets show the proposed approach outperforms other methods and achieves the desired level of recall with a lower cost than the existing method based on counting processes alone. |
Complexity-Guided Curriculum Learning for Text Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Curriculum learning is a systematic approach to training that refines training progressively and tailors training to task requirements. |
| Approach: | They propose a curriculum learning approach that employs "spaced repetition" and complexity formalisms to guide the training process. |
| Outcome: | The proposed model gains more and uses less data, and the best curricula are equally effective. |
CoVariance-based Causal Debiasing for Entity and Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition and Relation Extraction are key tasks of Information Extraction. |
| Approach: | They propose a causal framework called c ovariance and variance optimization framework (OVO) to optimize feature representations and conduct general debiasing. |
| Outcome: | The proposed framework minimizes characterizing features’ covariance for alleviating selection and distribution bias and enhances feature representation in the feature space. |
Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for data-driven annotations require domain-specific and task-aligned supervision. |
| Approach: | They propose a multi-label and multi-target sampling strategy to optimize the annotation quality. |
| Outcome: | The proposed method significantly improves performance and learning efficacy on the benchmark stance detection corpora. |
In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual generative language models (LMs) are fluent in a large variety of languages. |
| Approach: | They analyze formality distributions of XGLM and BLOOM’s predictions in 5 languages and classify 1,200 generations per language as formal, informal, or incohesive. |
| Outcome: | The proposed models generate a significant amount of informal predictions even when prompted with formal text. |
MaXM: Towards Multilingual Visual Question Answering (2023.findings-emnlp)
Copied to clipboard
Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, Radu Soricut
| Challenge: | Visual Question Answering (VQA) has been studied in the English language, but in other languages it would require a considerable amount of resources. |
| Approach: | They propose scalable solutions to multilingual visual question answering using an English language framework and an annotation protocol. |
| Outcome: | The proposed framework reduces human annotation efforts and creates a test-only VQA benchmark in 7 languages. |
Efficient Latent Variable Modeling for Knowledge-Grounded Dialogue Generation (2023.findings-emnlp)
Copied to clipboard
Gunsoo Han, Daejin Jo, Daniel Nam, Eunseop Yoon, Taehwan Kwon, Seungeun Rho, Kyoung-Woon On, Chang Yoo, Sungwoong Kim
| Challenge: | Existing knowledge-grounded dialogue generation algorithms require annotated knowledge to generate a response grounded on the retrieved knowledge. |
| Approach: | They propose an efficient algorithm for latent variable modeling that leverages large amount of dialogue data. |
| Outcome: | The proposed algorithm outperforms the supervised learning algorithm on knowledge-grounded dialogue datasets while maintaining efficiency and scalability. |
Ask To The Point: Open-Domain Entity-Centric Question Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | *entity-centric question generation (ECQG) is a task motivated by real-world applications such as topic-specific learning, assisted reading, and fact-checking. |
| Approach: | They propose a PLM-based framework GenCONE with two modules: content focusing and question verification. |
| Outcome: | The proposed framework outperforms baselines and is effective and complementary in generating high-quality questions. |
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open-domain question-answering methods lack quality assurance . existing methods lack scalability and poor diversity, hindering LLMs' capabilities . |
| Approach: | They propose an open-domain multi-hop reasoning framework to answer multi-choice questions . they propose an adaptive sampler for in-context selection and self-prompted inference . |
| Outcome: | The proposed framework surpasses the existing SOTA methods on large-scale datasets and doubles the zero-shot performance of small-scale LLMs. |
CASE: Commonsense-Augmented Score with an Expanded Answer Space (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive zero-shot performance on NLP tasks thanks to the knowledge they acquired in their training. |
| Approach: | They propose a Commonsense-Augmented Score with an Expanded Answer Space that assigns importance weights to words based on their semantic relations to other words in the input. |
| Outcome: | The proposed approach outperforms basic LM scores on 5 commonsense benchmarks and is complementary to previous approaches. |
GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for self-supervised representation learning on text-attributed graphs lack the full extent of structural context information or rely on task-specific training labels. |
| Approach: | They propose a Graph-Centric Language model that harnesses the synergy of pre-trained language model and graph neural network to optimize with graph-centric contrastive learning and graph-centered knowledge alignment. |
| Outcome: | The proposed model captures informative textual semantics as well as structural context information on text-attributed graphs. |
Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) |
| Approach: | They propose to use LLMs to probe their behavior using controlled experiments. |
| Outcome: | The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do. |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance textual entity prediction neglect the need for external knowledge or encounter high redundancy in the retrieved knowledge. |
| Approach: | They propose a framework that leverages ChatGPT as an implicit knowledge base and heuristically generates auxiliary knowledge for more efficient entity prediction. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two classic datasets and exhibits a stronger robustness and generalization capability. |
Understanding HTML with Large Language Models (2023.findings-emnlp)
Copied to clipboard
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust
| Challenge: | Large language models have shown exceptional performance on a variety of natural language tasks, but their capabilities for HTML understanding have not been fully explored. |
| Approach: | They propose to use HTML understanding models to parse HTML and perform HTML navigation tasks with a large-scale HTML dataset. |
| Outcome: | The proposed models perform 50% more tasks with 192x less data than the previous best supervised model. |
The PEACE-Reviews dataset: Modeling Cognitive Appraisals in Emotion Text Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have delved into its significance, yet the interplay between various forms of cognitive appraisal and specific emotions, such as joy and anger, remains an area of exploration in consumption contexts. |
| Approach: | They propose to construct a dataset to model the evaluations people make about their situations based on annotated autobiographical accounts of their emotional and appraisal experiences . |
| Outcome: | The proposed model incorporates emotion, cognition, individual traits, and demographic data. |
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model (2023.findings-emnlp)
Copied to clipboard
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, Fei Huang
| Challenge: | Existing studies for visually-situated language understanding have shown shallow zero-shot visual text recognition ability when fed a low-resolution image with salient text information. |
| Approach: | They propose a model for universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM) their model is jointly finetuned on a wide range of visually situated language understanding tasks via a unified instruction format. |
| Outcome: | The proposed model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks across 5 domains: documents, tables, charts, natural images, and webpage screenshots. |
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results prove that language models can learn from human feedback better, irrespective of sequence length . emergence of length bias often induces the model to favor longer outputs . |
| Approach: | They propose to separate reward modeling from the influence of sequence length by using the Product-of-Experts technique. |
| Outcome: | The proposed approach shows that language models perform better regardless of sequence length . the main expert is focused on understanding human intents, while the biased expert targets the identification and capture of length bias. |
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge in natural language tasks. |
| Approach: | They propose a framework that enables LLMs to ask relevant questions to uncover more details in the image, along with filters for refining the generated information. |
| Outcome: | The proposed framework boosts the performance of baseline methods by 2.15% on OK-VQA and achieves consistent improvements across different LLMs. |
Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated remarkable cross-lingual capability of pre-trained language models . however, semantic alignments may be the reason behind such capability but remain under-explored. |
| Approach: | They propose token-level and semantic-level code-switched masked language modeling to improve cross-lingual interactions over mono-mPLMs without parallel sentences. |
| Outcome: | The proposed method outperforms mono-mPLMs on natural language understanding and unsupervised machine translation tasks. |
LogiCoT: Logical Chain-of-Thought Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on self-instruction tuning has focused on enhancing the general proficiency of models. |
| Approach: | They propose a new instruction-tuning dataset for Logical Chain-of-Thought reasoning with GPT-4 that harvests instructions for prompting GPT to generate chain-of thought rationales. |
| Outcome: | The proposed dataset enables the model to generate chain-of-thought rationales with GPT-4. |
Hiding in Plain Sight: Tweets with Hate Speech Masked by Homoglyphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, there are almost 150,000 Unicode characters, which presents extensive substitution possibilities. |
| Approach: | They develop a character substitution scraping method to collect hate speech . they use an annotated dataset with 1,281 non-Latin characters to scrape out offensive words . |
| Outcome: | The proposed method can detect hate speech with annotated data, but it performs poorly in a zero-shot setting. |
Reducing Spurious Correlations in Aspect-based Sentiment Analysis with Explanation from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis models are susceptible to learning spurious correlations between words . a recent study shows that feature engineering is time-consuming and costly . |
| Approach: | They propose to use a template to prompt LLMs to generate an appropriate explanation for the sentiment polarity of each aspect to reduce spurious correlations. |
| Outcome: | The proposed methods improve ABSA models and their generalization ability. |
High-quality argumentative information in low resources approaches improve counter-narrative generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning improves the performance of language models . large language models generate acceptable texts in a number of scenarios, a study shows . |
| Approach: | They show that fine-tuning improves the task of hate speech counter-narrative generation . they provide a subset of arguments and a good base model is required for the fine-uning to have a positive impact. |
| Outcome: | The proposed model produces counter-narratives that are as satisfactory as the whole set. |
A Reference-free Segmentation Quality Index (SegReFree) (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Topic segmentation is the process of finding boundaries in sentences that separate groups of adjacent sentences at shifts in semantic meaning. |
| Approach: | They propose a reference-free segmentation quality index to assess segmentation . metric uses a modified cluster validity metric with semantic embeddings of the sentences . |
| Outcome: | The proposed metric uses a modified cluster validity metric with semantic embeddings of the sentences to determine the quality of the segmentation. |
In-context Learning for Few-shot Multimodal Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition are time-consuming and laborintensive. |
| Approach: | They propose a few-shot multimodal named entity recognition task that uses few examples to locate and identify named entities for a text-image pair. |
| Outcome: | The proposed framework outperforms baselines under several few-shot settings. |
On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications. |
| Approach: | They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty. |
| Outcome: | The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. |
Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on sign language recognition encodes videos without acknowledging phonological attributes of signs. |
| Approach: | They propose a single-encoder network and a dual-encoding network for handshape-inclusive sign language recognition. |
| Outcome: | The proposed methods outperform baseline methods in the PHOENIX14T-HS dataset . the proposed methods consistently outperformed baseline methods . |
SimCKP: Simple Contrastive Learning of Keyphrase Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for keyphrase generation and keyphrase extraction use a token level to generate keyphrases that do not appear in a document. |
| Approach: | They propose a simple contrastive learning framework that generates keyphrases that do not appear in a document and a reranker that adapts the scores for each generated phrase. |
| Outcome: | The proposed model outperforms the state-of-the-art models on multiple benchmark datasets. |
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in legal NLP have led to a rapid growth of the field . however, many benchmarks are available only in English and no multilingual benchmark exists . |
| Approach: | They propose to use 11 datasets covering 24 languages to compare NLP models. |
| Outcome: | The proposed benchmarks show that even the best baseline only achieves modest results and ChatGPT struggles with many tasks. |
Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | After the pandemic, e-learning has become part of mainstream education. |
| Approach: | They propose to integrate large language models (LLMs) into educational settings to enhance students' mathematical problem-solving skills by providing adaptive feedback. |
| Outcome: | The proposed model can generate free-text rationalizations and misinterpret meanings and can also misinterprét students' answers. |
Simultaneous Machine Translation with Tailored Reference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing SiMT models are trained using the same reference disregarding the varying amounts of available source information at different latency. |
| Approach: | They propose a method that provides tailored reference for the SiMT models trained at different latency by rephrasing ground-truth to the tailored reference. |
| Outcome: | The proposed method achieves state-of-the-art translation performance on three translation tasks. |
Dynamic Voting for Efficient Reasoning in Large Language Models (2023.findings-emnlp)
Copied to clipboard
Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Xingzhang Ren, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, Jiancheng Lv
| Challenge: | Multi-path voting methods generate multiple reasoning paths for each problem, causing factual errors and illusion generation. |
| Approach: | They propose a multi-path voting technique that effectively reduces the number of reasoning paths during multi-path voting while preserving accuracies. |
| Outcome: | The proposed method outperforms Self-consistency using 24.7% of the number of paths on the LetterConcat task. |
On Surgical Fine-tuning for Language Encoders (2023.findings-emnlp)
Copied to clipboard
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan
| Challenge: | preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements. |
| Approach: | They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning. |
| Outcome: | The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders. |
AutoPlan: Automatic Planning of Interactive Decision-Making Tasks With Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for making decisions in grounded environments require costly gradient computation or lengthy in-context demonstrations. |
| Approach: | They propose an approach to guide LLM-based agents to accomplish interactive decision-making tasks by using an LLM prompt and a task-solving plan. |
| Outcome: | The proposed approach outperforms human-written demonstrations on ALFWorld and HotpotQA by 8%. |
Measuring Faithful and Plausible Visual Grounding in VQA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lack of visual grounding (VG) in VQA systems can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. |
| Approach: | They propose a new metric that measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
| Outcome: | The proposed metric measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) enable zero-shot approaches in open domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. |
| Approach: | They propose to use a distraction-aware answer selection framework to mitigate the impact of irrelevant documents in the retrieved set and the overconfidence of the generated answers to enhance the performance of zero-shot readers. |
| Outcome: | The proposed approach handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. |
Can you Summarize my learnings? Towards Perspective-based Educational Dialogue Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Increasing use of virtual tutors has allowed for more efficient, personalized, and interactive AI-based learning experiences. |
| Approach: | They propose a task of Multi-modal Perspective based Dialogue Summarization (MM-PerSumm) that summarizes educational dialogues from three unique perspectives: the Student, the Tutor, and a Generic viewpoint. |
| Outcome: | The proposed model can summarize educational dialogues from three perspectives, while student-oriented summaries should distill learning points, track progress, and suggest scope for improvement. |
Adaptive Textual Label Noise Learning based on Pre-trained Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning with noisy labels are limited due to the time and labor costs involved. |
| Approach: | They propose an adaptive warm-up and hybrid training frameworks to learn with noisy labels based on pre-trained models. |
| Outcome: | The proposed approach performs comparable or even surpasses state-of-the-art methods in various noise scenarios, including scenarios with the mixture of multiple types of noise. |
Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have impressive capabilities in open-ended text generation. |
| Approach: | They propose a dynamic knowledge-guided informative open-ended text generation approach that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts. |
| Outcome: | The proposed approach generates more informative texts than baselines. |
Novel Relation Detection: Discovering Unknown Relation Types via Multi-Strategy Self-Supervised Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to relation extraction can only recognize predefined relation types . new or out-of-scope relation types may continually emerge after the model is deployed . |
| Approach: | They propose a novel relation detection task that uses self-supervised learning to handle shallow semantic similarity problem. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two datasets. |
Ask Language Model to Clean Your Noisy Translation Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models exhibit a noticeable decline in translation quality when exposed to noisy input. |
| Approach: | They use a dataset to evaluate the robustness of NMT models against noisy inputs. |
| Outcome: | The proposed dataset cleaners the noise from the target sentences while preserving the semantic integrity of the original sentences. |
Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users (2023.findings-emnlp)
Copied to clipboard
Yohan Jo, Xinyan Zhao, Arijit Biswas, Nikoletta Basiou, Vincent Auvray, Nikolaos Malandrakis, Angeliki Metallinou, Alexandros Potamianos
| Challenge: | a dataset of task-oriented dialogues assume conversations between the agent and one user at a time . but multi-user task-orientated dialogues are richer, containing deliberation and deliberations . a novel task is proposed to rewrite a task-focused query that retains only task-relevant information . |
| Approach: | They propose to rewrite a task-oriented chat between two users as a concise task-orientated query that retains only task-relevant information and is directly consumable by the dialogue system. |
| Outcome: | The proposed method surpasses existing models on multi-user dialogues and generalizes to unseen domains. |
Extractive Summarization via ChatGPT for Faithful Summary Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summarization methods struggle with generating ungrammatical or even nonfactual contents. |
| Approach: | They evaluate ChatGPT's performance on extractive summarization and compare it with traditional fine-tuning methods on benchmark datasets. |
| Outcome: | The proposed pipeline performs better than abstractive methods on summary faithfulness and in-context learning. |
MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization (2023.findings-emnlp)
Copied to clipboard
Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, Yanghua Xiao
| Challenge: | Existing research emphasizes the importance of adapting prompts to specific tasks, rather than specific LLMs. |
| Approach: | They propose a model-adaptive prompt optimizer method that optimizes original prompts for each LLM in downstream tasks. |
| Outcome: | The proposed method can optimize prompts for an LLM in downstream tasks. |
PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable zero-shot performance across various NLP tasks. |
| Approach: | They propose a method which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner and prompts an LLM to rate individual items at each turn. |
| Outcome: | The proposed method improves the performance and robustness of the standard GPT-3.5 personality detection task on two benchmark datasets. |
Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have shattered the ceiling of human-like text generation. |
| Approach: | They compared human-AI interaction types in LLM-assisted news headline generation to determine whether humans can best leverage them for writing. |
| Outcome: | The guiding and selecting model outputs added the most benefit with the lowest cost (in time and effort) Furthermore, AI assistance did not harm participants’ perception of control compared to freeform editing. |
NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a widely adopted NLP task . authors present three variants of NER task, with dataset to support them . |
| Approach: | They propose three variants of the NER task, together with a dataset to support them . they propose a move towards more fine-grained entities and zero-shot recognition . |
| Outcome: | The proposed model matches or surpasses existing models in NER tasks . the proposed model is based on a large, silver-annotated corpus of 4 million paragraphs . |
SWEET - Weakly Supervised Person Name Extraction for Fighting Human Trafficking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | SWEET is a weak supervision pipeline for extracting person names from noisy escort ads . it does not require any human annotators and labeling, which is incredibly important . |
| Approach: | They propose a weak supervision pipeline SWEET: Supervise Weakly for Entity Extraction to fight Trafficking for extracting person names from noisy escort ads. |
| Outcome: | The proposed weak supervision pipeline outperforms the previous method by 9% on domain data and generalizes to common benchmark datasets. |
Watermarking LLMs with Weight Quantization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are being deployed at an astonishing speed, exposing users to high risks. |
| Approach: | They propose a method that plants watermarks in quantization process of large language models without pre-defined triggers during inference. |
| Outcome: | The proposed method protects model weights without pre-defined triggers . it works when the model is used in the fp32 mode and remains hidden when the models are quantized to int8 . |
Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies highlight the struggles even large language models encounter when it comes to performing spatial reasoning over text. |
| Approach: | They propose to disentangle spatial reasoning over text and compare them to state-of-the-art models with no explicit design for these parts. |
| Outcome: | The proposed models show that they can perform spatial reasoning over text and can generalize within real data domains. |
PsyAttention: Psychological Attention Model for Personality Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Personality detection has incorporated psychological features from different personality models, such as the BigFive and MBTI. |
| Approach: | They propose to use psychological models to encode personality features to reduce their number by 85%. |
| Outcome: | The proposed model outperforms state-of-the-art methods on the BigFive and MBTI models and achieves average accuracy of 65.66% and 86.30%, respectively. |
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training (2023.findings-emnlp)
Copied to clipboard
Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, Madian Khabsa
| Challenge: | Several perspectives of robustness for pre-trained language models have been studied independently, but lacking a unified consideration in multiple perspectives. |
| Approach: | They propose a technique to enhance the multi-perspective robustness of LMs by introducing adversarial perturbation while the model parameters are selectively updated upon their relative importance. |
| Outcome: | The proposed technique improves the robustness of LMs by incorporating four perspectives on model robustness. |
The Law and NLP: Bridging Disciplinary Disconnects (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Legal practitioners and scholars have been slow to adopt tools from natural language processing (NLP) the legal system is experiencing an access to justice crisis, which could be partially alleviated with NLP. |
| Approach: | They argue that legal practitioners are slow to adopt natural language processing (NLP) they argue that there is a disconnect between legal needs and NLP research . |
| Outcome: | The proposed tasks bridge disciplinary disconnects and highlight interesting areas for legal NLP research that remain underexplored. |
Symbolization, Prompt, and Classification: A Framework for Implicit Speaker Identification in Novels (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for speaker identification in novel dialogues are limited to handling explicit narrative patterns and complex cases. |
| Approach: | They propose a framework which identifies implicit speakers in novels via symbolization, prompt, and classification. |
| Outcome: | The proposed framework outperforms existing methods by 4.8% accuracy on the web novel collection, which reduces 47% of speaker identification errors, and outperfies the emerging ChatGPT. |
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained and frozen LLMs can effectively map simple scene re-arrangement instructions to programs over a robot’s visuomotor functions, but fixed prompts fall short. |
| Approach: | They propose an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting. |
| Outcome: | The proposed agent parses human-robot dialogue into action programs using retrieval-augmented LLM prompting. |
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have focused on the development of semantic parsers within the framework of cross-domain analysis. |
| Approach: | They propose a method to generate auto-CoT exemplars using ACT-SQL and extend it to multi-turn text-to-Sql tasks. |
| Outcome: | The proposed method achieves SOTA performance on the Spider dev set among existing in-context learning approaches. |
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-head self-attention-based Transformers have shown promise in different learning tasks . but encoders of Transformers and their variants fail to preserve layer-wise contextual information . |
| Approach: | They propose an encoder model that guarantees a theoretical bound for layer-wise distance preservation between a pair of tokens. |
| Outcome: | The proposed model preserves equivalence between tokens and performs better than Transformers. |
ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unlike previous methods, large language models produce entity-agnostic templates instead of copying the given example entities or validating/crafting the templates manually. |
| Approach: | They propose an approach for structured data verbalisation into short template sentences in zero to few-shot settings that prompts Large Language Models to directly produce entity-agnostic templates. |
| Outcome: | The proposed approach averages 66% parsing error rate reduction in generated verbalisations of RDF triples on the DART dataset. |
Detecting Syntactic Change with Pre-trained Transformer Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a fine-tuned BERT model can distinguish between text from the early 1800s and late 1900s . we use it to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. |
| Approach: | They propose to use a BERT-based model to find syntactic differences between English of the early 1800s and that of the late 1900s. |
| Outcome: | The proposed model can distinguish between English of the early 1800s and that of the late 1900s using only syntactic information. |
Can Word Sense Distribution Detect Semantic Changes of Words? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to detect semantic variations of words are not accurate for time-sensitive predictions. |
| Approach: | They propose to use pretrained static sense embeddings to annotate a word's occurrence with a sense id to compare its distributions. |
| Outcome: | The proposed method compares word sense distributions across two corpora to predict meaning change . the results show that pretrained LLMs can detect changes in words over time . |
Gold: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to construct CSKGs with large semantic coverage are expensive and introduce spurious noise. |
| Approach: | They propose a denoising framework that incorporates entity semantic information, global rules, and local structural information from the CSKG. |
| Outcome: | The proposed framework outperforms baseline methods in noise detection tasks on synthetic noisy CSKG benchmarks. |
Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conversational Recommendation System (CRS) is a rapidly growing research area, along with advancements in language modelling techniques. |
| Approach: | They propose to use a benchmark dataset to develop CRS models and address biases arising from feedback loop inherent in multi-turn interactions to enhance model performance while mitigating biase. |
| Outcome: | The proposed strategies improve on ReDial and TG-ReDial benchmark datasets and offer additional insights on addressing multiple newly formulated biases. |
Exploring Graph Pre-training for Aspect-based Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies tend to extract the sentiment elements in a generative manner to avoid complex modeling of sentiment elements. |
| Approach: | They propose a generative model with an Element-level Graph Pre-training paradigm and a Task Decomposition Pre- training paradigm to make it generalizable and robust against irregular sentiment quadruples. |
| Outcome: | The proposed model is generalizable and robust against irregular sentiment quadruples. |
DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Temporal Language Grounding (TLG) is a task to determine temporal boundaries of video moments that correspond to a language query. |
| Approach: | They propose an energy-based model framework to explicitly learn moment-query distributions. |
| Outcome: | The proposed model outperforms the state-of-the-art models on four public temporal language grounding datasets. |
Test-time Augmentation for Factual Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve factual probing are relation-specific and do not generalize to unseen relation types. |
| Approach: | They propose to use test-time augmentation to augment and ensemble prompts at test time to reduce sensitivity to prompt variations. |
| Outcome: | The proposed method improves model confidence, but for other models, it leads to degradation. |
Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study comparing static and contextualized language models for subtle semantic shifts in Dutch and English shows that they can detect political connotations and associations. |
| Approach: | They propose a method for detecting subtle semantic shifts between political communities in Dutch and English using static and contextualized language models. |
| Outcome: | The proposed method outperforms static models on a Russian and Spanish task . it relies on behavioral information, specifically the most probable substitutions, instead of geometrical comparison of representations. |
Disfluent Cues for Enhanced Speech Understanding in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large number of language models struggle to handle disfluencies, authors say . when a speaker hesitates, interrupts themselves, repeats or corrects words, or abandons phrases, it can make their speech fragmented. |
| Approach: | They propose to use disfluent queries to “clean” spontaneous speech . they propose to apply disfluencies to models that use different types of speech repairs . |
| Outcome: | The proposed model improves on a reading comprehension task using disfluent queries . the results suggest that disfluencies can improve model performance, rather than their removal . |
Watermarking PLMs on Classification Tasks by Combining Contrastive Learning with Weight Perturbation (2023.findings-emnlp)
Copied to clipboard
Chenxi Gu, Xiaoqing Zheng, Jianhan Xu, Muling Wu, Cenyuan Zhang, Chengsong Huang, Hua Cai, Xuanjing Huang
| Challenge: | Large pre-trained language models (PLMs) are highly valuable intellectual property due to their expensive training costs. |
| Approach: | They propose to embed backdoors that can be triggered by specific inputs into models by model watermarking. |
| Outcome: | The proposed method can be used to protect the intellectual property of large pre-trained language models without knowledge about downstream tasks. |
BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla Lemmatizer (2023.findings-emnlp)
Copied to clipboard
Sadia Afrin, Md. Shahad Mahmud Chowdhury, Md. Islam, Faisal Khan, Labib Chowdhury, Md. Mahtab, Nazifa Chowdhury, Massud Forkan, Neelima Kundu, Hakim Arif, Mohammad Mamun Or Rashid, Mohammad Amin, Nabeel Mohammed
| Challenge: | Lemmatization holds significance in both natural language processing (NLP) and linguistics due to the highly inflected nature and morphological richness of Bangla text. |
| Approach: | They propose linguistic rules for lemmatization and utilize a dictionary along with the rules to design a lemma specifically for Bangla. |
| Outcome: | The proposed system achieves 96.36% accuracy when tested against a manually annotated test dataset. |
Exploring the Sensitivity of LLMs’ Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies have compared the decision-making abilities of large language models with those of humans from a psychological perspective. |
| Approach: | They examine LLMs' performance on the Horizon decision-making task studied by Binz and Schulz (2023) they observe that the decision- making abilities fluctuate based on input prompts and temperature settings. |
| Outcome: | The results show that LLMs display a human-like exploration–exploitation tradeoff after simple adjustments to the prompt. |
Search Augmented Instruction Learning (2023.findings-emnlp)
Copied to clipboard
Hongyin Luo, Tianhua Zhang, Yung-Sung Chuang, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, James Glass
| Challenge: | Large language models (LLMs) have been significantly improved by instruction fine-tuning, but still lack transparency and the ability to utilize up-to-date knowledge and information. |
| Approach: | They propose a search-augmented instruction learning model which grounds the language generation and instruction following abilities on complex search results generated by in-house and external search engines. |
| Outcome: | The proposed model outperforms plain LLMs on zero-shot language tasks and can generate both natural and programming languages following natural language guidance and requests. |
“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are an effective tool to assist individuals in writing documents. |
| Approach: | They examine gender biases in large language models (LLMs)-generated reference letters . they find that models are biased because they are hallucinated . |
| Outcome: | The proposed model-generated reference letters are evaluated on 2 popular LLMs- ChatGPT and Alpaca. |
TextMixer: Mixing Multiple Inputs for Privacy-Preserving Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are often deployed as cloud services, enabling users to upload textual data and perform inference remotely. |
| Approach: | They propose a privacy-preserving inference framework called MixPi which aims to obfuscate a user's private input by mixing it with multiple other inputs. |
| Outcome: | The proposed framework surpasses existing privacy-preserving methods on token and sentence classification tasks. |
FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4 (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models such as GPT-4 have demonstrated impressive capability to solve textual understanding problems at a level parallel to or surpassing state-of-the-art taskspecific models. |
| Approach: | They propose to transfer task-specific inductive biases from finetuned models to prompts to improve GPT-4's compositional reasoning capabilities. |
| Outcome: | The proposed prompt scheme shows competitive zero-shot and few-shot performances compared to existing prompts on complicated reasoning tasks. |
Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language teachers need to be accessible and have the necessary resources to create effective content for their students. |
| Approach: | They propose to extract grammar descriptions from a natural text corpus that answer questions about morphosyntax and semantics from lexical corpus. |
| Outcome: | The proposed method is applied to two Indian languages, Kannada and Marathi, which, unlike English, do not have well-developed resources for second language learning. |
Allies: Prompting Large Language Model with Beam Search (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to build LLMs with stacking are limited by their information coverage and low fault tolerance. |
| Approach: | They propose a method that leverages large language models to iteratively generate new queries from an input query. |
| Outcome: | The proposed method outperforms baselines on open-domain question answering benchmarks. |
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown human-like reasoning abilities but struggle with complex logical problems. |
| Approach: | They propose a framework which integrates large language models with symbolic solvers to improve logical problem-solving by combining them with a self-refinement module. |
| Outcome: | The proposed framework improves logical problem-solving by integrating LLMs with symbolic solvers. |
SiMFy: A Simple Yet Effective Approach for Temporal Knowledge Graph Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for temporal knowledge graph reasoning suffer from low training efficiency and insufficient generalization ability. |
| Approach: | They propose a temporal knowledge graph reasoning approach that uses multilayer perceptron to model the structural dependencies of events and adopts a fixed-frequency strategy to incorporate historical frequency during inference. |
| Outcome: | The proposed model achieves state-of-the-art performance with faster convergence speed and better generalization ability. |
Understanding Translationese in Cross-Lingual Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets involve translation, but translationese is distinguished from original text . previous studies have shown that translationeses in CLS are not a problem in training sets . |
| Approach: | They propose to use cross-lingual summarization to generate a concise summary in a target language from a document in . existing datasets typically involve translation in their creation, but the translated text is distinguished from the original written in that language. |
| Outcome: | The proposed method systematically investigates how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. |
The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new dataset evaluates the credibility of statements made by Israeli public figures and politicians . a dataset of 1021 statements is used to assess the credibility and accuracy of statements . |
| Approach: | They propose a dataset to evaluate the credibility of statements by Israeli politicians . they use annotated statements manually annotating them for their credibility status . |
| Outcome: | The proposed model outperforms models based on statement and context, and achieves a 48.3 F1 score. |
IndiSocialFT: Multilingual Word Representation for Indian languages in code-mixed environment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on word embeddings for Indian languages focus on monolingual corpora with limited reach to social media setups. |
| Approach: | They propose a generalized representation vector for diverse text characteristics . they use a FastText model to gather text from social media and well-formed sources . |
| Outcome: | The proposed representation vector surpasses baselines in most cases and languages, demonstrating suitability for various NLP applications. |
Adaptive Hinge Balance Loss for Document-Level Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-Level Relation Extraction is a multi-label classification task . however, most entity pairs do not express any relations . |
| Approach: | They propose to use a distance between the classification threshold and predicted score to reduce the imbalance problem by balancing the easy negatives. |
| Outcome: | The proposed method is superior to other methods, the authors show . it reduces the threshold to an appropriate value, and increases quadratically with the number of entities. |
Answer-state Recurrent Relational Network (AsRRN) for Constructed Response Assessment and Feedback Grouping (2023.findings-emnlp)
Copied to clipboard
| Challenge: | STEM educators must trade off the ease of assessing selected response (SR) questions with constructed response (CR) questions, where students articulate their own reasoning. |
| Approach: | They propose to use multiple choice questions to construct response questions . they use a relational neural network to relate context, questions, reference responses, and students’ answers to a new dataset of 6,532 student responses to three, two-part CR questions. |
| Outcome: | The proposed network outperforms classifiers based on LLMs and few-shot learning with GPT-3.5 on a new dataset of 6,532 student responses to three, two-part CR questions. |
Low-Resource Comparative Opinion Quintuple Extraction by Data Augmentation with Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Comparative Opinion Quintuple Extraction (COQE) aims to predict comparative opinion quintuples from comparative sentences. |
| Approach: | They propose a low-resource approach to extract comparative opinion quintuples from comparative sentences . they propose augmentation using ChatGPT and a data-centric approach . |
| Outcome: | The proposed approach improves the existing pipeline-based method and achieves state-of-the-art results. |
A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of working with humans in real-world scenarios, but they are prone to generate hallucinations and misinformation when deployed for mission-critical tasks. |
| Approach: | They propose a self-check approach to detect factual errors in a zero-resource fashion by using reverse validation to generate a hallucination detection benchmark. |
| Outcome: | The proposed method outperforms baseline methods while costing fewer tokens and less time. |
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show draft-then-verify paradigm can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding. |
| Approach: | They propose to use Spec-Drafter and Spec Verification to accelerate autoregressive (AR) decoding by combining a model optimized for efficient and accurate drafting and a reliable method for verifying the drafted tokens efficiently. |
| Outcome: | The proposed method achieves 5x speedup on seq2seq tasks with comparable generation quality to beam search decoding, refreshing the impression that draft-then-verify paradigm introduces only 1.4x2x speed up. |
APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection (2023.findings-emnlp)
Copied to clipboard
Pei Wang, Keqing He, Yutao Mou, Xiaoshuai Song, Yanan Wu, Jingang Wang, Yunsen Xian, Xunliang Cai, Weiran Xu
| Challenge: | Existing methods for detecting out-of-domain (OOD) intents are hard to label . previous studies use labeled in-domain data to learn intent representations . |
| Approach: | They propose a prototypical pseudo-labeling method for few-shot OOD detection . they propose 'protoOOD' framework and adaptive pseudo-labeled method . |
| Outcome: | The proposed method is able to detect out-of-domain (OOD) intents from user queries. |
2INER: Instructive and In-Context Learning on Few-Shot Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks are a fundamental task of natural language processing (NLP). |
| Approach: | They propose a text-to-text framework for Few-Shot Named Entity Recognition (NER) that employs instruction finetuning and auxiliary tasks to enhance the model's understanding of entity types in the overall semantic context of a sentence. |
| Outcome: | The proposed framework outperforms existing Few-Shot NER methods and remains competitive with state-of-the-art NER algorithms. |
Generative Emotion Cause Triplet Extraction in Conversations with Commonsense Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on ECTEC focus on Causal Emotion Entailment and Emotion-Cause Pair Extraction in Conversations. |
| Approach: | They propose to decompose the ECTEC task into multiple subtasks and solve them in a pipeline manner. |
| Outcome: | The proposed model outperforms competing systems on two benchmark datasets. |
Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for interpreting LLMs are post hoc and focus on low-level features and lack of explainability at higher-level text units. |
| Approach: | They propose a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings during the fine-tuning stage while maintaining competitive performance. |
| Outcome: | The proposed framework can learn interpretable embeddings during the fine-tuning stage while maintaining competitive performance. |
GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating stories with complex plots rely on detailed prompts, which inadvertently limit the creative potential of the generated stories. |
| Approach: | They propose a retrieval-auGmented stoRy generation framework with a fOrest of eVidEnce to enhance stories’ complexity. |
| Outcome: | The proposed framework enables generating more diverse plotlines from human-written stories. |
KAPALM: Knowledge grAPh enhAnced Language Models for Fake News Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of fake news detection focus on news entity information and ignore structured knowledge among news entities. |
| Approach: | They propose a model that fuses coarse- and fine-grained representations of entity knowledge from Knowledge Graphs (KGs) they identify entities in news content and link them to entities in KGs. |
| Outcome: | The proposed model outperforms state-of-the-art models on two benchmark datasets and is competitive in the few-shot scenario. |
Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that loophole-seeking is frequent and intuitive in children . a large number of models capture the pragmatic understanding required for loopholes, says a researcher . |
| Approach: | a study compares large language models to humans to examine loophole behavior . they found that models struggle to recognize humor in creative exploitation of loopholes . |
| Outcome: | a study compares state-of-the-art models to humans to examine loophole behavior in humans . a large language model can generate loopholes, but only two are capable of generating them . |
InstructExcel: A Benchmark for Natural Language Instruction in Excel (2023.findings-emnlp)
Copied to clipboard
Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri
| Challenge: | Large Language Models (LLMs) can solve increasingly complex NLP tasks such as Excel specific tasks. |
| Approach: | They propose a large-scale benchmark to test whether Large Language Models can generate code that solves Excel specific tasks provided via natural language user instructions. |
| Outcome: | The proposed model outperforms existing models and provides a hard benchmark for state of the art models like GPT-4. |
Hallucination Detection for Grounded Instruction Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for generating instructions for navigation generate references to objects or actions that are inconsistent with what a human follower would perform or encounter along the path. |
| Approach: | They propose a weakly supervised approach that detects hallucinated references by using a pre-trained vision-language model. |
| Outcome: | The proposed model outperforms baseline models and supervised models on generating navigation instructions. |
Definitions Matter: Guiding GPT for Multi-label Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent success of Large Language Models (LLMs) is due to their superior performance on various tasks such as text generation, summarization, question answering, and inductive reasoning. |
| Approach: | They propose to generate definitions from examples and use them for zero-shot classification and to investigate how an LLM makes use of the definitions. |
| Outcome: | The proposed method improves the definitions of class labels and improves their understanding of the definition. |
ECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ECHo is a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios. |
| Approach: | They propose a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios. |
| Outcome: | The proposed framework examines the reasoning capability of current AI systems on three human-centric tasks. |
An Empirical Study of Instruction-tuning Large Language Models in Chinese (2023.findings-emnlp)
Copied to clipboard
| Challenge: | emergence of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI) however, the closed source of LLMs coupled with the requirement for massive computing resources has deterred researchers from reaching the LLM training stage. |
| Approach: | They propose to use Chinese instruction-tuning LLMs as a cookbook for customizing LLM models that can better respond to Chinese instructions. |
| Outcome: | The proposed LLM can be used to customize Chinese LLMs that can better respond to Chinese instructions. |
Debiasing Multimodal Models via Causal Information Minimization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for debiasing multimodal models use approximate heuristics to represent the biases, such as shallow features from early stages of training or unimodal features for multimodal tasks like VQA, which may not be accurate. |
| Approach: | They propose a method that leverages causally-motivated information minimization to learn the confounder representations of a causal graph for multimodal data. |
| Outcome: | The proposed method improves out-of-distribution performance on multiple multimodal datasets without sacrificing in-distance performance. |
Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Emotion arcs capture how an individual (or a population) feels over time. |
| Approach: | They compare machine-learning and Lexicon-Only methods to generate emotion arcs . they run experiments on 18 diverse datasets in 9 languages . |
| Outcome: | The proposed method is poor at instance level emotion classification, but highly accurate when aggregating information from hundreds of instances. |
Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | With the rapid evolution of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. |
| Approach: | They propose to integrate ChatGPT and Bing GPT3 into their applications to create a set of LLMs that can be used to generate NLP tasks with appropriate prompts. |
| Outcome: | The proposed models can be zero-shot or few-shot learners to solve specified tasks and can even be zero or few shot learners. |
Chain-of-Thought Embeddings for Stance Detection on Social Media (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Stance detection on social media platforms like Twitter is challenging for Large Language Models (LLMs), as emerging slang and colloquial language in online conversations often contain deeply implicit stance labels. |
| Approach: | They propose to embed COT reasonings into a traditional RoBERTa-based stance detection pipeline by embedding COT stance reasonings and integrating them into slang-based models. |
| Outcome: | The proposed model achieves SOTA performance on multiple stance detection datasets collected from social media. |
Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using hierarchical Dirichlet processes, we characterize news articles associated with key events from news streams. |
| Approach: | They propose a generic framework for news stream clustering that analyzes the temporal trend of news articles to automatically extract the underlying key news events that draw significant media attention. |
| Outcome: | The proposed framework produces more coherent clusters based on event summaries . the proposed framework is a first step in a new field of news analysis . |
Descriptive Prompt Paraphrasing for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current researches mainly work on either of two types of targets in a decentralized manner. |
| Approach: | They propose a model to perform sentiment polarity on a target jointly considering its corresponding multiple modalities including text, image, and others. |
| Outcome: | The proposed model performs well on four datasets spanning the above two target types and is prompt-based language modelling. |
Joint Semantic and Strategy Matching for Persuasive Dialogue (2023.findings-emnlp)
Copied to clipboard
Chuhao Jin, Yutao Zhu, Lingzhen Kong, Shijie Li, Xiao Zhang, Ruihua Song, Xu Chen, Huan Chen, Yuchong Sun, Yu Chen, Jun Xu
| Challenge: | Persuasive dialogue models rely on utterance semantic matching and a key aspect has been ignored . compared with utterrance semantics, conversation strategies are high-level concepts, which can be informative and provide complementary information to achieve effective persuation. |
| Approach: | They propose to model conversation semantics and strategies to match them using a BERT-like module and an auto-regressive predictor. |
| Outcome: | The proposed model improves state-of-the-art by 5% on a small and 37% on 'large' datasets. |
Non-Autoregressive Sentence Ordering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing sentence ordering approaches only leverage unilateral dependencies during decoding and cannot fully explore the semantic dependency between sentences. |
| Approach: | They propose a non-autoregressive ordering network that explores bilateral dependencies between sentences and predicts sentences for each position in parallel. |
| Outcome: | The proposed model outperforms existing autoregressive sentence ordering approaches and yields competitive performance compared with the state-of-the-arts. |
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 are popular as evaluation metric for complex generative tasks . however, they are not ready as human replacements due to significant limitations . |
| Approach: | They conduct extensive analysis to examine the stability and reliability of LLMs as automatic evaluators for abstractive summarization. |
| Outcome: | The proposed methods outperform the commonly used automatic metrics but are not ready for human evaluation due to significant limitations. |
Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to visual understanding of image captioning rely on transformers and pre-trained paradigms to learn cross-modal representation. |
| Approach: | They propose a visual semantic-based gender score that measures the degree of bias and can be used as a plug-in for any image captioning system. |
| Outcome: | The proposed score can measure the bias relation between a caption and its related gender and can be used as an additional metric to the existing Object Gender Co-Occ approach. |
FREDSum: A Dialogue Summarization Corpus for French Political Debates (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in deep learning have improved the performance of abstractive summarization systems. |
| Approach: | They present a dataset of french political debates to enhance resources for multi-lingual dialogue summarization. |
| Outcome: | The proposed dataset will be made publicly available for use by the research community. |
Towards Zero-shot Relation Extraction in Web Mining: A Multimodal Approach with Relative XML Path (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot relation extraction do not take into account relationships between text nodes within and across web pages. |
| Approach: | They propose a new approach for zero-shot relation extraction in web mining that encodes the shortest relative paths in the Document Object Model tree of the web page. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on public benchmarks on semi-structured web pages. |
Narrative Style and the Spread of Health Misinformation on Twitter (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using a narrative style is an effective way to communicate health information on and off social media platforms. |
| Approach: | They annotate health misinformation tweets and classify them into narrative and non-narrative . they then use supervised fine-tuning and in-context learning to detect narratives . |
| Outcome: | The proposed model analyzes health misinformation tweets and finds that narrative use is linked to increased tweet engagement and can lead to increased misinformation use. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for mental health support are shallow and heuristic, e.g., analyzing emotions and generating comforting responses. |
| Approach: | They propose to use cognitive distortion detection to perform diagnosis on the patient’s speech via three stages: subjectivity assessment to separate the facts and the thoughts; contrastive reasoning to elicit the reasoning processes supporting and contradicting the thoughts and schema analysis to summarize the cognition schemas. |
| Outcome: | The proposed system improves on ChatGPT for cognitive distortion detection while generating high-quality rationales approved by human experts. |
Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has demonstrated that pre-trained language models acquire a broad range of knowledge about linguistic structures, encyclopedic relations, levels of commonsense, and even coding and reasoning rules. |
| Approach: | They propose a systematic framework to measure parametric knowledge utilization in pre-trained language models by extracting parametric information from a PLM and constructing a downstream task around this extracted knowledge. |
| Outcome: | The proposed framework extracts parametric knowledge from a PLM and constructs a downstream task around this extracted knowledge. |
Non-compositional Expression Generation Based on Curriculum Learning and Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Non-compositional expressions are a classic ‘pain in the neck’ for NLP systems because of their non-composibility and limited data resources. |
| Approach: | They propose a dynamic curriculum learning framework which learns training examples from easy ones to harder ones but suffers from the forgetting problem. |
| Outcome: | The proposed framework improves on idiomatic expression generation and metaphor generation. |
Information Extraction from Legal Wills: How Well Does GPT-4 Do? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using information extraction from legal wills is an important application of artificial intelligence (AI) |
| Approach: | They propose a manually annotated dataset for Information Extraction (IE) from legal wills . they also use it to evaluate the performance of large language models (LLMs) |
| Outcome: | The proposed dataset can be used to evaluate large language models on IE from legal wills . it shows that the model performs reasonably well, but inconsistent outputs and overgeneralization are observed . |
Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to train, evaluate and interpret neural language models uses artificial, language-like data. |
| Approach: | They propose a setup for training, evaluating and interpreting neural language models that uses artificial, language-like data. |
| Outcome: | The proposed model is based on a massive probabilistic grammar and a large natural language corpus, and provides complete control over the generative process. |
Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, the generalized intent classification system only considers one stage of OOD learning and requires all IND data for joint training. |
| Approach: | They propose a task that detects OOD intents from dynamic OOD data streams . they propose CGID method that bootstraps new intent discovery through class prototypes . |
| Outcome: | The proposed task can detect out-of-domain (OOD) queries and extend them to the in-domain classifier . it can safely and efficiently detect out of-domain queries and avoid wrong operations . |
Frugal Prompting for Dialog Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used in natural language processing tasks with an unrealistic speed and effectiveness. |
| Approach: | They propose more compact ways of providing dialog history information while ensuring good performance and reducing model’s inference-API costs. |
| Outcome: | The proposed models have the optimal usable-information density while maintaining good performance and reducing model’s inference-API costs. |
The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern artificial intelligence is characterized by large pretrained language models with strong language capabilities to be adapted to various downstream tasks. |
| Approach: | They propose to use the task of speech translation (ST) to pretrain speech models for end-to-end SLU on intra- and cross-lingual scenarios. |
| Outcome: | The proposed model achieves higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. |
MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multi-aspect controllable text generation require expensive iteration / searching within the discrete text space during the decoding stage, resulting in a degradation of text quality due to the domain discrepancies between different aspects. |
| Approach: | They propose a framework that estimates compact latent space for multiple aspects and performs efficient Sampling with a fast sampler to eliminate domain discrepancies. |
| Outcome: | The proposed framework outperforms baselines on attribute relevance and textual quality while maintaining a high inference speed. |
HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end neural networks excel at answering natural language questions but fail on complex ones . a proposed framework for question parsing and execution on textual QA is designed to combine the strengths of neural and symbolic methods. |
| Approach: | They propose a framework for question parsing and execution on textual QA . they parse questions into an intermediate representation and use deterministic rules to translate them . |
| Outcome: | The proposed framework outperforms existing methods in supervised, few-shot, and zero-shot settings while preserving its underlying reasoning process. |
Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token Pruning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to accelerate inference speed are model compression and dynamic computation (e.g., dynamic token pruning). |
| Approach: | They propose a two-stage knowledge distillation framework that produces a customized small language model for dynamic token pruning. |
| Outcome: | The proposed framework can make the small language model more customized for dynamic token pruning and achieve better speed-performance trade-off. |
Toxicity, Morality, and Speech Act Guided Stance Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies that focus on stance detection ignore the speech act, toxic, and moral features of tweets or lack an efficient architecture to detect the attitudes across targets. |
| Approach: | They propose a multitasking model that extracts valence, arousal, and dominance aspects hidden in tweets and injects the emotional sense into the embedded text followed by an efficient attention framework to correctly detect the tweet’s stance. |
| Outcome: | The proposed model exploits the toxicity, morality, and speech act features of the tweets to detect the public's stance. |
Reasoning about Ambiguous Definite Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing resources to evaluate reasoning are not well suited to investigate the capability of resolving ambiguities by explicit reasoning. |
| Approach: | They propose to use ambiguous definite descriptions to create a benchmark dataset which requires models to resolve ambiguity by explicit reasoning. |
| Outcome: | The proposed model includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. |
A Framework for Bidirectional Decoding: Case Study in Morphological Inflection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing encoder-decoders that generate sequences from left to right are prone to errors due to the "snowballing" effect. |
| Approach: | They propose a transformer-based encoder-decoder framework that produces sequences from the "outside-in" they argue that this approach is more principled than prior bidirectional decoders . |
| Outcome: | The proposed model beats the current system by over 4.7 and 2.7 points in accuracy on 2022 and 2023 tasks. |
Text-guided 3D Human Generation from 2D Collections (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 3D human modeling is used for engaging interaction in gaming, film, and animation. however, the customization of characters is crucial for creativity and scalability. |
| Approach: | They propose a 3D human generation using fashion descriptions to enhance 3D geometry transformation and fine-grained consistency. |
| Outcome: | The proposed model can generate a 3D human, guided by a fashion description, with high efficiency. |
Statistically Profiling Biases in Natural Language Reasoning Datasets and Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate NLP models' weaknesses are limited by “hypothesis-only” tests and CheckLists. |
| Approach: | They propose a lightweight general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. |
| Outcome: | The proposed framework assesses the extent to which models exploit these biases through black-box testing, confirming prior findings and revealing new insights. |
Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on LMs' internal vector representations has demonstrated the existence of syntactic information in those vectors, but has not described how LM models use this information. |
| Approach: | They propose a causal intervention paradigm to show that some linguistic features are represented in a linear format. |
| Outcome: | The proposed model can generate verb conjugations 91% correct before and 85% incorrect after the intervention. |
MUX-PLMs: Data Multiplexing for High-throughput Language Models (2023.findings-emnlp)
Copied to clipboard
Vishvak Murahari, Ameet Deshpande, Carlos Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, Karthik Narasimhan
| Challenge: | MUX-PLMs are high-throughput pre-trained language models that can be fine-tuned for any downstream task to yield high-performance. |
| Approach: | They propose to train language models with data multiplexing to achieve 2x/5x inference speedup . they use multiplexers to entangle and disentangle inputs to achieve the same performance . |
| Outcome: | MUX-PLMs achieve 2x/5x inference speedup with 1-4 % drop on broad suite of tasks. |
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for translation of ambiguous text use context to disambiguate meaning . current models for MTs consistently translate English idioms literally, whereas LMs are context-aware . |
| Approach: | They use a dataset of 512 pairs of English sentences to study semantic ambiguities . they use literal and figurative idioms to disambiguate intended meaning . |
| Outcome: | The results show that current models translate English idioms literally, even when the context suggests a figurative interpretation. |
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. |
| Approach: | They propose to use dynamic epistemic logic to isolate a particular component of ToM and generate controlled problems in English natural language. |
| Outcome: | The proposed language model scales from 70M to 6B and 350M to 174B do not consistently yield better results than random chance. |
LATENTLOGIC: Learning Logic Rules in Latent Space over Knowledge Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for learning logic rules for knowledge graph reasoning face limitations such as searching in vast search space and inefficient optimization. |
| Approach: | They propose a framework to efficiently mine logic rules by controllable generation in the latent space by a pre-trained VAE and a discriminator. |
| Outcome: | The proposed framework efficiently mines logic rules by controllable generation in the latent space. |
RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing PLMs suffer from poor robustness in adversarial scenarios, despite their success with unseen samples. |
| Approach: | They propose a self-supervised sentence embedding framework that enhances generalization and robustness in various text representation tasks and against diverse adversarial attacks. |
| Outcome: | The proposed framework improves generalization and robustness in various representation tasks and against diverse adversarial attacks. |
More than Votes? Voting and Language based Partisanship in the US Supreme Court (2023.findings-emnlp)
Copied to clipboard
| Challenge: | partisanship and ideology have been a key topic in legal studies of the US Supreme Court . most research quantifies partisan behavior based on voting behavior, and oral arguments have not been well studied for this purpose. |
| Approach: | They propose a framework for analyzing justices' oral arguments for partisan signals and how they align with voting patterns. |
| Outcome: | The proposed framework shows that the affiliated party of justices can be predicted reliably from their oral contributions. |
Automatic Evaluation of Attribution by Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) incorporate external references to generate and support claims. however, evaluating the attribution remains an open problem. |
| Approach: | They investigate automatic evaluation of attribution given by large language models . they define different types of attributed errors and then explore two approaches . |
| Outcome: | The proposed methods highlight promising signals and challenges. |
Modeling Highlighting of Metaphors in Multitask Contrastive Learning Paradigms (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to identify highlighted aspects of a metaphorical sentence are not yet available. |
| Approach: | They propose to identify the main aspect highlighted in a metaphorical sentence by using a joint learning approach and a continual learning approach. |
| Outcome: | The proposed approach improves predicting highlighted aspects and source domains compared to baselines on an existing corpus. |
LDM2: A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Extensive experiments conducted in two interactive environments have shown that our LDM2 outperforms the baselines in terms of both score and success rate. |
| Approach: | They propose a large decision model with memory that leverages a dynamic memory mechanism to construct dynamic prompts, guiding the LLMs in making proper decisions according to the faced state. |
| Outcome: | The proposed model outperforms baseline models in two interactive environments in terms of score and success rate. |
ZARA: Improving Few-Shot Self-Rationalization for Small Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies demonstrate great performance gain for self-rationalization by few-shot prompting LMs with rationale-augmented exemplars. |
| Approach: | They propose to leverage explanations for small LMs to improve few-shot self-rationalization by reducing the problem of plausibility judgement to natural language inference. |
| Outcome: | The proposed approach achieves SOTA performance on the FEB benchmark, for both the task accuracy and the explanation metric. |
ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | toxicity detection has been largely based on social media content, leaving the unique challenges inherent to real-world user-AI interactions insufficiently explored. |
| Approach: | They propose a benchmark to detect toxicity in real-world user-AI conversations . they compare existing models with social media content to find toxicity . |
| Outcome: | The proposed benchmark reveals that existing models fail to recognize toxicity in real-world user-AI conversations. |
Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Argumentative writing is an essential skill that can be challenging to acquire. |
| Approach: | They propose two new tasks to identify gaps in arguments and fill such gaps by deleting ADUs from argumentative text. |
| Outcome: | The proposed methods reduce argument quality and produce arguments that are natural to those written by learners. |
Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing variational dialog models have pre-trained, restricting diversity of responses . a diffusion model increases complexity of prior distribution and its compatibility with PLMs . |
| Approach: | They propose a hierarchical conditional variational autoencoder with diffusion priors to address these challenges. |
| Outcome: | The proposed method generates more diverse responses without dialog pre-training. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE (2023.findings-emnlp)
Copied to clipboard
| Challenge: | SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. |
| Approach: | They propose an unsupervised contrastive learning framework that combines SimCSE with hard negative mining to enhance the quality of sentence embeddings. |
| Outcome: | The proposed framework improves sentence embeddings on various STS benchmarks in terms of Spearman’s correlation, representation alignment and uniformity. |
The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Open-source dataset of code-text pairs for training large language models to understand code is outperforms other datasets for code generation and understanding tasks. |
| Approach: | They propose to extract high-quality code-text pairs from a dataset of 43 million pairs . they use rules and deep learning to ensure that the code-sampled samples contain high-quality pairs a . |
| Outcome: | The Vault dataset outperforms existing models on common coding tasks . authors hope the results will propel AI research and software development forward . |
SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinant from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. |
| Approach: | They propose a dataset that extracts social and behavioral determinants from clinical notes and uses them to form a natural language inference task. |
| Outcome: | The proposed dataset is based on publicly available notes and is more challenging than standard NLI benchmarks. |
On the Zero-Shot Generalization of Machine-Generated Text Detectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | rampant proliferation of large language models generates text indistinguishable from human-written language. |
| Approach: | They train neural detectors on outputs of a new generator and test their performance on held-out generators. |
| Outcome: | The proposed detectors can be built on training data from medium-sized models. |
Complex Event Schema Induction with Knowledge-Enriched Diffusion Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on event schema induction have been hindered by errors and data quality issues. |
| Approach: | They propose a knowledge-enriched discrete diffusion model that distills event scenario knowledge from LLMs. |
| Outcome: | The proposed model achieves outstanding performance across evaluation metrics. |
Exploiting Emotion-Semantic Correlations for Empathetic Response Generation (2023.findings-emnlp)
Copied to clipboard
Zhou Yang, Zhaochun Ren, Wang Yufeng, Xiaofei Zhu, Zhihao Chen, Tiecheng Cai, Wu Yunbing, Yisong Su, Sibo Ju, Xiangwen Liao
| Challenge: | Empathetic response generation aims to generate empathetic responses by understanding the speaker’s emotional feelings from the language of dialogue. |
| Approach: | They propose a dynamical Emotion-Semantic Correlation Model (ESCM) which constructs dynamic emotion-semantics through the interaction of context and emotions. |
| Outcome: | The proposed model understands emotions more accurately and expresses fluent and informative empathetic responses. |
Long-Range Language Modeling with Selective Cache (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use transformers to model language cost quadratically increase with sequence length. |
| Approach: | They propose a selective cache which stores key-value pairs from previous contexts. |
| Outcome: | The proposed selective cache outperforms XL cache and compressive cache by considerable margins. |
Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text simplification has emerged as an increasingly useful application of AI for bridging the communication gap in specialized fields such as medicine, where the lexicon is often dominated by technical jargon and complex constructs. |
| Approach: | They propose a unlikelihood loss that encourages generation of simpler terms and a reranked beam search decoding method that optimizes for simplicity. |
| Outcome: | The proposed methods achieve better performance on readability metrics on three datasets. |
FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on pre-trained models, such as BERT and RoBERTa. |
| Approach: | They propose a lightweight method for personalized NLP classification tasks post-backbone replacement using a personalized matrix calculated from documents corresponding to users' old and new backbones. |
| Outcome: | The proposed method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods. |
Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that multilingual models outperform monolingual ones. |
| Approach: | They propose a single model that can capture which language is given as input speech . they use a pre-trained model to fine-tune the model so it can recognize the language class as well as the speech with the corresponding language. |
| Outcome: | The proposed model can recognize which language is given as input speech . it can accurately recognize speech in noisy environments, such as crowded restaurants . |
Controllable Chest X-Ray Report Generation from Longitudinal Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Radiology reports are detailed text descriptions of the content of medical scans. |
| Approach: | They propose a method to align, concatenate and fuse the current and prior visual information into a joint longitudinal representation which can be provided to a multimodal report generation model. |
| Outcome: | The proposed method achieves state-of-the-art results while enabling anatomy-wise controllable report generation. |
Is ChatGPT a Good Multi-Party Conversation Solver? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful tools for multi-party conversations, but their capacity to handle multi-parties remains unexplored. |
| Approach: | They propose to evaluate ChatGPT and GPT-4's zero-shot learning capabilities within the context of multi-party conversations (MPCs) they also propose to incorporate MPC structures, encompassing both speaker and addressee architecture. |
| Outcome: | The proposed models perform poorly on a number of MPC tasks while GPT-4 performs well on speaker and addressee architecture. |
Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Synthesis is an efficient textual data utilization framework for end-to-end speech processing models . labeled speech data are scarcer and more expensive for collection compared to textual ones . |
| Approach: | They propose a textual data utilization framework for E2E speech processing models . they train a latent synthesizer to convert textual information into an intermediate latent representation . |
| Outcome: | The proposed framework improves on low-resource speech recognition and spoken language understanding tasks. |
Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to rank sentences using pre-trained embeddings create a gap due to different optimization objectives. |
| Approach: | They propose a pre-trained embedding process that optimizes informative sentences . they use sentence-word bipartite graphs to model intra-sentential distinctive features . |
| Outcome: | The proposed model outperforms heavy BERT- or RoBERTa-based sentence ranking methods by providing summary-worthy representations. |
Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have been fine-tuned for various NLP tasks . prompt tuning is a method that optimizes the output of the model to adapt to downstream tasks based on the posterior distribution of the source task. |
| Approach: | They propose a Bayesian approach to prompt tuning that optimizes for adapting pre-trained language models to downstream tasks rather than fine-tuning full model parameters. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on benchmark NLP tasks. |
CCIM: Cross-modal Cross-lingual Interactive Image Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on text image machine translation (TIMT) lacks recognized source language information resulting in a decrease in translation performance. |
| Approach: | They propose a cross-modal cross-lingual interactive model which incorporates source language information by synchronizing source and target language results. |
| Outcome: | The proposed model outperforms end-to-end models and has faster decoding speed with smaller model size than cascade models. |
TRAMS: Training-free Memory Selection for Long-range Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods like Transformer-XL are plagued by ineffective memory selections due to the high number of tokens involved in attention calculation. |
| Approach: | They propose a plug-and-play strategy that selects tokens participating in attention calculation based on one simple metric and ignores the other ones. |
| Outcome: | The proposed strategy keeps tokens with high attention scores and ignores the other ones on word-level and character-level benchmarks without additional training or adding additional parameters. |
A Critical Analysis of Document Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova
| Challenge: | Existing document understanding models focus on single-modal inputs such as images or texts. |
| Approach: | They propose to use a spatial-aware adapter to adapt transformer-based language models to document domain to exploit multi-modal information. |
| Outcome: | The proposed model significantly improves the OOD detection performance compared to using a standard language model and to competitive baselines. |
Improving Neural Machine Translation by Multi-Knowledge Integration with Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on how to integrate multiple types of knowledge into NMT models . |
| Approach: | They propose a framework that integrates multiple types of knowledge into NMT models . they use multiple types as prefix-prompts of input for the encoder and decoder . |
| Outcome: | The proposed framework outperforms baselines on English-Chinese and English-German translation. |
Active Learning Principles for In-Context Learning with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has significantly enhanced predictive performance in few-shot learning settings. |
| Approach: | They propose to use pool-based Active Learning to identify the most informative demonstrations for few-shot learning over a single iteration to identify best demonstrations. |
| Outcome: | The proposed model outperforms all other methods, including random sampling, in the analysis of 24 classification and multi-choice tasks. |
InteMATs: Integrating Granularity-Specific Multilingual Adapters for Cross-Lingual Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work relies on full-model fine-tuning on large parallel datasets to enhance cross-lingual alignment of MLLMs. |
| Approach: | They propose an approach that integrates multilingual adapters trained on texts of different levels of granularity into multilingual models. |
| Outcome: | The proposed approach improves the performance of multilingual language models on low-resource languages. |
PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning is a key task in health conversational assistants, but it is difficult to guarantee the specificity of the responses. |
| Approach: | They propose a plug-and-play medical dialogue system that provides a patient-centered medical interpretation service to users who are less knowledgeable about medical knowledge. |
| Outcome: | The proposed model improves the specificity of the patient-centered medical dialogues by providing them with real dialogues from similar patients as prompts. |
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing code translation datasets focus on a single pair of programming languages . early software systems are developed using programming languages such as Fortran and COBOL . |
| Approach: | They propose a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. |
| Outcome: | The proposed framework supports translations between multiple programming languages and a cross-framework dataset for deep learning code across different frameworks. |
impact of sample selection on in-context learning for entity extraction from scientific writing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based use of Large Language Models is becoming popular . specialized domains such as entity extraction are expensive to annotate . |
| Approach: | They propose to use a prompt set-up to provide training examples along with the inference request. |
| Outcome: | The proposed methods improve on a fully supervised transformer-based baseline. |
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for mitigating toxicity require drastic modifications to model parameters or the use of computationally intensive auxiliary models. |
| Approach: | They propose a method that matches current state-of-the-art toxicity mitigation while achieving 43% relative latency reduction during inference. |
| Outcome: | The proposed method achieves 43% latency reduction during inference and is more computationally efficient. |
Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | BERTScore, BLEURT, and COMET are automatic evaluation metrics that are often underperformed on adversarially-synthesized texts. |
| Approach: | They examine MT evaluation metric performance on adversarially-synthesized texts . they validate that automatic metrics tend to overpenalize adversarial-degraded translations . |
| Outcome: | The results show that automatic metrics tend to overpenalize adversarially-degraded translations. |
Time-Considerable Dialogue Models via Reranking by Time Dependency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dialogue models do not consider time information that humans are constantly aware of. |
| Approach: | They propose to categorize responses by their naturalness at different times and introduce a new metric to classify responses into categories. |
| Outcome: | The proposed model categorizes responses by their naturalness at different times and evaluates them subjectively. |
Non-Compositionality in Sentiment: New Data and Analyses (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Many studies on sentiment analysis focus on the fact that sentiment computations are compositional . linguistic utterances often do not adhere to strict patterns and can be surprising when looking at the individual words involved. |
| Approach: | They propose a method for obtaining non-compositionality ratings for phrases with respect to their sentiment . they also propose evaluating computational models for sentiment analysis using the rating resource . |
| Outcome: | The proposed method enables non-compositional ratings for phrases with respect to their sentiment . the results are compared with a new resource of ratings for 259 phrases . |
MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing soft prompt methods focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset. |
| Approach: | They propose a multi-level prompt tuning method that utilizes prompts at task-specific, domain-specific and context-specific levels to enhance the comprehension of input semantics. |
| Outcome: | The proposed method improves on 12 benchmarks on various QA formats and achieves an average improvement of 1.94% over the state-of-the-art methods. |
DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document AI models that can read visually rich documents have a long way to go before they can read them as accurately, continuously, and flexibly as humans do. |
| Approach: | They propose a visually-rich document dataset that aligns with human eye-movement information using eye-tracking technology. |
| Outcome: | The proposed dataset can help in designing better document AI models and human reading robots in the future. |
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive capabilities in many tasks, including natural language understanding and generation. |
| Approach: | They propose a framework for adaptation with self-evaluation to improve selective prediction performance of large language models. |
| Outcome: | The proposed framework outperforms state-of-the-art selective prediction methods on QA datasets and improves the AUACC from 91.23% to 92.63% and AUROC from 74.61% to 80.25%. |
Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models can be fine-tuned on limited training data, which can overfit and thus diminish performance. |
| Approach: | They propose a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout. |
| Outcome: | The proposed method outperforms existing methods on the GLUE benchmark and exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios. |
ClozEx: A Task toward Generation of English Cloze Explanation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tasks and datasets specifically designed for generating language learner explanations for cloze questions are lacking . clozing questions are used to assess language proficiency and enhance language learning . |
| Approach: | They propose a task ClozEx to generate explanations for cloze questions in LA . they use a curated dataset of clozing questions paired with explanations . |
| Outcome: | The proposed task generates fluent explanations for cloze questions in English as a second language learners. |
Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing probing tasks are designed to evaluate the information existing in representations by training a simple classification model. |
| Approach: | They propose to use indicators to query embedding spaces for the existence of certain properties to determine whether a property exists in an embeddable space. |
| Outcome: | The proposed indicators provide a more accurate picture of the information captured and removed compared to probes. |
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on LLM compression focuses on general metrics like perplexity or downstream task accuracy. |
| Approach: | They propose to quantify the effect of pruning and quantization on model quality . they use the LAMA and LM-Harness benchmarks to quantify compression techniques . |
| Outcome: | The proposed compression techniques provide faster inference, smaller memory footprints, and enables local deployment. |
CoEdIT: Text Editing by Task-Specific Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | We present a large language model for writing assistance that is fine-tuned on task-specific instructions. |
| Approach: | They propose a large language model that is fine-tuned on task-specific instructions and outputs the edited text. |
| Outcome: | The proposed model performs better than other state-of-the-art models on various editing benchmarks while being 60x smaller. |
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. |
| Approach: | They propose to apply world knowledge to enhance OOD detection performance through selective generation from large language models (LLMs) they propose to extract visual objects from each image to fully capitalize on the aforementioned world knowledge. |
| Outcome: | The proposed method outperforms the state-of-the-art on visual OOD detection on in-distribution (ID) samples. |
Better Together: Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge graph completion (KGC) methods are computationally intensive and impractical for large-scale KGs. |
| Approach: | They propose to include node neighborhoods as additional information to improve KGC methods based on language models. |
| Outcome: | The proposed method outperforms KGT5 and conventional methods on inductive and transductive Wikidata subsets and shows its importance. |
DeltaScore: Fine-Grained Story Evaluation with Perturbations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for stories are limited in assessing intricate aspects of storytelling, such as fluency and interestingness. |
| Approach: | They propose a novel method that uses perturbation techniques to evaluate story aspects . they compare fluency, coherence, relatedness, logicality, interestingness and interestingness to existing metrics . |
| Outcome: | The proposed method shows that one specific perturbation is highly effective in capturing multiple aspects. |
MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multimodal classification systems use tabular, textual, and visual data to provide efficient and scalable services. |
| Approach: | They propose a multimodal classification benchmark MuG with eight datasets . they analyze label balance ratios, percentages of missing features, distributions of data within each modality . |
| Outcome: | The proposed benchmark is available on https://github.com/lujiaying/MUG-Bench . it includes eight datasets that allow researchers to evaluate and improve their models . |
Don’t waste a single annotation: improving single-label classifiers through soft labels (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for annotating data are limited by ambiguity and lack of context in data samples. |
| Approach: | They challenge the traditional approach of annotating data by only providing a single label for each sample and annotator disagreement is discarded . instead, they use additional annotation information such as confidence, secondary label and disagreement to generate soft labels. |
| Outcome: | The proposed method improves model performance and calibration on the hard label test set. |
Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large vision-language models are often not open-source due to preventing abuse or commercial factors. |
| Approach: | They propose a method for parameter-efficient fine-tuning to improve model accessibility . large models are often not open-source due to preventing abuse or commercial factors . they propose implementing a lightweight adapter over the output feature of an inaccessible model . |
| Outcome: | The proposed methods improve on 11 benchmarks and are made publicly available. |
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transferability estimation has been a topic of great interest in computer vision fields . a lack of a comprehensive comparison between these estimation methods is a problem . |
| Approach: | They conduct a thorough survey of existing methods to find the most suitable model . they also outline difficulties of consideration of training details and applicability to text generation . |
| Outcome: | The proposed methods perform well with superiorities in effectiveness and efficiency. |
Licon: A Diverse, Controllable and Challenging Linguistic Concept Learning Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Concept Learning focus on visual information, but visual information cannot present abstract concepts exactly, which struggles the introduction of novel concepts related to known concepts. |
| Approach: | They propose a benchmark where concepts in diverse forms are defined by linguistic descriptions and an entailment-based concept learning method to model the relationship among concepts. |
| Outcome: | The proposed benchmark is based on the existing visual concepts learning benchmarks and will be released to the public soon. |
InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on NLP explainability methods lacks a dialogue-based interpretability framework that can convey faithful explanations in human-understandable terms. |
| Approach: | They adapt the conversational explanation framework TalkToModel to the NLP domain and add new NLP-specific operations such as free-text rationalization to illustrate its generalizability. |
| Outcome: | The proposed framework can be used to explain models on three NLP tasks and is generalizable to different datasets, use cases and models. |
INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large language models have enabled them to hold free form conversations over multiple turns, but they exhibit a tendency to make unfounded and incorrect statements, commonly labeled as hallucinations. |
| Approach: | They propose a framework to test large language models for hallucinations using automatically generated INValId questions. |
| Outcome: | The proposed framework is based on a testbed of automatically generated INValId questions to evaluate large language models for hallucinations. |
Multimodal Automated Fact-Checking: A Survey (2023.findings-emnlp)
Copied to clipboard
Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, Andreas Vlachos
| Challenge: | Existing studies on automated fact-checking focus on text, but they focus on a single modality, text . multimodal misinformation is perceived as more credible by humans and spreads faster than text-only counterparts. |
| Approach: | They propose a framework for automated fact-checking that includes subtasks unique to multimodal misinformation. |
| Outcome: | The proposed framework includes subtasks unique to multimodal misinformation. |
PROTEGE: Prompt-based Diverse Question Generation from Web Articles (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a popular format for knowledge bases is question-answer pairs (Q&As) specialized knowledge bases that extract and store question-annwer pairs are prevalent . |
| Approach: | They propose a framework for question generation that generates diverse questions from text . they propose 'protege' framework that can generate diverse questions using a variety of prompts . |
| Outcome: | The proposed framework improves diversity and fidelity over diverse beam search and prompt-based baselines on three public Q&A datasets. |
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms that generate captions for scientific figures are costly and dependent on author-written captions. |
| Approach: | They constructed a human evaluation dataset that contains human judgments for 3,600 scientific figure captions for 600 arXiv figures. |
| Outcome: | The proposed model outperforms all other models and outperformed undergraduates in achieving a Kendall correlation score of 0.401 with Ph.D. students’ rankings. |
Mulan: A Multi-Level Alignment Model for Video Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on visual-language alignment at the video level, but they do not account for fine-grained semantic interaction between video and text. |
| Approach: | They propose a multi-level Alignment Model for Video Question Answering that establishes alignment between visual and textual modalities at the object-level, frame-level and video-level. |
| Outcome: | The proposed model outperforms state-of-the-art methods even with a small amount of extra visual-language pre-training data and a reduced number of trainable parameters. |
HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks have attempted to identify and explain hate speech but lack the reasoning to supervise detection models. |
| Approach: | They propose a framework that uses large language models to fill in the gaps in hate speech explanations by using existing annotations. |
| Outcome: | The proposed framework outperforms baselines on SBIC and Implicit Hate using model-generated data and improves generalization to unseen datasets. |
ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for predicting chemical reactions are limited by insufficient training data and inability to utilize textual information. |
| Approach: | They propose a framework that leverages chemical knowledge encoded in language models to assist GNNs, thereby enhancing the accuracy of real-world chemical reaction predictions. |
| Outcome: | The proposed framework improves state-of-the-art GNN-based methods across chemical reaction datasets especially in out-of distribution settings. |
Decomposing Complex Queries for Tip-of-the-tongue Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Tip-of-the-tongue retrieval is a retrieval setting in which a user is unable to formulate a precise query that identifies a sought item . a framework that decomposes complex queries into subqueries can improve gold book recall . |
| Approach: | They propose a framework for handling tip-of-the-tongue queries by decomposing queries into individual clues routing them to specialized retrievers. |
| Outcome: | The proposed framework improves gold book recall up to 6% on a new query-book pair . it takes advantage of off-the-shelf retrievers or incorporates retriever-specific logic . |
Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have focused on the ethical aspects of NLP, but little to no discussion of the terminology and theories underpinning those efforts and their implications. |
| Approach: | They propose to provide an overview of some important ethical concepts stemming from philosophy and to survey the existing literature on moral NLP w.r.t. their findings show that most papers neither provide a clear definition of the terms they use nor adhere to definitions from philosophy. |
| Outcome: | The findings show that most papers neither provide a clear definition of the terms they use nor adhere to definitions from philosophy. |
Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text game agents are often modeled using reinforcement learning, but their performance is limited. |
| Approach: | They propose a self-supervised behavior cloning transformer for text games . they explore trajectories that lead to reward within the games and then train small models . their approach consistently uncovers generalizable training data, achieving 90% performance of supervised systems across three benchmark text games. |
| Outcome: | The proposed model achieves 90% performance on three text games. |
Adapting Pretrained Text-to-Text Models for Long Text Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing short-context models are limited in their domain coverage and can be used for long-sequence inputs. |
| Approach: | They propose to replace full attention in transformers with pooling-augmented blockwise attention and pretrain the model with a masked-span prediction task with spans of varying lengths. |
| Outcome: | The proposed model outperforms existing models on long-sequence summarization tasks and achieves competitive performance on long document corpora. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |
MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems (2023.findings-emnlp)
Copied to clipboard
Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan
| Challenge: | Existing models for automatic dialogue tutoring fail to provide accurate feedback or reveal solutions to students too early. |
| Approach: | They propose a framework to generate one-to-one teacher-student tutoring dialogues by pairing human teachers with a Large Language Model (LLM) they use scaffolding questions and annotations to fine-tune models to be more effective tutors . |
| Outcome: | The proposed framework can generate 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems. |
Towards Making the Most of ChatGPT for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies have shown that ChatGPT achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g., low-resourced and distant-language-pairs translation. |
| Approach: | They propose task-specific prompts and domain-specific prompts which are based on task information and domain information and a task-specific prompt. |
| Outcome: | The proposed prompts improve the performance of ChatGPT in complex tasks and generate hallucinations for non-English-centric tasks. |
Enhancing Reasoning Capabilities by Instruction Learning and Chain-of-Thoughts for Implicit Discourse Relation Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for implicit discourse relation recognition are based on generative models, but some studies suggest they do not perform as well as generic encoder-only models for NLU tasks. |
| Approach: | They propose a classification method that is solely based on generative models and utilize Chain-of-Thoughts to partition the inference process into a sequence of three successive stages. |
| Outcome: | The proposed model outperforms existing models on a natural language understanding task. |
Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for opinion summarization are deficient in epitomizing extensive reviews and offering opinion summaries from various angles. |
| Approach: | They propose a supervised opinion summarization framework that takes sentiment orientation into account and trains the summarizer to learn from sub-optimal and optimal review subsets. |
| Outcome: | The proposed framework generates pros, cons, and verdict summaries from hundreds of input reviews. |
Topic-Informed Dialogue Summarization using Topic Distribution and Prompt-based Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that deal with multiple topics are prone to topic drift . experimental results show that our model outperforms state-of-the-art methods on ROUGE scores . |
| Approach: | They propose a dialogue summarization model that reflects dialogue topic distribution to consider all topics present in a dialog. |
| Outcome: | The proposed model outperforms state-of-the-art methods on ROUGE scores and generates comprehensive summaries. |
Disentangling Structure and Style: Political Bias Detection in News by Inducing Document Hierarchy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method to detect political bias in news articles overcomes this domain dependency . partisan bias exists in various social issues, including the 2016 presidential election . |
| Approach: | They propose a multi-head hierarchical attention model that encodes the structure of long documents through a diverse ensemble of attention heads. |
| Outcome: | The proposed model outperforms existing methods for detecting political bias in news articles. |
Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
Unsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for candidate answer extraction are reliant on linguistic rules or annotated data and face partial annotation issue and challenges in generalization. |
| Approach: | They propose an unsupervised approach that leverages the inherent structure of context passages through a Differentiable Masker-Reconstructor (DMR) Model with the enforcement of self-consistency for picking up salient information tokens. |
| Outcome: | The proposed model outperforms supervised and unsupervised methods in two datasets with exhaustively-annotated answers and shows that it is comparable to supervised methods. |
HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science (2023.findings-emnlp)
Copied to clipboard
| Challenge: | LLaMa-based language model for materials science is first of its kind in the world . |
| Approach: | They propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct) they then apply this process to finetune a LLaMa-based language model targeted for materials science. |
| Outcome: | The proposed model outperforms existing language models on materials science tasks and improves in successive stages of refinement. |
Prompt-Based Editing for Text Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text style transfer is a type of textual prompt that generates style-transferred texts word by word . early prediction errors may affect future word predictions. |
| Approach: | They propose a prompt-based editing approach to text style transfer using a pretrained language model. |
| Outcome: | The proposed approach outperforms existing systems with 20 times more parameters on three style-transfer benchmark datasets. |
Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions. |
| Approach: | They propose to use existing CSW data sets to improve the representativeness of CSW datasets. |
| Outcome: | The proposed model lacks representativeness due to location-based, socio-demographic and register variation in CSW data. |
NERvous About My Health: Constructing a Bengali Medical Named Entity Recognition Dataset (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is used in a variety of downstream tasks in the biomedical domain, but is difficult when working with consumer health questions (CHQs). |
| Approach: | They propose to use a dataset to identify named entities in health-related texts in Bengali to address the scarcity of available data. |
| Outcome: | The proposed dataset captures the diverse range of linguistic styles and dialects used by native speakers from various regions in their day-to-day lives. |
Sparse Black-Box Multimodal Attack for Vision-Language Adversary Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adversarial attacks using imperceptible perturbations are challenging to simulate . e-commerce product restrictions and hate speech monitoring are examples of such attacks . |
| Approach: | They propose a black-box adversarial attack that leverages sparse perturbations to simulate adversarials exhibited by illegal merchants in the black- box scenario. |
| Outcome: | The proposed method outperforms existing attacks and unimodal attacks by treating images and text in discrete space and outperforming existing models. |
Towards a Unified Framework for Reference Retrieval and Related Work Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for related work generation use human-annotated references as information sources. |
| Approach: | They propose a model which combines reference retrieval and related work generation processes in a unified framework based on the large language model. |
| Outcome: | The proposed model outperforms the state-of-the-art models on two wide-applied datasets. |
Visual Storytelling with Question-Answer Plans (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models focus on enhancing the representation of image sequences, but the stories are repetitive, illogical, and lacking in detail. |
| Approach: | They propose a framework which integrates visual representations with pretrained language models and planning. |
| Outcome: | The proposed framework combines visual representations with pretrained language models and planning. |
Investigating Online Community Engagement through Stancetaking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale computational work on stancetaking has explored community similarities in their preferences for stance markers without considering the stance-relevant properties of the contexts in which stance marker use is carried out. |
| Approach: | They propose to use stance-relevant properties of Reddit communities to capture community identity patterns distinct from textual or marker similarity measures. |
| Outcome: | The proposed representations capture community identity patterns distinct from textual or marker similarity measures and relate them to broader inter- and intra-community engagement patterns. |
ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not provide robustness evaluations for large language models, but we find that they are inconsistent in performance. |
| Approach: | They propose to use semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection to generate a test suite of prompts covering diverse robustness settings. |
| Outcome: | The proposed system generates a set of prompts covering diverse settings covering semantic equivalence, related scenarios, and adversarial. |
Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that noise correction in fine-grained entity typing improves quality of training samples. |
| Approach: | They propose a method that leverages multiple prediction results to correct noisy labels . they integrate prediction results and utilize a differentiated margin to identify inaccurate labels a . |
| Outcome: | The proposed model improves quality of training samples annotated using distant supervision, ChatGPT, and crowdsourcing. |
Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models can encode unfair social biases from large pre-training corpora and even amplify biase in downstream applications. |
| Approach: | They propose a *debias-while-prompt tuning* method for mitigating biases via counterfactual contrastive prompt tuning on downstream tasks. |
| Outcome: | The proposed method can mitigate biases on three extrinsic bias benchmarks and adapt to existing debiased language models. |
A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have been used for abstractive single-document summarization (SDS) but they may not be suitable for multi-document summary (MDS) |
| Approach: | They propose to enforce hierarchy on both encoder and decoder to facilitate multi-document interactions for MDS. |
| Outcome: | Xiao et al. (2019) outperforms or is competitive with the previous best models. |
Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP (2023.findings-emnlp)
Copied to clipboard
Hyuhng Kim, Hyunsoo Cho, Sang-Woo Lee, Junyeob Kim, Choonghyun Park, Sang-goo Lee, Kang Yoo, Taeuk Kim
| Challenge: | Despite advances in computer vision, its application on language input still needs to be explored despite its feasibility. |
| Approach: | They propose a universal domain adaptation (uniDA) benchmark for natural language that offers thorough viewpoints of the model’s generalizability and robustness. |
| Outcome: | The proposed model can handle spoken language in the real world while also detecting unprocessable inputs from the target domain. |
Aligning Language Models to User Opinions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Personality is a defining feature of human beings, shaped by a complex interplay of demographic characteristics, moral principles, and social experiences. |
| Approach: | They use public opinion surveys to model past user opinions in addition to user demographics and ideology to achieve up to 7 points accuracy gains in predicting public opinions from survey questions. |
| Outcome: | The proposed model achieves 7 points accuracy gains in predicting public opinions from public opinion surveys across a broad set of topics. |
CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech-to-text translation models can extract features from speech inputs, but they may include non-linguistic speech factors such as pitch, timbre and speaker identity. |
| Approach: | They propose a content-centric speech representation disentanglement learning framework for speech translation that decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning. |
| Outcome: | The proposed framework outperforms state-of-the-art speech translation models and cascaded models on five translation directions. |
Miracle: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Personalized dialogue systems aim to endow the chatbot agent with more anthropomorphic traits for human-like interactions. |
| Approach: | They propose a method to generate personalized dialogues using latent-space energy-based models by using a latent space energy-model. |
| Outcome: | The proposed method outperforms baselines in personality controllability and response quality. |
Towards Multilingual Interlinear Morphological Glossing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Interlinear Morphological Glosses are annotations produced in the context of language documentation. |
| Approach: | They propose to use a conditional random field to label morphs in L1 and then align them to L2 words to facilitate the process. |
| Outcome: | The proposed method outperforms baselines in several under-resourced languages and is effective and data-efficient. |
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models of working memory are limited by self-attention, but we propose a new variant of regular programming that can model regular languages. |
| Approach: | They propose a Transformer variant that constructs working memory along the depth dimension . they find it rediscovers the local windowed attention effect needed for length extrapolation . |
| Outcome: | The proposed model rediscovers the local windowed attention effect needed for length extrapolation. |
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to rewrite context-dependent queries lack sufficient information for optimal retrieval performance. |
| Approach: | They propose to use large language models (LLMs) as query rewriters to generate informative queries through well-designed instructions. |
| Outcome: | The proposed approach improves performance on the QReCC dataset compared to human rewrites . |
Distilling ChatGPT for Explainable Automated Student Answer Assessment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing automated student answer assessment models lack explainable and faithful feedback. |
| Approach: | They propose a framework that leverages ChatGPT for student answer scoring and rationale generation. |
| Outcome: | The proposed method improves the overall QWK score by 11% compared to ChatGPT. |
Grammatical Error Correction via Mixed-Grained Weighted Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical evaluation shows that MainGEC achieves consistent and significant performance improvements on two benchmark datasets. |
| Approach: | They propose to use mixed-grained weighted training to improve the training effect for GEC by analyzing the inherent discrepancies in annotated training data. |
| Outcome: | Empirical results show that the proposed method achieves significant performance improvements on two benchmark datasets. |
A Unified Framework for Synaesthesia Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Synaesthesia is a cognitive phenomenon structuring human thought and action, which makes understanding it challenging. |
| Approach: | They propose a framework for annotating synaesthetic elements and exploring their relationship . they propose to include sensory modalities, cues and stimuli in the framework . |
| Outcome: | The proposed framework yields state-of-the-art results, demonstrating its effectiveness. |
Domain Private Transformers for Multi-Domain Dialog Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large general purpose language models have demonstrated impressive performance across many different domains, but their outputs are not guaranteed to stay within the domain of a given input prompt. |
| Approach: | They propose to quantify how likely a conditional language model will leak across domains by defining domain privacy as a way to fine-tune a model's privacy. |
| Outcome: | The proposed method has comparable resiliency to methods adapted from recent literature on differentially private language models. |
Visual Elements Mining as Prompts for Instruction Learning for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | VEMP uses visual elements with text symbols embedded in the image to classify sentiment polarity towards a given opinion target. |
| Approach: | They propose a visual element mining as prompts method to fuse visual and text semantic information into instruction prompts for TMSC. |
| Outcome: | The proposed method achieves state-of-the-art performance on two benchmark datasets. |
NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures. |
| Approach: | They propose a framework that narrows the encoder and shortens the decoder networks of encoder-decoder models. |
| Outcome: | The proposed framework reduces the number of decoder layers and improves generation quality. |
GBT: Generative Boosting Training Approach for Paraphrase Identification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Paraphrase Identification (PI) is a fundamental natural language understanding task with non-trivial challenges. |
| Approach: | They propose a Generative Boosting Training approach for Paraphrase Identification (PI) they use a seq2seq model to perform DA on misclassified instances periodically . |
| Outcome: | The proposed method outperforms state-of-the-art PI models on English and Chinese PI tasks with good efficiency and effectiveness. |
DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies utilize social media platforms such as Twitter to build models for crisis event analysis, but semi-supervised approaches require annotating vast amounts of data and are impractical due to limited response time. |
| Approach: | They propose a method that stores and performs equal sampling for generated pseudo-labels from each class at each training iteration. |
| Outcome: | The proposed method performs better than existing methods in both in-distribution and out-of-difference settings. |
Probing LLMs for hate speech detection: strengths and vulnerabilities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to detect hateful or toxic language using large language models have not used explanation, additional context and victim community information in the detection process. |
| Approach: | They use different prompt variations, input information and victim community information to evaluate large language models in zero shot setting without adding any in-context examples. |
| Outcome: | The proposed models perform significantly better when included in the pipeline than baseline models. |
From Simple to Complex: A Progressive Framework for Document-level Informative Argument Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document-level event argument extraction use memory to store the results of already predicted events. |
| Approach: | They propose a simple-to-complex progressive framework for document-level event argument extraction . they first calculate the difficulty of each event and then conduct the extraction following a simpler order . |
| Outcome: | The proposed model outperforms previous methods by 1.4% in the document-level EAE task. |
MultiCMET: A Novel Chinese Benchmark for Understanding Multimodal Metaphor (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on multimodal metaphors does not address categorizing the source and target domains in metaphors beyond the English language. |
| Approach: | They propose a Cascading Domain Knowledge Integration benchmark to detect metaphors by introducing domain-specific lexical features. |
| Outcome: | The proposed dataset includes 13,820 text-image pairs of advertisements with manual annotations of the occurrence of metaphors, domain categories, and sentiments metaphors convey. |
GlotLID: Language Identification for Low-Resource Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing web-mined datasets for low-resource languages have been useful for low resource NLP. |
| Approach: | They propose a model that identifies 1665 low-resource languages and a new model that is rigorously evaluated and reliable. |
| Outcome: | The proposed model outperforms baselines when balancing F1 and false positive rate (FPR). |
Finding Support Examples for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning is a new learning paradigm where a language model observes a few examples and directly outputs the test input’s prediction. |
| Approach: | They propose a method to find “support examples” for in-context learning by filtering a training dataset and a progressive filtering process to filter out uninformative examples. |
| Outcome: | The proposed method outperforms baselines and shows that each component contributes critically to the improvements. |
Uncovering the Root of Hate Speech: A Dataset for Identifying Hate Instigating Speech (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive datasets specifically annotated for hate instigating speech hinders research . lack of reliable models for hate triggering makes it difficult to apply off-the-shelf models to the problem. |
| Approach: | They propose to use a multilingual dataset to identify hate instigating speech . lack of comprehensive datasets specifically annotated for hate instigators hinders their work . |
| Outcome: | The proposed dataset identifies hate instigating speech across languages . lack of comprehensive datasets makes it difficult to train and evaluate models . |
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examines research and reporting practices for text summarization tasks . text summaries are often overlooked by the responsible AI community . |
| Approach: | They examine research and reporting practices in the context of text summarization . they find that relatively few papers engage with possible stakeholders . |
| Outcome: | The findings highlight current research practices and provide recommendations on research directions. |
Improving Speech Translation by Fusing Speech and Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In speech translation, multimodal data to address limitations of individual modalities has shown significant effectiveness. |
| Approach: | They propose a cross-modal model which supports three input modalities for speech, text and fused speech-text. |
| Outcome: | The proposed model achieves an average of 34.0 BLEU on MuST-C, GigaST and newstest benchmark. |
Narrative Order Aware Story Generation via Bidirectional Pretraining Model with Optimal Transport Reward (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing storytelling systems suffer from insufficient understanding of event correlations and inadequate awareness of event temporal order. |
| Approach: | They propose a narrative order aware framework to generate coherent stories with flashbacks . they propose 'bidirectional pretraining model with Optimal Transport Reward' to improve quality . |
| Outcome: | The proposed framework generates coherent stories with flashbacks with a novel optimal transport reward. |
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing claims verification models rely on annotated data, which is expensive to create at a large scale. |
| Approach: | They propose a model that can verify complex claims without annotated data . they leverage the in-context learning ability of Large Language Models to translate a claim into a First-Order-Logic clause . |
| Outcome: | The proposed model outperforms baseline models on three datasets . it performs well on the datasets, and the results are published online. |
Strong and Efficient Baselines for Open Domain Conversational Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current Open Domain Question Answering (ODQA) approaches to reevaluating baselines for efficiency and effectiveness have been limited due to various limitations. |
| Approach: | They propose and evaluate strong yet simple and efficient baselines by introducing a fast reranking component between the retriever and the reader and performing targeted finetuning steps. |
| Outcome: | The proposed method improves the SotA results while reducing the reader’s latency by 60%. |
Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. |
| Approach: | They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens. |
| Outcome: | The proposed method outperforms existing temporal language models on two datasets and three tasks. |
Retrieval-Augmented Parsing for Complex Graphs by Exploiting Structure and Uncertainty (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval augmentation is effective for large graph parsing tasks, but can fail to identify the most informative exemplars . structure-aware and uncertainty-guided adaptive retrieval (SUGAR) exploits two unique sources of information: structural similarity and model uncertainty. |
| Approach: | They propose a structure-aware and uncertainty-guided adaptive retrieval approach that exploits structural similarity and model uncertainty to improve retrieval-augmented parsing for complex graph problems. |
| Outcome: | The proposed method improves retrieval-augmented parsing for graph parsers with large output graphs and non-trivial structure. |
When it Rains, it Pours: Modeling Media Storms and the News Ecosystem (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Occasionally, an event triggers a media storm, with coverage lasting weeks instead of days. |
| Approach: | They develop a pairwise article similarity model to identify story clusters in news corpora and build a corpus of media storms over a nearly two year period. |
| Outcome: | The proposed model validates theories about storm evolution and topical distribution and supports hypotheses about storm influence on media coverage and intermedia agenda setting. |
Intra-Event and Inter-Event Dependency-Aware Graph Network for Event Argument Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not build dependency information among event argument roles . Existing methods do not learn the interactions between different roles based on event structure . |
| Approach: | They propose an intra-event and inter-e event dependency-aware graph network to model dependencies between roles . they use event structure as the fundamental unit to construct role dependencies within events . |
| Outcome: | The proposed model improves on the ACE05, RAMS, and WikiEvents datasets. |
From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence retrieval models are based on probability ranking principle . existing models do not align with retrieval-enhanced verification frameworks . |
| Approach: | They propose a feedback-based evidence retriever that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. |
| Outcome: | Empirical studies show that the proposed method is superior to baseline methods. |
How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
| Challenge: | Existing techniques to improve dense retrieval suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, some argue due to the limited model capacity. |
| Approach: | They propose to use diverse queries and sources of supervision to train a generalizable DR to achieve high accuracy in both supervised and zero-shot retrieval. |
| Outcome: | The proposed DR can achieve state-of-the-art in supervised and zero-shot evaluations without increasing model size. |
Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. |
| Approach: | They propose a method to quantify the severity of shortcut reasoning by leveraging out-of-distribution data. |
| Outcome: | The proposed method quantifies the severity of the discovered shortcut reasoning using out-of-distribution data. |
Schema-adaptable Knowledge Graph Construction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Knowledge Graph Construction (KGC) tasks rely on static information extraction with a closed set of pre-defined schemas. |
| Approach: | They propose a static knowledge Graph Construction task that extracts entity, relation, and event based on dynamically changing schema graph without retraining. |
| Outcome: | The proposed system outperforms existing methods but still has room for improvement . it can extract entity, relation, and event based on dynamically changing schema graph without re-training . |
Evaluating the Knowledge Base Completion Potential of GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC) however, their ability to do this at scale and with high accuracy remains an open question. |
| Approach: | They propose to use language models to complete a large public KB, Wikidata, with 90% precision. |
| Outcome: | The proposed models can extend Wikidata by 27M facts at 90% precision. |
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks or datasets require only a few steps of reasoning, making it difficult to analyse AI’s behaviour with reference to different problems within a specific topic in detail. |
| Approach: | They propose a conic10K math problem dataset that requires only a few steps of reasoning to be analysed. |
| Outcome: | The proposed dataset shows that existing language models exhibit weak performance on complex reasoning. |
DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for spatial reasoning in text overlook the gap between natural language and symbolic structures. |
| Approach: | They propose a novel depth-wise Graph Neural Network to aggregate spatial information over the depth dimension instead of the breadth dimension of the graph. |
| Outcome: | The proposed model outperforms existing methods on two multi-hop spatial reasoning datasets. |
TK-KNN: A Balanced Distance-Based Pseudo Labeling Approach for Semi-Supervised Intent Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-supervised methods for detecting intent generate a large amount of unlabeled data . labeling data requires substantial human effort, and picking an imbalanced set of examples could lead to poor labels. |
| Approach: | They propose a balanced distance-based pseudo-labeling approach for semisupervised intent classification . they use a ranking-based approach to select samples with a model prediction confidence . |
| Outcome: | The proposed method outperforms existing models on popular datasets. |
Late Fusion of Transformers for Sentiment Analysis of Code-Switched Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Code-switching is a common phenomenon in multilingual communities . however, sentiment analysis of code-switch data is elusive and unexplored . |
| Approach: | They propose to combine two transformers using logits of their output and feed them to a neural network for sentiment analysis. |
| Outcome: | The proposed system achieves an F1 score of 73.66% for English-Hi and 61.24% for English . it outperforms the best model reported for the GLUECoS benchmark dataset. |
Inductive Relation Inference of Knowledge Graph Enhanced by Ontology Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to inference knowledge graphs lack ontology information, which is often too sparse. |
| Approach: | They propose a knowledge graph inductive inference method that fuses ontology information to learn the semantic information of entities. |
| Outcome: | The proposed method outperforms large language models like ChatGPT on two benchmark datasets and improves the MRR metrics by 15.4% and 44.1%, respectively. |
Dynamic Stance: Modeling Discussions by Labeling the Interactions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Stance detection is a popular task that has been modeled as a static task, but its limitations are strong topic-dependent. |
| Approach: | They propose to model stance as a dynamic task by focusing on interactions between a message and their replies. |
| Outcome: | The proposed model shows portability across topics and languages. |
Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empathetic dialogue is an essential part of building harmonious social relationships and contributes to the development of a helpful AI. |
| Approach: | They propose three methods to improve the performance of large language models (LLMs) they propose semantically similar in-context learning, two-stage interactive generation and combination with the knowledge base. |
| Outcome: | The proposed methods achieve state-of-the-art in automatic and human evaluations and the possibility of GPT-4 simulating human evaluators. |
GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves (2023.findings-emnlp)
Copied to clipboard
| Challenge: | We use GPT-4 to quantify dissent among members on the topic of inflation . transcripts and minutes reflect the diversity of member views in a way that is lost or omitted from the public statements. |
| Approach: | They use transcripts and minutes to quantify dissent among FOMC members . they find that transcripts reflect diversity of member views in a way that is lost or omitted . |
| Outcome: | The proposed method better captures extremes, which mirror human annotations, and suggests that Large Language Models can avoid noise in this nuanced context. |
DialogQAE: N-to-N Question Answer Pair Extraction from Customer Service Chatlog (2023.findings-emnlp)
Copied to clipboard
Xin Zheng, Tianyu Liu, Haoran Meng, Xu Wang, Yufan Jiang, Mengliang Rao, Binghuai Lin, Yunbo Cao, Zhifang Sui
| Challenge: | Existing work on question-answer extraction fails to integrate incomplete utterances from dialog context for composite QA retrieval. |
| Approach: | They propose a task where questions and corresponding answers might be separated across different utterances. |
| Outcome: | The proposed methods perform well on 5 customer service datasets and set a benchmark for N-to-N DialogQAE with utterance and session level evaluation metrics. |
Inverse Reinforcement Learning for Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that inverse reinforcement learning (RL) training has certain disadvantages such as object mismatch and exposure bias. |
| Approach: | They propose inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models. |
| Outcome: | The proposed model outperforms MLE and RL baselines on ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations. |
MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent knowledge-based visual question answering approaches miss visual information captured by captions and cannot fully utilize the visual information required to answer the question. |
| Approach: | They propose a framework that extracts visual information from an image and prompts an LLM to extract query-specific knowledge from the extracted textual information. |
| Outcome: | Empirical results show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |
From Chaos to Clarity: Claim Normalization to Empower Fact-Checking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social media posts are noisy and pervasive, resulting in difficult to identify precise and prominent claims that require verification. |
| Approach: | They propose a task called Claim Normalization that decomposes complex and noisy social media posts into more straightforward and understandable forms, termed normalized claims. |
| Outcome: | The proposed model outperforms baselines across evaluation measures and errors. |
Mitigating Biases in Hate Speech Detection from A Causal Perspective (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to detect hate speech are prone to spurious correlations between training data and labels, which could lead to biased treatment of vulnerable and minority groups. |
| Approach: | They propose to use grammar induction to find grammar patterns for hate speech and analyze this phenomenon from a causal perspective. |
| Outcome: | The proposed methods can detect hate speech from a causal perspective and are robust to different datasets. |
Unmasking the Hidden Meaning: Bridging Implicit and Explicit Hate Speech Embedding Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to detect explicit hate speech (HS) are focusing on detecting explicit forms of hateful expressions on user-generated content. |
| Approach: | They propose to examine the differences between embedding implicit and explicit hateful messages . they compare and link explicit and implicit hateful message across datasets . |
| Outcome: | The proposed model improves on explicit hate speech detection while retaining high performance on borderline cases. |
PerturbScore: Connecting Discrete and Continuous Perturbations in NLP (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language processing (NLP) applications are growing rapidly due to discrete nature of texts. |
| Approach: | They propose to connect discrete perturbations with continuous perturbations to help understand discrete ones in NLP models. |
| Outcome: | The proposed method surpasses methods used in discrete perturbation measuring and can be generalized to different datasets, perturbation methods. |
InstructoR: Instructing Unsupervised Conversational Dense Retrieval with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for conversational retrieval only fine-tune on limited supervised data, making it difficult for the retriever to fully grasp the entire conversation. |
| Approach: | They propose a method to instruct unsupervised conversational dense retrieval with large language models (LLMs) they use supervised data to discover the user's query intent from the conversation context . |
| Outcome: | The proposed method can bring significant improvements across various ad-hoc retrievers, surpassing the current state-of-the-art method. |
The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system, but its quality is often brought into question. |
| Approach: | They argue that the generation of more esoteric forms of language constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance. |
| Outcome: | The proposed system generates coherent and well-formed text of a particular type, usually given an input such as a prompt, outline, topic, or data. |
INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models (2023.findings-emnlp)
Copied to clipboard
H S V N S Kowndinya Renduchintala, Krishnateja Killamsetty, Sumit Bhatia, Milan Aggarwal, Ganesh Ramakrishnan, Rishabh Iyer, Balaji Krishnamurthy
| Challenge: | Pre-trained language models have a remarkable improvement in generalization capability . however, this leads to prohibitively long training times and a detrimental environmental impact . |
| Approach: | They propose to use submodular optimization to select highly informative subsets of training data to train multiple PTLMs using only fractions of data. |
| Outcome: | The proposed framework achieves 99% of the performance of fully-trained models using only fraction of training data. |
Towards General Error Diagnosis via Behavioral Testing in Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing behavioral testing approaches only evaluate translation quality without references, restricting diagnosis to specific types of errors. |
| Approach: | They propose a bilingual translation pair generation based behavior testing framework that auto-generates test cases and pseudo-references to facilitate general error diagnosis. |
| Outcome: | The proposed framework can provide comprehensive and accurate behavioral testing results for general error diagnosis on machine translation systems. |
Retrieval-Augmented Few-shot Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for retrieval-augmented text classification are successful in the few-shot scenario with limited retrieval space. |
| Approach: | They propose to use EM-L and R-L to provide task-specific guidance to retrieval metric . they also propose to incorporate retrieved memory alongside parameters for better generalization . |
| Outcome: | The proposed methods perform better on the few-shot scenario with limited retrieval space. |
Temporal Extrapolation and Knowledge Transfer for Lifelong Temporal Knowledge Graph Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for lifelong TKG reasoning only address part of the challenges. |
| Approach: | They propose a temporal-path-based reinforcement learning framework for lifelong TKG reasoning . they add temporal displacement into the action space of RL to extrapolate for the future . |
| Outcome: | The proposed model outperforms existing models against well-adapted baselines on three lifelong TKG reasoning benchmarks. |
Comparing Prompt-Based and Standard Fine-Tuning for Urdu Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have demonstrated the efficacy of pre-trained language models for various downstream tasks. |
| Approach: | They compare prompt-based fine-tuning with standard fine-uning for text classification in Urdu and Roman Urdu languages. |
| Outcome: | The proposed approach improves up to 13% in accuracy in low-resource languages with limited labeled examples over standard fine-tuning approaches. |
Explore the Way: Exploring Reasoning Path by Bridging Entities for Effective Cross-Document Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on capturing implicit relations between entities, but humans use explicit information chains to find the relations between two entities. |
| Approach: | They propose a cross-document relation extraction task that captures the relation between two entities mentioned in different documents. |
| Outcome: | The proposed model outperforms baseline models in the CodRED task. |
The student becomes the master: Outperforming GPT3 on Scientific Factual Error Correction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Factual Claim Correction rely on a verification model to guide the correction process. |
| Approach: | They propose a claim correction system that does not require a verifier but outperforms existing methods by a considerable margin. |
| Outcome: | The proposed system outperforms existing methods by a considerable margin on the SciFact dataset, 77% on SciFACT-Open and 72.75% on the CovidFact data set. |
Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models, including large language models (LLMs), achieve superior performance on multi-hop question-answering tasks. |
| Approach: | They propose to use the chain-of-thought mechanism to generate both the reasoning chain and the answer. |
| Outcome: | Empirical results show that the proposed framework generates more faithful reasoning chains and significantly improves the QA performance on two benchmark datasets. |
Hierarchical Catalogue Generation for Literature Review: A Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Scientific literature review generation aims to extract and organize important information from an abundant collection of reference papers and produces corresponding reviews while lacking a clear and logical hierarchy. |
| Approach: | They propose a task to generate a hierarchical catalogue of a review paper given various references by using a database of 7.6k literature review catalogues and 389k reference papers. |
| Outcome: | The proposed method produces a hierarchical catalogue of a review paper given various references. |
MCC-KD: Multi-CoT Consistent Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable abilities in complex reasoning through chain of thought (CoT) prompting. |
| Approach: | They propose to generate multiple rationales for each question and enforce consistency among their predictions by minimizing the bidirectional KL-divergence between the answer distributions. |
| Outcome: | The proposed model achieves superior performance on in-distribution and commonsense reasoning benchmarks. |
An Empirical Study of Frame Selection for Text-to-Video Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for text-to-video retrieval select a subset of frames to represent video content . current methods only explore video contents while ignoring relevancy to texts . |
| Approach: | They propose to use a subset of frames to represent video content for TVR . they analyze six different frame selection methods to determine their effectiveness . |
| Outcome: | The proposed method improves retrieval efficiency without sacrificing visual details . the proposed method explores the video contents while ignoring relevancy to texts . |
Conditional Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | False positive: widely-used feature-attribution explanation models are not suitable for finding conditions when sentences are long and are written independently. |
| Approach: | They propose a task of conditional natural language inference to explain sentence pairs that provide contradictory (different) information for different conditions. |
| Outcome: | The proposed model outperforms the full cross-encoder and other baselines in extracting conditions while maintaining comparable accuracy. |
Contrastive Distant Supervision for Debiased and Denoised Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Distant supervision (DS) is a promising learning approach for machine reading comprehension (MRC) however, the annotated dataset will inevitably lead to mislabeled instances, resulting in answer bias and context noise problems. |
| Approach: | They propose an algorithm that can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. |
| Outcome: | The proposed algorithm can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. |
KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awareness (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown their superiority by pre-training on unstructured text corpus and then fine-tuning on downstream tasks. |
| Approach: | They propose a Knowledge-Enhanced Pre-trained LanguagE model with Topic entity awareness that incorporates the interactions between tokens and mentioned entities in pre-training. |
| Outcome: | The proposed model incorporates the interactions between tokens and mentioned entities in pre-training and is more effective on entity-centric tasks. |
Revisiting Large Language Models as Zero-shot Relation Extractors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that large language models (LLMs) transfer well to new tasks out-of-the-box . relationship extraction (RE) involves a certain degree of labeled or unlabeled data even under zero-shot setting. |
| Approach: | They propose a simple prompt recursively using LLMs to transform RE inputs to QA format . they propose qq prompting and qt prompting to improve their results . |
| Outcome: | The proposed method improves on different model sizes, benchmarks and settings. |
Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for dialogue summarization only apply to specific scenarios and domains. |
| Approach: | They propose a pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization. |
| Outcome: | The proposed model significantly outperforms state-of-the-art models on three dialogue summarization datasets from different scenarios and domains. |
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning. |
| Approach: | They propose a framework that enables LLMs to create their own tools using documentation and code realization. |
| Outcome: | The proposed framework outperforms existing chain-of-thought, program-of thought, and tool-using baselines on MATH and TabMWP benchmarks. |
Query-based Image Captioning from Multi-context 360cdegree Images (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing image captioning datasets follow salient images with limited camera field of view, ignoring minor details. |
| Approach: | They propose a task where a query specifies the context to describe in 360-degree images and construct a dataset for the task that contains 3,940 360- degree images and 18,459 pairs of queries and captions annotated manually. |
| Outcome: | The proposed task is more challenging than the conventional image captioning task, which describes salient objects in images. |
Auto Search Indexer for End-to-End Document Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative retrieval heavily relies on the “preprocessed” document identifiers, thus limiting its retrieval performance and ability to retrieve new documents. |
| Approach: | They propose a fully end-to-end retrieval paradigm that can learn the best docids for existing and new documents automatically via a semantic indexing module. |
| Outcome: | The proposed model outperforms baselines on public and industrial datasets and can handle new documents. |
‘Person’ == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using CLIP-cosine similarity for zero-shot classification of images, we chronicle results from 136 prompts (50 results/prompt) of front-facing images of faces from 6 different continents, 27 countries and 3 genders. |
| Approach: | They use CLIP-cosine similarity for zero-shot classification of images generated by CLIP based Stable Diffusion v2.1 verified by manual examination to determine what gender and nationality/continental identity is assigned to ‘a person’. |
| Outcome: | The results show that the image generator Stable Diffusion displays gender and nationality/continental identity in the absence of such information. |
Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms for learning unimodal vision-only or language-only tasks are limited by the size and computational load of fine-tuning large-scale pre-trained neural networks. |
| Approach: | They propose a transformer-based CL architecture for learning bimodal vision-and-language tasks by increasing the number of the learnable parameters dynamically and using knowledge distillation. |
| Outcome: | The proposed model reaches state-of-the-art on vision-and-language tasks. |
Evaluating Verifiability in Generative Search Engines (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing generative search engines are rapidly gaining users, according to a new study . existing systems are poorly cited and lack reliability, a study finds . |
| Approach: | They conduct human evaluations of four popular generative search engines . they find that existing generative engines are fluent and appear informative . |
| Outcome: | The results show that existing generative search engines are not reliable and often contain unsupported statements and inaccurate citations. |
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are slow and abstractive, but we propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries. |
| Approach: | They propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries by exposing diverse pseudo summary with two supervision to the student model. |
| Outcome: | The proposed method outperforms previous methods in abstractive summarization distillation, producing highly abstractive and informative summaries. |
Visually Grounded Continual Language Learning with Selective Specialization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the role of model components in learning specialized or generalizable representations are lacking. |
| Approach: | They propose to analyze selection strategies for visually grounded continual language learning using two diagnostic datasets. |
| Outcome: | The proposed models outperform existing models and provide enough control and flexibility for a thorough model analysis. |
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. |
| Approach: | They introduce RoMQA, the first benchmark for robust, multi-evidence, multianswer question answering (QA) RoMQ contains clusters of related questions that are derived from the Wikidata knowledge graph . |
| Outcome: | The proposed model is the first benchmark for robust, multi-evidence, multianswer question answering (QA) compared to prior QA datasets, it has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. |
Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches focus on language-guided classifiers that can generalize in zero-shot settings, but their performance varies significantly between different language explanations in unpredictable ways. |
| Approach: | They propose a framework that uses data programming to adapt a language-guided classifier for a new task when provided with multiple teachers and unlabeled test examples. |
| Outcome: | The proposed framework outperforms a baseline from previous work by 9.3%. |
Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for abstractive text summarization do not provide explicit interdocument relationships among source documents. |
| Approach: | They propose a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features. |
| Outcome: | The proposed model outperforms baseline models in terms of evaluation metrics but struggle to handle conflicts in source documents. |
VIPHY: Probing “Visible” Physical Commonsense Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that vision-language models can retain and generalize knowledge, but they do not measure their ability to retain it. |
| Approach: | They build an automatic pipeline to derive a knowledge resource for calibrating and probing vision-language models. |
| Outcome: | The proposed model outperforms the pretrained model on size and spatial tasks. |
Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can generate natural language texts for various domains and tasks, but their potential for clinical text mining is under-explored. |
| Approach: | They propose a pragmatic taxonomy for AD sign and symptom progression based on expert knowledge and train a system to detect AD-related signs and symptoms from EHRs. |
| Outcome: | The proposed taxonomy outperforms existing methods using only the gold dataset and silver datasets. |
Stylized Dialogue Generation with Feature-Guided Knowledge Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods synthesize pseudo data through back translation but lack guidance on target style features. |
| Approach: | They propose a knowledge-augmented stylized dialogue generation model with a feature-guided style knowledge selection module that utilizes context and response features. |
| Outcome: | The proposed model produces a satisfactory performance on two public benchmarks on both semantic and stylized levels. |
Probing LLMs for Joint Encoding of Linguistic Categories (2023.findings-emnlp)
Copied to clipboard
Giulio Starace, Konstantinos Papakostas, Rochelle Choenni, Apostolos Panagiotopoulos, Matteo Rosati, Alina Leidinger, Ekaterina Shutova
| Challenge: | Existing research suggests that a linguistic hierarchy emerges in large language models . little is known about how encodings of different linguistic phenomena interact within the models - and to what extent processing of linguistically-related categories relies on the same, shared model representations. |
| Approach: | They propose a framework for testing the joint encoding of linguistic categories in large language models. |
| Outcome: | The proposed framework shows that the same patterns hold across languages in multilingual LLMs. |
On Robustness of Finetuned Transformer-based NLP Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Transformer-based language models have been finetuned for a large number of tasks. |
| Approach: | They characterize changes between pretrained and finetuned models with CKA and STIR metrics. |
| Outcome: | The proposed models are more robust to perturbations than BERT and T5 on classification tasks and generation tasks. |
Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In task-oriented semantic parsing, the system aims to translate users’ utterances in natural language to machine-interpretable programs (API calls) However, Large Language Models (LLMs) are known to hallucinate and therefore pose a formidable challenge in constraining generated content. |
| Approach: | They propose to use large language models to translate user's utterances to machine-interpretable programs (API calls) they identify constraints violations in task-oriented utterrances and define fine-grained metrics that complement traditional ones. |
| Outcome: | The proposed methods reduce constraints violations and improve quality of the generated API calls, but require careful consideration given their implementation complexity and latency. |
Entity Disambiguation on a Tight Labeling Budget (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to training entity disambiguation models require a small labeling budget . a defense research analyst might need to map military equipment to a knowledge base describing emergent defense technologies. |
| Approach: | They propose a method that combines feature diversity with low rank correction . they use bilinear tensor models to train a model that uses a rich representation of context . |
| Outcome: | The proposed approach reduces the amount of labeled data necessary to achieve a given performance. |
Topic-DPR: Topic-based Prompts for Dense Passage Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research focused on optimizing a single prompt as a continuous prompt, but this approach leads to a semantic space collapse, preventing differentiation between relevant and irrelevant passages. |
| Approach: | They propose a dense passage retrieval model that uses topic-based prompts and propose 'positive and negative sampling strategies' to boost dense retrieval efficiency. |
| Outcome: | The proposed model surpasses state-of-the-art retrieval techniques and improves space uniformity. |
Quantifying the Dialect Gap and its Correlates Across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Historically, studies investigating minority variants of languages have been limited to a select few languages. |
| Approach: | They evaluate state-of-the-art large language models for regional dialects of several high- and low-resource languages and analyze how regional dialect gap is correlated with economic, social, and linguistic factors. |
| Outcome: | The proposed model is compared with two high-use applications and shows that it can solve the regional dialect gap. |
RECAL: Sample-Relation Guided Confidence Calibration over Tabular Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Various machine learning methods for tabular data lack accurate confidence estimation, which is needed for high-risk sensitive applications such as credit modeling and financial fraud detection. |
| Approach: | They propose a general post-training confidence calibration framework to calibrate the confidence of current machine learning models by employing graph neural networks to model the relationships between different samples. |
| Outcome: | The proposed framework improves the confidence estimation on tabular datasets by using graph neural networks to model the relationships between different samples. |
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages. |
| Approach: | They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities. |
| Outcome: | The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters. |
Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. |
| Approach: | They propose to partition test set based on lexical similarity of reference test summaries with training summary to determine model competencies. |
| Outcome: | The proposed evaluation protocol improves generalization and generalization on novel test cases while maintaining average performance. |
Pseudointelligence: A Unifying Lens on Language Model Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies claim that language models surpass human performance on new benchmarks within a few years. |
| Approach: | They propose a framework for model evaluation that casts as a dynamic interaction between a model and a learned evaluator. |
| Outcome: | The proposed framework can be used to reason about two case studies in language model evaluation, and analyze existing evaluation methods. |
GDA: Grammar-based Data Augmentation for Text Classification using Slot Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies suggest data augmentation approaches to resolve the low-resource problem in natural language processing tasks. |
| Approach: | They propose to use slot information to augment sentences using a set of injective relations between a sentence’s semantics and its syntactical structure to augment the dataset. |
| Outcome: | The proposed approach outperforms all other data augmentation methods by 19.38%. |
Implicit Sense-labeled Connective Recognition as Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for identifying implicit discourse relations are limited by the number of possible categories and sense labels. |
| Approach: | They propose a method for identifying the sense label of an implicit connective between adjacent text spans by using an encoder-decoder model. |
| Outcome: | The proposed method outperforms the conventional classification-based method on a shallow discourse parsing dataset. |
VISTA: Visual-Textual Knowledge Graph Representation Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding methods only consider the structure of a knowledge graph, but some recent proposed methods utilize images or text descriptions of entities in a VTKG. |
| Approach: | They propose a visual-textual knowledge graph (VTKG) where triplets can be explained using images and entities and relations can accompany text descriptions. |
| Outcome: | The proposed method outperforms state-of-the-art knowledge graph completion methods in real-world knowledge graphs. |
Dynamic Stashing Quantization for Efficient Transformer Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. |
| Approach: | They propose a dynamic quantization strategy that reduces the amount of memory operations and reduces arithmetic cost by 20.95 on two translation tasks and three classification tasks. |
| Outcome: | The proposed model reduces the amount of arithmetic operations by 20.95 and the number of DRAM operations by 2.55 on two translation tasks and three classification tasks. |
A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. |
| Approach: | They propose a framework to investigate LLMs' competence in the law domain by using similar cases and multi-choice options. |
| Outcome: | The proposed solutions can be extended to other domains to facilitate evaluations in other domain. |
A Lightweight Method to Generate Unanswerable Questions in English (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to build robust question answering models are too complex . antonym and entity swaps on answerable questions are used to build models . |
| Approach: | They propose a method for performing antonym and entity swaps on unanswerable questions. |
| Outcome: | The proposed method outperforms the previous state-of-the-art and has higher human-judged relatedness and readability. |
Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations of dialogue quality rely on human judgments, which are time-consuming, labor-intensive, prone to biases, and lacking objectivity. |
| Approach: | They propose a method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses. |
| Outcome: | The proposed method proves that human judgments are time-consuming, labor-intensive, and lacking objectivity. |
TabPrompt: Graph-based Pre-training and Prompting for Few-shot Table Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of Table Understanding (TU) focus on the textual content within the tabular data, disregarding the topological information of the table. |
| Approach: | They propose a framework that uses tabs to understand tabular data without ignoring the topological information of the table. |
| Outcome: | The proposed framework outperforms baselines in few-shot table understanding tasks. |
Towards Formality-Aware Neural Machine Translation by Leveraging Context Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Formality is one of the most important linguistic properties to determine the naturalness of translation. |
| Approach: | They propose a method to explicitly inform neural machine translation models by pinpointing key informative tokens using a formality classifier. |
| Outcome: | The proposed method improves translation quality and conforms to the appropriate syntax. |
Improving Seq2Seq Grammatical Error Correction via Decoding Interventions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to grammatical error correction (GEC) are sequence-to-sequence and sequence-edit. |
| Approach: | They propose a unified decoding intervention framework that employs an external critic to assess the appropriateness of the token to be generated incrementally. |
| Outcome: | The proposed framework outperforms baselines and state-of-the-art methods on English and Chinese datasets. |
Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we generate code-tracing questions based on code snippets and descriptions. |
| Approach: | They propose to use large language models to generate code-tracing questions in introductory programming courses by using GPT4 prompts. |
| Outcome: | The proposed model generates code-tracing questions based on code snippets and descriptions. |
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Model-agnostic meta-learning has garnered attention as a promising technique for enhancing few-shot cross-lingual transfer learning in low-resource scenarios. |
| Approach: | They propose a Meta-Task Collector-based Cross-lingual Meta-Transfer framework to adapt data selection strategies to construct cross-lingual meta-tasks to reduce language gaps. |
| Outcome: | The proposed framework significantly improves model performance in the target language with minimal annotation costs. |
Reasoning Makes Good Annotators : An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract knowledge from unlabeled data generate noise labels. |
| Approach: | They propose an automatic task-specific rules distilling framework to generate a logic rule from unlabeled data. |
| Outcome: | The proposed framework could power the labeling ability by discovering reliable model-labeled data. |
Co-training and Co-distillation for Quality Improvement and Compression of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) compresses expensive pre-trained language models . however, most smaller models fail to surpass performance of larger model . |
| Approach: | They propose a framework that co-trains two models while mutually distilling knowledge to improve performance and inference speed together. |
| Outcome: | The proposed framework outperforms the original larger model by 1.66 on the GLUE benchmark. |
ReadPrompt: A Readable Prompting Method for Reliable Knowledge Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to prob pre-trained language models (PLMs) lack readability and credibility. |
| Approach: | They propose a method to identify meaningful sentences to serve as prompts to assess the knowledge encoded within pre-trained language models (PLMs). |
| Outcome: | The proposed method achieves state-of-the-art on the current knowledge probing benchmark. |
Coherent Entity Disambiguation via Modeling Topic and Categorical Dependency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing entity disambiguation methods struggle to capture explicit discourse-level dependencies, resulting in incoherent predictions at the abstract level. |
| Approach: | They propose an unsupervised variational autoencoder to extract latent topic vectors of context sentences to enhance coherence of entity predictions. |
| Outcome: | The proposed system achieves state-of-the-art on popular ED benchmarks with an average improvement of 1.3 F1 points. |
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that large language models can be used to predict performance on new configurations. |
| Approach: | They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard . |
| Outcome: | The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set. |
POSQA: Probe the World Models of LLMs with Size Comparisons (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embodied language comprehension emphasizes that language understanding is not only mental processing in the brain but also involves interactions with the physical and social environment. |
| Approach: | They propose to use a physical object size question to examine the extremity of large language models to test their embodied comprehension. |
| Outcome: | The proposed dataset shows that even the largest LLMs perform poorly under the zero-shot setting. |
Hierarchical Fusion for Online Multimodal Dialog Act Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fusion. |
| Approach: | They propose a framework for online multimodal dialog act (DA) classification based on raw audio and ASR-generated transcriptions of current and past utterances. |
| Outcome: | The proposed model achieves a significant increase in the F1 score relative to current state-of-the-art models on two prominent DA classification datasets, MRDA and EMOTyDA. |
STEER: Unified Style Transfer with Expert Reinforcement (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show unified style transfer models outperform the 175B instruction-tuned GPT-3 on overall style transfer quality. |
| Approach: | They propose a unified style transfer framework that can transfer to multiple target styles from an arbitrary source style. |
| Outcome: | The proposed method outperforms the 175B instruction-tuned GPT-3 on overall style transfer quality despite being 226 times smaller in size . |
Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Argument structure extraction (ASE) aims to identify the discourse structure of arguments within documents. |
| Approach: | They propose an Efficient Context-aware ASE model that fully exploits contextual information by augmenting modeling capacity and augmenting training data. |
| Outcome: | The proposed model can extract argumentative discourse structure from documents and reduce reliance on specific words or less informative sentences. |
Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on inconsistency issues within a single LLM, while we explore the inter-consistencies among multiple LLMs for collaboration. |
| Approach: | They propose a formal debate framework to examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal. |
| Outcome: | The proposed framework enables LLMs to achieve consensus in three real-world debate scenarios with real-time scenarios aligned to the LLM's goals. |
Culturally Aware Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Cultural norms are behavioral rules and conventions shared within specific groups, connecting cultural symbols and values. |
| Approach: | They propose a task that operationalizes cultural variations in language understanding through a natural language inference task that surfaces cultural variations as label disagreement between annotators from different cultural groups. |
| Outcome: | The proposed model can be evaluated at which levels it is culturally aware. |
End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Reply suggestion systems are poorly suited for out-of-the-box retrieval architectures, which only consider individual message-reply similarity. |
| Approach: | They propose an autoregressive text-to-text retrieval model that learns the smart reply task end-to end from a dataset of (message, reply set) pairs obtained via bootstrapping. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on three datasets and shows that it is more diverse and relevant to the user. |
Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on editing LLMs neglects the dependency between a fact and its logical implications. |
| Approach: | They propose an evaluation protocol that assesses the editing process using StandUp . they aim to ensure that the editing respects internal logical constraints . |
| Outcome: | The proposed evaluation protocol assesses the editing process using a standup question-answering dataset. |
Effects of Human Adversarial and Affable Samples on BERT Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Limiting quantities of training data is considered a key impediment to achieving generalizability in machine learning. |
| Approach: | They examine the impact of training data quality, not quantity, on a model’s generalizability by comparing human-adversarial and human-affable training samples. |
| Outcome: | The proposed model performance improves with 10-30% h-adversarial instances in text classification and relation extraction tasks. |
Logic Unveils Truth, While Disguise Obscures It: Transition Logic Augmented Response Selection for Multi-Turn Dialogue (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of negative samples tend to yield false negatives due to one-to-many property in open-domain dialogue. |
| Approach: | They propose a sequential variational ladder auto-encoder to capture one-to-many transition pattern of multiple characteristics in open-domain dialogue. |
| Outcome: | The proposed approach improves the performance of a retrieval dialogue system on two benchmarks. |
Are Language Models Worse than Humans at Following Prompts? It’s Complicated (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models perform surprisingly well when given intentionally irrelevant or misleading prompts. |
| Approach: | They challenge an assumption that humans would perform badly when given pathological instructions by ignoring irrelevant prompts and following them faithfully when given misleading instructions. |
| Outcome: | The proposed model performs well when given intentionally irrelevant or misleading prompts, whereas models do not. |
A Sequence-to-Structure Approach to Document-level Targeted Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) has received wide attention in NLP for nearly two decades . previous studies focused on sentence-level ABSA, but document-level research has not received enough attention. |
| Approach: | They propose a Sequence-to-Structure approach to address the document-level targeted sentiment analysis task, which aims to extract the opinion targets consisting of multi-level entities from a review document and predict their sentiments. |
| Outcome: | The proposed approach outperforms baselines on six domains on the document-level targeted sentiment analysis task. |
Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conversational question answering (CQA) requires models to extract answers from given contents to answer follow-up questions according to conversation history. |
| Approach: | They propose a novel architecture that integrates extractive MRC models into a generalized sequence-to-sequence framework. |
| Outcome: | The proposed architecture can use less storage space and consider historical memory deeply and selectively. |
Text2Tree: Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to medical text classification are struggling with imbalanced data distribution and rare labels. |
| Approach: | They propose a framework-agnostic algorithm that only utilizes internal label hierarchy in training deep learning models. |
| Outcome: | The proposed approach performs better on public datasets and real-world medical records than existing methods. |
Impact of Co-occurrence on Factual Knowledge of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often make factually incorrect responses despite their success in various applications. |
| Approach: | They propose to fine tune large language models to mitigate the co-occurrence bias by filtering out biased samples with high subject-object co-occurring counts. |
| Outcome: | The proposed model scales up to debiased datasets to mitigate the co-occurrence bias, but is not effective in recalling rare facts unseen during finetuning. |
CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have demonstrated the capability to perform on machine translation when the input is prompted with a few examples. |
| Approach: | They propose a regression model that combine features influencing example selection to maximize translation quality. |
| Outcome: | The proposed model outperforms random selection and strong single-factor baselines on multiple language pairs and language models. |
Swap and Predict – Predicting the Semantic Changes in Words across Corpora by Context Swapping (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Detecting semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. |
| Approach: | They propose a method that randomly swaps contexts between two different corpora to detect whether a given word changes its meaning . they then use a pretrained masked language model to generate contextualised word embeddings of w, which are then used to predict the semantic changes of words in four languages . |
| Outcome: | The proposed method achieves significant performance improvements compared to baselines for the English semantic change prediction task. |
Beyond Layout Embedding: Layout Attention with Gaussian Biases for Structured Document Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for encoding layout information rely on millions of learnable parameters . polar coordinates provide superior choice for layout modeling, study finds . |
| Approach: | They propose to model layout attention with Gaussian biases by feeding polar coordinates into 2-D Gausssian kernels. |
| Outcome: | The proposed model improves on three widely used benchmarks. |
ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for acquiring local visual information are limited . existing methods for named entity recognition are redundant or insufficient . |
| Approach: | They propose an Entity Spans Position Visual Regions module to obtain visual regions corresponding to entities in the text. |
| Outcome: | The proposed method achieves the SOTA on Twitter-2017 and competitive results on Twitter 2015 . previous efforts have yielded promising results, but they still fall short in selecting visual information. |
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Manually "engineering" prompts for large language models can be laborious and time-intensive. |
| Approach: | They propose a new metric to quantify the expected utility of a language prompt. |
| Outcome: | The proposed metric outperforms previous prompt selection metrics with 10% increase in Pearson correlation across 6 classification benchmarks and the prompt selected by the proposed meter gains 5% higher accuracy than previous metrics. |
Detecting Erroneously Recognized Handwritten Byzantine Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Handwritten text recognition (HTR) produces textual output that contains errors, which are much higher than recognised printed text. |
| Approach: | They investigate the properties of handwritten texts that lead post-correction systems to this adversarial behaviour in Byzantine Greek. |
| Outcome: | The proposed model achieves an average precision score of 95% in Byzantine Greek and 97% in modern and ancient Greek. |
Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment (2023.findings-emnlp)
Copied to clipboard
Boyang Xue, Weichao Wang, Hongru Wang, Fei Mi, Rui Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Experimental results show that pretrained language models generate inconsistent factual knowledge in many conversational tasks. |
| Approach: | They propose a method which explicitly introduces extended feedforward networks (FFNs) in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs. |
| Outcome: | The proposed methods improve the factual expression capability of feedforward networks (FFNs) in knowledge-grounded dialogue systems by knowledge enhancement and alignment respectively. |
TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets (2023.findings-emnlp)
Copied to clipboard
Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, Zhaochuan Gao, Anthony Aue, Arul Menezes, Furu Wei
| Challenge: | Existing approaches to multilingual sequence-to-sequence pre-training rely on monolingual corpora and sometimes synthetic document-level bilingual corporata. |
| Approach: | They propose to leverage document-level trilingual parallel corpora to improve sequence-to-sequence multilingual pre-training by using a novel method called Grafting. |
| Outcome: | The proposed method achieves strong state-of-the-art (SOTA) scores on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark. |
Frequency Balanced Datasets Lead to Better Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence that high-frequency tokens in pretraining data might bias learning, causing undesired effects, is not clear. |
| Approach: | They propose a sampling algorithm that iteratively assesses token frequencies and removes sentences that contain still high-frequency tokens, resulting in a balanced dataset. |
| Outcome: | The proposed method reduces the amount of pre-training data required for training attention-based transformer language models by up to three times. |
Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for pre-trained language models rely on noisy data, which can be expensive if all parameters are updated. |
| Approach: | They propose a self-training framework that incorporates Monte Carlo dropouts into the model and judiciously selects reliable pseudo-labeled examples based on confidence and certainty. |
| Outcome: | The proposed framework improves performance and efficiency over multiple tasks over multiple datasets. |
TR-Rules: Rule-based Model for Link Forecasting on Temporal Knowledge Graph Considering Temporal Redundancy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models suffer from temporal redundancy when leveraged under dynamic settings. |
| Approach: | They propose a temporal knowledge graph extrapolation method which solves temporal redundancy issues by using cyclic rules to capture more information lurking in TKGs. |
| Outcome: | The proposed model captures more information lurking in TKGs, and also mines and properly leverages acyclic rules, which has not been explored by existing models. |
On the Transferability of Visually Grounded PCFGs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on visually grounded grammar induction have not evaluated text domains that are different from the training domain. |
| Approach: | They extend visually grounded grammar induction model to transfer across text domains . they find that benefits transfer to text in a domain similar to the training domain . |
| Outcome: | The proposed model can transfer across text domains but fails to transfer to remote domains. |
Analysis of Style-Shifting on Social Media: Using Neural Language Model Conditioned by Social Meanings (2023.findings-emnlp)
Copied to clipboard
Seiya Kawano, Shota Kanezaki, Angel Fernando Garcia Contreras, Akishige Yuguchi, Marie Katsurai, Koichiro Yoshino
| Challenge: | Using a personalized neural language model, we predict an individual’s conversational style based on surprisals predicted by a personal neural language modeling model. |
| Approach: | They propose a personalized neural language model that predicts changes in an individual’s conversational style based on surprisals predicted by a neural language modeling model. |
| Outcome: | The proposed model outperforms existing models in predicting conversational style-shifting in a test set and shows correlations between it and various conversation factors as well as human evaluation of style- shifting. |
Linguistic Compression in Single-Sentence Human-Written Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Summarizing texts involves significant cognitive efforts to compress information. |
| Approach: | They analyze linguistic patterns in human-written single-sentence summaries to find that they are generally written with morphological expansion, increased lexical diversity, and similar positional arrangements of specific words . |
| Outcome: | The results show that human summarizers are written with morphological expansion, increased lexical diversity, and similar positional arrangements of specific words compared to the source across different genres. |
MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Trending ASR-robust SLU systems have seen impressive improvements through global contrastive learning, but they can easily lead to severe semantic changes. |
| Approach: | They propose a two-stage multi-grained contrastive learning framework to improve ASR robustness . they first adapt pre-trained language models to downstream SLU datasets and then fine-tune it on the corresponding dataset. |
| Outcome: | The proposed framework improves on four datasets and four BERT-like backbone models. |
Beyond Candidates : Adaptive Dialogue Agent Utilizing Persona and Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a previous study suggested that human dialogue systems ground persona and knowledge but they require incomplete candidate sets. |
| Approach: | They propose an adaptive dialogue agent that uses persona and knowledge without candidate sets . their model generates consistent and relevant persona descriptions and identifies relevant knowledge . |
| Outcome: | The proposed model outperforms baselines that ground persona and knowledge candidates even with fragmentary information. |
SmartSpanNER: Making SpanNER Robust in Low Resource Scenarios (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is one of the most fundamental tasks in natural language processing. |
| Approach: | They propose a method which introduces a Named Entity Head (NEH) prediction task to SpanNER and performs multi-task learning together with task of span classification. |
| Outcome: | The proposed method improves the robustness of SpanNER in low resource scenarios on the CoNLL03, Few-NERD, GENIA and ACE05 benchmark datasets. |
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM. |
| Approach: | They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets . |
| Outcome: | The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks. |
Data Selection Curriculum for Abstractive Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive Text Summarization (ATS) models are commonly trained using large-scale data that is randomly shuffled. |
| Approach: | They propose a data selection curriculum scoring system that measures the learning difficulty of an ATS model and expected performance on an instance. |
| Outcome: | The proposed system surpasses baselines on CNN/DailyMail dataset, utilizing 20% of available instances. |
Romanization-based Large-scale Adaptation of Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models are limited by their vocabulary size and parameter budget. |
| Approach: | They explore the potential of leveraging transliteration on a massive scale to improve performance for multilingual pretrained language models. |
| Outcome: | The proposed transliteration tool outperforms other methods on low-resource languages. |
Measuring bias in Instruction-Following models with P-AT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-Following Language Models (IFLMs) are promising and versatile tools for solving many downstream, information-seeking tasks. |
| Approach: | They propose a resource to test whether IFLMs are prone to biases . they cast WEAT word tests in promptized classification tasks and associate a metric - the bias score . |
| Outcome: | The proposed resource consists of 2310 prompts and tests gender and race biases in all the analyzed models. |
Open-ended Commonsense Reasoning with Unrestricted Answer Candidates (2023.findings-emnlp)
Copied to clipboard
Chen Ling, Xuchao Zhang, Xujiang Zhao, Yanchi Liu, Wei Cheng, Mika Oishi, Takao Osaki, Katsushi Matsuda, Haifeng Chen, Liang Zhao
| Challenge: | Current approaches to commonsense reasoning are limited due to limited answer scope. |
| Approach: | They propose to solve a commonsense question without a pre-defined answer scope . they leverage pre-trained language models to iteratively retrieve reasoning paths on the external knowledge base . |
| Outcome: | The proposed method achieves better performance on two commonsense benchmark datasets. |
Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units (2023.findings-emnlp)
Copied to clipboard
| Challenge: | DISSC is a lightweight voice conversion method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. |
| Approach: | They propose a method that converts rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. |
| Outcome: | The proposed method outperforms baseline methods on quantitative and qualitative evaluations. |
Knowledge-Selective Pretraining for Attribute Value Extraction (2023.findings-emnlp)
Copied to clipboard
Hui Liu, Qingyu Yin, Zhengyang Wang, Chenwei Zhang, Haoming Jiang, Yifan Gao, Zheng Li, Xian Li, Chao Zhang, Bing Yin, William Wang, Xiaodan Zhu
| Challenge: | Existing methods for AVE are limited on rare attributes due to poor generalization ability. |
| Approach: | They propose to leverage pretraining and transfer learning to address weaknesses in existing methods. |
| Outcome: | The proposed method achieves new state-of-the-art performance without pretraining on rare attributes with limited training resources. |
New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In bilingual or multilingual settings, code-switching ASR has greater challenges and research value. |
| Approach: | They propose a controllable iterative method for improving the performance of mainstream automatic speech recognition systems by using Chinese-English code-switching dialogues. |
| Outcome: | The proposed method achieves the best performance compared with the rule-based, back-translation-based data augmentation methods and large language model ChatGPT. |
Efficient k-NN Search with Cross-Encoders using Adaptive Multi-Round CUR Decomposition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ANNCUR uses a cross-encoder only to perform k-NN search, but the approximation of the distances is often detrimental to the retrieval of top-k items. |
| Approach: | They propose a method that minimizes approximation error for k-nearest neighbor searches . they propose to use a cross-encoder only to perform k NN search . |
| Outcome: | The proposed method reduces approximation error for top-k neighbors by up to 70% . iteratively performs k-NN search using the available anchors, then adds them to the next set . |
Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing mPLMs can align representations well for myriads of cross-lingual transfer tasks. |
| Approach: | They propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by anisotropic representations. |
| Outcome: | The proposed method improves on three zero-shot cross-lingual transfer tasks and over existing methods. |
Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Can Large Language Models pass the test? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate large language models for generalization lack generalization ability . current methods for evaluating LLMs are based on tests of human intelligence . |
| Approach: | They propose to use a language task to evaluate large language models' generalisation ability . they propose to ask LLMs to solve simple variants of the RAVEN IQ test . |
| Outcome: | The proposed task can be used to evaluate the generalisation ability of large language models . it shows that current generative models can handle the task in the sense that they understand instructions . |
DistillCSE: Distilled Contrastive Learning for Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings are based on contrastive learning (CL) . |
| Approach: | They propose a framework which performs contrastive learning under the self-training paradigm with knowledge distillation and propose 'Group-P shuffling strategy' and averaging logits from multiple teacher components. |
| Outcome: | The proposed framework outperforms many strong baseline methods and yields a new state-of-the-art performance. |
GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models are crucial for academic writing . existing ground truth datasets do not treat fine-grained types like ML model and model architecture as separate entity types . |
| Approach: | They propose to annotate 100 full-text scientific publications and a first baseline model for 10 entity types centered around ML models and datasets. |
| Outcome: | The proposed model can be used to identify 10 entity types in scientific articles . existing models cannot recognize fine-grained models like ML models and model architecture . |
Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-document summarization (MDS) assumes a set of topic-related documents is provided as input. |
| Approach: | They formalize the task and bootstrap it using existing datasets, retrievers and summarizers. |
| Outcome: | The proposed method reduces the sensitivity of summarizers to imperfect retrieval, but is highly sensitive to other errors. |
Few-shot Unified Question Answering: Tuning Models or Prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Question-answering (QA) tasks investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. |
| Approach: | They propose to use model and prompt tuning for unified QA in a low-resource setting to overcome drawbacks of unified models. |
| Outcome: | The proposed model and prompt tuning paradigms outperform model tuning in a few-shot setting with a good initialization and achieve a significant performance boost from pre-training in 'low-resource' setting. |
Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Creating and updating common ground (CG) between interlocutors is the key to a successful conversation. |
| Approach: | They propose a new annotation and corpus to capture common ground in human communication . they then conduct experiments to extract propositions from dialog and track their status in common ground from the perspective of each speaker . |
| Outcome: | The proposed corpus captures common ground from the perspective of two speakers in a dialog. |
Getting MoRE out of Mixture of Language Model Reasoning Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) have poor generalizability on question types beyond those seen in the prompt. |
| Approach: | They propose a framework that integrates specialized language models to generalize across question types that require distinct reasoning abilities. |
| Outcome: | The proposed framework gives higher accuracy than any single specialized model on a collection of 12 QA datasets from four reasoning types. |
“You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate proficiency and fluency in the use of language, but do they have the linguistic knowledge to serve as an expert linguistic annotator? |
| Approach: | They examine the successes and limitations of large language models using the Abstract Meaning Representation (AMR) parsing formalism. |
| Outcome: | The proposed models can reproduce the basic format of AMR, as well as some core event, argument, and modifier structure, but they have virtually no fully accurate parses. |
Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to diagnose large annotated datasets require the fitting of a strong model to the dataset. |
| Approach: | They propose a new approach to compute confidence and variability over an ensemble of zero-shot models constructed with different but semantically equivalent label descriptions. |
| Outcome: | The proposed method can be used to diagnose large annotated datasets with accuracy up to 14x faster than the current method. |
Isotropy-Enhanced Conditional Masked Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive models with auto-regressing decoding paradigms have been used for various text generation tasks to accelerate inference but at the cost of generation quality to some extent. |
| Approach: | They propose to use Look Neighbors strategy to enhance learning of target token representations during training to achieve a good balance between inference speedup and generation quality. |
| Outcome: | The proposed models outperform current models on 4 WMT datasets and outperformed the current SoTA results. |
Scaling Law for Document Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) methods fail to capture discourse phenomena such as pronominal anaphora, lexical consistency, and document coherence as the input text exceeds a single sentence. |
| Approach: | They examine the effects of model scale, data scale, and sequence length on translation quality when model size is limited. |
| Outcome: | The proposed model scales and data scales are compared with the existing models and show that increasing sequence length improves translation quality when model size is limited. |
Automatic Pronunciation Assessment - A Review (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. |
| Approach: | They review methods employed in computer-aided pronunciation training for both phonemic and prosodic pronunciations. |
| Outcome: | The proposed system should be able to automatically score non-native speech segments and give meaningful feedback. |
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown dominant performance across a range of domains including language and vision, but their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. |
| Approach: | They propose a segmented recurrent transformer that combines segmente recursion with recursive attention to reduce the computational cost. |
| Outcome: | The proposed model achieves higher ROUGE1 scores and lower computational complexity than current approaches. |
PUNR: Pre-training with User Behavior Modeling for News Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing news recommendation methods use pre-trained language models to produce news vectors and user vectors. |
| Approach: | They propose an unsupervised pre-training paradigm with two tasks for user behavior modeling. |
| Outcome: | The proposed model improves on the real-world news benchmark. |
Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a goal-driven combinatorial search using large language models has not been explored in detail. |
| Approach: | They propose a Monte Carlo Tree Search-based approach that improves beyond state-of-the-art chain-of thought prompting variants to augment scientific reasoning. |
| Outcome: | The proposed approach improves over the best baseline by 25.8% and can augment scientist’s reasoning and discovery process with novel insights. |
Measure Children’s Mindreading Ability with Machine Reading (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing scoring models do not take the features of the stories and video clips into account when scoring, which will reduce the accuracy of the models. |
| Approach: | They propose to leverage the features extracted from stories and videos related to the questions being asked during the children’s mindreading evaluation. |
| Outcome: | The proposed framework agrees well with human experts on scores produced by the models. |
Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on colexification patterns relies on annotated word lists, limiting scalability and usefulness in NLP. |
| Approach: | They propose two methods to train multilingual graphs from colexification patterns using an unannotated parallel corpus. |
| Outcome: | The proposed methods achieve high recall on CLICS and transfer learning in multilingual graphs. |
Injecting structural hints: Using language models to study inductive biases in language learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examines the cognitive inductive biases that make language learning possible. |
| Approach: | They structurally bias transformer language models by pretraining on synthetic data . they then evaluate their inductive biases by fine-tuning on three different languages . |
| Outcome: | The proposed method predisposes transformer models to three types of inductive biases . it also fine-tunes the models on three typologically-distant human languages . |
Machine Reading Comprehension using Case-based Reasoning (2023.findings-emnlp)
Copied to clipboard
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
| Challenge: | Current state-of-the-art machine readers do not support case-based reasoning . |
| Approach: | They propose a method that extracts a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers. |
| Outcome: | The proposed method outperforms baselines on NaturalQuestions and NewsQA by 11.5 and 8.4 EM. |
Unleashing the Power of Language Models in Text-Attributed Graph (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on graph learning on text-attributed graphs have been limited by memory cost and underutilization of relationships between nodes and words. |
| Approach: | They propose a Node Representation Update Pre-training Architecture based on Co-modeling text and graph to learn representations of papers and words simultaneously. |
| Outcome: | The proposed model outperforms baselines on the ogbn-arxiv benchmark dataset. |
Locally Differentially Private Document Generation Using Zero Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing privacy protections are limited by pretrained large language models . a new mechanism that leverages the power of large language model de-anonymization is proposed . |
| Approach: | They propose a locally differentially private mechanism that leverages pretrained large language models to counter author de-anonymization attacks by minimizing the impact on downstream utility. |
| Outcome: | The proposed mechanism reduces the success rate of de-anonymization attacks against IMDB datasets while minimizing the impact on downstream utility. |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study suggests that linear models with interpretable features are more reliable than opaque models. |
| Approach: | They propose an approach for natural-language specification of features for linear models . they prompt LLMs with expert-crafted queries to generate interpretable features from health records . |
| Outcome: | The proposed approach can be used to craft features clinically meaningful for downstream tasks . it is based on a risk prediction task and standard predictive tasks based upon this data . |
Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent applications of LLMs in Machine Reading Comprehension systems have shown impressive results, but the use of shortcuts has emerged as a potential threat to their reliability. |
| Approach: | They propose a framework that guides an editor to add potential shortcuts-triggers to samples. |
| Outcome: | The proposed framework can edit trigger shortcuts in samples that fool LLMs . it also shows that GPT4 can be deceived by its own edits (15% drop in F1). |
Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that can create open-domain dialogue agents lack character representation and annotations. |
| Approach: | They propose a dataset to study character alignment and character representation . it includes all dialogue sessions from the Harry Potter series and includes annotations . |
| Outcome: | The proposed dataset can be used as a universal benchmark for character-driven LLMs. |
Quick Back-Translation for Unsupervised Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised machine translation models are limited by the run-time of autoregressive inference during back-translation and lack of synthetic data efficiency. |
| Approach: | They propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder. |
| Outcome: | Experiments on various WMT benchmarks show that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities. |
SIR-ABSC: Incorporating Syntax into RoBERTa-based Sentiment Analysis Models with a Special Aggregator Token (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to integrate syntactic dependency information into language models capture syntax . aspect-based sentiment classification tasks require a complex model to handle different aspects of a sentence . |
| Approach: | They propose a method to incorporate syntactic dependency information directly into transformer-based language models for Aspect-Based Sentiment Classification. |
| Outcome: | The proposed model outperforms existing models for aspect-based sentiment analysis tasks. |
Citance-Contextualized Summarization of Scientific Papers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current automatic summarization approaches generate abstracts, but abstracts do not show relationship between paper and references. |
| Approach: | They propose a contextualized summarization approach that generates an informative summary . they extract and model the citances of a paper, retrieve relevant passages from cited papers, and generate abstractive summaries tailored to each citance. |
| Outcome: | The proposed method extracts and models the citances of a paper, retrieves relevant passages from cited papers, and generates abstractive summaries tailored to each citance. |
SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end Speech Translation models are limited by a data bottleneck . end-to end models can address several shortcomings of cascaded models . |
| Approach: | They propose a data augmentation strategy to augment sentence-level datasets by using an Audio Segmentation system to re-segment the speech of each document with different length constraints. |
| Outcome: | The proposed method achieves state-of-the-art results in MuST-C and in mTEDx. |
Intersectional Stereotypes in Large Language Models: Dataset and Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on intersectional stereotypes focus on broader, individual categories . current studies focus on single-group stereotypes, such as racial bias against African Americans . |
| Approach: | They propose to use a dataset of intersectional stereotypes curated with the ChatGPT model to analyze propagation in three contemporary LLMs. |
| Outcome: | The proposed dataset enables analysis of stereotype propagation in three contemporary LLMs. |
Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond (2023.findings-emnlp)
Copied to clipboard
Zhecan Wang, Long Chen, Haoxuan You, Keyang Xu, Yicheng He, Wenhao Li, Noel Codella, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing studies have examined dataset biases in VQA benchmarks with short-phrase answers Multiple-choice Question with the LONG Answers (VCR, VLEP, etc.) |
| Approach: | They propose to use Adversarial Data Synthesis (ADS) to generate synthetic training and debiased evaluation data and introduce Intra-sample Counterfactual Training (ICT) to assist models in utilizing synthesized training data. |
| Outcome: | The proposed approach improves model performance even in domain-shifted scenarios. |
The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Automated fact-checking is often presented as an epistemic tool fact-seekers, social media consumers, and other stakeholders can use to fight misinformation. |
| Approach: | They analyse 100 highly-cited papers and annotate epistemic elements related to intended use, i.e., means, ends, and stakeholders. |
| Outcome: | The proposed strategies are often left out of the literature and lack empirical backing. |
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. |
| Approach: | They propose a new compression paradigm that extracts knowledge from pre-trained language models to construct a knowledge store from which the model can leverage it for effective inference. |
| Outcome: | The proposed model extracts knowledge from LLMs to construct a knowledge store, which the model can leverage for effective inference. |
COUNT: COntrastive UNlikelihood Text Style Transfer for Text Detoxification (2023.findings-emnlp)
Copied to clipboard
Mohammad Mahdi Abdollah Pour, Parsa Farinneya, Manasa Bharadwaj, Nikhil Verma, Ali Pesaranghader, Scott Sanner
| Challenge: | Text detoxification is a task to ensure the generation of non-toxic and safe text. |
| Approach: | They propose a novel contrastive unlikelihood objective that combines rephrasing and identity mapping to effectively isolate and focus learning on non-toxic style transfer. |
| Outcome: | The proposed method achieves significant improvements in fluency, content preservation, and detoxification on two parallel datasets. |
KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph completion methods struggle with long-tail entities due to limited structural information and imbalanced distributions of entities. |
| Approach: | They propose a framework that integrates a large language model and a triple-based KGC retriever to alleviate the long-tail problem without incurring additional training overhead. |
| Outcome: | The proposed model reduces training overhead and finetuning costs on benchmark datasets. |
Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work typically encodes all tokens in articles using pre-trained language models, however, many named entities are difficult to accurately recognize and predict by language models. |
| Approach: | They propose an ENtity-aware article GeneratIoN and rEtrieval framework to explicitly incorporate named entities into language models. |
| Outcome: | The proposed framework can boost article generation and retrieval performance, with a 4-5 perplexity improvement in article generation, and a 3-4% boost in recall@1 in article retrieval. |
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theories of human sentence processing can be divided into two broad categories: expectation-based theories and memory-based ones. |
| Approach: | They propose to integrate expectations and retrieval from working memory into a unified cognitive model that can capture syntactic and semantic interference effects observed in human experiments. |
| Outcome: | The proposed model captures syntactic and semantic interference effects observed in human experiments. |
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on temporal sentence grounding rely on expensive video-query paired annotations . despite this, there are no ground-truth annotations in the current work . |
| Approach: | They propose to use paired video-query and segment boundary annotations to generate temporal sentence grounding without training. |
| Outcome: | The proposed model outperforms existing unsupervised methods and beats supervised ones on two challenging datasets. |
Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)
Copied to clipboard
Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang
| Challenge: | Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models. |
| Approach: | They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt. |
| Outcome: | The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets. |
A Rewriting Approach for Gender Inclusivity in Portuguese (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has focused on gender-inclusive and gender-neutral language . however, current approaches to gender- neutral rewriting for gendered languages rely on large datasets . |
| Approach: | They propose a rule-based and a neural-based tool for gender-neutral rewriting for Portuguese, a heavily gendered Romance language. |
| Outcome: | The proposed model fine-tunes large multilingual machine translation models on examples generated by the rule-based model. |
EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to measure semantic similarity between biomedical texts are inefficient due to too many biomedically-related entities. |
| Approach: | They propose an entity-aligned, attention-based and retrieval-augmented PLM that aligns the same type of fine-grained entity information in each sentence pair with an entity alignment matrix with an auxiliary loss. |
| Outcome: | The proposed model can achieve state-of-the-art on both in-domain and out-of domain datasets. |
Neuro-Symbolic Sentiment Analysis with Dynamic Word Sense Disambiguation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Traditional neural network models represent word senses as vectors that are uninterpretable for humans. |
| Approach: | They propose a framework that incorporates word Sense Disambiguation (WSD) by identifying and paraphrasing ambiguous words to improve sentiment predictions. |
| Outcome: | The proposed framework improves sentiment analysis accuracy and interpretability on a downstream task without ground-truth word sense labels. |
Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised learning of word representations involves capturing the contextual information surrounding word occurrences. |
| Approach: | They propose to use text-based dialog act tags to compare content- and context-oriented sentence representations inferred on telephone conversations to examine whether a contextual signal is of any significant benefit to general-purpose sentence representation. |
| Outcome: | The proposed model outperforms content-based and context-oriented representations on telephone conversations and shows that it increases the dimensionality of the vectors. |
CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve procedural reasoning tasks are limited by the prior art. |
| Approach: | They propose a domain-specific, continual pre-training framework that learns from a large set of procedural recipes. |
| Outcome: | The proposed framework outperforms baselines on recipes (in-domain) but is able to generalize to open-domain procedural NLP tasks. |
Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that large language models (LLMs) rank documents based on the probability of generating the query given the content of a document. |
| Approach: | They propose a ranking system that integrates LLMs with a hybrid zero-shot retriever. |
| Outcome: | The proposed system shows exceptional ranking in both zero-shot and few-shot scenarios. |
On General Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent paper suggests that the evidence underspecifies the understanding of large language models. |
| Approach: | They propose to use a "general language understanding" benchmark to examine what it could mean in machines. |
| Outcome: | The proposed model can be used to ground questions of the adequacy of benchmarking methods. |
USB: A Unified Summarization Benchmark Across Tasks and Domains (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing summarization benchmarks lack the rich annotations needed to address important problems related to control and reliability. |
| Approach: | They propose a Wikipedia-derived summarization benchmark with crowd-sourced annotations . they find that fine-tuned models outperform larger few-shot prompted language models . |
| Outcome: | The proposed model outperforms many-shot prompted language models on multiple tasks . the proposed model is based on Wikipedia annotations and can be used in other domains . |
tagE: Enabling an Embodied Agent to Understand Human Instructions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for natural language understanding (NLU) are limited due to the inherent ambiguity and incompleteness inherent in natural language. |
| Approach: | They propose a system to extract tasks from natural language instructions and map them to robots' established collection of skills. |
| Outcome: | The proposed system outperforms baseline models in the training and evaluation of a dataset featuring complex instructions. |
Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to hierarchical multi-label text classification (HMTC) ignore the correlation between similar samples and introduce noise . |
| Approach: | They propose a semi-supervised method that uses a label hierarchy to bring text and label embeddings closer to each other by supervised contrastive learning. |
| Outcome: | The proposed method bridges the gap between supervised contrastive learning and HMTC by bringing text and label embeddings closer. |
Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for text-to-image generation models is proposed to address these limitations . SSA focuses on learning structured semantic embeddings across different modalities . |
| Approach: | They propose a method to evaluate text-to-image generation models using structured semantic embeddings . they propose to learn mutated prompts by substituting words with equivalent or nonequivalent alternatives . |
| Outcome: | The proposed method improves the measurement of semantic consistency of text-to-image generation models. |
An Intent-based and Annotation-free Method for Duplicate Question Detection in CQA Forums (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detecting duplicate questions in CQA rely on generic text-pair matching models, overlooking the intent behind the questions. |
| Approach: | They propose a new intent-based duplication detector that leverages intent information to address the problem of duplicate question detection in CQA. |
| Outcome: | The proposed detector leverages the characteristics of CQA forums and extracts training labels to recognize and match intents without human annotation. |
Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive Spoken Language Understanding models suffer from multi-modality problem . current methods have little prior knowledge about the reference during inference . |
| Approach: | They propose a Targeted Knowledge Distillation Framework (TKDF) for multi-intent SLU that utilizes the knowledge distillation method to improve the performance. |
| Outcome: | The proposed model outperforms existing models on two public multi-intent datasets while speeding up by over 4.5 times. |
Type-Aware Decomposed Framework for Few-Shot Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition (NER) methods can detect entity spans and classify them into pre-defined categories. |
| Approach: | They propose a type-aware decomposed framework to filter out false spans . they propose 'type-against-type' learning strategy to construct more accurate prototypes based on type names as references. |
| Outcome: | The proposed framework yields state-of-the-art on several benchmarks. |
A Closer Look into Using Large Language Models for Automatic Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies explore the use of large language models to evaluate text quality, but they differ in some details of the evaluation process. |
| Approach: | They propose to use large language models to evaluate text quality by giving LLMs instructions to evaluate samples by giving them a rating. |
| Outcome: | The auto Chain-of-Thought (CoT) used in G-Eval does not always make it more aligned with human ratings. |
Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Graph Neural Networks have been used for text classification, but only in domains with limited data characteristics. |
| Approach: | They compare graph representation methods for text classification using different architectures and setups. |
| Outcome: | The proposed graph representation methods outperform other models in document comprehension tasks. |
Natural Language Annotations for Reasoning about Program Semantics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Xu et al., 2022) and Tafjord eet . al. 2021) have shown that programming assistants can explain their work by grounding natural language inference in code. |
| Approach: | They propose a dataset and protocol for annotating programs with natural language predicates at a finer granularity than code comments without relying on internal compiler representations. |
| Outcome: | The proposed method can be used to ground natural language inference in code without static analysis and without internal compiler representations. |
Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work has established that a person’s demographics and speech style affect how well speech processing models perform for them. |
| Approach: | They propose a method to detect bias in pre-trained models by using word embedding association tests in natural language processing to quantify bias in models' representations of different concepts. |
| Outcome: | The proposed method detects bias in pre-trained models and can have real-world effects. |
Text Classification via Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale Language Models (LLMs) have shown the ability for in-context learning. |
| Approach: | They propose a progressive reasoning strategy tailored to addressing complex linguistic phenomena such as intensification, contrast, irony and limited number of tokens allowed in in-context learning. |
| Outcome: | The proposed model performs better on 4 out of 5 widely-used text-classification benchmarks, while demonstrating comparable performance to SOTA on MR. |
On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Visually-rich document entity retrieval (VDER) is an important topic in industrial NLP applications. |
| Approach: | They propose a task-aware meta-learning framework to tackle the problem of visually-rich document entity retrieval (VDER) they adopt a hierarchical decoder and employ contrastive learning to achieve this goal. |
| Outcome: | The proposed framework significantly improves the robustness of popular meta-learning baselines. |
Semi-Structured Object Sequence Encoders (2023.findings-emnlp)
Copied to clipboard
Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara, Siva Patel, Hui Wan, Tejas Dhamecha, Danish Contractor, Marina Danilevsky
| Challenge: | Semi-structured object sequences are often represented as a sequence of key-value pairs over time . authors propose a two-part approach that takes each key independently and encodes a representation of its values over time. |
| Approach: | They propose a two-part approach that first considers each key independently and encodes a representation of its values over time. |
| Outcome: | The proposed approach outperforms existing methods on multiple prediction tasks using real-world data. |
DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Topic Models and Large Language Models (LLMs) primarily use contextual embeddings from LLMs, which are not optimal for clustering or topic generation. |
| Approach: | They propose a framework that leverages Encoder-Decoders to generate highly clusterable embeddings that could generate topics that exhibit enhanced clusterability and enhanced semantic coherence compared to existing methods. |
| Outcome: | The proposed framework is efficient to train and exhibits high adaptability, demonstrating its potential for a wide array of applications. |
Energy and Carbon Considerations of Fine-Tuning BERT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work quantifying energy costs and associated carbon emissions has focused on pretraining and fine-tuning. |
| Approach: | They perform an empirical study to quantify the energy requirements of language model fine-tuning in the context of pretraining and inference. |
| Outcome: | The proposed model fine-tuning energy and carbon footprints are compared with pre-training and inference energy requirements and outline recommendations for NLP researchers and practitioners. |
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been experiencing seismic growth in size and capabilities, radically transforming the field of NLP. |
| Approach: | They propose a generalized variant of iterative self-critique and self-refinement devoid of external influence and a ranking metric to find the optimal model for a given task considering refined performance and cost. |
| Outcome: | The proposed model improves 8.2% from baseline and even with extremely small memory footprints, outperforms ChatGPT post-refinement. |
Chinese Metaphorical Relation Extraction: Dataset and Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Metaphor identification is a core task in metaphor processing, which involves recognizing and analyzing metaphorical expressions in text. |
| Approach: | They propose a new formulation of metaphor identification as a relation extraction problem . they use a dataset to analyze metaphorical relations between two spans, a target and a source . |
| Outcome: | The proposed model can capture the properties of the target and source in Chinese sentences. |
Example-based Hypernetworks for Multi-source Adaptation to Unseen Domains (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In order to achieve unprecedented performance, many out-of-distribution generalization approaches use unlabeled data from the target distribution. |
| Approach: | They propose a framework that leverages labeled data from multiple source domains to generalize to unknown target domains at training. |
| Outcome: | The proposed framework outperforms existing models in two tasks, and it is compared to few-shot GPT-3. |
Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for harmful meme detection ignore in-depth cognition of meme text and image . authors propose a framework for learning reasonable thoughts from LLMs for better multimodal fusion . |
| Approach: | They propose to use large language models to learn reasonable thoughts from LLMs for better multimodal fusion and lightweight fine-tuning. |
| Outcome: | The proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task. |
Domain Adaptation for Conversational Query Production with the RAG Model Feedback (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on human-annotated search queries but they can not cover conversations of various domains. |
| Approach: | They propose a domain adaptation framework that uses retrieval-augmented generation to improve the model's robustness. |
| Outcome: | The proposed model is more robust and performs significantly better in a more challenging setting over strong baselines. |
LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Causality explanation generation is a generative task that aims to explain why a given cause-effect pair is true using natural language. |
| Approach: | They propose a multi-agent framework with role-playing and iterative feedback for causality explanation generation. |
| Outcome: | The proposed framework is superior to existing frameworks on WIKIWHY and e-CARE datasets. |
Ranking LLM-Generated Loop Invariants for Program Verification (2023.findings-emnlp)
Copied to clipboard
Saikat Chakraborty, Shuvendu Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, Nikhil Swamy
| Challenge: | Large Language Models (LLMs) are capable of synthesizing inductive loop invariants for a class of programs in a 0-shot setting, yet require several samples to generate the correct invariant. |
| Approach: | They propose a re-ranking approach to generate inductive loop invariants using Large Language Models . they propose reranking rankers that can distinguish between correct and incorrect attempts . |
| Outcome: | The proposed method reduces the number of calls to a verifier by comparing the generated results with the original model. |
WordNet Is All You Need: A Surprisingly Effective Unsupervised Method for Graded Lexical Entailment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a simple unsupervised method for predicting graded lexical entailment in English relies on WordNet . despite its simplicity, our method outperforms all previous methods using WordNet as weak supervision. |
| Approach: | They propose an unsupervised method which relies exclusively on WordNet for predicting graded lexical entailment in English. |
| Outcome: | The proposed method outperforms existing methods on the largest GLE dataset using WordNet. |
Knowledge Corpus Error in Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in open-domain question answering (QA) has explored generating context passages from large language models (LLMs) however, it is not well understood why generated passages can be more effective than retrieved ones. |
| Approach: | They propose to generate context passages from large language models by paraphrasing human-annotated gold context using LLMs to observe knowledge corpus error. |
| Outcome: | The proposed framework shows that paraphrasing human-annotated gold contexts improves performance over retrieval steps. |
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in machine translation (MT) have shown that minimum bayes risk decoding can be a powerful alternative to beam search. |
| Approach: | They propose to use epsilon-sampling to prune away all tokens with a smaller probability mass. |
| Outcome: | The proposed method outperforms beam search decoding and other methods in four languages. |
The language of prompting: What linguistic properties make a prompt successful? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pretraining and instruction-tuned LLMs can achieve impressive performance on a multitude of tasks. |
| Approach: | They propose to use a standard for prompting research to better understand linguistic properties of LLMs. |
| Outcome: | The proposed standard would improve the performance of pre-trained and instruction-tuned LLMs on a multitude of tasks. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation. |
| Approach: | They propose to have large language models actively involved in retrieval to guide retrieval with generation. |
| Outcome: | The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations. |
Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models. |
| Approach: | They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels . |
| Outcome: | The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates . |
Non-parallel Accent Transfer based on Fine-grained Controllable Accent Modelling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing accent transfer methods rely on parallel data or speech recognition models. |
| Approach: | They propose to use mutual information learning to disentangle accent features and control the accent of the generated speech during the inference time. |
| Outcome: | The proposed framework achieves superior performance to baseline models in accentedness and audio quality. |
Compositional Generalization for Data-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Data-to-text generation models can be used to generate textual descriptions from structured data . despite advances, systems struggle when confronted with unseen combinations of predicates . |
| Approach: | They propose a data-to-text generation model that addresses compositional generalization by clustering predicates into groups. |
| Outcome: | The proposed model outperforms T5-baselines in all evaluation metrics. |
In-Context Learning Creates Task Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful new learning paradigm for Large Language Models (LLMs). |
| Approach: | They propose to use a model with a prompt and a query to learn a mapping based on two examples to produce the output. |
| Outcome: | The proposed model can learn functions from a simple structure based on a training set and a single task vector calculated from the training set. |
TalkUp: Paving the Way for Understanding Empowering Language (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empowerment has rarely been studied in NLP because of its implicit nature . linguistics and psychology research shows how empowerment can impact people by increasing their sense of self-efficacy and self-esteem. |
| Approach: | They crowdsource Reddit posts labeled for empowerment and use it to train language models that capture empowering and disempowering language. |
| Outcome: | The proposed dataset can be used to train language models that capture empowering and disempowering language. |
Unifying Text, Tables, and Images for Multimodal Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal question answering rely on single-modal or bi-modal models, which limit their ability to integrate information across all modalities. |
| Approach: | They propose a framework that unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. |
| Outcome: | The proposed framework unifies three input modalities into a text-to-text format using position-enhanced table linearization and diversified image captioning techniques. |
Unsupervised Lexical Simplification with Context Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing unsupervised lexical simplification methods only use monolingual data and pre-trained models. |
| Approach: | They propose an unsupervised method that generates substitutes based on monolingual data and pre-trained language models. |
| Outcome: | The proposed method outperforms existing models on the TSAR-2022 task in English, Portuguese, and Spanish. |
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages. |
| Approach: | They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data . |
| Outcome: | The proposed model performs well on multilingual summarization and question-answering tasks. |
Multilingual Lottery Tickets to Pretrain Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual pretrained language models (mPLMs) have become the de-facto standard for multilingual tasks. |
| Approach: | They propose to search for a per-language subnetwork with comparable performance to the full model by scaling the model to reduce interference and then redistributing parameters to keep the parameters reduced. |
| Outcome: | The proposed model reduces the inference cost of models for each language while keeping the capacity per language more or less the same. |
Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamic Audio-Visual Scenarios (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Audio-visual question answering requires multistep spatio-temporal reasoning over multimodal contexts. |
| Approach: | They propose a new target-aware joint spatio-temporal grounding network for audio-visual question answering . the proposed system integrates audio-vision fusion and question-awful temporal grounding into one module . |
| Outcome: | The proposed method over existing state-of-the-art methods is effective over existing methods . it can focus on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question . |
KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models for complex reasoning tasks on knowledge graphs remains unexplored. |
| Approach: | They propose a multi-purpose framework leveraging large language models for complex reasoning tasks on knowledge graphs. |
| Outcome: | The proposed framework outperforms fully-supervised models in KG-based fact verification and KGQA benchmarks. |
Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that multilingual language models (MultiLMs) are capable of logically reasoning over natural language statements, reasoning with their implicit knowledge, and performing multi-step reasoning when the model size is large enough. |
| Approach: | They propose a mechanism that encourages cross-lingual attention in code-switched sequences and improves reasoning performance by up to 14%. |
| Outcome: | The proposed approach improves reasoning performance by 14% and 4% on the RuleTaker and LeapOfThought datasets. |
CITB: A Benchmark for Continual Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for instruction tuning do not leverage the rich natural language instructions. |
| Approach: | They propose to use a benchmark to study how instruction tuning works in CL tasks. |
| Outcome: | The proposed method can achieve similar or better results than existing CL methods. |
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (NLP). |
| Approach: | They propose a method that injects linguistic structures into pre-trained language models in the parameter-efficient fine-tuning setting. |
| Outcome: | The proposed approach outperforms state-of-the-art methods with a comparable number of parameters. |
Towards Better Representations for Multi-Label Text Classification with Multi-granularity Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models generate word frequency-oriented text representations, causing texts with different labels to be closely distributed in a narrow region, which is difficult to classify. |
| Approach: | They propose a framework to refine the text representation for multi-label text classification using contrastive learning and multi-task learning modules. |
| Outcome: | The proposed framework improves the quality of the representations and yields stable and competitive improvements. |
PCMID: Multi-Intent Detection through Supervised Prototypical Contrastive Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to intent detection assume that each utterance represents only a single intent. |
| Approach: | They propose a framework for intent detection that can learn multiple representations of a given user utterance under the context of different intent labels in an optimized semantic space. |
| Outcome: | The proposed framework achieves state-of-the-art on multiple public benchmark datasets and a private real-world dataset for the multi-intent detection task. |
Is GPT-4 a Good Data Analyst? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc. |
| Approach: | They propose to use GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. |
| Outcome: | The proposed framework compares GPT-4 with human data analysts to perform end-to-end data analysis with databases from a wide range of domains. |
DiffusionRet: Diffusion-Enhanced Generative Retriever using Constrained Decoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative retrieval methods have suffered from the lack of the intermediate reasoning step . generative retrieval uses sequence-to-sequence diffusion models to map a query to relevant docids . |
| Approach: | They propose a novel method that uses query as an intermediate step before retrieval . they propose to use sequence-to-sequence diffusion models to map a query to relevant docids . |
| Outcome: | Experiments show that proposed method outperforms existing methods on MARCO and Natural Questions datasets. |
Estimating Large Language Model Capabilities without Labeled Test Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive ability to perform in-context learning from only a few examples, but their accuracy varies widely from task to task. |
| Approach: | They propose a method that trains a meta-model using LLM confidence scores as features to perform ICL accuracy estimation. |
| Outcome: | The proposed method improves over baselines across 7 out of 12 settings and achieves the same accuracy as evaluating on 40 sampled examples per task. |
A Novel Contrastive Learning Method for Clickbait Detection on RoCliCo: A Romanian Clickbait Corpus of News Articles (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Clickbait detection is a task that aims to automatically detect misleading news titles . despite the importance of the task, there is no publicly available clickbait corpus for Romanian . |
| Approach: | They propose a Romanian Clickbait Corpus that automatically detects misleading news titles . they propose four machine learning methods to establish competitive baselines . |
| Outcome: | The proposed model can learn to encode news titles and contents into a deep metric space . the proposed model is available for download on github.com/dariabroscoteanu/RoCliCo. |
Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues (2023.findings-emnlp)
Copied to clipboard
Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, Kam-Fai Wong
| Challenge: | Existing knowledge-grounded dialogue systems focus on a single knowledge source or ignore the dependency between multiple knowledge sources. |
| Approach: | They propose a framework that integrates multiple knowledge sources and dependencies between them. |
| Outcome: | The proposed framework can produce persona-consistent and knowledge-enhanced responses on a knowledge-grounded dialogue dataset. |
Toxicity in Multilingual Machine Translation at Scale (2023.findings-emnlp)
Copied to clipboard
Marta Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, Carlos Escolano
| Challenge: | In this paper, we evaluate and analyze added toxicity when translating a large dataset from English into 164 languages. |
| Approach: | They evaluate added toxicity when translating a large dataset from English into 164 languages. |
| Outcome: | The results show that added toxicity is more prevalent in low-resource languages than in high-resolution translations. |
Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue (2023.findings-emnlp)
Copied to clipboard
Yuanxing Liu, Weinan Zhang, Yifan Chen, Yuchi Zhang, Haopeng Bai, Fan Feng, Hengbin Cui, Yongbin Li, Wanxiang Che
| Challenge: | E-commerce pre-sales dialogues elicit user needs and preferences for items . large language models lack domain-specific knowledge for accurate recommendations . |
| Approach: | They propose two collaboration strategies to integrate CRS and large language models in pre-sales dialogues. |
| Outcome: | The proposed methods can be very effective in some cases, the authors say . |
VIP5: Towards Multimodal Foundation Models for Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in foundation models have impeded the ability for these fields to benefit from each other’s advancements. |
| Approach: | They propose to use a multimodal foundation model to unify various modalities and recommendation tasks under the P5 recommendation paradigm to implement personalized prompts. |
| Outcome: | The proposed model will unify visual, textual, and personalization modalities under the P5 recommendation paradigm and will improve recommendation performance and efficiency. |
A Spectral Viewpoint on Continual Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve the Continual Relation Extraction problem have been proposed . |
| Approach: | They propose a class-wise regularization method that preserves eigenvectors for each class shape . they propose spectral regularization to preserve eenvector shape after learning new tasks . |
| Outcome: | The proposed method improves performance on two benchmark datasets. |
Learning to Follow Object-Centric Image Editing Instructions Faithfully (2023.findings-emnlp)
Copied to clipboard
| Challenge: | avrahami et al., 2022b,a): natural language instructions are often underspecified, requiring models to uncover their implicit meaning. |
| Approach: | They propose to use paired data to model the implicit meaning of instructions . they also propose to ground the model to localize where the edit has to be performed . |
| Outcome: | The proposed model performs better than state-of-the-art baselines on paired data, showing improvements in quality and faithfulness. |
Zero-shot Topical Text Classification with LLMs - an Experimental Study (2023.findings-emnlp)
Copied to clipboard
Shai Gretz, Alon Halfon, Ilya Shnayderman, Orith Toledo-Ronen, Artem Spector, Lena Dankin, Yannis Katsis, Ofir Arviv, Yoav Katz, Noam Slonim, Liat Ein-Dor
| Challenge: | Topical text classification is an ancient, yet timely research area in natural language processing. |
| Approach: | They compare the zero-shot performance of a variety of LMs over a large dataset of 23 publicly available TTC datasets. |
| Outcome: | The proposed models outperform their counterparts over a large dataset and show that they perform better in a zero-shot scenario. |
Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models enable them to follow freeform instructions, including imitating generic or specific demographic personas in conversations. |
| Approach: | They propose to investigate persona biases by experimenting with UNIVERSALPERSONA, a model that incorporates both generic and specific personas. |
| Outcome: | The proposed model systematically measures persona biases in harmful expression and harmful agreement. |
A Black-Box Attack on Code Models via Representation Nearest Neighbor Search (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating adversarial code examples face challenges such as limted availability of substitute variables and the creation of adversarials with noticeable perturbations. |
| Approach: | They propose a search seed based on historical attacks to find adversarial substitutes . they employ a pre-trained variable name encoder to map the search seed to a continuous vector space . |
| Outcome: | The proposed approach outperforms baseline methods in terms of ASR and QT. |
How Well Do Text Embedding Models Understand Syntax? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text embedding models have not addressed syntactic understanding challenges, highlighting ineffectiveness and enhancing generalization ability. |
| Approach: | They propose to examine the ability of text embedding models to generalize across syntactic contexts. |
| Outcome: | The proposed models exhibit high similarity socres at this simple task. |
CASSI: Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for text augmentation suffer from annotation corruption for token-level tasks like NER. |
| Approach: | They propose a novel augmentation scheme that generates high-quality contextually diverse augmentations while avoiding annotation corruption. |
| Outcome: | The proposed scheme outperforms existing methods at multiple low resource levels, in multiple languages, and for noisy and clean text. |
NEWTON: Are Large Language Models Capable of Physical Reasoning? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have been shown to encapsulate syntactic, semantic, word sense, and common-sense knowledge, but limited exploration of their physical reasoning abilities has been conducted. |
| Approach: | They propose a repository and benchmark to evaluate LLMs' physical reasoning skills . they use a pipeline to generate a variant of the benchmark customized to the objects and attributes relevant for their application. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of language models across reasoning tasks. |
Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Counterspeech, i.e. responses to counteract potential harms of hateful speech, has become an increasingly popular solution to address online hate speech without censorship risks of deletion-based content moderation. |
| Approach: | They draw from psychology and philosophy literature to craft six psychologically inspired strategies to challenge the underlying stereotypical implications of hateful language. |
| Outcome: | The strategies used in human- and machine-generated counterspeech datasets are convincing, whereas human-written counterspech uses less specific strategies compared to machine-produced counters. |
On the Calibration of Large Language Models and Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are becoming more popular and are proving to be reliable . however, their reliability is often understudied due to their uncertainty and complex structure . |
| Approach: | They conduct a systematic examination of the calibration of aligned language models throughout the entire construction process including pretraining and alignment training. |
| Outcome: | The results shed light on whether popular large language models are well-calibrated and how the training process influences model calibration. |
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 are commercial large language models (LLMs) however, they may produce vague responses or incorrect answers in certain specialized domains. |
| Approach: | They propose a token compression scheme that uses summarization and semantic compression to reduce the token size of LLMs. |
| Outcome: | The proposed method reduces token size by doing summarization and semantic compression while reducing token size with only 1.6% accuracy drop. |
Identifying Conspiracy Theories News based on Event Relation Graph (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conspiracy theories are narratives that explains an event or situation in an irrational or malicious manner. |
| Approach: | They propose to integrate an event relation graph into conspiracy theory identification by using soft labels. |
| Outcome: | The proposed approach improves precision and recall of conspiracy theory identification, and generalizes well for new unseen media sources. |
Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing conversational recommender systems focus on a single-shot approach to understand user preferences and provide recommendations. |
| Approach: | They propose a problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog. |
| Outcome: | The proposed framework can simulate salesbot and shopperbot agents and provide both product recommendations and educational value through mixed-type mixed-initiative dialog. |
Dynamic Open-book Prompt for Conversational Recommender System (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for prompt learning use only training samples for parameter training, limiting the performance of existing methods. |
| Approach: | They propose a Dynamic Open-book Prompt approach where the open book stores user's experiences in historical data and dynamically constructs the prompt to memorize the user' s current utterance. |
| Outcome: | The proposed model improves on the existing methods on the ReDial dataset and shows that it can be used to learn contextually relevant recommendations. |
Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models (2023.findings-emnlp)
Copied to clipboard
Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, Meng Jiang
| Challenge: | Large language models can perform a wide range of tasks by following natural language instructions without task-specific fine-tuning. |
| Approach: | They propose a method to automatically improve the quality of LLM instructions . they leverage the generative ability of LMS to generate diverse candidate instructions based on a scoring model trained on 575 existing NLP tasks. |
| Outcome: | The proposed method surpasses human-written and LLM-generated instructions on 118 out-of-domain tasks. |
DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to text generation use discrete text within a continuous diffusion space, which incurs substantial computational overhead during training and results in slower sampling speeds. |
| Approach: | They propose a soft absorbing state that facilitates diffusion models in learning to reconstruct discrete mutations based on the underlying Gaussian space. |
| Outcome: | The proposed method accelerates training convergence by 4x and generates samples of similar quality 800x faster, rendering it closer to practical application. |
M2C: Towards Automatic Multimodal Manga Complement (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features. |
| Approach: | They propose a task to enhance manga understanding with visual and textual features by providing a shared semantic space for vision and language understanding. |
| Outcome: | The proposed task provides a shared semantic space for vision and language understanding. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Towards Detecting Contextual Real-Time Toxicity for In-Game Chat (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ToxBuster is a simple and scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. |
| Approach: | They propose a model that detects toxic content in real-time for a line of chat by including chat history and metadata. |
| Outcome: | The proposed model outperforms conventional toxicity models across popular multiplayer games including Rainbow Six Siege, For Honor, and DOTA 2 and 6% of unreported toxic players can be proactively moderated. |
JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing sign language datasets are limited and skewed towards high-income sign languages, mainly those from high-risk countries. |
| Approach: | They propose a large and highly multilingual dataset for sign language translation: JWSign. |
| Outcome: | The proposed dataset consists of 2,530 hours of Bible translations in 98 sign languages, featuring more than 1,500 individual signers. |
Do Stochastic Parrots have Feelings Too? Improving Neural Detection of Synthetic Text via Emotion Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative AI have shone a spotlight on high-performance synthetic text generation technologies. |
| Approach: | They propose to use emotion-driven pretrained language models to generate synthetic text that lacks emotional coherence. |
| Outcome: | The proposed detector achieves significant improvements across a range of synthetic text generators, various sized models, datasets, and domains. |
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules (2023.findings-emnlp)
Copied to clipboard
Chaojun Xiao, Yuqi Luo, Wenbin Zhang, Pengle Zhang, Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Large language models (LLMs) have been successful on NLP tasks but require huge parameter sizes and computational resources. |
| Approach: | They propose a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. |
| Outcome: | The proposed method saves 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%. |
PivotFEC: Enhancing Few-shot Factual Error Correction with a Pivot Task Approach using Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Factual Error Correction (FEC) use mask-then-correct paradigms . however, the lack of datasets containing false claims has impeded progress . |
| Approach: | They propose a method that enhances few-shot FEC with a pivot task approach using large language models. |
| Outcome: | The proposed method outperforms its few-shot counterpart by 7.9 points in SARI . it improves widely-adopted SARI metrics by 11.3 compared to the best-performing methods . |
Semantic Similarity Covariance Matrix Shrinkage (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to estimate covariance matrix relied on historical price data and ignored company fundamental data. |
| Approach: | They propose to use semantic similarity to improve covariance estimations by using a shrinkage target. |
| Outcome: | The proposed method is compared with the prior art estimate for covariance shrinkage using semantic similarity and price history. |
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research shows that large language models can replicate human-like behavior in various tasks. |
| Approach: | They propose a framework for human-LLM collaboration to conduct TA with in-context learning (ICL) they propose to use survey data to frame discussions with an LLM to generate a final codebook for TA. |
| Outcome: | The proposed framework outperforms crowd workers on text-annotation tasks and yields similar coding quality to that of human coders but reduces TA’s labor and time demands. |
LLM aided semi-supervision for efficient Extractive Dialog Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a method to extract dialog summarization data from unlabeled data is currently expensive to build. |
| Approach: | They propose a method to extract dialog summarization using unlabeled data . they frame summarizing as a question-answering problem and use pseudo-labels to fine-tune a chat summarisation model . |
| Outcome: | The proposed method achieves 65.9/57.0/61.0 ROUGE-1/-2/-L on a TWEETSUMM dataset compared with current state-of-the-art methods on the entire training dataset. |
Investigating Multilingual Coreference Resolution by Universal Annotations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for multilingual coreference resolution have been challenging due to linguistic diversity and complexity of different languages. |
| Approach: | They propose a multilingual coreference dataset with universal morphosyntactic and coreference annotations. |
| Outcome: | The proposed dataset improves the baseline system by 0.9% . the proposed dataset is based on the framework of Universal Dependencies 2 . |
FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Graph-to-text (G2T) generation is an important task in natural language generation as it renders graphs accessible to non-technical users in downstream applications such as question answering. |
| Approach: | They propose a metric that correctly identifies factual faithfulness and uses it to determine if a triple is present in a generated text. |
| Outcome: | The proposed metric achieves highest correlation with human annotations on data correctness, data coverage, and relevance. |
LayoutDIT: Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods struggle to capture the visual layout in complex document images. |
| Approach: | They propose to integrate layout knowledge into document image translation by using a layout-aware encoder and a multi-step conductive decoder to achieve the translation step by step. |
| Outcome: | The proposed model outperforms state-of-the-art methods with better parameter efficiency. |
Balaur: Language Model Pretraining with Lexical Semantic Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. |
| Approach: | They propose a model that models LSRs directly in the LM’s hidden states throughout pretraining by allowing them to be controlled by the model. |
| Outcome: | The proposed model improves performance on hypernymy-informed tasks and on the original LM objective. |
Exploring In-Context Learning for Knowledge Grounded Dialog Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge grounded dialog generation models are prone to hallucination and produce factually inaccurate outputs. |
| Approach: | They propose a retrieval-based framework which leverages in-context learning and retrieval techniques to enhance LLMs on knowledge grounded dialog generation. |
| Outcome: | The proposed framework outperforms existing training-based models on a large-scale knowledge graph with 1M+ facts and is expected to perform knowledge-intensive tasks. |
Towards Enhancing Relational Rules for Knowledge Graph Link Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph reasoning methods are inadequate for missing knowledge . Various methods are explored to facilitate reasoning for missing information . |
| Approach: | They propose a novel knowledge graph reasoning approach that uses a query-related fusion gate unit to model the sequentiality of relation composition and a buffering update mechanism to alleviate lagged entity information propagation. |
| Outcome: | Experimental results show that the proposed approach is superior on both transductive and inductive link prediction tasks. |
Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel in generating coherent texts, but their ability to comprehend the author’s thoughts remains uncertain. |
| Approach: | They conduct a comprehensive survey of narrative understanding tasks, examining their key features, definitions, taxonomy, associated datasets, evaluation metrics, and limitations. |
| Outcome: | The proposed framework could be extended to address novel narrative understanding tasks. |
Who is Speaking? Speaker-Aware Multiparty Dialogue Act Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Identifying how speakers interact with each other in a conversation is difficult when more than two interlocutors take part in . To overcome this challenge, we propose to explicitly add speaker awareness to each utterance representation. |
| Approach: | They propose to add speaker awareness to each utterance representation to model how each speaker is behaving within the local context of a conversation. |
| Outcome: | The proposed approach is able to model multiparticipant and dyadic conversations on the MRDA and SwDA datasets and shows that it is more efficient than previous approaches. |
Demystifying Prompts in Language Models via Perplexity Estimation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models can be prompted to perform a wide variety of tasks with zero- and few-shot learning. |
| Approach: | They propose a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
| Outcome: | The proposed method extends a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
C2D2 Dataset: A Resource for the Cognitive Distortion Analysis and Its Impact on Mental Health (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Cognitive distortions refer to patterns of irrational thinking that can lead to distorted perceptions of reality and mental health problems in individuals. |
| Approach: | They propose to use the C2D2 dataset to detect cognitive distortions in everyday life scenes to improve existing models of mental health detection. |
| Outcome: | The proposed dataset contains 7,500 cognitive distortion thoughts in everyday life scenes. |
MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Various data augmentation strategies have been proposed to improve GEC models . high-quality parallel data for GEC is not as widely available . |
| Approach: | They propose a data augmentation approach that strategically augments real data by generating pseudo data. |
| Outcome: | The proposed approach significantly improves GEC models on English and Chinese datasets. |
CCEval: A Representative Evaluation Benchmark for the Chinese-centric Multilingual Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual machine translation (MMT) has gained more importance due to international business development and cross-cultural exchanges. |
| Approach: | They propose to use Chinese-centric MMT evaluation dataset to build an impartial and representative evaluation benchmark. |
| Outcome: | The proposed dataset covers more diverse linguistic features than other benchmarks and is highly representative and humancorrelated. |
ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a vision-language model with commonsense knowledge can reason beyond common sense . however, pre-trained vision-linguistic models are incapable of interpreting counter-intuitive content . |
| Approach: | They introduce a probing dataset to evaluate vision-language models' reasoning abilities . they use images that defy commonsense knowledge to test their reasoning abilities. |
| Outcome: | The proposed dataset evaluates whether pre-trained vision-language models can reason beyond common sense . it contains images that defy commonsense knowledge with regards to color, shape, material, size and position . |
Automatic Analysis of Substantiation in Scientific Peer Reviews (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems to analyze peer reviews' quality are inadequate due to the increasing workload of reviewers and the lack of domain experts . |
| Approach: | They propose to use a claim-evidence pair extraction problem to analyze substantiation in peer reviews and train an argument mining system to do the same. |
| Outcome: | The proposed system could be used by conference managers and reviewers to analyze the quality of peer reviews. |
Hierarchical Prompting Assists Large Language Model on Web Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle on processing complicated observations in interactive decision making tasks. |
| Approach: | They propose a hierarchical prompting approach that constructs an action-aware observation and a Summarizer prompt. |
| Outcome: | The proposed method outperforms the current state-of-the-art prompting mechanism by 6.2% on task success rate. |
Can Large Language Models Fix Data Annotation Errors? An Empirical Study Using Debatepedia for Query-Focused Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Debatepedia dataset limited by noise and most queries do not have relevance to document . |
| Approach: | They harness the language generation capabilities of two LLMs to regenerate queries in a Debatepedia dataset. |
| Outcome: | The proposed model can regenerate queries from the Debatepedia dataset. |
TSTR: Target Similarity Tuning Meets the Real World (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs). |
| Approach: | They propose to use sentences from a larger language model to improve similarity between two NL inputs and associated code outputs. |
| Outcome: | The proposed model can be trained on a small number of training examples and is cost-effective. |
RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms (2023.findings-emnlp)
Copied to clipboard
Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing studies on human-like behaviors in foundation models do not verify their faithfulness . a simple application of psychological tools cannot faithfully characterize all human-type behaviors . |
| Approach: | They propose a framework to characterize humanoid behaviors in foundation models . they argue that a simple application of psychological tools cannot faithfully characterize all human-like behaviors . |
| Outcome: | The proposed framework assesses the faithfulness of results based on reproducibility, internal consistency, and generalizability. |
Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. |
| Approach: | They conduct a large-scale user study with 231 students writing business case peer reviews in german. |
| Outcome: | The proposed model does not carry bias in the feedback loops of the students . |
VERVE: Template-based ReflectiVE Rewriting for MotiVational IntErviewing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | During the Covid-19 pandemic, the number of people living with anxiety and depression rose more than four times . counselor training is difficult to speed up due to several factors, such as the need for expert supervision and the laborious and time-extensive process needed to provide evaluative feedback. |
| Approach: | They propose a template-based rewriting system that transforms non-reflective statements into reflective responses using paraphrase-augmented training and adaptive template updating. |
| Outcome: | The proposed model transforms non-reflective statements into more reflective responses while achieving a good content preservation-reflection style trade-off. |
Self-Knowledge Guided Retrieval Augmentation for Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown superior performance without task-specific fine-tuning due to the computational costs. |
| Approach: | They propose a method which lets LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. |
| Outcome: | The proposed method outperforms chain-of-thought based and fully retrieval-based methods on multiple datasets and outperformed chain- of-though, chatGPT and InstructGPT. |
Pretraining Language Models with Text-Attributed Heterogeneous Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pretraining tasks for Language Models (LMs) focus on learning the textual information of each entity and overlook the crucial aspect of capturing topological connections among entities in TAHGs. |
| Approach: | They propose a topology-aware pretraining task that explicitly considers the topological and heterogeneous information in TAHGs by optimizing an LM and an auxiliary heterogenous graph neural network. |
| Outcome: | The proposed framework explicitly considers the topological and heterogeneous information in TAHGs. |
CReTIHC: Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in natural language processing, but their ability to establish causal relationships remains challenging. |
| Approach: | They propose a novel dataset to test and enhance the causal reasoning abilities of large language models (LLMs) by integrating elements of verbal hallucinations and temporal interventions into existing causal inference datasets. |
| Outcome: | The proposed dataset is designed to test and enhance the causal reasoning abilities of large language models. |
On the Dimensionality of Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on improving the quality of sentence embeddings, but the exploration of sentence dimension is limited. |
| Approach: | They propose a two-step training method where the encoder and pooler are optimized separately to mitigate the overall performance loss in low-dimension scenarios. |
| Outcome: | The proposed method significantly improves the performance of low-dimensional sentence embeddings on seven STS tasks and seven sentence classification tasks. |
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements. |
| Approach: | They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms. |
| Outcome: | The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks. |
Entity-Based Evaluation of Political Bias in Automatic Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that NLP systems may encode social biases, but the *political* bias of summarization models remains relatively unknown. |
| Approach: | They use an entity replacement method to examine the portrayal of politicians in automatically generated summaries. |
| Outcome: | The proposed model can control for the content of the source document and can be used to predict the ideal quality of summarization models. |
StyleBART: Decorate Pretrained Model with Style Adapters for Unsupervised Stylistic Headline Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on unsupervised headline generation focus on a standard dataset and mono-style corpora. |
| Approach: | They propose an unsupervised approach for stylistic headline generation using a pretrained BART model decorated with adapters responsible for different styles. |
| Outcome: | The proposed method separates the task of style learning and headline generation, allowing for the generation of diverse headlines with diverse styles. |
RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing intent detection approaches have relied on adaptively pre-training language models with large-scale datasets, yet the predominant cost of data collection may hinder their superiority. |
| Approach: | They propose a self-supervised framework dedicated to task-oriented dialogues which incorporates agent responses for pre-training in a two-stage manner. |
| Outcome: | The proposed framework outperforms the state-of-the-art frameworks for task-oriented dialogues on two real-world customer service datasets. |
Improving Low-resource Question Answering by Augmenting Question Information (2023.findings-emnlp)
Copied to clipboard
Andong Chen, Yuan Sun, Xiaobing Zhao, Rosella Galindo Esparza, Kehai Chen, Yang Xiang, Tiejun Zhao, Min Zhang
| Challenge: | Low-resource questions pose a significant challenge within the field of Question-Answering (QA) tasks. |
| Approach: | They propose a method that leverages large models' internal knowledge to enhance the quality of augmented data by Prompt Answer, Question Generation, and Question Filter. |
| Outcome: | The proposed method outperforms existing augmentation strategies on high-resource QA tasks like SQUAD1.1 and TriviaQA. |
InstructSafety: A Unified Framework for Building Multidimensional and Explainable Safety Detector through Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing safety detection systems have limitations in terms of their versatility and interpretability. |
| Approach: | They introduce a safety detection framework that unifies 7 common sub-tasks into a uniform formulation and process 39 human-annotated datasets for instruction tuning. |
| Outcome: | The proposed framework unifies 7 common sub-tasks into a uniform formulation and then runs on 39 human-annotated datasets to fine-tune it. |
“A Tale of Two Movements’: Identifying and Comparing Perspectives in #BlackLivesMatter and #BlueLivesMatter Movements-related Tweets using Weakly Supervised Graph-based Structured Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a weakly supervised graph-based approach to model #BLM-related tweets is difficult to obtain . |
| Approach: | They propose a weakly supervised graph-based approach that explicitly models perspectives in #BackLivesMatter-related tweets. |
| Outcome: | The proposed model outperforms multitask baselines by a large margin. |
ClusterPrompt: Cluster Semantic Enhanced Prompt Learning for New Intent Discovery (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for identifying new intent categories focus on relations between utterances and clusters, while neglecting the usage of semantics. |
| Approach: | They propose a method that leverages contrastive learning and label semantic alignment to learn meaningful representations of intent clusters. |
| Outcome: | The proposed method outperforms existing methods and suggests meaningful intent labels. |
Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study of presupposition, discourse and sarcasm suggests that pre-finetuning can improve models' performance on presimplified cases. |
| Approach: | They propose to leverage the connection between presupposition, discourse and sarcasm to improve models' performance. |
| Outcome: | The proposed model improves on cases involving presupposition by pre-finetuning on additional tasks and datasets. |
MRRL: Modifying the Reference via Reinforcement Learning for Non-Autoregressive Joint Multiple Intent Detection and Slot Filling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive models for multiple intent detection and slot filling have limited overall accuracy due to multi-modality problem and lack of alignment between correct predictions. |
| Approach: | They propose a method for multiple intent detection and slot filling that introduces a modifier and employs reinforcement learning to modify the reference. |
| Outcome: | The proposed method outperforms the previous best approach by 3.6 overall accuracy on MixATIS dataset. |
DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task (2023.findings-emnlp)
Copied to clipboard
Guanting Dong, Tingfeng Hui, Zhuoma GongQue, Jinxu Zhao, Daichi Guo, Gang Zhao, Keqing He, Weiran Xu
| Challenge: | Recent prompt-based generative frameworks have shown impressive capabilities in sequence labeling tasks. |
| Approach: | They propose a multi-task demonstration-based generative framework for noisy slot filling that captures input perturbations at different granularities. |
| Outcome: | The proposed framework outperforms baseline methods and achieves strong generalization. |
SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Several methods have been proposed to improve the inference efficiency of transformer-based models. |
| Approach: | They propose a new adaptive inference method that takes into account the hardness of input samples. |
| Outcome: | The proposed model outperforms or complements existing per-sample adaptive inference methods in terms of accuracy vs. FLOPs and can be applied to compressed and efficient transformer encoders to further improve their efficiency. |
Always the Best Fit: Adaptive Domain Gap Filling from Causal Perspective for Few-Shot Relation Extraction (2023.findings-emnlp)
Copied to clipboard
Ge Bai, Chenji Lu, Jiaxiang Geng, Shilong Li, Yidong Shi, Xiyan Liu, Ying Liu, Zhang Zhang, Ruifang Liu
| Challenge: | Existing approaches to cross-domain relation extraction have been limited by domains . data bias between domains can be difficult to fill, especially in few-shot scenarios . |
| Approach: | They propose a framework to bridge the semantic gap caused by data bias between domains . they use syntactic structure, label distribution, and entities to calculate causal effects . |
| Outcome: | The proposed framework fills the domain gap and yields better results on the few-shot task. |
MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for text classification use human annotations or a set of class seed words for supervision, which can be costly, especially in emerging domains. |
| Approach: | They propose a weakly-supervised method that leverages mutually-enhancing text granularities to learn a contextualized document representation that captures the most discriminative class indicators. |
| Outcome: | Extensive experiments on seven benchmark datasets show that MEGClass outperforms other weakly and extremely weakly supervised methods. |
Causal Inference from Text: Unveiling Interactions between Variables (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for estimating causal effects from text only account for latent covariates that affect both treatment and outcome. |
| Approach: | They propose to disentangle non-confounding covariates from text to minimize selection bias . they conduct experiments on two different treatment factors under various scenarios . |
| Outcome: | The proposed model outperforms strong baselines on earnings call transcripts . the proposed model is based on a randomized controlled trial . |
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples! (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have made remarkable strides in various tasks, but whether they are competitive few-shot solvers remains an open question. |
| Approach: | They propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. |
| Outcome: | The proposed system achieves promising improvements on various IE tasks with acceptable time and cost investment. |
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that ChatGPT has limitations such as failing to ask clarifying questions to ambiguous queries or refusing problematic user requests. |
| Approach: | They propose a Proactive Chain-of-Thought prompting scheme which augments LLMs with the goal planning capability over descriptive reasoning chains to trigger proactivity. |
| Outcome: | The proposed scheme augments LLMs with the goal planning capability over descriptive reasoning chains to trigger the proactivity of LLM-based proactive dialogue systems. |
Ecologically Valid Explanations for Label Variation in NLI (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation exists in many natural language processing tasks, including NLI . |
| Approach: | They build an English dataset of 1,415 ecologically valid explanations for 122 MNLI items . they find that people can systematically vary on their interpretation . |
| Outcome: | The proposed dataset contains 1,415 ecologically valid explanations for 122 items . the results show that people can vary on interpretation and highlight differences . |
A Benchmark for Semi-Inductive Link Prediction in Knowledge Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-inductive link prediction (LP) is a task of predicting facts for new, previously unseen entities based on context information. |
| Approach: | They propose to use Wikidata5M to evaluate semi-inductive link prediction (LP) in knowledge graphs. |
| Outcome: | The proposed benchmark provides a test bed for further research into semi-inductive link prediction (LP) in knowledge graphs. |
SummIt: Iterative Text Summarization via ChatGPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text summarization systems generate summaries in a single step, but are often inadequate due to the issue of hallucination and the lack of accuracy. |
| Approach: | They propose an iterative text summarization framework based on large language models like ChatGPT that refines the generated summary iterativly through self-evaluation and feedback. |
| Outcome: | The proposed framework refines the generated summary iteratively through self-evaluation and feedback, closely resembling the iteration humans undertake when drafting and revising summaries. |
Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks. |
| Approach: | They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks. |
Attention-Enhancing Backdoor Attacks Against BERT-based Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing textual backdoor attacks focus on generating stealthy triggers or modifying model weights. |
| Approach: | They propose a Trojan Attention Loss (TAL) which enhances the Trojan behavior by directly manipulating attention patterns. |
| Outcome: | The proposed method improves the effectiveness of the backdoor attacks on different backbone models and tasks. |
Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. |
| Approach: | They propose a higher-order theory of mind benchmark and introduce a new deception mechanism to evaluate ToM reasoning. |
| Outcome: | The proposed benchmarks show that the LLMs are not performing well on higher-order tasks. |
Image and Text: Fighting the same Battle? Super Resolution Learning for Imbalanced Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using high-resolution images to overcome the problem of low resolution has never been used in NLP. |
| Approach: | They propose a super-resolution learning method that uses high-res images to overcome the problem of low resolution images. |
| Outcome: | The proposed method is efficient when compared to state-of-the-art methods on several benchmarks datasets in two languages. |
SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for supervised OOD detection require expensive manual annotation of in-distribution and OOD samples. |
| Approach: | They propose a self-supervised OOD detection method that requires only in-distribution samples as supervision. |
| Outcome: | Experiments with multiple classifiers on coarse- and fine-grained datasets show the proposed method performs well in both coarse-and fine-grid settings. |
Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a gap between conversations can be weeks, months or years, and dialogue systems which do not explicitly model time may generate unnatural responses. |
| Approach: | They propose to model the passage of time between conversations by exposing time information to a multi-session dialogue dataset and comparing different representations of time and event progress. |
| Outcome: | The proposed model is based on a real-time dataset showing that it can predict topics and information gained from conversations over a long time span. |
A Structure-Aware Generative Adversarial Network for Bilingual Lexicon Induction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Bilingual lexicon induction (BLI) is the task of inducing word translations with a learned mapping function that aligns monolingual word embedding spaces in two different languages. |
| Approach: | They propose a model that explicitly captures multiple topological structure information to achieve accurate bilingual lexicon induction. |
| Outcome: | The proposed model captures multiple topological structure information to achieve accurate BLI on a public dataset. |
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating large language models using annotated benchmarks are in trouble . data contamination can cause wrong scientific conclusions being published . |
| Approach: | They argue that the evaluation of NLP tasks using annotated benchmarks is in trouble . they define different levels of data contamination and propose a community effort . |
| Outcome: | The proposed measures should detect when data from a benchmark was exposed to a model and flag papers with conclusions compromised by data contamination. |
Improving Pacing in Long-Form Story Planning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for writing long-form stories suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details. |
| Approach: | They propose a system that uses a concreteness evaluator to judge which of two events is more concrete. |
| Outcome: | The proposed system improves pacing when automatically generating story outlines. |
Argument mining as a multi-hop generative machine reading comprehension task (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Argument mining is a natural language processing task that aims to generate an argumentative graph given an unstructured argumentative text. |
| Approach: | They propose a new approach which transfers the argument mining task into a multi-hop reading comprehension task by incorporating a "chain of thought" information into the model. |
| Outcome: | The proposed approach surpasses SOTA results on two arguments mining benchmarks. |
HuatuoGPT, Towards Taming Language Model to Be a Doctor (2023.findings-emnlp)
Copied to clipboard
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li
| Challenge: | Experimental results show that the distilled language model outperforms its teacher model (ChatGPT) in most cases. |
| Approach: | They propose a Large Language Model (LLM) that leverages both distilled data from **ChatGPT** and real-world data from**doctors** in the supervised fine-tuning stage. |
| Outcome: | The proposed model outperforms the teacher model in most cases by using additional real-world data and RLMF to align the language model with the merits of both sources. |
Debias NLU Datasets via Training-free Perturbations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to debiase NLU models capture biased features that are independent of the task but spuriously correlated to labels. |
| Approach: | They propose a framework that conducts training-free perturbations on samples containing biased features to Debias NLU Datasets. |
| Outcome: | The proposed framework shows competitive performance with previous state-of-the-art debiasing strategies. |
Aspect-to-Scope Oriented Multi-view Contrastive Learning for Aspect-based Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Aspect-based sentiment analysis (ABSA) focus on mining syntactic or semantic information, which suffers from noisy interference when multiple aspects exist in a sentence. |
| Approach: | They propose a scope-assisted multi-view graph contrastive learning framework that captures correlation and difference between aspect and syntactic/semantic information. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on five benchmark datasets and verifies its effectiveness and robustness. |
Robustness of Named-Entity Replacements for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Saeed Goodarzi, Nikhil Kagita, Dennis Minn, Shufan Wang, Roberto Dessi, Shubham Toshniwal, Adina Williams, Jack Lanchantin, Koustuv Sinha
| Challenge: | Modern large language models perform in-context learning, where query- answer demonstrations are shown before the final query. |
| Approach: | They propose to use in-context learning to prompt queries before they are answered . they find that the choice of demonstrations can affect model performance . |
| Outcome: | The proposed model performance improves on named entity replacements across three reasoning tasks and two popular LLMs. |
Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing. |
| Approach: | They propose to use contrastive loss to fine-tune sentences by inverse word frequency . they also show that more informative words receive greater weight than less informative ones . |
| Outcome: | The proposed method improves the performance of sentence embeddings by weighing them based on information-theoretic quantities. |
Legally Enforceable Hate Speech Detection for Public Forums (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work does not align systems with enforceable hate speech definitions, which can make outputs inconsistent with the goals of regulators. |
| Approach: | They propose a task for enforceable hate speech detection centred around legal definitions and an annotated dataset of violations by legal experts. |
| Outcome: | The proposed method can be used to detect hate speech in public forums on a large scale. |
ConPrompt: Pre-training a Language Model with Machine-Generated Data for Implicit Hate Speech Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models for hate speech detection are not specialized in implicit hate speech. |
| Approach: | They propose a pre-trained language model for implicit hate speech detection that leverages machine-generated data to train the model. |
| Outcome: | The proposed model can be trained on a massive hate speech dataset with positive samples . it can be generalized and reduce identity term bias, the authors show . |
Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models lack syntactic knowledge for many tasks that handle complex or long sentences. |
| Approach: | They propose to use pre-trained language models to incorporate syntactic knowledge into a model by adding additional syntatic knowledge to the model. |
| Outcome: | The proposed model can be easily applied to downstream tasks that require syntactic knowledge. |
Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. |
| Approach: | They propose a human readable prompt tuning method that incorporates a fluency constraint to find a distribution of effective and fluent prompts. |
| Outcome: | The proposed method outperforms baselines by 7.0% across three tasks. |
Chain-of-Thought Reasoning in Tabular Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to extend chain-of-thought reasoning into large language models are not viable in the scenario of privatization deployment or limited resources. |
| Approach: | They propose a framework that extends chain-of-thought reasoning into tabular language models . framework coordinates two TaLMs responsible for CoT generation and answer inference . |
| Outcome: | The proposed framework outperforms the state-of-the-art ChatGPT on the TABMWP dataset by 9.55% (82.60%92.15% in accuracy) with less parameters (0.8B). |
Diffusion Language Model with Query-Document Relevance for Query-Focused Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Query-Focused Summarization (QFS) aims to generate summaries that address specific queries by extracting crucial information from source documents. |
| Approach: | They propose a non-autoregressive diffusion language model that incorporates query-document fragment relevance and query-doctoral global relevance to enhance the adaptability of QFS tasks. |
| Outcome: | The proposed model achieves state-of-the-art performance on Debatepedia and PubMedQA datasets in ROUGE scores, GPT-4, and human evaluations. |
Grounded and well-rounded: a methodological approach to the study of cross-modal and cross-lingual grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on grounding have focused on qualitatively different generalizations, but limited empirical evidence supports either position. |
| Approach: | They propose a methodological framework for studying the effects of grounding on NLP systems . they use a sample of models trained on different input modalities to tease out qualitative differences . |
| Outcome: | The proposed framework teases out qualitative differences in model behavior between models trained on different input sources from quantifiable models. |
EMO-KNOW: A Large Scale Dataset on Emotion-Cause (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets focus on extracting parts of the document that contain the emotion cause and fail to provide more abstractive, generalizable root cause. |
| Approach: | They propose to use 9.8 million cleaned tweets to create a large-scale dataset of emotion causes, derived from 9.8 millions tweets over 15 years. |
| Outcome: | The proposed dataset comprises over 700,000 tweets with corresponding emotion-cause pairs spanning 48 emotion classes, validated by human evaluators. |
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments. |
| Approach: | They propose a method to enhance the inference efficiency of parameter-shared PLMs by pre-training models that can achieve even greater acceleration. |
| Outcome: | The proposed method improves inference efficiency on autoregressive and autoencoding models. |
Natural Response Generation for Chinese Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | MRC models trained on labeled answers are limited in generating human-like responses in real QA scenarios. |
| Approach: | They construct a dataset called Penguin to promote machine reading comprehension . they use 200k training data with fluent, well-informed responses to train models . |
| Outcome: | The proposed dataset is the first benchmark towards natural response generation in Chinese MRC on a relatively large scale. |
Treepiece: Faster Semantic Parsing via Tree Tokenization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive (AR) encoder-decoder neural networks are slow in sequential prediction of natural language to machine-readable parse trees. |
| Approach: | They propose a technique that tokenizes a parse tree into subtrees and generates one subtrea per decoding step. |
| Outcome: | The proposed approach shows 4.6 times faster decoding speed and comparable speed but significantly higher accuracy compared to non-autoregressive (NAR) models. |
Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot Dialogue State Tracking have focused on domaintransfers and have not yielded satisfactory results. |
| Approach: | They propose a new In-Context Learning method to introduce additional updating strategies in zero-shot DST by leveraging powerful Large Language Models and translating the original dialogue to JSON through semantic parsing as an intermediate state. |
| Outcome: | The proposed method outperforms existing zero-shot DST methods on MultiWOZ, showing significant improvements in JGA and slot accuracy compared to existing methods. |
Mitigating Framing Bias with Polarity Minimization Loss (2023.findings-emnlp)
Copied to clipboard
| Challenge: | polarity is a pervasive problem in modern media, misleading the understanding of what really happened via a skewed selection of information and language. |
| Approach: | They propose a loss function that encourages the model to minimize the polarity difference between the skewed input articles to reduce framing bias. |
| Outcome: | The proposed loss improves the model's ability to map polarity ends bidirectionally. |
Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate ChatGPT's causal reasoning abilities are based on pre-trained language models, but they rely on supervised training. |
| Approach: | They conduct the first comprehensive evaluation of ChatGPT’s causal reasoning capabilities using four state-of-the-art (STA) simulations. |
| Outcome: | The proposed model is not a good causal reasoner, but a great causal interpreter. |
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, Andre Martins
| Challenge: | Large language models (LLMs) are a promising avenue for machine translation (MT) however, their effectiveness depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. |
| Approach: | They propose a method that incorporates few-shot examples during finetuning to improve performance on MT tasks. |
| Outcome: | The proposed method outperforms few-shot prompting while eliminating the need for in-context examples. |
How Many Demonstrations Do You Need for In-context Learning? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are capable of complex reasoning when given a few input-output demos. |
| Approach: | They use fewer input-output demos for each test query to study ICL . they do not observe significant degradation when using only one randomly chosen demo . |
| Outcome: | The proposed model outperforms multi-demo models on the tasks in 2022. |
Improving word mover’s distance by leveraging self-attention matrix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Word mover's distance (WMD) computes semantic similarity via the optimal alignment between word embeddings, but does not consider word order, making it challenging to distinguish sentences with significant overlaps of similar words. |
| Approach: | They propose to incorporate the sentence structure represented by BERT’s self-attention matrix (SAM) to improve WMD and its variants in paraphrase identification. |
| Outcome: | The proposed method improves WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. |
Improving Span Representation by Efficient Span-Level Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality span representations are limited by subset of tokens . span-span interactions should play an important role in span encoding, authors argue . |
| Approach: | They propose to introduce span-span interactions and more comprehensive span-token interactions to improve span representations. |
| Outcome: | The proposed model outperforms baseline models on span-related tasks and shows superior performance. |
Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models (2023.findings-emnlp)
Copied to clipboard
Simon Stepputtis, Joseph Campbell, Yaqi Xie, Zhengyang Qi, Wenxin Zhang, Ruiyi Wang, Sanketh Rangreji, Charles Lewis, Katia Sycara
| Challenge: | Deception and persuasion play a critical role in long-horizon multi-party dialogues, especially when the interests, goals, and motivations of the participants are not aligned. |
| Approach: | They propose a game in which players must determine each other’s hidden identities to complete their team’s objective. |
| Outcome: | The proposed model can be used to determine the true player identities of six human players in a cooperative-competitive game. |
Improving Sequential Model Editing with Fact Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to fix erroneous knowledge in Pre-trained Language models experience a performance decline when the number of edits increases. |
| Approach: | They propose a framework that leverages factual information to enhance editing generalization and guide the identification of edits by retrieving related facts from the fact-patch memory. |
| Outcome: | The proposed framework can improve model generalization and accuracy even with thousands of edits. |
Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a number of open-source large language models claim to be performing better than commercial ones . however, these models fall short of the performance achieved by closed-source models like GPT-3.5 . |
| Approach: | They evaluate six popular large language models against each other to evaluate their performance . authors say open-source models are not as effective as those built by commercial models . |
| Outcome: | a new set of models claim to match or surpass the language understanding abilities of commercial models . the results show that the models performed far below the performance of closed-source models compared to open-source ones . |
KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for multilingual biomedical training are monolingual, resulting in limited cross-lingual capability. |
| Approach: | They propose a model that transforms a multilingual biomedical corpus into a biomedically domain using a knowledge-anchored approach. |
| Outcome: | The proposed model outperforms monolingual and multilingual models in cross-lingual scenarios. |
Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies using LLMs on psycholinguistic data have gone unverified . a growing body of research is using word-level prediction as a computational proxy . |
| Approach: | They compare morphological, morphologic, and BPE tokenization estimates with reading time data. |
| Outcome: | The proposed method could be used to evaluate morphological prediction. |
A Zero-Shot Language Agent for Computer Control with Structured Reflection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent works require a model to learn from trace examples of a task via supervised learning or few/many-shot prompting. |
| Approach: | They propose a model that iteratively learns from its mistakes via self-reflection and structured thought management. |
| Outcome: | The proposed model outperforms previous models on easy tasks with more efficient reasoning and self-reflection. |
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF (2023.findings-emnlp)
Copied to clipboard
| Challenge: | supervised fine-tuning and reinforcement learning from human feedback (RLHF) are not effective in generating useful and high-quality responses. |
| Approach: | They propose a supervised fine-tuning method that empowers end-users to control responses during inference. |
| Outcome: | Experiments show that supervised fine-tuning and reinforcement learning from human feedback (RLHF) can generate helpful and high-quality responses while maintaining customizability. |
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models (2023.findings-emnlp)
Copied to clipboard
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing approaches to decompose VL reasoning rely on domain-specific sub-question decomposing models. |
| Approach: | They propose a framework that iteratively decomposes VL reasoning using large language models. |
| Outcome: | The proposed framework outperforms existing models on multiple VL reasoning tasks. |
GRI: Graph-based Relative Isomorphism of Word Embedding Spaces (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attempts to control relative isomorphism of different spaces fail to consider lexical variations of semantically similar words . Existing methods for building bilingual dictionaries rely on geometric similarity of individual spaces . |
| Approach: | They propose a method that incorporates the impact of lexical variations of semantically similar words into the training objective. |
| Outcome: | The proposed method outperforms existing research by improving the average P@1 by 63.6%. |
PersonaLM: Language Model Personalization via Domain-distributed Span Aggregated K-Nearest N-gram Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language modeling tools for automatic speech recognition (ASR) are difficult to personalize. |
| Approach: | They propose a domain-distributed Span-Aggregated K-nearest N-gram retrieval augmentation to improve language modeling for automatic speech recognition (ASR) personalization. |
| Outcome: | The proposed model outperforms baselines on Wikitext-103, UserLibri, and ASAP datasets with a 10-16% improvement in perplexity and a 5-8% reduction in word error rates. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Aspect-Category Enhanced Learning with a Neural Coherence Model for Implicit Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) is a major research topic in NLP since social networking services have increased . but the recognition of implicit sentiments that do not contain obvious opinion words remains unexplored . elcom captures document-level coherence by using contrastive learning and sentence-level by a hypergraph . |
| Approach: | They propose aspect-category enhanced learning with a neural coherence model . it captures document-level coherency by contrastive learning and sentence-level by a hypergraph . |
| Outcome: | The proposed model captures document-level coherence by using contrastive learning and sentence-level by a hypergraph to mine opinions from explicit sentences to aid implicit sentiment classification. |
End-to-end Adversarial Sample Generation for Data Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating adversarial samples have deceived many neural inference models, such as text classification and machine translation. |
| Approach: | They propose an adversarial sample generator that consists of a conditioned paraphrasing model and a condition generator and introduce a pretrained discriminator to help the adversarial sample generator adapt to the data characteristics. |
| Outcome: | The proposed approach improves the performance of the trained model on several tasks and is robust for various attacking techniques. |
Query2Triple: Unified Query Encoding for Answering Diverse Complex Queries over Knowledge Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Complex Query Answering (CQA) is a challenge task of Knowledge Graphs due to incompleteness of KGs. |
| Approach: | They propose a query embedding approach that decouples the training for simple and complex queries. |
| Outcome: | The proposed approach decouples training for simple and complex queries and achieves state-of-the-art performance over three public benchmarks. |
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023.findings-emnlp)
Copied to clipboard
Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Jia Liu, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing prompting methods have been used to enhance multistep reasoning capabilities of large language models, but they have overlooked the potential of formulating higher-quality problems. |
| Approach: | They propose a method that starts from the problem side and refines problems to be more comprehensible and solvable for models. |
| Outcome: | The proposed method achieves notable and consistent effectiveness on five reasoning benchmarks across different models. |
Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pruning methods focus on a single pruning criterion and lack variety. |
| Approach: | They propose a model pruning strategy that generates several pruning masks randomly and then chooses the optimal mask from the pool of mask candidates. |
| Outcome: | The proposed pruning strategy achieves state-of-the-art performance across eight datasets from GLUE, particularly excelling at high levels of sparsity. |
Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection depend on knowledge sources that are explicit such as Wikipedia or knowledge graphs. |
| Approach: | They propose a cognitive approach that leverages gaze signals from humans to detect hallucinations in natural language processing (NLP) they collect and introduce an eye tracking corpus consisting of 500 instances, annotated by five annotators for hallucinism detection. |
| Outcome: | The proposed approach achieves a balanced accuracy of 87.1% on a FactCC dataset. |
Noisy Pair Corrector for Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dense retrieval models assume that query-document pairs are exactly matched, resulting in mismatched-pair noise. |
| Approach: | They propose a novel approach to train an effective model with mismatched-pair noise. |
| Outcome: | The proposed model performs well on natural question and triviaQA, code-search benchmarks and SO-DS. |
Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for translating European Portuguese into LGP glosses rely on hand-crafted rules . current systems rely only on toy examples, disregarding non-manual movements . |
| Approach: | They propose a corpora-driven rule-based machine translation system between European Portuguese and LGP glosses and two neural machine translation models. |
| Outcome: | The proposed system improves on existing translation systems and annotates a gold collection of the results. |
Diversifying language models for lesser-studied languages and language-usage contexts: A case of second language Korean (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing morpheme parsers/taggers do not work reliably and optimally for L2 data. |
| Approach: | They train a neural network model on varying L2 datasets and measure its morpheme parsing/POS tagging performance on L2 test sets. |
| Outcome: | The proposed model excels in domain-specific tokenization and POS tagging compared to the baseline model. |
Improving generalization in large langue model by learning prefix subspaces (2023.findings-emnlp)
Copied to clipboard
| Challenge: | emergence of large language models has significantly transformed the applications of deep learning methods in natural language processing. |
| Approach: | They propose to improve LLMs' generalization by optimizing entire models in parameter space by learning entire simplexes of continous prefixes. |
| Outcome: | The proposed method improves generalization of large language models in the scarce data regime. |
Domain Adaptation for Sentiment Analysis Using Robust Internal Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-domain sentiment analysis methods reduce the domain gap by training generalizable classifiers for each domain . large interclass margins in source domain help to reduce the effect of "domain shift" in the target domain. |
| Approach: | They propose a domain adaptation method which induces large margins between data representations that belong to different classes in an embedding space. |
| Outcome: | The proposed method reduces the domain gap by training cross-domain generalizable classifiers . large interclass margins in the source domain help reduce the effect of "domain shift" the proposed method is available in the u.s. |
KeFVP: Knowledge-enhanced Financial Volatility Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current studies ignore the role of financial metrics knowledge in earnings calls and little consideration is given to integrating text and price information. |
| Approach: | They propose to integrate financial metrics knowledge into text comprehension by knowledge-enhanced adaptive pre-training and effectively incorporating text and price information by introducing a conditional time series prediction module. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three real-world datasets and is effective and reliable. |
A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years, Chinese Spelling Check (CSC) has been greatly improved by designing task-specific pre-training methods or introducing auxiliary tasks. |
| Approach: | They propose to decompose Chinese Spelling Check into detection, reasoning, and searching subtasks and to train a module that is compatible with existing CSC models. |
| Outcome: | The proposed module can be trained for one model and benefit other models. |
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ambiguous questions persist in open-domain question answering because formulating a precise question with a unique answer is often challenging. |
| Approach: | They propose to ask a clarification question where the user’s response will help identify the interpretation that best aligns with the user's intention. |
| Outcome: | The proposed approach achieves F1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work. |
Addressing the Length Bias Challenge in Document-Level Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level neural machine translation (DNMT) models incorporate context information through increased maximum lengths of source and target sentences. |
| Approach: | They propose a sliding decoding strategy that limits the length of target sentences . they propose 'length-normalized attention mechanism' to aid the model in focusing on target information . |
| Outcome: | The proposed method can achieve state-of-the-art results on open datasets. |
EconBERTa: Towards Robust Extraction of Named Entities in Economics (2023.findings-emnlp)
Copied to clipboard
Karim Lasri, Pedro Vitor Quinta de Castro, Mona Schirmer, Luis Eduardo San Martin, Linxi Wang, Tomáš Dulka, Haaya Naushan, John Pougué-Biyong, Arianna Legovini, Samuel Fraiberger
| Challenge: | Adapting general-purpose language models to specific domains has proven to be effective in tackling downstream tasks such as impact evaluation. |
| Approach: | They propose to use EconBERTa, a large language model pretrained on scientific publications in economics, and ECON-IE, based on an expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). |
| Outcome: | The proposed model outperforms EconBERTa on the downstream NER task and ECON-IE on the economics abstracts. |
Consonant is all you need: a compact representation of English text for efficient NLP (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In natural language processing, the representation of text plays a crucial role in various tasks such as language modeling, sentiment analysis, and machine translation. |
| Approach: | They propose a method to represent English text with only consonants that is more discriminative than vowels and a technique to retrieve vowel information from it. |
| Outcome: | The proposed representation significantly reduces the overall memory and compute footprint required for storing and processing textual data. |
Detrimental Contexts in Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using the whole passages in QA datasets can improve model accuracy by 10% . |
| Approach: | They analyze how passages can have a detrimental effect on retrieve-then-read architectures used in question answering when evaluated on common question answering datasets. |
| Outcome: | The proposed model accuracy can be improved by 10% on two popular QA datasets by filtering out detrimental passages. |
PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for Indian languages are limited in terms of coverage and size. |
| Approach: | They propose a multilingual and massively parallel summarization corpus focused on languages in India that provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. |
| Outcome: | The proposed dataset provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. |
Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture (2023.findings-emnlp)
Copied to clipboard
Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, Dakuo Wang
| Challenge: | Existing low-resource learning techniques focus on label annotation while neglecting the natural language explanation of a data point. |
| Approach: | They propose a novel architecture that leverages an explanation-generation model to produce explanations guided by human explanations and a prediction model that utilizes generated explanations toward prediction faithfully. |
| Outcome: | The proposed architecture produces explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a data diversity-based AL sampling strategy that benefits from the explanation annotations. |
Decoding Stumpers: Large Language Models vs. Human Problem-Solvers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to the development of systems 2 models that can solve complex tasks and predict human behavior. |
| Approach: | They compare the performance of four state-of-the-art LLMs to human participants and compare their results to stumpers, a unique single-step intuition problem that humans can easily verify. |
| Outcome: | The proposed models excel in solving stumpers and surpass human performance on stumpers, while humans exhibit superior skills in verifying solutions to the same problems. |
Efficient Cross-Task Prompt Tuning for Few-Shot Conversational Emotion Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation (ERC) models are often expensive to train and fine-tune . |
| Approach: | They propose a derivative-free optimization method for few-shot conversational emotion recognition that leverages sharable cross-task knowledge by exploiting external knowledge from other source tasks. |
| Outcome: | The proposed method improves on few-shot scenarios and zero-shot transfers on five different contextual conversation datasets. |
SYMPTOMIFY: Transforming Symptom Annotations with Language Model Knowledge Harvesting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new dataset of annotated vaccine adverse reaction reports is aimed at improving human annotators . a continual evolution in language models and strides in few-shot learning offer promise for improvement. |
| Approach: | They propose a resource to help human annotators improve their efficiency . they evaluate performance across various methods and learning paradigms . |
| Outcome: | The proposed resource outperforms existing systems and learning paradigms in evaluating their performance. |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |
Unified Representation for Non-compositional and Compositional Expressions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing representations of non-compositional language are based on BART, but they are not as accurate as the state-of-the-art IE representation model, GIEA. |
| Approach: | They propose a language model, PIER+, that builds on BART and can generate semantically meaningful and contextually appropriate representations for English potentially idiomatic expressions. |
| Outcome: | The proposed model achieves 33% higher homogeneity score on embedding clustering than BART, while sacrificing performance on NLU tasks (+/- 1% accuracy) |
Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that the quantity and quality of context affect retrieval-augmented generation models during training. |
| Approach: | They propose a method to mitigate overfitting to specific context quality by introducing bias to the cross-attention distribution. |
| Outcome: | The proposed method improves retrieval-augmented generation models on different context quality. |
Error Detection for Text-to-SQL Semantic Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-SQL parsers are often over-confident, thus casting doubt on their trustworthiness when deployed for real use. |
| Approach: | They propose a parser-independent error detection model for text-to-SQL semantic parsing . they use a language model of code as its bedrock and graph neural networks to learn structural features of queries . |
| Outcome: | The proposed model outperforms parser-dependent uncertainty metrics on three strong parsers . it could improve the performance and usability of text-to-SQL semantic parsing, it is shown . |
Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ultra-fine entity typing is a task of inferring the semantic types from a large set of fine-grained candidates that apply to a given entity mention. |
| Approach: | They propose to use pre-trained label embeddings to cluster the labels into semantic domains and treat them as additional types. |
| Outcome: | The proposed method improves the performance of existing models with high quality embeddings. |
Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Several platforms currently rate news outlets according to their political bias, but they are not ready for use in the near future. |
| Approach: | They use ratings of authentic news outlets to create a multilingual corpus of news with coarse stance annotations along with automatically extracted topic annotations. |
| Outcome: | The proposed model can identify the editorial line of most unseen newspapers in English, German, Spanish and Catalan. |
Do “English” Named Entity Recognizers Work Well on Global Englishes? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Most of English named entity recognition datasets contain American or British English data . multiple problems may occur in low-resource English contexts, such as confusion of named entities with regionspecific meanings . |
| Approach: | They build a newswire dataset to analyze NER model performance on low-resource English variants . they find that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops . |
| Outcome: | The results show that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops. |
Affective and Dynamic Beam Search for Story Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | AffGen introduces ‘intriguing twists’ in narratives by employing two novel techniques—Dynamic Beam Sizing and Affective Reranking. |
| Approach: | They propose to use dynamic beam sizing and affective reranking to generate interesting stories using two novel techniques. |
| Outcome: | The proposed method outperforms baseline models in generating affectively charged and interesting narratives. |
Multiview Clickbait Detection via Jointly Modeling Subjective and Objective Preference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing clickbait detection models rely on analyzing the objective semantics of posts or correlating posts with article content only, but fail to identify and exploit the manipulation intention of clickbaiting from a user’s subjective perspective. |
| Approach: | They propose a multiview clickbait detection model to model subjective and objective preferences simultaneously to capture clickbaiting from a user's subjective perspective. |
| Outcome: | The proposed model outperforms state-of-the-art models on two real-world datasets and shows that it integrates subjective and objective preferences simultaneously. |
Let’s Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | *Data Synthesis* is a promising way to train a small model with very little labeled data. |
| Approach: | They propose a framework that iteratively extrapolates the errors of a small model trained on a real-world validation dataset using a large language model. |
| Outcome: | The proposed framework reduces the gap between the synthesized dataset and the real data . it improves on multiple NLP tasks and on large models with human-annotated data. |
Identifying Early Maladaptive Schemas from Mental Health Question Texts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for identifying maladaptive schemas are not practical in 'offline' counseling scenarios such as community QA forums. |
| Approach: | They propose to use large language models and non-LLM methods to identify early maladaptive schemas (EMS) in Schema Therapy. |
| Outcome: | The proposed methods show complementary properties and can be used to further devise techniques for EMS identification. |
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023.findings-emnlp)
Copied to clipboard
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
| Challenge: | Existing methods for image-to-text generation store all knowledge within parameters, thus requiring computational-expensive fine-tuning. |
| Approach: | They propose a Retrieval-augmented Visual Language Model that stores all the knowledge within parameters and can be used to retrieve it from the external database. |
| Outcome: | The proposed model significantly boosts performance for image-to-text generation tasks with 4x less parameters compared with baseline methods. |
Syntax Matters: Towards Spoken Language Understanding via Syntax-Aware Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on SLU systems have focused on integrating syntactic information into language models. |
| Approach: | They propose a model where attention scopes are constrained based on syntactic relationships. |
| Outcome: | The proposed model improves on three datasets and can be integrated into other language models to further boost their performance. |
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive performance in complex reasoning tasks, but it is difficult to know whether they are reasoning based on deep understandings of truth and logic or leveraging their vast previously-seen patterns in a relatively shallow way. |
| Approach: | They propose to test large language models by engaging with them in a debate-like conversation where the user and LLM need to discuss to make the correct decision starting from opposing arguments. |
| Outcome: | The proposed model can achieve the correct answer on its own, but can also hold and defend its belief instead of blindly believing or getting misled by the user’s (invalid) arguments and critiques. |
Using In-Context Learning to Improve Dialogue Safety (2023.findings-emnlp)
Copied to clipboard
Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, Dilek Hakkani-Tur
| Challenge: | Recent work has highlighted safety issues with large neural-based conversational models. |
| Approach: | They propose a retrieval-based approach for reducing bias and toxicity in chatbot responses . they retrieve demonstrations of safe responses to similar dialogue contexts to generate a response . |
| Outcome: | The proposed method reduces bias and toxicity in three chatbot models . it can be used in compliment to existing dialogue safety approaches, such as RLHF. |
HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems are competent only to incorporate information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. |
| Approach: | They propose to perform sensible listening by selectively attending to audio whenever the question requires it. |
| Outcome: | The proposed framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. |
Improving Consistency for Text Summarization with Energy Functions (2023.findings-emnlp)
Copied to clipboard
Qi Zeng, Qingyu Yin, Zheng Li, Yifan Gao, Sreyashi Nag, Zhengyang Wang, Bing Yin, Heng Ji, Chao Zhang
| Challenge: | Current abstractive summarization models generate inconsistent content due to the inherently noisy dataset and the discrepancy between maximum likelihood estimation based training objectives and consistency measurements. |
| Approach: | They propose a new consistency taxonomy that categorizes inconsistent content into faithfulness, factuality, and self-supportiveness. |
| Outcome: | Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency. |
Defining a New NLP Playground (2023.findings-emnlp)
Copied to clipboard
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel Tetreault, Eduard Hovy, Heng Ji
| Challenge: | Recent explosion of performance of large language models (LLMs) has changed the field more abruptly and seismically than any other shift in the field’s 80 year history. |
| Approach: | They propose 20+ PhD-dissertation-worthy research directions to define a new NLP playground by combining theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications. |
| Outcome: | The proposed research will cover theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications. |
UPTON: Preventing Authorship Leakage from Public Text Release via Data Poisoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent authorship attribution models can reveal the true authorship of unseen texts with high accuracies, with some cases up to 95% accuracy. |
| Approach: | They propose a solution that weakens authorship features in training samples and makes released texts unlearnable by exploiting black-box data poisoning methods. |
| Outcome: | The proposed model weakens authorship features in training samples and makes released texts unlearnable. |
IAEval: A Comprehensive Evaluation of Instance Attribution on Natural Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Instance attribution (IA) aims to identify the training instances leading to the prediction of a test example. |
| Approach: | They propose a systematic and comprehensive evaluation scheme covering four significant requirements: sufficiency, completeness, stability and plausibility. |
| Outcome: | The proposed evaluation scheme covers four significant requirements: sufficiency, completeness, stability and plausibility. |
Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Referring expression comprehension is a visual-linguistic task that involves localizing objects in images based on textual referring expressions. |
| Approach: | They propose a scene graph-based framework that generates high-quality pseudo region-query pairs . their method captures relationships between objects in images and generates expressions enriched with relation information. |
| Outcome: | The proposed framework outperforms existing methods by 10%, 12%, and 11% on RefCOCO, RefCoCO+, and Ref COCOg datasets. |
Noisy Self-Training with Synthetic Queries for Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing neural retrieval models require training on a sufficient number of human-labelled query-passage pairs to work well. |
| Approach: | They propose a noisy self-training framework with synthetic queries to improve retrieval methods. |
| Outcome: | The proposed method outperforms baselines on general-domain and out-of-domain retrieval benchmarks on low-resource settings and is data efficient and data efficient. |
Leveraging GPT-4 for Automatic Translation Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models still require translation post-editing to rectify errors and enhance quality under critical settings. |
| Approach: | They use GPT-4 to automatically post-edit NMT outputs across several language pairs . they show that GPT4 is adept at translation post- editing, producing meaningful edits . |
| Outcome: | The proposed translation post-editor improves on state-of-the-art language models on English-Chinese, English-German, Chinese-English and German-English language pairs. |
Uniform Complexity for Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not capture factors that contribute to producing consistent text. |
| Approach: | They propose a benchmark test to evaluate text complexity in generative models by observing linguistic properties of input prompts. |
| Outcome: | The proposed model fails to preserve complexity of input prompts even if finetuned with professionally written texts. |
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LLMs generate responses based on the dialogue context, overlooking the underlying linguistic cues about the user status exhibited in the context. |
| Approach: | They propose a linguistic cue-based chain-of-thoughts method which enhances the LLMs inference with an intermediate reasoning step to find cues exhibited in the dialogue. |
| Outcome: | The proposed method outperforms standard prompting methods on in-depth dialogue questions and linguistic cues exhibited in the context. |
CONTRASTE: Supervised Contrastive Pre-training With Aspect-based Prompts For Aspect Sentiment Triplet Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on Aspect Sentiment Triplet Extraction focus on developing more efficient techniques for the task, but our proposed approach can improve the downstream performance of multiple ABSA tasks simultaneously. |
| Approach: | They propose a novel approach that uses contrastive learning to enhance the ASTE performance by masked sentiments. |
| Outcome: | The proposed approach improves the performance of multiple ABSA tasks simultaneously. |
Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Continual pre-training has been used for a multitude of domains and tasks . a continually pre-trained model can show a non-decreasing performance on unseen domains . |
| Approach: | They propose a method that generates domain-specific prompts by agreement and disagreement losses. |
| Outcome: | The proposed method achieves improvements of 3.57% and 3.4% on two real-world datasets. |
Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Visual question answering (VQA) is a task that requires an understanding of both the image and the question to provide a natural language answer. |
| Approach: | They propose a multimodal framework that leverages language guidance to answer questions more accurately. |
| Outcome: | The proposed framework improves on the multi-choice question-answering task using CLIP and BLIP models. |
XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method to segment speech into words is needed to overcome the lack of explicit word boundaries in the speech stream. |
| Approach: | They propose to fine-tune a self-supervised speech model to predict word boundaries . they use XLS-R to fine tune the models and infer new word boundary labels . |
| Outcome: | The proposed model outperforms existing models and sets a new state-of-the-art on five corpora with different languages. |
Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a new approach to prompt large language models (LLMs) but most studies rely on human-annotated rational chains to prompt LLMs . |
| Approach: | They propose a method that augments rational chains from a small labeled dataset and pruning low-quality chains to construct a pool of machine generated rationale chains based on the labels. |
| Outcome: | The proposed method can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and pruning low-quality chains to construct a candidate pool of machine generated rationale chains based on the labels. |
What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations (2023.findings-emnlp)
Copied to clipboard
Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, Yejin Choi
| Challenge: | Moral or ethical judgments rely heavily on the contexts in which they occur . a student model that produces defeasible contexts with improved validity, diversity, and defasibility is superior to intermediate student models . |
| Approach: | a new study uses a student model to provide contextualizations that make an action morally acceptable . the model is based on a dataset of 115K defeasible moral actions rated highly by human annotators . |
| Outcome: | The proposed model outperforms all intermediate models in a high-quality dataset . the model is based on 1.2M entries of contextualizations and rationales for 115K moral actions . |
An Empirical Study on Multiple Knowledge from ChatGPT for Emotion Recognition in Conversations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts in ERC focus on context- and speaker-sensitive dependencies, but lack of annotated data and high cost of obtaining such knowledge is a blank slate. |
| Approach: | They propose a Multiple Knowledge Fusion Model to integrate multiple knowledge generated by Large Language Models (LLMs) they analyze the contribution and complementarity of this knowledge into the model. |
| Outcome: | The proposed model integrates multiple knowledge generated by LLMs and analyzes its contribution and complementarity on three public datasets. |
Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies fail to distinguish different classification errors with a standard cross-entropy classification loss and ignore the numbers in the fact description for predicting the term of penalty. |
| Approach: | They propose to extract crime amounts from fact description and use them to learn distinguishable representations to exploit the numbers in the fact description for predicting the term of penalty. |
| Outcome: | The proposed method achieves state-of-the-art results on real-world datasets and ablation studies demonstrate the effectiveness of each component. |
One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot cross-lingual transfer are unreliable due to the lack of pretraining data. |
| Approach: | They propose to accumulatively average model snapshots from different runs into a single model. |
| Outcome: | The proposed protocol decouples performance maximization from hyperparameter tuning. |
Dimensions of Online Conflict: Towards Modeling Agonism (2023.findings-emnlp)
Copied to clipboard
Matt Canute, Mali Jin, Hannah Holtzclaw, Alberto Lusoli, Philippa Adams, Mugdha Pandya, Maite Taboada, Diana Maynard, Wendy Hui Kyong Chun
| Challenge: | agonism fosters robust discussions, but hateful antagonism undermines constructive dialogue . a new study analyzes Twitter conversations to identify different dimensions of conflict . |
| Approach: | They annotated Twitter conversations related to trending controversial topics to model conflict on a richly annotized dataset. |
| Outcome: | The proposed model can help to moderate online conflicts and improve content monetization. |
Learning under Label Proportions for Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using a set of data samples with ground truth labels, learning from label proportions is infeasible in many cases. |
| Approach: | They propose a setup where the data is provided in an aggregate form called bags and only the proportion of samples belonging to each class as the ground truth. |
| Outcome: | The proposed model achieves better results than baselines in almost 87% of the configurations. |
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn compositional concepts from previous experience are based on systematic generalization, productivity and substitutivity. |
| Approach: | They propose a retrieval-enhanced meta-learning model to solve the visually grounded compositional concept learning problem by meta-training retrieved primitive concepts from episodes constructed by the retriever. |
| Outcome: | The proposed model outperforms other baselines and the retrieval module plays an important role in this compositional learning process. |
PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing image captioning metrics are vulnerable to lexical perturbations, but they are not robust to such perturbations. |
| Approach: | They propose a perturbation-robust multilingual CLIPScore which is a reference-free image captioning metric for multiple languages. |
| Outcome: | The proposed metric outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages while maintaining a strong correlation with human judgments. |
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have shown effectiveness in literature understanding tasks, especially when tuned via contrastive learning. |
| Approach: | They propose a multi-task contrastive learning framework that enables common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. |
| Outcome: | The proposed framework outperforms state-of-the-art pre-trained language models on a comprehensive dataset. |
BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current NLP models are achieving performance comparable to human capabilities on well-established benchmarks. |
| Approach: | They propose a BLM task to identify a missing element in a linguistic pattern from a list of candidate options based on a given matrix. |
| Outcome: | The proposed framework is based on the spray-load verb alternations in English as a case study. |
Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that adding a instruction tuning stage to training large language models can improve zero-shot task generalization. |
| Approach: | They propose a method that retrieves promptspecific source prompt embeddings from training instances . they train soft prompt embeds for each prompt through prompt tuning and store the samples . |
| Outcome: | The proposed method outperforms hard prompts on unseen tasks by 2.39% points and outperformed 10 out of 11 datasets. |
Geographical Erasure in Language Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models encode vast amounts of world knowledge but are at risk of inordinately capturing information about dominant groups. |
| Approach: | They propose to operationalise a form of geographical erasure wherein language models underpredict certain countries. |
| Outcome: | The proposed model underpredicts certain countries by a factor 3 . the model is based on large datasets and is able to mitigate the effects . |
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? (2023.findings-emnlp)
Copied to clipboard
Yuwei Bao, Keunwoo Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alex de la Iglesia, Megan Su, Xiao Zheng, Joyce Chai
| Challenge: | despite advances in AI, it remains a challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. |
| Approach: | They propose to use a multimodal benchmark dataset to study whether interactive task guidance systems can be quickly adapted to perceptually enabled tasks. |
| Outcome: | The proposed models demonstrate fair performances in some cases with no training . the results will provide a stepping stone for future work on situated task guidance . |
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)
Copied to clipboard
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler
| Challenge: | Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour. |
| Approach: | They conduct extensive experiments to understand scaling behaviour of ten different model architectures. |
| Outcome: | The results show that the best performing model can fluctuate at different scales. |
Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate impressive multilingual capability, but their performance varies substantially across different languages. |
| Approach: | They propose a generic template prompt that stimulates cross-lingual and logical reasoning skills to enhance task performance across languages. |
| Outcome: | The proposed method improves multilingual capability across languages and covers high-resource and low-resourced languages. |
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models generate huge amounts of text, making it impractical to manually distinguish whether a text is machine-generated. |
| Approach: | They propose two methods to detect machine-generated text by leveraging Log-Rank information and propose a faster method that uses less perturbations to achieve the same level of performance. |
| Outcome: | The proposed methods improve over the state of the art by 3.9 and 1.75 AUROC points absolute and require less perturbations to achieve the same level of performance. |
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve complex logical reasoning problems are cumbersome for language models. |
| Approach: | They propose to use iterative methodology to construct a cognitive tree using language models . they propose to generate multiple responses by utilizing in-context examples . |
| Outcome: | The proposed model achieves a performance level comparable to that of GPT-3.5 . the proposed model contains fewer parameters than 5% of the model with 175B parameters . |
Macedon: Minimizing Representation Coding Rate Reduction for Cross-Lingual Natural Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual models require limited data to perform cross-linguistic tasks. |
| Approach: | They propose a method to remove language-associated information via minimizing representation coding rate reduction. |
| Outcome: | The proposed model outperforms state-of-the-art models on cross-lingual tasks. |
Adversarial Robustness for Large Language NER models using Disentanglement and Word Attributions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks are becoming more challenging due to the introduction of complex tagsets, which often leads to the failure of existing NER systems in accurately recognizing these entities. |
| Approach: | They propose a novel attack which relies on disentanglement and word attribution techniques to learn an embedding and identifying important words across both components. |
| Outcome: | The proposed approach improves the F1 score over the original LLM model by 8% and 18% on CoNLL-2003 and Ontonotes 5.0 datasets respectively. |
LLMs – the Good, the Bad or the Indispensable?: A Use Case on Legal Statute Prediction and Legal Judgment Prediction on Indian Court Cases (2023.findings-emnlp)
Copied to clipboard
Shaurya Vats, Atharva Zope, Somsubhra De, Anurag Sharma, Upal Bhattacharya, Shubham Nigam, Shouvik Guha, Koustav Rudra, Kripabandhu Ghosh
| Challenge: | Large Language Models have touched upon many real-life tasks. |
| Approach: | They apply Large Language Models to two popular tasks: Statute Prediction and Judgment Prediction. |
| Outcome: | The proposed model performs well in Statute Prediction and Judgment Prediction on Indian Supreme Court cases. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |
Large Language Models Are Better Adversaries: Exploring Generative Clean-Label Backdoor Attacks Against Text Classifiers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Backdoor attacks manipulate model predictions by inserting malicious "poison" instances that contain a specific pattern or "trigger." |
| Approach: | They propose an attack that inserts style-based triggers into training and test data by using a poison selection technique to improve the effectiveness of both LLMBkd and existing backdoor attacks. |
| Outcome: | The proposed attack achieves high success rates across a wide range of styles with little effort and no model training. |
Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) are advanced but data labels are noisy due to the complex annotation process. |
| Approach: | They propose a framework for fine-tuning PLMs using noisy labels that incorporates guidance from Large Language Models like ChatGPT. |
| Outcome: | Experiments on synthetic and real-world noisy datasets show that the proposed framework outperforms the state-of-the-art framework. |
Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are capable of answering knowledge-intensive complex questions with chain-of-thought reasoning. |
| Approach: | They propose a method to solve complex questions with a tree-of-thought approach using parametric knowledge and retrieved external knowledge to augment CoT reasoning. |
| Outcome: | The proposed approach outperforms SOTA methods on three Complex QA datasets under the open-domain setting. |
Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs (2023.findings-emnlp)
Copied to clipboard
Young-Suk Lee, Md Sultan, Yousef El-Kurdi, Tahira Naseem, Asim Munawar, Radu Florian, Salim Roukos, Ramón Astudillo
| Challenge: | Empirical studies with different instruction-tuned LMs show that our proposed method yields higher-quality instruction tuning data than Self-Instruct. |
| Approach: | They propose to use in-context learning techniques to train strong conversational agents . they propose to categorize and simplify ICL templates to make prompt learning easier . |
| Outcome: | Empirical results show that the proposed method yields higher-quality instruction tuning data than Self-Instruct and improves performance of both vanilla and instruction-tuned LMs. |
The Less the Merrier? Investigating Language Representation in Multilingual Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual models can be used to integrate multiple languages into one model and use cross-language transfer learning to improve performance for different NLP tasks. |
| Approach: | They propose to include languages in popular multilingual models and to use cross-language transfer learning to improve performance for different NLP tasks. |
| Outcome: | The proposed models perform better on downstream tasks for seen and unseen languages than community-centered models for low-resource languages. |
SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research (2023.findings-emnlp)
Copied to clipboard
Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa-Anke, Jiaxin Pei, Jose Camacho-Collados
| Challenge: | specialised language models (LMs) have shown to exhibit lower perplexity and higher downstream performance across the board. |
| Approach: | They propose a benchmark for NLP evaluation in social media, SuperTweetEval. |
| Outcome: | The proposed benchmark shows that social media models perform better when compared to general-purpose models, metrics and benchmarks. |
Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation methods have been observed to make particular errors in comparison to supervised machine translation, such as confusing nouns that pertain to the same semantic category. |
| Approach: | They propose a method that incorporates images at the word level to augment lexical mappings. |
| Outcome: | Experiments on a multi-lingual dataset show that the proposed method generates more accurate translations with only monolingual data. |
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches (2023.findings-emnlp)
Copied to clipboard
| Challenge: | People rely heavily on context to enrich meaning beyond what is literally said. |
| Approach: | They analyze how task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. |
| Outcome: | The proposed frameworks are based on linguistic goals, environmental contexts, and communicative affordances to enrich linguistic meaning. |
MISCA: A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for detecting multiple intents and filling slots are based on graphs but face problems . a joint model can exploit the correlations between intents, slots and slot filling tasks . |
| Approach: | They propose a joint model that captures correlations between intents and slot labels . they propose MISCA to incorporate an intent-slot co-attention mechanism and a label attention mechanism . |
| Outcome: | The proposed model outperforms previous models on two benchmark datasets. |
Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Emotion recognition in conversation (ERC) is an advanced capability of conversational AI systems. |
| Approach: | They propose a semi-parametric paradigm for Emotion Recognition in conversation that uses supervised contrastive learning to align semantic-view and context-view features. |
| Outcome: | The proposed model achieves state-of-the-art on four widely used benchmarks. |
Mandarin classifier systems optimize to accommodate communicative pressures (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that gendered languages are inherently optimized to accommodate communicative pressures on language learning and processing. |
| Approach: | They propose to use grammatical or probabilistic modifiers to smooth the entropy of nouns in context to find the same frequency, similarity, and co-occurrence interactions that structure gender systems. |
| Outcome: | The proposed noun classification device is sensitive to frequency, similarity, and co-occurrence interactions that structure gender systems. |
Probing Representations for Document-level Event Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level information extraction tasks require a more comprehensive understanding that often extends to the entire input document. |
| Approach: | They propose to use probing to analyze document-level information extraction representations by embedding probes into a standard dataset. |
| Outcome: | The proposed models improve argument detections but struggle with document length and cross-sentence discourse. |
Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Features (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current knowledge is limited on whether cultural features can predict cross-cultural transfer learning success for subjective tasks. |
| Approach: | They advocate integration of cultural information into datasets and cultural adaptability . findings suggest cultural features can predict cross-cultural transfer learning success . |
| Outcome: | The findings suggest that cultural features can predict cross-cultural transfer learning success in OLD tasks. |
Linguistically Motivated Sign Language Segmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Sign language segmentation is a crucial task in sign language processing systems. |
| Approach: | They propose to combine two kinds of segmentation: segmentation into individual signs and segmentation to segment into phrases, larger units comprising several signs. |
| Outcome: | The proposed model is based on linguistic cues observed in sign language corpora and replaces the predominant IO tagging scheme with BIO taging to account for continuous signing. |
Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing active learning approaches focus on information-rich sequences, reducing the need for expert annotation. |
| Approach: | They propose a re-weighting-based active learning strategy that assigns dynamic weights to individual tokens. |
| Outcome: | The proposed strategy improves on multiple corpora and validates its effectiveness. |
Language-Agnostic Bias Detection in Language Models with Bias Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) contain strong social biases, which are difficult to quantify because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input. |
| Approach: | They propose a bias probing technique called LABDet to evaluate social bias in pretrained language models with a language-agnostic method. |
| Outcome: | The proposed method “surfaces” nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection. |
CompleQA: Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on Knowledge Graph Completion as an end in itself, neglecting its potential impact on subsequent applications. |
| Approach: | They propose a benchmark to assess the impact of representative KGC methods on Knowledge Graph Question Answering (KGQA) they use a knowledge graph with 3 million triplets across 5 distinct domains to evaluate their results. |
| Outcome: | The proposed benchmark compares four well-known methods with two state-of-the-art systems to assess the impact of incomplete knowledge graphs on KGQA. |
Improving Multi-Criteria Chinese Word Segmentation through Learning Sentence Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent Chinese word segmentation models tend to learn the segmentation knowledge through in-vocabulary words rather than understanding the meaning of the entire context. |
| Approach: | They propose a context-aware approach that incorporates unsupervised sentence representation learning over different dropout masks into the multi-criteria training framework. |
| Outcome: | The proposed approach achieves state-of-the-art (SoTA) performance on six of the nine CWS benchmark datasets and out-of vocabulary (OOV) recalls for eight of nine. |
A Joint Matrix Factorization Analysis of Multilingual Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that pre-trained models acquire and incorporate linguistic knowledge in their multilingual representations. |
| Approach: | They propose a tool for comparing latent representations of multilingual and monolingual models . they use joint matrix factorization to analyze multiple sets of representations in a joint manner . |
| Outcome: | The proposed tool analyzes latent representations of multilingual and monolingual models . it shows that language properties influence the factorization outputs . |
Don’t Add, don’t Miss: Effective Content Preserving Generation from Pre-Selected Text Spans (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing CTR models are mediocre and lack reliable performance . authors propose an explicit decomposition of these two subtasks into a single task . |
| Approach: | They propose an isolated task that challenges models to generate coherent text conforming to pre-selected content within the input text ("highlights") authors propose a high-quality, open-source CTR model that tackles two prior key limitations: inadequate enforcement of the content-preservation constraint, and suboptimal silver training data. |
| Outcome: | The proposed model significantly improves silver training data quality over the existing model, with up to 30 ROUGE-L points. |
A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on interpreting the high-level strategic intent of humans has focused on enabling them to specify their strategy to an AI system. |
| Approach: | They propose to translate unstructured language strategies into actionable intent in the form of goals and constraints by using a game environment to build a computational interface. |
| Outcome: | The proposed model significantly outperforms human interpreters in inferring strategic intent from language (p 0.05) and in a low-data setting. |
HFMRE: Constructing Huffman Tree in Bags to Find Excellent Instances for Distantly Supervised Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to extract sentence-level features are labor-intensive and time-consuming. |
| Approach: | They propose a distantly supervised relation extraction algorithm that uses circular cosine similarity to show intrinsic associations between sentences within a bag. |
| Outcome: | The proposed method outperforms baselines on the popular DSRE datasets. |
DISCO: A Large Scale Human Annotated Corpus for Disfluency Correction in Indo-European Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on disfluency correction has primarily focused on English due to the unavailability of large-scale open-source datasets. |
| Approach: | They propose to use an annotated human-annotated corpus to analyze disfluency correction in four important Indo-European languages to demonstrate the benefits. |
| Outcome: | The proposed model improves BLEU scores by 5.65 points when used with a state-of-the-art machine translation system. |
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have established that Mixture-of-experts models are parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. |
| Approach: | They propose a mix-of-experts model that uses sparse activation to increase the number of parameters while maintaining low computational requirements per token. |
| Outcome: | The proposed models outperform state-of-the-art models on three multilingual machine translation benchmarks with 4, 15, and 94 language pairs. |
Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a negative emotion is a cognitive bias that affects how we express thoughts and opinions online . a recent study shows that negative words generate more engagement and clicks than positive ones . |
| Approach: | They propose to use readability and linguistic complexity metrics to better understand emotions . they propose to fine-tune three state-of-the-art transformers to detect emotions based on a dataset . |
| Outcome: | The proposed model fails to predict emotions on complex texts, the authors show . they also show that more advanced models fail to predict complex texts . |
Probing the “Creativity” of Large Language Models: Can models produce divergent semantic association? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models possess remarkable capacity for processing language, but it remains unclear whether they can further generate creative content. |
| Approach: | They utilize the divergent association task (DAT) to examine the creative thinking of large language models through a cognitive perspective. |
| Outcome: | The proposed model outperforms the greedy search strategy while outperforming the average human level. |
Code-Switching with Word Senses for Pretraining in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT) many state-of-the-art (SOTA) NMT systems struggle to handle polysemous words . |
| Approach: | They propose an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. |
| Outcome: | The proposed approach improves translation quality and scales to various data and resource-strapped scenarios. |
DiffusionSL: Sequence Labeling via Tag Diffusion Process (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Sequence Labeling (SL) is a long-standing field of natural language processing. |
| Approach: | They propose a framework that utilizes a conditional discrete diffusion model for generating discrete tag data. |
| Outcome: | The proposed framework outperforms gpt-3.5-turbo on multiple benchmark datasets and tasks. |
COMET-M: Reasoning about Multiple Events in Complex Sentences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing commonsense models that generate event-centric inferences for simple sentences struggle with the complexity of multi-event sentences prevalent in natural text. |
| Approach: | They propose a commonsense model that generates inferences for a target event within a complex sentence using a multi-event inference dataset. |
| Outcome: | The proposed model produces inferences for a target event within a complex sentence taking the complete context into account. |
On Event Individuation for Document-Level Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a bomb exploded in a restaurant in Lima, and a second device was deactivated by the police . |
| Approach: | They argue that the task demands definitive answers to thorny questions of *event individuation* they argue that even human experts disagree on the task . |
| Outcome: | The proposed task demands definitive answers to thorny questions of *event individuation* . the proposed task also raises concerns about the usefulness of template filling metrics . |
AniEE: A Dataset of Animal Experimental Literature for Event Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Event extraction (EE) is a crucial information extraction task in biomedical domain . existing biomedically EE datasets focus on cell experiments or overall procedures . |
| Approach: | They propose an animal experiment customized entity and event scheme for event extraction . they create an expert-annotated high-quality dataset containing discontinuous entities and nested events . |
| Outcome: | The proposed dataset is based on the animal experiment stage and a NER and EE model. |
From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models have a capacity for electronic circuit design from high-level textual descriptions, akin to code generation. |
| Approach: | They propose two benchmarks to assess a model's ability to design common microcontroller circuits and code in the Arduino ecosystem. |
| Outcome: | The proposed model knowledge of electrical components and the model's ability to generate full circuits and code in the Arduino ecosystem are evaluated. |
Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured prediction is a fundamental problem in NLP, wherein the label space consists of complex structured outputs with groups of interdependent variables. |
| Approach: | They propose a partial annotation approach that selects only the most informative sub-structures for annotation and a method that incorporates the current model's automatic predictions as pseudo-labels for un-annotated sub-structurals. |
| Outcome: | The proposed approach reduces annotation cost over strong full annotation baselines under a fair comparison scheme that takes reading time into consideration. |
Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has explored how to improve the abilities of decision-making and question generation. |
| Approach: | They propose a pipeline framework that aligns the document and user-provided information in an explicit way, makes decisions using a lightweight many-to-many entailment reasoning module and generates follow-up questions based on the document. |
| Outcome: | The proposed framework achieves state-of-the-art in micro-accuracy and ranks the first place on the public leaderboard of the CMR benchmark dataset ShARC. |
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to understanding compositional generalization of models have focused on novel architectures and alternative learning paradigms. |
| Approach: | They propose a method that harnesses the power of dataset cartography to improve model accuracy by strategically identifying a subset of compositional generalization data. |
| Outcome: | The proposed method improves model accuracy by 10% on CFQ and COGS datasets. |
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown strong abilities to understand natural language, but how these factors affect the models’ language perception is unclear. |
| Approach: | They compare the self-attention of several existing large language models in different sizes to assess the effect of scaling and instruction tuning on language perception. |
| Outcome: | The proposed models are closer to non-native speakers than native speakers in attention, suggesting a sub-optimal language perception of all models. |
Efficient Data Learning for Open Information Extraction with Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the time (3 minutes) to achieve comparable results. |
| Approach: | They propose a framework that transforms OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. |
| Outcome: | The proposed framework transforms OpenIE into the pre-training task form of the T5 model, reducing the need for extensive training data and significantly reducing training time. |
Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning has been an effective paradigm for large pretrained language models (LLMs), enabling few-shot or even zero-shot learning. |
| Approach: | They propose a black-box prompt search method that clusters and prunes the search space to focus exclusively on influential prompt tokens. |
| Outcome: | The proposed method achieves state-of-the-art performance across tasks and LLMs while significantly reducing search costs. |
Towards Zero-shot Learning for End-to-end Cross-modal Translation Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end zero-shot speech translation model is based on a zero-shot approach, but it is less competitive because of the limited amount of data available for multiple modalities. |
| Approach: | They propose an end-to-end zero-shot speech translation model that connects two pre-trained uni-modality modules via word rotator’s distance. |
| Outcome: | The proposed model performs better than or as well as those of the CTC-based models and can be trained in an end-to-end style to avoid error propagation. |
LLMaAA: Making Large Language Models as Active Annotators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing supervised learning methods in natural language processing require large amounts of data. |
| Approach: | They propose an active learning loop that takes LLMs as annotators and puts them into an active loop to determine what to annotate efficiently. |
| Outcome: | The proposed model outperforms existing models with few-shot performance in two NLP tasks. |
NLMs: Augmenting Negation in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Negation is the fundamental component in a natural language that reverses the semantic meaning of a sentence. |
| Approach: | They propose a language model objective with a weighted cross-entropy loss and elastic weight consolidation regularization to improve negation understanding. |
| Outcome: | The proposed model reduces the error rate of the existing models by 8% and outperforms them on original and negation benchmarks. |
Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt tuning is a technique that updates few parameters in pre-trained models for language understanding and generation tasks. |
| Approach: | They propose to leverage prompt tuning for neural text retrieval to improve generalization and cross-domain generalization. |
| Outcome: | The proposed approach can mitigate the two issues faced by fine-tuning retrieval methods and improve the out-of-domain zero-shot generalization of the retrieval models. |
X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance when evaluated in non-English languages. |
| Approach: | They propose to use sub-network similarity between two languages as a proxy for XLT prediction. |
| Outcome: | The proposed method shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3. |
Noise-Robust Semi-Supervised Learning for Distantly Supervised Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Distantly supervised relation extraction (DSRE) methods are not capable of extracting relation labels for individual sentences. |
| Approach: | They propose a semi-supervised learning relation extraction framework for sentence-level DSRE . they discard only the labels of the noisy samples and utilize them as unlabeled samples . |
| Outcome: | The proposed framework achieves significant performance enhancements on two real-world datasets. |
Towards Concept-Aware Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Concepts play a pivotal role in various human cognitive functions, including reasoning and communication. |
| Approach: | They analyze how well contemporary large language models capture human concepts and their structure . they propose a method for pretraining LLMs using concepts and a simpler approach . |
| Outcome: | The proposed method matches human intuition and improves robustness of predictions. |
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have led to significant breakthroughs in the field. |
| Approach: | They evaluate ChatGPT over multiple tasks with diverse languages and large datasets to provide more comprehensive information for multilingual NLP applications. |
| Outcome: | The proposed model can process and generate texts for multiple languages due to its multilingual training data. |
Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Contemporary advances in NLP are built on the representational power of latent embedding spaces learned by self-supervised language models (LMs). |
| Approach: | They use a new information theoretic probing suite to analyze representational subspaces in language models. |
| Outcome: | The proposed approach compared performance of nine tasks across 2M pre-training steps and five seeds. |
Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have recently gained the In-Context Learning ability . however, the quality of demonstration examples is usually uneven . |
| Approach: | They propose to determine optimal weights for demonstration examples and apply them during ICL. |
| Outcome: | The proposed approach outperforms conventional ICL on 8 classification tasks. |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
Learn From One Specialized Sub-Teacher: One-to-One Mapping for Feature-Based Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical results show that our proposed approach outperforms the state-of-the-art methods by maintaining higher performance on most benchmark datasets. |
| Approach: | They propose to break down the global feature distillation task into N local sub-tasks and make each focused sub-student learn from one specialized sub-teacher. |
| Outcome: | The proposed method outperforms state-of-the-art methods on most benchmark datasets while maintaining higher performance. |
IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to align motion sensors with text and video are limited in their scale and limited in the use of IMU models. |
| Approach: | They propose to project IMU motion sensor recordings into the joint representation space of Contrastive Language-Image Pre-training (CLIP) they introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data. |
| Outcome: | The proposed approach significantly improves downstream performance when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource. |
Conditioning on Dialog Acts improves Empathy Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research has focused on empathetic response generation, but it is not applicable to more sensitive cases such as medicine and therapy where the content of the responses requires the supervision of medical experts. |
| Approach: | They propose two new style transfer strategies that use only examples of the target style and dialog-act-conditioned prompting to make a sentence more empathetic. |
| Outcome: | The proposed methods improve empathy more effectively while maintaining semantic similarity and preserving both semantics and the dialog-act type. |
Systematic Assessment of Factual Knowledge in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing question-answering benchmarks for large language models have limitations regarding factual knowledge coverage, as they focus on generic domains and overlap with pretraining data. |
| Approach: | They propose a framework to assess the factual knowledge of large language models by leveraging knowledge graphs. |
| Outcome: | The proposed framework generates questions and expected answers from the facts stored in a given knowledge graph and evaluates them with KGs in generic and specific domains. |
From Speculation Detection to Trustworthy Relational Tuples in Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on speculation detection are defined at sentence level, but not all factual tuples extracted from a sentence are speculative. |
| Approach: | They propose to study speculations in OIE tuples and determine whether a tample is speculative. |
| Outcome: | The proposed model is based on the LSOIE dataset and provides labels for speculative tuples. |
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained sequence-to-sequence (seq2sequ) models have been widely used to solve extractive tasks, where parts of the input are extracted to form the desired output. |
| Approach: | They propose a simple fix to tokenization inconsistency that damages extractive nature of generative models by causing performance drop and hallucination. |
| Outcome: | The proposed model performs better in both in-domain and out-of-domain datasets with a notable average of +1.7 F1 gain when a BART model is trained on SQuAD and evaluated on 8 QA datasets. |
Dialogue Medical Information Extraction with Medical-Item Graph and Dialogue-Status Enriched Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches ignore relationships between medical items and statuses in the multi-turn doctor-patient dialogue. |
| Approach: | They propose a task to extract structured medical information from free text dialogues . they propose 'Dialogue Medical Information Extraction' to model relationships between items . |
| Outcome: | The proposed model outperforms previous models and achieves state-of-the-art performance on the public benchmark data set. |
LogicAttack: Adversarial Attacks for Evaluating Logical Consistency of Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated impressive performance on Natural Language Inference (NLI) tasks. |
| Approach: | They propose a method to attack NLI models using diverse logical forms of premise and hypothesis using propositional logic to generate effective adversarial attacks. |
| Outcome: | The proposed method achieves an average 53% Attack Success Rate (ASR) across multiple logic-based attacks. |
Decomposed Prompt Tuning via Low-Rank Reparameterization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable performance on various tasks. |
| Approach: | They propose a decomposed prompt tuning approach that utilizes low-rank matrices to initialize the soft prompt. |
| Outcome: | The proposed method significantly reduces the number of trainable parameters while maintaining effectiveness. |
SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that SGP-TOD provides state-of-the-art zero-shot performance . prevailing approach for creating task bots is to fine-tune pre-trained language models . |
| Approach: | They propose a Schema-Guided Prompting for building Task-Oriented Dialog systems . they use predefined task schema and dialog policy to instruct fixed LLMs to generate appropriate responses . |
| Outcome: | The proposed system outperforms few-shot approaches on multiwoz, RADDLE, and STAR datasets. |
Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a paper by a team of researchers proposes that large language models should be morally aligned to ethical principles . a moral compass is a model that integrates moral dilemmas with moral principles pertaining to different foramlisms of normative ethics . |
| Approach: | They propose to infuse generic ethical reasoning capabilities into large-scale models . they argue that LLMs should take a moral stance on value pluralism . |
| Outcome: | a new ethical reasoning framework integrates moral dilemmas with moral principles . the framework is based on the results of a hypothetical case study on a large-scale model . |
Vector-Quantized Prompt Learning for Paraphrase Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for paraphrase generation are difficult to understand and generate. |
| Approach: | They propose to generate diverse paraphrases by using instance-dependent prompts to control the generation of pre-trained models. |
| Outcome: | The proposed method achieves state-of-the-art on three benchmark datasets, including Quora, Wikianswers, and MSCOCO. |
Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models are expected to map input text to a set of vectors while preserving the inherent relationships within the text. |
| Approach: | They propose a new line to constructing metrics for understanding the mechanisms of pretrained language models. |
| Outcome: | The proposed model is referred to as the tree topological probe and is based on a tree topology probe. |
PARROT: Zero-Shot Narrative Reading Comprehension via Parallel Reading (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to narrative comprehension require extensive annotation of data. |
| Approach: | They propose a zero-shot approach for narrative comprehension through parallel reading using two parallel narratives that tell the same story. |
| Outcome: | The proposed approach surpasses previous zero-shot approaches and comparable performance to fully supervised models. |
BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance (2023.findings-emnlp)
Copied to clipboard
Karel D’Oosterlinck, François Remy, Johannes Deleu, Thomas Demeester, Chris Develder, Klim Zaporojets, Aneiss Ghodsi, Simon Ellershaw, Jack Collins, Christopher Potts
| Challenge: | pharmacovigilance (PV) is a tool for analyzing adverse drug events from biomedical literature . pharmacologists use natural language processing to extract core information from papers . |
| Approach: | They propose a resource for biomedical adverse drug event eXtraction using natural language processing. |
| Outcome: | The proposed model achieves 59.1% F1 (validation) and estimates human performance to be 72.0% F1 . the proposed model could be used to improve drug safety monitoring, also called pharmacovigilance, in the future. |
Coarse-to-Fine Dual Encoders are Better Frame Identification Learners (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to model frame definitions lack sufficient representation learning of definitions or lack efficient frame modeling. |
| Approach: | They propose a frame-target-encoder architecture that uses coarse-to-fine learning to model alignment between frames and targets. |
| Outcome: | The proposed framework outperforms existing models by 0.93 overall scores and 1.53 R@1 without lf. |
Sound of Story: Multi-modal Storytelling with Audio (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on storytelling with sound have focused on visuals and sounds, but little attention has been given to sound. |
| Approach: | They propose to establish a new component called background sound which is story context-based audio without any linguistic information. |
| Outcome: | The proposed dataset is the largest well-curated dataset for storytelling with sound . it contains 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio . |
Synthesize, if you do not have: Effective Synthetic Dataset Creation Strategies for Self-Supervised Opinion Summarization in E-commerce (2023.findings-emnlp)
Copied to clipboard
Tejpalsingh Siledar, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera, Pushpak Bhattacharyya
| Challenge: | Existing approaches to generate general and aspect-specific opinion summarization are limited due to their reliance on human-specified aspects and seed words. |
| Approach: | They propose synthetic dataset creation approaches for general and aspect-specific opinion summarization . general opinion summaries struggle to generate faithful to the input reviews, they say . aspect- specific opinion summarisation models are limited due to reliance on human-specified aspects . |
| Outcome: | The proposed approach outperforms existing models on three e-commerce test sets on general and aspect-specific opinion summarization. |
Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing rumor detection models neglect the semantic coherence between text and image components in multimodal posts . Existing models neglect incomplete modalities in single modal posts, such as missing text or images . |
| Approach: | They propose a framework for incomplete modality rumor detection that captures semantic consistency between text and image pairs while enhancing model generalization to incomplete modalities within individual posts. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two English and two Chinese benchmark datasets for rumor detection in social media. |
Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs (2023.findings-emnlp)
Copied to clipboard
Chenyang Yang, Rishabh Rustogi, Rachel Brower-Sinning, Grace Lewis, Christian Kaestner, Tongshuang Wu
| Challenge: | Identifying what to test is a step that is largely ignored and poorly supported. |
| Approach: | They propose an interactive tool that supports requirements elicitation for guiding model testing. |
| Outcome: | The proposed tool can help practitioners test models in real-world settings . |
CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot commonsense question answering use incomplete CSKBs . lack of human annotations makes sampled negative examples potentially uninformative and contradictory. |
| Approach: | They propose a framework that abstracts a commonsense knowledge triple to many higher-level instances, which increases the coverage of the CSKB and expands the ground-truth answer space. |
| Outcome: | Experiments show that CAR can generalize to zero-shot commonsense scenarios . lack of human annotations makes sampled negative examples potentially uninformative and contradictory. |
kNN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on text-discriminating properties of semi-parametric models have not been done on non-parameter models. |
| Approach: | They propose an inference-phase approach that incorporates a neighborhood search into a model to enhance the capacity of a pre-trained parametric text classifier. |
| Outcome: | The proposed model improves performance on eight SuperGLUE tasks, three adversarial natural language inference datasets, 11 question-answering (QA) datasets and two sentiment classification datasets. |
Cross-modality Data Augmentation for End-to-End Sign Language Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts without intermediate representations. |
| Approach: | They propose a cross-modality data-augmented framework to transfer gloss-to-text translation capabilities to end-to end sign language translation. |
| Outcome: | The proposed framework outperforms baseline models on two widely used SLT datasets. |
Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end approaches to speech translation suffer from data scarcity compared to machine translation (MT). |
| Approach: | They propose a method which combines knowledge distillation and consistency learning to break the dilemma of learning-forgetting. |
| Outcome: | The proposed method outperforms the previous methods on a MuST-C dataset even without additional data. |
Relation-Aware Question Answering for Heterogeneous Knowledge Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing retrieval-based approaches to solve multihop Knowledge Base Question Answering (KBQA) fail to utilize information from head-tail entities and the semantic connection between relations to enhance the information capturing of relations in KGs. |
| Approach: | They propose to use a dual relation graph to find the answer entity in a knowledge graph . they use primal entity graph reasoning, dual relation grafitment and interaction . |
| Outcome: | The proposed approach achieves significant performance gain over the prior state-of-the-art on two public datasets, WebQSP and CWQ. |
InstOptima: Evolutionary Multi-objective Instruction Optimization via Large Language Model-based Instruction Operators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on automating instruction generation but do not consider other objectives that impact instruction quality. |
| Approach: | They propose an approach that treats instruction generation as an evolutionary multi-objective optimization problem. |
| Outcome: | The proposed approach improves fine-tuning performance and the generation of high-quality instructions. |
Less than One-shot: Named Entity Recognition via Extremely Weak Supervision (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition (NER) problem is performed under extremely weak supervision . XWS setting is considered weaker than 1-shot since example entity is given in context-free way . |
| Approach: | They propose a method that uses extremely weak supervision to train named entity recognition models. |
| Outcome: | The proposed method outperforms the state-of-the-art few-shot methods with 1-shot supervision and ChatGPT annotations significantly. |
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance. |
| Approach: | They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands. |
| Outcome: | Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model. |
Semantic Decomposition of Question and SQL for Text-to-SQL Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-SQL models for complex queries are limited by the syntactic complexity of SQL. |
| Approach: | They propose a question decomposition language that decomposes SQL queries into simple and regular sub-queries. |
| Outcome: | The proposed language decomposes SQL queries into simple and regular sub-queries . it is more accessible to non-experts for complex queries, enabling interpretable output . |
Time-Aware Language Modeling for Historical Text Dating (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to automatic text dating ignore diachronic change of words, which may affect the efforts of text modeling. |
| Approach: | They propose a time-aware language model to learn temporal word representations by transferring language models of general domains to those of time-specific ones and build a hierarchical modeling approach to represent diachronic documents by encoding them with temporal representations. |
| Outcome: | The proposed model outperforms state-of-the-art approaches in historical text dating and other NLP tasks. |
A Read-and-Select Framework for Zero-shot Entity Linking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on the candidate retrieval stage and ignore the essential candidate ranking stage, which disambiguates among entities and makes the final linking prediction. |
| Approach: | They propose a read-and-select framework that models the main components of entity disambiguation . they use mention context to output mention-aware entity representations . |
| Outcome: | The proposed framework achieves state-of-the-art performance on established zero-shot entity linking dataset ZESHEL with 2.55% micro-average accuracy gain, with no need for laborious multi-phase pre-training used in most of the previous work. |
Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to identifying ambiguous questions as part of a conversation have not addressed this challenge. |
| Approach: | They propose a multi-task learning approach that uses a text generation model for question rewriting and classification. |
| Outcome: | The proposed approach outperforms single-task learning baselines on three LIF test sets. |
DepNeCTI: Dependency-based Nested Compound Type Identification for Sanskrit (2023.findings-emnlp)
Copied to clipboard
Jivnesh Sandhan, Yaswanth Narsupalli, Sreevatsa Muppirala, Sriram Krishnan, Pavankumar Satuluri, Amba Kulkarni, Pawan Goyal
| Challenge: | Multi-component compounding is a prevalent phenomenon in Sanskrit, and understanding the implicit structure of a compound is crucial for deciphering its meaning. |
| Approach: | They propose a task to identify nested spans of a multi-component compound and decode the implicit semantic relations between them. |
| Outcome: | The proposed framework surpasses the best baseline framework with an average improvement of 13.1 points in terms of Labeled Span Score and 5-fold enhancement in inference efficiency. |
HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. |
| Approach: | They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem. |
| Outcome: | The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. |
HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Authorship Analysis is an essential aspect of Natural Language Processing (NLP) for a long time. |
| Approach: | They propose to use 17 human speech datasets and 3 LLMs to create a benchmark for spoken texts. |
| Outcome: | The proposed benchmark encompasses 17 human datasets and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. |
Data Augmentation for Code Translation with Comparable Corpora and Multiple References (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for translating code between programming languages are limited by parallel training data. |
| Approach: | They propose a data augmentation technique that builds comparable corpora and augments existing parallel data with multiple reference translations. |
| Outcome: | The proposed techniques improve CodeT5 translation between Java, Python, and C++ by an average of 7.5% Computational Accuracy (CA@1) . |
Multilingual Generation and Answering of Questions from Texts and Knowledge Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for QG-QA are limited to English, but can be used in other languages. |
| Approach: | They propose to bring multilinguality to multimodal QG-QA by using Brazilian Portuguese and Russian data. |
| Outcome: | The proposed approach outperforms a baseline on English and can handle both languages. |
InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text diffusion models have failed to capture the difference between the “easy-first” process and the “keyword-first" process of humans. |
| Approach: | They propose a non-autoregressive text diffusion model that incorporates a "keyinfo-first" generation strategy and a noise schedule based on the amount of text information. |
| Outcome: | The proposed model outperforms the baseline model in terms of generation quality and diversity, and higher sampling efficiency. |
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scale pre-trained language models to a deeper model depth share all parameters or use extra blocks. |
| Approach: | They propose a parameter-efficient approach to scaling pre-trained language models to a deeper model depth using matrix product operator. |
| Outcome: | The proposed model scales pre-trained language models to a deeper model depth by 4x and achieves 0.1 points higher than BERT-large for GLUE score. |
Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent prompt-based fine-tuning techniques have garnered considerable interest as a core technique for few-shot text classification tasks. |
| Approach: | They propose a prompt-based fine-tuning approach that reformulates the fine-uning objective to align with the Masked Language Modeling objective. |
| Outcome: | The proposed method has shown superior performance on five multi-class classification datasets. |
On the Impact of Cross-Domain Data on German Language Models (2023.findings-emnlp)
Copied to clipboard
Amin Dada, Aokun Chen, Cheng Peng, Kaleb Smith, Ahmad Idrissi-Yaghir, Constantin Seibold, Jianning Li, Lars Heiliger, Christoph Friedrich, Daniel Truhn, Jan Egger, Jiang Bian, Jens Kleesiek, Yonghui Wu
| Challenge: | Traditionally, large language models have been trained on general web crawls or domain-specific data. |
| Approach: | They present a German dataset and a dataset aimed at containing high-quality data to examine the importance of data diversity over quality. |
| Outcome: | The proposed model outperforms models trained on quality data on multiple downstream tasks. |
Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text normalization is a range of tasks that consist in replacing non-standard spellings with their standard equivalents. |
| Approach: | They introduce dialect-to-standard normalization as a sentence-level character transduction task and provide a large-scale analysis of these methods. |
| Outcome: | The proposed model performs best for Finnish, Swiss German and Slovene while the pre-trained model using full sentences performs the best for Norwegian. |
Re-Examining Summarization Evaluation across Multiple Quality Criteria (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a number of automated evaluation metrics are evaluated by multiple quality criteria, such as relevance, consistency, fluency and coherence. |
| Approach: | They propose a method that removes the confounding variable and detects unreliable correlations. |
| Outcome: | The proposed method detects unreliable correlations between QCs and human scores . it is based on a multi-QC setup, but it fails to detect summary corruptions . |
A Parallel Corpus for Vietnamese Central-Northern Dialect Text Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Among these, the northern dialect is often treated as the standard i.e. the defacto text style of the language. |
| Approach: | They propose a parallel corpus for Vietnamese central-northern dialect text transfer to facilitate research on this domain. |
| Outcome: | The proposed model improves existing models on the central dialect domain with dedicated results in translation and text-image retrieval tasks. |
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Various few-shot tool-usage strategies have been proposed to overcome LMs' shortcomings. |
| Approach: | They propose to augment language models with tools to overcome their shortcomings . they find strong no-tool baselines are competitive to tool-assisted strategies . |
| Outcome: | The proposed strategies outperform those that refine incorrect outputs with tools in knowledge-retrieval tasks, the study finds . the findings suggest few-shot tool integration is still an open challenge . |
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized the way summarization is generated. |
| Approach: | They propose a summarization model derived from GPT-3.5 through distillation that is compact and has comparable summarizing capabilities to GPT-3. |
| Outcome: | The proposed model outperforms the established best small models in prefix-tuning and full-data fine-tuned scenarios. |
Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Xu et al., 2018: chatbots generate offensive utterances, which must be avoided . he proposes a solution that can learn from user interactions in a way that is robust to trolls . |
| Approach: | They propose a method to learn from user feedback in a way that is robust to trolls . they propose multiple users rate each utterance, then perform latent class analysis to infer correct labels. |
| Outcome: | The proposed solution can infer training labels with high accuracy when trolls are consistent, even when a majority are trolled. |
Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have attracted a lot of attention in the legal domain due to their ability to tackle a variety of legal tasks. |
| Approach: | They constructed a corpus consisting of two legal scenarios using the IRAC method and used it to perform analysis on the corpus. |
| Outcome: | The proposed model can analyze a contract act in Malaysia and the Australian Social Act for Dependent Child using the IRAC method. |
Coverage-based Example Selection for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a training-free paradigm of fewshot inference that can generalize to novel tasks by conditioning on a few task examples. |
| Approach: | They show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects of the test input. |
| Outcome: | The proposed model outperforms methods that leverage task or LLM-specific training on compositional tasks. |
Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have implicitly transfer knowledge across languages, but not all languages have such generalization capabilities. |
| Approach: | They propose a meta-learning-based method to learn to align conceptual spaces of different languages to enhance cross-lingual generalization. |
| Outcome: | The proposed method achieves competitive results with state-of-the-art methods and narrows the performance gap between languages. |
Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Dual use is a problem in the context of natural language processing, says aaron eliotta . eelisa et al.: it is important to examine their rightful use and potential misuse . |
| Approach: | They propose a definition and checklist for dual-use in natural language processing based on a survey of NLP researchers and practitioners. |
| Outcome: | The proposed checklist focuses on dual-use in NLP based on a survey of NLP researchers and practitioners. |
BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to text classification require large annotated corpora to train or long context to fit many examples. |
| Approach: | They propose a method to few-shot text classification using an LLM. |
| Outcome: | The proposed approach yields high accuracy classifiers within 79% of the performance of models trained with larger datasets while using only 1% of their training sets. |
Approximating CKY with Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | CKY algorithm is a cubic dependence on sentence length, but transformers can be used to approximate it. |
| Approach: | They propose a transformer-based approach that approximates the CKY algorithm by directly predicting a sentence's parse and avoiding its cubic dependence on sentence length. |
| Outcome: | The proposed approach achieves better performance than comparable parsers that make use of CKY, while being faster. |
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2023.findings-emnlp)
Copied to clipboard
Prakhar Gupta, Yang Liu, Di Jin, Behnam Hedayatnia, Spandana Gella, Sijia Liu, Patrick Lange, Julia Hirschberg, Dilek Hakkani-Tur
| Challenge: | Dialogue models are able to generate fluent and interesting responses, but they can be difficult to control and may produce non-engaging, unsafe results. |
| Approach: | They propose a framework for controlling dialogue model behavior using natural language rules, or guidelines, which provide information about the context they are applicable to and what should be included in the response. |
| Outcome: | The proposed framework is effective in three open-domain dialogue response generation tasks and is consistent with the developer's expectations and intent. |
RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)
Copied to clipboard
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
| Challenge: | recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability. |
| Approach: | They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. |
| Outcome: | The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models. |
Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing AV techniques, including stylometric and deep learning, face limitations in terms of data requirements and lack of explainability. |
| Approach: | They propose a technique that leverages Large-Language Models (LLMs) to provide step-by-step stylometric explanation prompts to verify authorship. |
| Outcome: | The proposed technique outperforms state-of-the-art baselines, operates effectively with limited training data, and enhances interpretability through intuitive explanations. |
Transitioning Representations between Languages for Cross-lingual Event Detection via Langevin Dynamics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for event detection (ED) are limited to a small set of popular languages due to the high cost of data annotation. |
| Approach: | They propose a method to develop cross-lingual transfer learning models in high-resource source languages . they aim to transition the representations for target-language examples into the source-language space . |
| Outcome: | Extensive experiments over three languages demonstrate the state-of-the-art in CLTL. |
VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in interpretability suggests we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary space, a transformation that makes them more human interpretable. |
| Approach: | They propose a tool to visualize a forward pass of Generative Pre-trained Transformers as an interactive flow graph with nodes representing neurons or hidden states and edges representing interactions between them. |
| Outcome: | The proposed visualization simplifies huge amounts of data into easy-to-read graphs that can reflect the models’ internal processing, uncovering the contribution of each component to the models' final prediction. |
Is Robustness Transferable across Languages in Multilingual Neural Machine Translation? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on bilingual machine translation with a single translation direction. |
| Approach: | They propose a robustness transfer analysis protocol to analyze the transferability of robustness across different languages in multilingual neural machine translation. |
| Outcome: | The proposed protocol shows that the robustness gained in one translation direction can transfer to other translation directions. |
Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM (2023.findings-emnlp)
Copied to clipboard
Sahal Mullappilly, Abdelrahman Shaker, Omkar Thawakar, Hisham Cholakkal, Rao Anwer, Salman Khan, Fahad Khan
| Challenge: | Recent large language models like ChatGPT and Bard excel in a wide variety of NLP tasks but are not specifically tailored for climate related domain specific information. |
| Approach: | They propose a lightweight Arabic Mini-ClimateGPT that is built on an open-source LLM and specifically fine-tuned on a conversational-style instruction tuning curated Arabic dataset Clima500-Instruct. |
| Outcome: | The proposed model surpasses the baseline LLM in 88.3% of cases during ChatGPT-based evaluation and human expert prefers it over other open-source models. |
Interpreting Answers to Yes-No Questions in User-Generated Content (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on yes-no questions outside social media have found that yes and no keywords are rare in answers. |
| Approach: | They propose a corpus of 4,442 yes-no question-answer pairs from twitter . they find that yes and no keywords are rare in answers and poor indicators of correct interpretation . |
| Outcome: | The proposed corpus of 4,442 yes-no question-answer pairs shows that large language models are far from solving the problem. |
Task-Aware Self-Supervised Framework for Dialogue Discourse Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing discourse parsing approaches are constrained by predefined relation types, which can impede the adaptability of the parser for downstream tasks. |
| Approach: | They propose to introduce a task-aware paradigm to improve the versatility of the parser. |
| Outcome: | Empirical studies on dialogue discourse parsing datasets and a downstream task demonstrate the proposed framework. |
Selective Demonstrations for Cross-domain Text-to-SQL (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models with in-context learning have demonstrated impressive generalization capabilities in the cross-domain text-to-SQL task without the use of in-domain annotations. |
| Approach: | They propose a demonstration selection framework that utilizes both out-of-domain examples and synthetically generated in-domain demonstration examples to construct demonstrations. |
| Outcome: | The proposed framework outperforms baseline methods on two cross-domain text-to-SQL datasets with improvements of 1.1 and 11.8 points in execution accuracy. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that conversational Large Language Models (LLMs) can perform ill-defined complex tasks with different prompt types/styles and different degrees of detail. |
| Approach: | They propose a general taxonomy that can be used to design prompts with specific properties to perform a wide range of complex tasks. |
| Outcome: | The proposed taxonomy will allow future benchmarking studies to report specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. |
IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Intent-related tasks are typically modeled as separate tasks, but a unified approach is proposed . INTENDD uses an entirely unsupervised contrastive learning strategy for representation learning . |
| Approach: | They propose a unified approach to identifying intents from dialogue utterances . they propose an unsupervised contrastive learning strategy for representation learning . |
| Outcome: | The proposed approach outperforms baselines on three intent-related tasks on multiple datasets. |
INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for word-level autocompletion (WLAC) only use human typed sequences as prefixes in decoding module. |
| Approach: | They propose a novel iterative nonautoregressive instruct generation model for WLAC task . it uses human typed sequences and iterating decoding with subwords to fully utilize input information. |
| Outcome: | The proposed model is more competent in dealing with low-frequency words, and achieves state-of-the-art results on the WMT22 and benchmark datasets. |
Is the Answer in the Text? Challenging ChatGPT with Evidence Retrieval from Instructive Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative language models have recently shown remarkable success in generating answers to questions in a given textual context, but they suffer from hallucination, wrongly cite evidence, and spread misleading information. |
| Approach: | They propose a benchmark to evaluate an annotated WikiHow article and use it to retrieve answers to questions from trustworthy texts. |
| Outcome: | The proposed model can retrieve answers to lexically varied and open-ended questions from trustworthy instructive texts. |
PaRaDe: Passage Ranking using Demonstrations with LLMs (2023.findings-emnlp)
Copied to clipboard
Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, Kai Hui
| Challenge: | Existing studies show that large language models can be instructed to perform zero-shot passage re-ranking . Existing work like UPR demonstrate promising results for zero- shot ranking using LLMs . |
| Approach: | They propose a demonstration selection strategy based on difficulty rather than semantic similarity . they propose to include only one demonstration in the prompt to improve re-ranking . |
| Outcome: | The proposed method improves LLM-based re-ranking by adding one demonstration to the prompt. |
Learning Dynamic Representations for Discourse Dependency Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models characterize transition states by examining a certain number of elementary discourse units (EDUs) Existing work neglects the arcs obtained from the transition history. |
| Approach: | They propose to employ GAT-based encoder to learn dynamic representations for sub-trees constructed in previous transition steps. |
| Outcome: | The proposed model retains access to parsed EDUs through the obtained arcs, especially when handling lengthy text spans with complex structures. |
K-HATERS: A Hate Speech Detection Corpus in Korean with Target-Specific Ratings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets on hate speech detection focus on overt forms of hate . however, a majority of these resources are English-centric, focusing on overtones of hate. |
| Approach: | They propose a new corpus for hate speech detection in Korean with target-specific offensiveness ratings that offer a three-point Likert scale. |
| Outcome: | The proposed corpus is the largest offensive language corpus in Korean and offers target-specific ratings on a three-point Likert scale. |
Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multilingual neural machine translation (MNMT) are limited in their ability to handle large amounts of data. |
| Approach: | They propose a framework which only requires target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. |
| Outcome: | The proposed framework is more effective than baselines in long-tail and high-resource languages. |
BotPercent: Estimating Bot Populations in Twitter Communities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to bot detection are agnostic to social environments the bots operate in . however, standard approaches are not a good fit for the social environments they operate in. |
| Approach: | They propose a method that estimates the percentage of Twitter bots given a community . they use Twitter bot detection datasets and feature-, text-, and graph-based models adjusted to a particular community based on Twitter . |
| Outcome: | The proposed method achieves state-of-the-art in community-level Twitter bot detection across balanced and imbalanced class distribution settings. |
The Locality and Symmetry of Positional Encodings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work uses fixed or learned positional encodings to infuse word order information into transformer-based language models. |
| Approach: | They conduct a systematic study of positional encodings in Bidirectional Masked Language Models. |
| Outcome: | The proposed model improves performance across a wide range of natural language understanding tasks. |
Towards a Deep Understanding of Multilingual End-to-End Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have witnessed the rapid development of end-to-end speech-totext translation (ST) which has demonstrated remarkable performance and outperformed conventional cascaded systems. |
| Approach: | They employ Singular Value Canonical Correlation Analysis to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. |
| Outcome: | The proposed approach outperforms existing cascaded systems in predicting phonetic features and improves translation quality. |
An Empirical Investigation of Implicit and Explicit Knowledge-Enhanced Methods for Ad Hoc Dataset Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for ad hoc dataset retrieval are lexical and cannot capture semantic similarity. |
| Approach: | They propose to implement and evaluate a set of implicit and explicit knowledge-enhancement retrieval methods on two test collections to find semantic matches for ad hoc dataset retrieval. |
| Outcome: | The proposed methods are compared with existing methods on two test collections and reveal the unique features of the task and suggest an interpolation of different kinds of methods as the current best practice. |
A Multi-Modal Multilingual Benchmark for Document Image Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing document image classification datasets have several limitations and we present two new datasets that overcome these limitations. |
| Approach: | They propose to use two newly curated multilingual datasets that overcome these limitations and propose to develop multilingual Document AI models. |
| Outcome: | The proposed datasets overcome limitations in document image classification and open the door for future research into improving Document AI models. |
Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
Investigating the Effectiveness of Multiple Expert Models Collaboration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using multi-domain MT, we compare the performance of a single model with a multi-expert model in a fair condition. |
| Approach: | They propose to combine a multi-domain machine translation model with a aggregation strategy to investigate their results. |
| Outcome: | The proposed approach outperforms the current multi-domain model and aggregation methods in a fair condition on multiple domain datasets. |
Gradually Excavating External Knowledge for Implicit Complex Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have gained attention for their human-comparable capabilities but they may not solve open-domain implicit questions due to out-of-date domain knowledge, one-shot generation and restricted comprehensiveness. |
| Approach: | They propose a gradual knowledge excavation framework for open-domain complex question answering using extrinsic knowledge and historical knowledge. |
| Outcome: | The proposed framework achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA in the 10B LLM class. |
Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on automatic prediction of cognitive appraisals has focused on physiological aspects of emotions. |
| Approach: | They present a dataset that assesses 24 appraisal dimensions across 241 Reddit posts . they find that open-source models fail to automatically assess and explain cognitive appraisals . |
| Outcome: | The proposed dataset assesses 24 appraisal dimensions across 241 reddit posts. |
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages (2023.findings-emnlp)
Copied to clipboard
Elena Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Zanzotto
| Challenge: | a recent study examined how models for typologically similar languages encode structural information. |
| Approach: | They propose to layer-wise compare transformers for typologically similar languages to observe similarities . they use a domain adaptation on semantically equivalent texts to measure similarity . |
| Outcome: | The proposed model outperforms all other models on unseen sentences . the proposed model is based on a typologically similar language . |
Discourse Sense Flows: Modelling the Rhetorical Style of Documents across Various Domains (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on shallow discourse parsing has given renewed attention to the role of discourse relation signals, in particular explicit connectives and alternative lexicalizations. |
| Approach: | They propose a model for extracting and classifying discourse relation signals from the Penn Discourse Treebank v3 corpus and introduce a new way of modeling rhetorical style by the linear order of coherence relations. |
| Outcome: | The proposed models are based on the Penn Discourse Treebank v3 corpus and employ n-gram patterns to predict genre/domain discrimination. |
HierarchicalContrast: A Coarse-to-Fine Contrastive Learning Framework for Cross-Domain Zero-Shot Slot Filling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing zero-shot slot filling methods show limited generalization ability in target domain . et al., 2018: empirical results show that proposed method is better than existing methods . |
| Approach: | They propose a hierarchical contrastive learning framework for zero-shot slot filling . they use Gaussian-distributed embedding to learn generalized deep semantic relations . |
| Outcome: | The proposed method performs better than existing methods on unseen slot types . empirical results show that the proposed method can generalize to unseense slot types. |
A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a range of recent LLMs on English creative writing have been evaluated . we examine the implications and limitations of our study and suggest directions for future research. |
| Approach: | They evaluate a range of recent LLMs on English creative writing . they ask LLM writers and humans to write a story using an open-ended scenario . |
| Outcome: | The results show that state-of-the-art commercial LLMs outperform writers in most dimensions . human creativity is retained, while humor shows a binary divide between LLM that can handle it . |
1-PAGER: One Pass Answer Generation and Evidence Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 1-Pager is the first system that answers a question and retrieves evidence using a single Transformer-based model and decoding process. |
| Approach: | They propose a system that partitions the corpus using constrained decoding to select a document and answer string, and a method that uses a single Transformer-based model to generate evidence. |
| Outcome: | The proposed system outperforms the equivalent ‘closed-book’ question answering model by grounding predictions in evidence corpus. |
Context-faithful Prompting for Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models encode parametric knowledge about world facts but overly rely on it can cause incorrect predictions in context-sensitive NLP tasks. |
| Approach: | They propose to use opinion-based prompts and counterfactual demonstrations to improve LLM faithfulness to contexts. |
| Outcome: | The proposed methods improve faithfulness to contexts using opinion-based prompts and counterfactual demonstrations. |
InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. |
| Approach: | They propose a replay-based continual text classification method that uses fast-slow and current-past contrastive learning to perform mutual information maximization and better recover previously learned representations. |
| Outcome: | The proposed method achieves state-of-the-art on three text classification tasks. |
Sparse Frame Grouping Network with Action Centered for Untrimmed Video Paragraph Captioning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for paragraph captioning videos without event ground truths generate one sentence for each event, but without event labels, it is difficult to locate the transitions between events and minimize repetition. |
| Approach: | They propose a module that dynamically groups event information with the help of action information for the entire video and excludes redundant frames within pre-defined clips. |
| Outcome: | The proposed module outperforms the state-of-the-art methods on all metrics. |
Unsupervised Binary Code Translation with Application to Code Clone Detection and Vulnerability Discovery (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that binary code analysis is a key problem in software security research. |
| Approach: | They propose to apply Neural Machine Translation to binary code analysis . they translate a binary in a low-resource ISA and train a model on the high-resourced ISA . |
| Outcome: | The proposed model can be used to analyze binary code across ISAs using a high-resource ISA. |
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Long document question answering requires locating relevant paragraphs within a document to answer a question. |
| Approach: | They propose to exploit the discourse structure commonly found in documents to create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. |
| Outcome: | The proposed approach retains 99.6% of the best zero-shot approach's performance while processing only 26% of tokens used by the best approach in the information seeking evidence retrieval setup. |
Emergent Inabilities? Inverse Scaling Over the Course of Pretraining (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has found that increased number of model parameters and increased size of the training dataset positively influence model performance. |
| Approach: | They investigate whether language models' performance on specific tasks can decrease over the course of training. |
| Outcome: | The proposed model size-based scaling is found on 8 tasks on which Pythia 12B shows decreased performance over the course of training. |
Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Continual learning and zero-shot learning approaches have not been adopted to scale to novel-emerging types. |
| Approach: | They propose a method to recognize entities in novel types by their textual names or descriptions. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on three challenging OVNER benchmarks by 9.7%, 9.5%, and 1.8% F1-score of novel types. |
Representation Projection Invariance Mitigates Representation Collapse (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning contextualized representations by pre-trained models can lead to representation degradation, which can result in instability, sub-optimal performance, and weak generalization. |
| Approach: | They propose a regularization method to maintain the information content of representations and reduce representation collapse during fine-tuning. |
| Outcome: | The proposed method outperforms baselines on most tasks and improves out-of-distribution performance. |
Tunable Soft Prompts are Messengers in Federated Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to protect model privacy in federated learning (FL) are limited. |
| Approach: | They propose a federated learning approach that provides model privacy protection via tunable soft prompts. |
| Outcome: | The proposed approach provides protection for the global model while reducing communication and computation costs. |
Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
Benjamin Yan, Ruochen Liu, David Kuo, Subathra Adithan, Eduardo Reis, Stephen Kwak, Vasantha Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, Michael Moor
| Challenge: | Existing methods for generating reports from medical images conflate the content of the report with its style, which can lead to inaccurate reports. |
| Approach: | They propose a two-step approach to generate radiology reports from medical images using large language models and a graph representation of reports. |
| Outcome: | The proposed approach improves the performance of human evaluations with clinical raters. |
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems exhibit decreased sensitivity to visual information when text inputs are complete. |
| Approach: | They propose to generate parallel VQA style pairs from source text to foster more robust cross-modal interaction. |
| Outcome: | The proposed approach generates parallel VQA style pairs from the source text, fostering more robust cross-modal interaction. |
GenKIE: Robust Generative Multimodal Document Key Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Key information extraction (KIE) is a key application for information retrieval and text mining. |
| Approach: | They propose a novel generative end-to-end model, named GenKIE, to address the KIE task. |
| Outcome: | The proposed model generalizes over different types of documents and achieves state-of-the-art results. |
Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. |
| Approach: | They propose a framework to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. |
| Outcome: | The proposed model improves discrimination and generalizability of the multimodal representation and overcomes biases in the fusion vector’s modality. |
Efficient Multilingual Language Model Compression through Vocabulary Trimming (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual language models (LMs) have become a powerful tool in NLP, especially for non-English languages. |
| Approach: | They propose a method to reduce a multilingual LM vocabulary to a target language by deleting potentially irrelevant tokens from its vocabulary. |
| Outcome: | The proposed method can retain the original performance of the multilingual LM while being considerably smaller in size than the original model. |
ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use multilingual captions for images have limited results due to the scarcity of training data. |
| Approach: | They propose a multilingual vision-and-language model that divides a V&L task into two stages . they propose IC, which takes the caption as the alt text and performs cross-lingual language understanding . |
| Outcome: | The proposed model can achieve state-of-the-art results for five languages and comparable results for the rest. |
GTA: Gated Toxicity Avoidance for LM Performance Preservation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Controllable Text Generation methods that generate toxic text can negatively impact the performance of the language model. |
| Approach: | They propose a gated Toxicity Avoidance method that can be applied to any CTG method and evaluate its effectiveness. |
| Outcome: | The proposed method reduces toxicity and preserves performance while preserving the language model's generation performance. |
LMGQS: A Large-scale Dataset for Query-focused Summarization (2023.findings-emnlp)
Copied to clipboard
Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter, Pengcheng He, Chenguang Zhu, Michael Zeng
| Challenge: | Lack of large-scale datasets for query-focused summarization hinders model development . lack of data limits the ability of QFS models to train robust neural models . |
| Approach: | They propose to generate a query for each summary sentence in a generic summarization annotation using a pretrained language model. |
| Outcome: | The proposed model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks. |
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have excellent performance in evaluation benchmarks, but struggle in complex reasoning tasks. |
| Approach: | They propose a tool-augmented chain-of-thought reasoning framework for chat-based LLMs . they model chain- of-thoughting reasoning as multi-turn conversations to utilize tools . |
| Outcome: | The proposed framework can outperform state-of-the-art models on complex reasoning tasks. |
Non-Autoregressive Document-Level Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive translation models struggle with document context and handling discourse phenomena. |
| Approach: | They propose a simple but effective design of sentence alignment between source and target to improve their performance on document-level machine translation. |
| Outcome: | The proposed model achieves high acceleration on documents and sentence alignment significantly enhances their performance. |
Exploring the Effectiveness of Multi-Lingual Commonsense Knowledge-Aware Open-Domain Dialogue Response Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that commonsense knowledge-aware models can improve informativeness while reducing the hallucination issue. |
| Approach: | They propose a task to use commonsense knowledge in other languages to enhance the current dialogue generation by using commonsensical knowledge in different languages. |
| Outcome: | The proposed model improves the current dialogue generation while reducing the hallucination issue. |
Mixture of Soft Prompts for Controllable Data Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate fluent text when the target output follows natural language patterns. |
| Approach: | They propose a method that uses large language models to generate fluent text from a limited ontology rather than direct prediction by using soft prompts. |
| Outcome: | The proposed method produces diverse and natural text while preserving label semantics. |
A Boundary Offset Prediction Network for Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental task in natural language processing . span-based methods assign entity types to text spans, resulting in imbalanced sample space . |
| Approach: | They propose a method that predicts boundary offsets between candidate and nearest spans . the method integrates entity type and span representations to generate type-aware boundary offset . |
| Outcome: | The proposed method outperforms existing methods on eight widely-used NER datasets. |
Prefix-Tuning Based Unsupervised Text Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised text style transfer is an important task in computer vision and natural language processing. |
| Approach: | They propose a method that uses pre-trained large language models to train a generative model that can alter the style of the input sentence without using any parallel data. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on well-known datasets. |
Evaluating and Enhancing the Robustness of Code Pre-trained Models through Structure-Aware Adversarial Samples Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained code models have made significant strides in the field of neural code intelligence, but they are susceptible to adversarial attacks that subtly modify the input sequence and can impair generalization. |
| Approach: | They propose a set of novel robustness evaluation methods based on the intrinsic structure of the code to explore the impact of imperceptible perturbation. |
| Outcome: | The proposed methods have demonstrated their effectiveness across a wide range of models and tasks, and are able to predict the performance of perturbed models. |
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using an annotation instrument, the design of the annotation instrument and the instructions given to annotators can impact training data. |
| Approach: | They investigate the impact of an annotation instrument on training data . they collect hate speech and offensive language annotations in a tweet corpus . |
| Outcome: | The proposed model performs better on holdout conditions than on the standard model. |
Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Qualitative coding is a content analysis method that assigns descriptive labels or qualitative codes to passages. |
| Approach: | They propose a qualitative code suggestion task where a ranked list of previously assigned qualitative codes is suggested from an identified passage. |
| Outcome: | The proposed method integrates previously ignored properties such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. |
D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. |
| Approach: | They propose a multimodal multimodal summarization task that aims to generate summaries in any language with document inputs in any languages and the corresponding image sequence. |
| Outcome: | The proposed task can generate summaries in any language with document inputs in any languages and the corresponding image sequence. |
Improving Input-label Mapping with Demonstration Replay for In-context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is an emerging capability of large autoregressive language models where a few demonstrations are appended to the input to enhance the model’s understanding of downstream NLP tasks without directly adjusting the model parameters. |
| Approach: | They propose a method where a few demonstrations are appended to the input to enhance the model's understanding of downstream NLP tasks without directly adjusting the model parameters. |
| Outcome: | The proposed method significantly improves the input-label mapping in ICL demonstrations. |
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)
Copied to clipboard
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev
| Challenge: | In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory. |
| Approach: | They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs. |
| Outcome: | The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset. |
Cross-lingual Open-Retrieval Question Answering for African Languages (2023.findings-emnlp)
Copied to clipboard
Odunayo Ogundepo, Tajuddeen Gwadabe, Clara Rivera, Jonathan Clark, Sebastian Ruder, David Adelani, Bonaventure Dossou, Abdou Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Kahira, Shamsuddeen Muhammad, Akintunde Oladipo, Abraham Owodunni, Atnafu Tonja, Iyanuoluwa Shode, Akari Asai, Anuoluwapo Aremu, Ayodele Awokoya, Bernard Opoku, Chiamaka Chukwuneke, Christine Mwase, Clemencia Siro, Stephen Arthur, Tunde Ajayi, Verrah Otiende, Andre Rubungo, Boyd Sinkala, Daniel Ajisafe, Emeka Onwuegbuzia, Falalu Lawan, Ibrahim Ahmad, Jesujoba Alabi, Chinedu Mbonu, Mofetoluwa Adeyemi, Mofya Phiri, Orevaoghene Ahia, Ruqayya Iro, Sonia Adhiambo
| Challenge: | Our Dataset is the first cross-lingual QA dataset with a focus on African languages. |
| Approach: | They propose to use African languages as the only high-coverage source of answer content for cross-lingual open-retrieval question answering systems. |
| Outcome: | Our Dataset includes 12,000+ XOR QA examples across 10 African languages. |
Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that translation quality alone is not sufficient for measuring knowledge transfer in multilingual neural machine translation. |
| Approach: | They propose a method that measures representational similarities between languages to measure knowledge transfer. |
| Outcome: | The proposed method improves translation quality for low- and mid-resource languages across multiple datasets and models. |
Aligning Predictive Uncertainty with Clarification Questions in Grounded Dialog (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Previous work bases the timing of questions on supervised models learned from interactions between humans. |
| Approach: | They propose to ground the need for questions in the acting agent's predictive uncertainty by using the T5 encoder-decoder architecture to solve a Minecraft Collaborative Building task. |
| Outcome: | The proposed model can detect ambiguous instructions and predict responses better than previous models. |
Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. |
| Approach: | They propose a framework that allows reducing calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. |
| Outcome: | The proposed framework reduces calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. |
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback (2023.findings-emnlp)
Copied to clipboard
Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, Zhaopeng Tu
| Challenge: | Large language models (LLMs) like ChatGPT are only accessible through restricted APIs, which creates barriers to new research and advancements in the field. |
| Approach: | They propose a framework to enhance and regulate the translation abilities during chat . they reformulate translation data into the instruction-following style and introduce a "Hint" field . |
| Outcome: | The proposed framework enhances and regulates the translation abilities during chat . it reformulates translation data into the instruction-following style and introduces a "Hint" field . |
Dense Retrieval as Indirect Supervision for Large-space Decision Making (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Dense Decision Retrieval (DDR) is a learning-to-retrieve task for discriminative natural language understanding (NLU) tasks with large label spaces. |
| Approach: | They propose a novel approach to learning large-space discriminative NLU tasks as a learning-to-retrieve task by adopting a dual-encoder architecture that learns to predict by retrieving from a decision thesaurus. |
| Outcome: | The proposed approach outperforms baselines greatly on multi-label classification tasks, 1.17% in F1 score ultra-fine entity typing, and 1.26% in accuracy on three few-shot intent classification tasks on average. |
One-Model-Connects-All: A Unified Graph Pre-Training Model for Online Community Modeling (2023.findings-emnlp)
Copied to clipboard
Ruoxue Ma, Jiarong Xu, Xinnong Zhang, Haozhe Zhang, Zuyu Zhao, Qi Zhang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Existing studies on integrating online community to solve social problems have not fully utilized these three components and the relationship among them. |
| Approach: | They propose a framework that simultaneously considers communities, users, and texts and can easily connect with a variety of downstream tasks related to social media. |
| Outcome: | The proposed model can be used to perform violation detection, sentiment analysis, and community recommendation across multiple tasks. |
In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-Image Machine Translation (IIMT) aims to convert images containing texts from one language to another. |
| Approach: | They propose an end-to-end model instead of the traditional cascade methods which use optical character recognition followed by neural machine translation and text rendering. |
| Outcome: | The proposed model outperforms both cascade methods and current model in translation quality and robustness across various dimensions. |
NarrativeXL: a Large-scale Dataset for Long-Term Memory Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 990,595 questions are needed to solve ultra-long-context reading comprehension problems. |
| Approach: | They propose a large-scale reading comprehension dataset using 1,500 hand-curated fiction books and a set of reading comprehension questions based on these summaries. |
| Outcome: | The proposed reading comprehension dataset is larger than the closest alternatives and has more questions than the existing models. |
Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Backchanneling is a form of feedback that is produced by listeners in a conversation . since the advent of ChatGPT, modern dialogue systems exhibit answer quality levels on par with humans in various professions. |
| Approach: | They propose a multi-task learning approach that learns textual representations for the task of backchannel prediction in tandem with dialogue act classification. |
| Outcome: | The proposed approach improves the prediction of specific backchannels by up to 2.0% in F1 . the audio encoder is pre-trained in a self-supervised fashion using voice activity projection . |
mReFinED: An Efficient End-to-End Multilingual Entity Linking System (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work assumed that entity mentions were given and skipped the entity mention detection step due to a lack of high-quality multilingual training corpora. |
| Approach: | They propose a bootstrapping mention detection framework that enhances the quality of training corpora. |
| Outcome: | The proposed framework outperforms existing work in the end-to-end MEL task while being 44 times faster. |
Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Continual Learning (CL) have limited KT and catastrophic forgetting . a new method overcomes CF by isolating the knowledge of each task . |
| Approach: | They propose a method to overcome catastrophic forgetting and encourage knowledge transfer . they propose to discover a sub-network for each task and a soft-masking mechanism to preserve the previous knowledge. |
| Outcome: | The proposed method outperforms baselines in classification, generation, information extraction and their mixture. |
PIVOINE: Instruction Tuning for Open-world Entity Profiling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for information extraction focus on a closed-world setting, but PIVOINE is a promising solution to tackle the open-world problem of entity profiling. |
| Approach: | They propose to develop an LLM that performs Open-world Entity Profiling with instruction tuning to extract desirable entity profiles . they construct INSTRUCTOPENWIKI, a substantial instruction-tuning dataset for Open-World Entity Profiles . |
| Outcome: | The proposed model outperforms existing methods and ChatGPT-based baselines on unseen and out-of-ontology cases. |
DiQAD: A Benchmark Dataset for Open-domain Dialogue Quality Assessment (2023.findings-emnlp)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Existing studies on dialogue quality assessment are uncapable of providing an end-to-end and human-epistemic assessment dataset . open-domain dialogue assessment is complicated and costly, but it can be done by recruiting human evaluators. |
| Approach: | They propose a large-scale dialogue quality assessment dataset for automatically assessing open-domain dialogue quality. |
| Outcome: | The proposed dataset is openly accessible at https://github.com/yukunZhao/Dialogue_quality_evaluation. |
Tuna: Instruction Tuning using Feedback from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | LLms like LLaMA have shown to be cost-effective for generating better responses . however, the instruction-tuned model has only seen one response per instruction . |
| Approach: | They propose to fine tune an instruction-tuned LLM using probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. |
| Outcome: | The proposed model improves on Super Natural Instructions, LMentry and Vicuna QA. |
Emptying the Ocean with a Spoon: Should We Edit Models? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has questioned the use of direct model editing for factual corrections in LLMs. aaron s. de stefano, a sociologist, says that model editing is not a systematic remedy for factuality. |
| Approach: | They argue that direct model editing cannot be trusted as a remedy for LLM disadvantages . authors call for cautious promotion and application of model editing as part of LLM deployment process . |
| Outcome: | The proposed method is not trusted as a remedy for the disadvantages inherent to LLMs, the authors argue . they argue that it opens risks by reinforcing the notion that models can be trusted for factuality . |
A Causal View of Entity Bias in (Large) Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Entity bias affects pretrained (large) language models, causing them to rely on (biased) parametric knowledge to make unfaithful predictions. |
| Approach: | They propose a structured causal model whose parameters are easier to estimate . they propose to perturb the original entity with neighboring entities . |
| Outcome: | The proposed model reduces biasing information pertaining to the original entity while still preserving sufficient semantic information from similar entities. |
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating text quality are discriminative and generative . current methods use manual annotation of human judgements to train them . |
| Approach: | They propose a framework that combines the best of both worlds by using supervised and unsupervised signals from whatever data we have available. |
| Outcome: | The proposed method outperforms existing metrics on 5 datasets, 19 languages and 280 systems. |
T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotation projection is a task of transporting labels from source to target language without high-quality training data. |
| Approach: | They propose an annotation projection approach that leverages pretrained text2text models and machine translation technology to generate annotated data. |
| Outcome: | The proposed approach outperforms existing methods on intrinsic and extrinsic tasks in 5 Indo-European and 8 low-resource African languages. |
MTGER: Multi-view Temporal Graph Enhanced Temporal Reasoning over Time-Involved Document (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work models time implicitly, making it difficult to handle complex relationships . a novel temporal reasoning framework explicitly models the temporal relationships among facts by multi-view temporal graphs . |
| Approach: | They propose a multi-view temporal graph-based temporal reasoning framework that explicitly models the temporal relationships among facts by multi-visit temporal charts. |
| Outcome: | The proposed framework gives more consistent answers under question perturbations. |
MSCFFN: A New FFN with Multi-Space Cross to Accelerate Transformer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models to reduce computation complexity are limited in some areas . a new structure to reduce the computation complexity is proposed to accelerate Transformers . |
| Approach: | They propose a new feed forward network structure which splits matrix space to smaller space to reduce computation complexity. |
| Outcome: | The proposed model can achieve a faster speed and better accuracy on the long-range arena benchmark. |
Dialect Transfer for Swiss German Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study of Swiss German speech translation systems focuses on dialect diversity and differences between Swiss German and Standard German. |
| Approach: | They focus on the impact of dialect diversity and differences between Swiss German and Standard German . they first review the Swiss German dialect landscape and the differences to Standard German. |
| Outcome: | The proposed model is based on the Swiss German dialect landscape and differences to Standard German. |
Masked Path Modeling for Vision-and-Language Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | A major challenge in vision-and-language navigation is the limited available training data, which hinders the models’ ability to generalize effectively. |
| Approach: | They propose a masked path modeling objective that pretrains an agent using self-collected data for subsequent navigation tasks. |
| Outcome: | The proposed model pretrains an agent using self-collected data for subsequent navigation tasks eliminating the need for external tools. |
Learning Interpretable Style Embeddings via Prompting LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has treated the style of a text as separable from the content. |
| Approach: | They use prompting to perform stylometry on a large number of texts to generate a synthetic stylometric dataset. |
| Outcome: | The proposed model trains human-interpretable representations on a large stylometric dataset and a linguistic model for style representation learning. |
Exploring Context-Aware Evaluation Metrics for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on machine translation evaluation focused on quality of individual sentences, while neglecting the importance of contextual information. |
| Approach: | They propose a context-aware machine translation evaluation metric called Cont-COMET . they use the COMET framework to consider the preceding and subsequent contexts of the sentence . |
| Outcome: | The proposed metric improves system-level and segment-level evaluations on the official WMT framework. |
GRACE: Discriminator-Guided Chain-of-Thought Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) can assign a high likelihood to incorrect steps . Existing models (LLMs), however, struggle with complex multi-step reasoning. |
| Approach: | They propose a stepwise decoding approach that steers the decoding process towards producing correct reasoning steps. |
| Outcome: | The proposed approach outperforms existing methods on math and symbolic reasoning tasks. |
QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to QA fine-tune language models on QA pairs constructed from CommonSense Knowledge Bases (CSKBs) however, current QA synthesis protocols introduce noise from the CSKB and generate ungrammatical questions and false negative options, which impede the model’s ability to generalize. |
| Approach: | They propose a framework to analyze the training dynamics of each QA pair at both the question level and option level, discarding machine-detectable artifacts and mislabeled or false-negative options. |
| Outcome: | The proposed framework outperforms baseline approaches while using only 33% of the synthetic data. |
RexUIE: A Recursive Method with Explicit Schema Instructor for Universal Information Extraction (2023.findings-emnlp)
Copied to clipboard
Chengyuan Liu, Fubang Zhao, Yangyang Kang, Jingyuan Zhang, Xiang Zhou, Changlong Sun, Kun Kuang, Fei Wu
| Challenge: | Named Entity Recognition (NER) and Relation Extraction (RE) models have limited success when extracting general schemas such as quadruples and quintuples. |
| Approach: | They propose a formal formulation that covers almost all extraction schemas and a Recursive Method with Explicit Schema Instructor for UIE. |
| Outcome: | The proposed method shows strong performance under full-shot and few-shot settings and achieves state-of-the-art results on the tasks of extracting complex schemas. |
PromptARA: Improving Deep Representation in Hybrid Automatic Readability Assessment with Prompt and Orthogonal Projection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Readability assessment aims to automatically classify texts based on readers’ reading levels. |
| Approach: | They propose a hybrid automatic readability assessment model using prompts to improve deep feature representations and an orthogonal projection layer to fuse both deep and linguistic features. |
| Outcome: | The proposed model outperforms state-of-the-art models over four English and two Chinese corpora and demonstrates that it is more efficient than existing models. |
Does Listener Gaze in Face-to-Face Interaction Follow the Entropy Rate Constancy Principle: An Empirical Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that nonverbal behaviours are rich in communicative functions, such as gaze, head movements, and speech-accompanying manual gestures. |
| Approach: | They train a transformer-based neural sequence model to process gaze data extracted from video-recorded conversations and compute its information density. |
| Outcome: | The proposed model computes listeners’ gaze behaviour and the information density of speech using a pre-trained language model. |
Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that fine-grained entity typing is superior to text-based methods. |
| Approach: | They propose a task called fine-grained entity typing to classify entities . they propose combining textual and visual contexts to capture fine-granular semantic information . |
| Outcome: | The proposed approach achieves superior classification performance compared to previous text-based approaches. |
Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks have assessed language models' numerical abilities . limitations include tokenization and representation of numbers in text, hallucination, and a lack of numerical commonsense knowledge. |
| Approach: | They propose a hierarchical taxonomy for numerical reasoning skills that includes representation, number sense, manipulation, and complex reasoning. |
| Outcome: | The proposed model outperforms other models on the tabular Natural Language Inference task. |
Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks. |
| Approach: | They investigate the membership inference attack by using model's API to determine if a sample was part of the training data. |
| Outcome: | The proposed model is able to identify if a sample was part of the training data and exploits its similarity and resistance to document modifications as potential MI signals on widely used datasets. |
BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
Extrapolating Multilingual Understanding Models as Multilingual Generators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual understanding models are not capable of generating high-quality text compared with decoder-based causal language models. |
| Approach: | They propose a method to adapt a multilingual encoder to a language generator with a small number of additional parameters. |
| Outcome: | The proposed approach outperforms initialization-based methods with 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation. |
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on self-consistency check alone . prominent LMs exhibit a tendency to produce exceedingly confident, but erroneous, assertions . |
| Approach: | They propose a sampling-based method that expands on the principle of self-consistency checking to detect hallucinations at question-level and model-level. |
| Outcome: | The proposed method outperforms the state of the art in detecting non-factual and factual statements across multiple question-answering and open-domain generation benchmarks. |
Test-Time Self-Adaptive Small Language Models for Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent instruction-finetuned large language models (LMs) have shown notable performances in various tasks, such as question-answering. |
| Approach: | They propose to use unlabeled test data to transfer smaller language models with limited knowledge. |
| Outcome: | The proposed strategy shows significant performance improvements on benchmark QA datasets with higher robustness across diverse prompts, enabling LMs to stay stable. |
ExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown great power in solving various tasks but fail in many specific tasks. |
| Approach: | They propose a framework to help black-box LLMs better adapt to unfamiliar tasks by reflecting and noting experiences from training data and retrieving them from external memory during testing. |
| Outcome: | The proposed framework improves the performance of black-box Large Language Models on multiple tasks and demonstrates that it is a good choice for the future. |
Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. |
| Approach: | They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods. |
| Outcome: | The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance. |
Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 are popular retriever-augmented language models for a variety of tasks. |
| Approach: | They evaluate the strengths and weaknesses of kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 in reasoning over retrieved statements across different tasks. |
| Outcome: | The proposed models do not exhibit strong reasoning even when provided with only the required statements. |
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |
Closed Boundary Learning for Classification Tasks with the Universum Class (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods treat the Universum class equally with the classes of interest, leading to problems such as overfitting, misclassification, and diminished model robustness. |
| Approach: | They propose a closed boundary learning method that applies closed decision boundaries to classes of interest and designates the area outside all closed boundaries as the Universum class. |
| Outcome: | The proposed method improves accuracy and robustness of classification models on six state-of-the-art tasks. |
Revisiting Entropy Rate Constancy in Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence supports the uniform information density hypothesis . however, we re-evaluate the hypothesis with neural language models . |
| Approach: | They propose to use n-gram language models to argue that English documents exhibit entropy rate constancy . they re-evaluate the claims of Genzel and Charniak with neural language models . |
| Outcome: | The proposed hypothesis fails to support the proposed hypothesis with language models. |
Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | CASENT predicts ultra-fine entities mentioned in text into types with calibrated confidence scores. |
| Approach: | They propose a model that predicts ultra-fine entities with calibrated confidence scores for entity typing. |
| Outcome: | The proposed model outperforms existing models in terms of F1 score and calibration error while achieving 50 times faster inference speed. |
Learning Semantic Role Labeling from Compatible Label Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. |
| Approach: | They propose a framework that jointly models VerbNet and PropBank labels as one sequence and enforcing Semlink constraints during decoding improves the overall F1 . |
| Outcome: | The proposed model outperforms the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank). |
QUADRo: Dataset and Models for QUestion-Answer Database Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tools do not consider answers (question-question similarity only) or their quality in the annotation process. |
| Approach: | They propose an open-domain annotated resource that retrieves questions from pre-computed databases . they use a binary label associated with each question/answer pair to determine relevance . |
| Outcome: | The proposed model trains and evaluates models on 443,000 examples with respect to key aspects of QA systems. |
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are trained on vast unlabeled data, rich in world knowledge. |
| Approach: | They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases . |
| Outcome: | The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work. |
Is ChatGPT the ultimate Data Augmentation Algorithm? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has examined the use of ChatGPT for data augmentation, but only in limited contexts. |
| Approach: | They use ChatGPT to create new data with paraphrasing and zero-shot generation to compare it to seven other algorithms. |
| Outcome: | The proposed model performs exceptionally well on some simpler data, but it does not perform better than the other algorithms. |
Enhanced Simultaneous Machine Translation with Word-level Policies (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies assume that operations are carried out at the subword level . a novel policy dictates whether to READ or WRITE at each step of the translation process . |
| Approach: | They propose a method to boost SiMT models using language models to address subword disparity . they propose implementing a word-level policy that dictates whether to READ or WRITE . |
| Outcome: | The proposed policy improves the performance of SiMT models by boosting them with language models . the proposed policy plays a vital role in addressing the subword disparity between LMs and SiMT systems. |
Causal Intervention-based Few-Shot Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to perform few-shot named entity recognition are limited and overfitting is caused by the spurious correlation resulting from the bias in selecting a few samples. |
| Approach: | They propose a causal intervention-based few-shot named entity recognition method that blocks the backdoor path between context and label. |
| Outcome: | The proposed method achieves state-of-the-art in a few-shot named entity recognition (NER) task. |
TADI: Topic-aware Attention and Powerful Dual-encoder Interaction for Recall in News Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent researches of news recall adopt dual-encoder architecture as it provides a much faster recall scheme and they encode each word equally. |
| Approach: | They propose a model which weights words according to news topics and a module which enhances dual-encoder interaction. |
| Outcome: | The proposed model outperforms state-of-the-art models in a series of experiments. |
Unveiling the Power of Argument Arrangement in Online Persuasive Discussions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that the CMV is the best time period in human history for the vast majority of people. |
| Approach: | They extend a semantic argumentation unit type model by clustering type sequences into different argument arrangement patterns and representing discussions as sequences of these patterns. |
| Outcome: | The proposed model outperforms existing classifiers on the change my view forum discussion data. |
FFAEval: Evaluating Dialogue System via Free-For-All Ranking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for open-domain dialogue systems show poor correlation with human assessment. |
| Approach: | They propose a free-for-all human evaluation framework that shares dialogue history with annotators for multi-turn scoring. |
| Outcome: | The proposed framework achieves a strong correlation with human assessment on English and Chinese dialogue systems. |
Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
Nuo Chen, Hongguang Li, Junqing He, Yinan Bao, Xinshi Lin, Qi Yang, Jianfeng Liu, Ruyi Gan, Jiaxing Zhang, Baoyuan Wang, Jia Li
| Challenge: | Existing benchmarks for conversational machine reading comprehension are inconsistent with real scenarios. |
| Approach: | They propose to use a Chinese CMRC benchmark to evaluate model's generalization ability towards diverse domains by using zero-shot/few-shot settings. |
| Outcome: | The proposed benchmarks are based on 831 hot-topic driven conversations with 4,742 turns and cover 33 domains. |
VER: Unifying Verbalizing Entities and Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new model for verbalizing entities and relations is proposed to help understand entities and relationships . a unified model for Verbalizing Entities and Relations is proposed . |
| Approach: | They propose a model that takes any entity or entity set as input and generates a sentence to represent entities and relations. |
| Outcome: | The proposed model can generate sentences describing entities and relations . it can be used to explain entities and relationships, and to perform commonsense reasoning tasks . |
The Linearity of the Effect of Surprisal on Reading Times across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large amount of insight into human language processing can be gleaned by studying word-by-word processing difficulty. |
| Approach: | They extend the study by examining eyetracking corpora of seven languages . they find evidence for superlinearity in some languages, but highly sensitive to language models . |
| Outcome: | The study extends existing studies on english to Danish, Dutch, English, German, Japanese, Mandarin, and Russian. |
Adversarial Text Generation by Search and Learning (2023.findings-emnlp)
Copied to clipboard
Guoyi Li, Bingkang Shi, Zongzhen Liu, Dehan Kong, Yulei Wu, Xiaodan Zhang, Longtao Huang, Honglei Lyu
| Challenge: | Existing text generation methods only use heuristic replacement strategies or language models to generate replacement words at the word level. |
| Approach: | They propose a search and learning framework for Adversarial Text Generation by Search and Learning to evaluate the robustness of natural language processing models. |
| Outcome: | The proposed methods are significantly superior to the most advanced methods in terms of attack efficiency and adversarial text quality. |
Measuring Pointwise 𝒱-Usable Information In-Context-ly (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. |
| Approach: | They propose to adapt a recently proposed hardness metric, pointwise V-usable information (PVI), to an in-context version. |
| Outcome: | The proposed hardness metric is compared with the original model and is more efficient because it requires only a few exemplars and does not require fine-tuning. |
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-modal large language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. |
| Approach: | They propose a large language model with intrinsic cross-modal conversational abilities . they construct a cross-text speech instruction dataset and employ a three-stage training strategy . |
| Outcome: | The proposed model can follow cross-modal human instructions and handle multiple modalities with one model. |
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research demonstrates that multilingual encoder models are capable of zero-shot cross-lingual learning by using cloze-style prompts. |
| Approach: | They propose to reformulate input examples into cloze-style prompts to perform zero-shot multilingual tasks or linguistic probing by predicting label words at the masked token position. |
| Outcome: | The proposed method performs zero-shot multilingual tasks without updating parameters. |
A Thorough Examination on Zero-shot Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
Ruiyang Ren, Yingqi Qu, Jing Liu, Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, Ji-Rong Wen
| Challenge: | Recent advances in dense retrieval (DR) models have been shown to be not as competitive as traditional sparse retrieval models in a zero-shot retrieval setting. |
| Approach: | They propose to examine the zero-shot capability of DR models by analyzing key factors related to source training set and potential bias from target dataset. |
| Outcome: | The proposed model is not as competitive as sparse retrieval models in a zero-shot retrieval setting. |
Contrastive Pre-training for Personalized Expert Finding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to expert finding are effective for a community question answering platform. |
| Approach: | They propose a CQA-domain Contrastive Pre-training framework for Expert Finding which could learn more comprehensive question representations. |
| Outcome: | The proposed framework could learn more comprehensive question representations on six real-world datasets. |
Mitigating Intrinsic Named Entity-Related Hallucinations of Abstractive Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive text summarization (ATS) is important and challenging, but some hallucinations remain a challenge. |
| Approach: | They propose an adaptive margin ranking loss to facilitate two entity-alignment learning methods to tackle named entity-related hallucinations. |
| Outcome: | The proposed method improves the baseline model on automatic evaluation scores. |
Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language models (LLMs) have the capability to engage In-context Learning (ICL) however, this particular learning paradigm suffers from high instability stemming from factors such as input distribution, order and prompt formats. |
| Approach: | They propose to quantify the information gain obtained in prediction after observing a given example candidate and to sample those with maximum IG. |
| Outcome: | The proposed method can yield an average relative improvement of 14.3% across six classification tasks using three LLMs. |