Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
IAG: Induction-Augmented Generation Framework for Answering Reasoning Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to QA using retrieval-augmented knowledge are limited by limited coverage and noisy information. |
| Approach: | They propose an induction-augmented generation framework that utilizes inductive knowledge along with retrieved documents for implicit reasoning. |
| Outcome: | The proposed framework outperforms RAG and ChatGPT on two Open-Domain QA tasks. |
Absolute Position Embedding Learns Sinusoid-like Waves for Attention Based on Relative Position (2023.emnlp-main)
Copied to clipboard
| Challenge: | Attention weight is a clue to interpret how a Transformer-based model makes an inference. |
| Approach: | They analyze the mechanism behind the concentration of attention on nearby tokens . they find that attention in some heads is largely determined by relative positions . |
| Outcome: | The attention weights of the self-attention in a Transformer-based model are analyzed . the model can learn relationships between tokens while allowing parallelization, they show . |
Chinese Lexical Substitution: Dataset and Method (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for lexical substitution (LS) are limited and limited in coverage . despite extensive research on Lexical Substitution in various languages, there is limited evidence for LS in Chinese. |
| Approach: | They propose to use human and machine collaboration to construct a Chinese LS dataset . they combine four unsupervised LS methods to generate candidate substitutes . |
| Outcome: | The proposed method outperforms existing benchmarks on the Chinese lexical substitution task. |
Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to forecast news media responses have limited exploration of how to best process and utilize these important features. |
| Approach: | They propose a framework that leverages a large language model to induce a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. |
| Outcome: | The proposed framework surpasses state-of-the-art in experimental evaluations for both zero-shot and supervised settings, demonstrating its effectiveness in response forecasting. |
Fine-grained Conversational Decoding via Isotropic and Proximal Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing text decoding methods are not tailoring for dialogue generation. |
| Approach: | They propose a fine-grained conversational decoding method that generates a semantic-concentrated response while maintaining informativeness and discrimination against the context. |
| Outcome: | The proposed method outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. |
Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper we examine the complexity of persuasion technique annotation in a multilingual annotation campaign involving 6 languages and approximately 40 annotators. |
| Approach: | They propose a word embedding-based annotator agreement metric and propose 'holistic IAA' metric to measure the coherence of the entire dataset. |
| Outcome: | The proposed method is compared with the existing IAA metrics and its correlation with the results. |
PHD: Pixel-Based Language Modeling of Historical Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen a boom in efforts to digitise historical documents in numerous languages and sources, leading to a transformation in the way historians work. |
| Approach: | They propose a method for generating synthetic scans to resemble real historical documents by pre-training a model to reconstruct masked patches instead of predicting token distributions. |
| Outcome: | The proposed model can reconstruct masked patches and understand language well. |
Primacy Effect of ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing machine learning models may lead to poor performance in discriminative natural language understanding tasks. |
| Approach: | They propose to use ChatGPT to query large amounts of human-written text to find the answer to a question. |
| Outcome: | The proposed model has a high chance to select labels at earlier positions as the answer. |
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models (e.g., InstructGPT) struggle to answer subquestions written for the incorrect options of the main questions, implying that they have a limited capability for explaining why incorrect alternatives should be eliminated. |
| Approach: | They crowdsource free-form rationale for main questions and generate subquestion texts that explain why we should select or eliminate answer options. |
| Outcome: | The proposed dataset shows that large language models struggle to answer the subquestions even if they are able to answer main questions correctly. |
Evaluating and Modeling Attribution for Cross-Lingual Question Answering (2023.emnlp-main)
Copied to clipboard
Benjamin Muller, John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Soares, Roee Aharoni, Jonathan Herzig, Xinyi Wang
| Challenge: | Open-retrieval question answering systems are lacking in attribution for cross-lingual question answering . open-research questions are available in 20 languages, but their raw generation often falls short in factuality . |
| Approach: | They are the first to study attribution for cross-lingual question answering . they collect data in 5 languages to assess the attribution level of a state-of-the-art QA system . |
| Outcome: | The proposed approach improves the attribution level of a state-of-the-art cross-lingual QA system. |
Better Quality Pre-training Data and T5 Models for African Languages (2023.emnlp-main)
Copied to clipboard
Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, Jimmy Lin
| Challenge: | Existing web crawls have demonstrated quality issues for low-resource languages . Existing pretraining corpora have numerous quality issues . |
| Approach: | They propose to audit existing pretraining corpora to understand and rectify quality issues . they pretrain a new T5-based model and evaluate its performance on multiple tasks . |
| Outcome: | The proposed model outperforms existing pretrained models on four NLP tasks. |
Sparse Universal Transformer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use VTs as their backbone model are based on UTs that share parameters across layers and have better compositional generalization. |
| Approach: | They propose to use Sparse Mixture of Experts to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. |
| Outcome: | The proposed model achieves strong generalization results on formal language tasks and impressive parameter and computation efficiency on standard natural language benchmarks. |
Theory of Mind for Multi-Agent Collaboration via Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated impressive accomplishments in reasoning and planning, but their abilities in multi-agent collaborations remain unexplored. |
| Approach: | They propose to use explicit belief state representations to enhance task performance and the accuracy of ToM inferences for LLM-based agents. |
| Outcome: | The proposed model improves performance and accuracy of ToM inferences for LLM-based agents. |
Establishing Trustworthiness: Rethinking Tasks and Model Evaluation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing community has striven to model computationally for decades. |
| Approach: | They propose to rethink what constitutes tasks and model evaluation in NLP and pursue a more holistic view on language, placing trustworthiness at the center. |
| Outcome: | The proposed models are based on generative models and are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups. |
Let’s Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023.emnlp-main)
Copied to clipboard
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Wang
| Challenge: | Existing studies show vision-language systems can reason about images using natural language, but their capacity for video reasoning remains underexplored. |
| Approach: | They propose to frame video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language systems' capacity to reason about images using natural language. |
| Outcome: | The proposed models can generate multiple intermediate keyframes and predict future keyframe, and they perform poorly on GPT-4, GPT-3, and VICUNA. |
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Our study examines ChatGPT’s performance on Arabic languages and dialectal varieties. |
| Approach: | They conduct a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. |
| Outcome: | The proposed model outperforms smaller models on Arabic dialects compared to GPT-4's Modern Standard Arabic and Dialectal Arabic (DA) |
Dual-Channel Span for Aspect Sentiment Triplet Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extract sentiment triplets are too noisy and enumerate all possible spans. |
| Approach: | They propose a dual-channel span generation method to constrain the search space of span candidates. |
| Outcome: | The proposed method reduces span enumeration by nearly half on two versions of public datasets. |
Cultural Concept Adaptation on Multimodal Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past methods focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. |
| Approach: | They propose an annotation-free method for cultural-concept adaptation and construct a concept mapping set to facilitate model's comprehension of cultural-consensual mappings. |
| Outcome: | The proposed method outperforms baseline models on zero-shot and few-shot settings on five languages and cultures. |
Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection (2023.emnlp-main)
Copied to clipboard
| Challenge: | a data augmentation technique that generates synthetic examples by randomly substituting stem characters in existing training examples is still poorly understood. |
| Approach: | They propose a data augmentation strategy that generates synthetic examples by randomly substituting stem characters in existing training examples. |
| Outcome: | The proposed method generates synthetic examples by randomly substituting stem characters in existing training examples. |
Evaluating Object Hallucination in Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) suffer from object hallucinations, i.e., they tend to generate objects inconsistent with the target images in the descriptions. |
| Approach: | They propose to integrate powerful large vision-language models (LVLMs) they propose a polling-based query method to evaluate object hallucination . |
| Outcome: | The proposed model can evaluate object hallucination in a more stable and flexible way. |
Event Ontology Completion with Hierarchical Structure Evolution Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event detection require predefined schemas, but manual defining is expensive and labor-intensive. |
| Approach: | They propose a task to achieve event clustering, hierarchy expansion and type naming . they propose 'neighbor Contrastive Clustering' module and a Hierarchy-Aware Linking module . |
| Outcome: | The proposed method outperforms baseline methods on three datasets. |
Parameter-efficient Tuning for Large Language Model without Calculating Its Gradients (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent parameter-efficient tuning methods can only save 30% of training memory . gradient computation and backpropagation are still necessary for these methods . |
| Approach: | They propose a parameter-efficient tuning method that can be used to fine-tune large language models without calculating gradients. |
| Outcome: | The proposed method saves 30% of training memory and improves performance on large language models. |
Discourse Structures Guided Fine-grained Propaganda Identification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using teacher-predicted probabilities and knowledge distillation frameworks to identify propaganda content is important. |
| Approach: | They propose to integrate local and global discourse structures for propaganda discovery and construct two teacher models for identifying PDTB-style discourse relations between nearby sentences and common discourse roles of sentences in a news article respectively. |
| Outcome: | The proposed models improve accuracy and recall of propaganda content identification at sentence-level and token-level. |
CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, there is no dataset containing compound and non-compound words across languages . however, current LLMs perform poorly on words tokenized unfavorably by subword tokenization. |
| Approach: | They propose to use a Wiktionary dataset to evaluate large language models on decompounding . they find that current LLMs perform poorly on words tokenized unfavorably . |
| Outcome: | The proposed model outperforms the best unsupervised models by 13.9% accuracy on average. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
GATITOS: Using a New Multilingual Lexicon for Low-resource Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new study explores the effectiveness of bilingual lexica in machine translation models . cross-lingual vocabulary alignment is still highly imperfect in these models, despite the success of supervised and self-supervised training. |
| Approach: | They use a resource to improve translation performance on 200-language models . they show that lexica is more reliable than human-translated data . |
| Outcome: | The proposed approach improves on 200-language translation models with lexical data augmentation . the proposed approach is open-source and has 168 tail languages . |
Continually Improving Extractive QA via Human Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of extractive question answering systems using human feedback shows promising potential for continual learning. |
| Approach: | They study extractive question answering system by using user feedback to improve it . they design and deploy an iterative approach where users ask questions and provide feedback . |
| Outcome: | The proposed model improves over time across different data regimes and domains . human user feedback is more affordable and abundant than annotations provided by trained experts . |
Using Interpretation Methods for Model Enhancement (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks for enhancing neural models with interpretation methods and gold rationales have not been fully explored. |
| Approach: | They propose a framework for utilizing interpretation methods and gold rationales to enhance neural models. |
| Outcome: | The proposed framework outperforms gradient-based methods in low-resource settings on a variety of tasks. |
An Expression Tree Decoding Strategy for Mathematical Equation Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate mathematical equations from natural language ignore parallel or dependent relations between math expressions. |
| Approach: | They propose to integrate tree structure into the expression-level generation and advocate an expression tree decoding strategy. |
| Outcome: | The proposed method outperforms baseline methods for generating mathematical equations from natural language. |
Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LMs) have shown promising ability on handling various downstream tasks including textual classification and question answering. |
| Approach: | They propose to use BabyBERTa to train child-directed speech without unmasking words while masking parameters to improve grammatical accuracy. |
| Outcome: | The proposed model achieves grammatical ability comparable to RoBERTa-base model, which is trained on 6,000 times more words and 15 times more parameters. |
Diversity Enhanced Narrative Question Generation for Storybooks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Question generation (QG) from a given context can enhance comprehension, engagement, assessment, and overall efficacy in learning or conversational environments. |
| Approach: | They propose a multi-question generation model which generates multiple, diverse questions by focusing on context and questions. |
| Outcome: | The proposed model shows promising results across evaluation metrics, among strong baselines. |
Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in weakly supervised text classification focus on designing sophisticated methods to turn high-level human heuristics into quality pseudo-labels. |
| Approach: | They propose to use a seed matching-based method to generate quality pseudo-labels by deleting the seed words present in the matched input text. |
| Outcome: | The proposed method can be improved significantly by deleting the seed words in the matched input text with a high deletion ratio. |
How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models excel at capturing semantic correlations within utterance embeddings but fail to determine specific causal relationships. |
| Approach: | They propose to incorporate i.i.d. noise terms into conversation process to build a structural causal model . they propose to use unstructured conversation data to facilitate deep learning . |
| Outcome: | The proposed approach can be implemented in unstructured conversation data and a synthetic dataset that includes i.i.d. noise. |
Compressing and Debiasing Vision-Language Pre-Trained Models for Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on VQA models have found that they suffer from dataset biases and inefficient memory footprints. |
| Approach: | They investigate whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. |
| Outcome: | The proposed compression and debiasing pipelines outperform the debiased full VLPs on VQA tasks. |
Selectively Answering Ambiguous Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown. |
| Approach: | They propose to use a sampled set of questions to calibrate answers to ambiguous questions with varying model scales. |
| Outcome: | The results show that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous ones. |
Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal knowledge graphs (TKGs) are used to represent real-world facts in a structured way. |
| Approach: | They propose to use in-context learning with large language models for TKG forecasting . they compare naive LLMs to state-of-the-art (SOTA) supervised models . |
| Outcome: | The proposed approach performs well against pre-trained large language models . the proposed approach is based on simple heuristics and state-of-the-art models compared with pre-trainers . |
Knowledge Graph Compression Enhances Diverse Commonsense Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models use commonsense knowledge graphs to extract subgraphs of relevant knowledge pertaining to concepts in the input but due to the large coverage and vast scale of ConceptNet, the extracted subgraph may contain loosely related, redundant and irrelevant information. |
| Approach: | They propose to apply a differentiable graph compression algorithm to extract subgraphs of relevant knowledge from input sentences. |
| Outcome: | The proposed algorithm achieves better quality-diversity tradeoff than a large language model with 100 times the number of parameters. |
Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generalized quantifiers are used to indicate the proportions predicates satisfy (e.g., some apples are red). |
| Approach: | They propose a framework to model quantifier semantics for textbased foundation models by combining natural language inference and the Rational Speech Acts framework. |
| Outcome: | The proposed framework shows a 20% improvement over a literal listener baseline in predicting percentage scopes for quantifier comprehension even with no training. |
LLM-FP4: 4-Bit Floating-Point Quantized Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quantization solutions are integer-based and struggle with bit widths below 8 bits. |
| Approach: | They propose a method for quantizing weights and activations in large language models down to 4-bit floating-point values in a post-training manner. |
| Outcome: | The proposed method outperforms existing methods on common sense zero-shot reasoning tasks by 12.7 points. |
Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models struggle to generate technical summaries that are on par with those produced by biomedical experts due to the lack of domain-specific background knowledge. |
| Approach: | They propose a attention-based citation aggregation model that integrates domain-specific knowledge from citation papers and a large-scale biomedical summarisation dataset to build on. |
| Outcome: | The proposed model outperforms state-of-the-art approaches and achieves substantial improvements in biomedical abstractive summarisation. |
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks. |
| Approach: | They propose to optimize explanation-infused prompts in a blackbox fashion by using leave-one-out schemes and a two-stage framework. |
| Outcome: | The proposed method improves prompts over crowdworker annotations and naive search strategies. |
HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation (2023.emnlp-main)
Copied to clipboard
David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loic Barrault, Marta Costa-jussà
| Challenge: | Previously available quality assessments do not distinguish between hallucinations and omissions. |
| Approach: | They propose to annotate hallucinations and omissions in machine translation using a single language pair. |
| Outcome: | The proposed dataset covers 18 translation directions with varying resource levels and scripts. |
Gradient-based Gradual Pruning for Language-Specific Multilingual Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation suffers from performance degradation in high-resource languages compared to bilingual counterparts. |
| Approach: | They propose a gradient-based gradual pruning technique for multilingual neural machine translation that allows for partial parameter sharing across language pairs to alleviate interference. |
| Outcome: | The proposed approach yields a notable performance gain on IWSLT and WMT datasets. |
LLM-powered Data Augmentation for Enhanced Cross-lingual Performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing training data for multilingual commonsense reasoning datasets is limited. |
| Approach: | They propose to use large language models for data augmentation in multilingual datasets . they use Dolly-v2, StableVicuna, ChatGPT, and GPT-4 to augment three datasets. |
| Outcome: | The proposed model outperforms larger general-purpose, zero-shot models when training in smaller models. |
Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for identifying discourse relations without explicit connectives are limited by the availability of annotated data. |
| Approach: | They propose a method that injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. |
| Outcome: | The proposed method achieves outstanding performance against the current state-of-the-art models. |
VLIS: Unimodal Language Models Guide Multimodal Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models face challenges in tasks that require complex linguistic understanding. |
| Approach: | They propose a framework that combines visual conditioning and linguistic understanding of unimodal text-only language models without further training to improve vision-language models. |
| Outcome: | The proposed framework improves vision-language models on diverse tasks including commonsense understanding and complex text generation. |
Conceptual structure coheres in human cognition but not in large language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural network models of language have long been used to develop hypotheses about conceptual representation in the mind and brain. |
| Approach: | They propose to use three techniques borrowed from cognitive psychology to estimate lexical-semantic structure in humans and a large language model. |
| Outcome: | The proposed models show that human-like models can estimate lexical-semantic structure robustly to cultural, language, and method of estimation. |
Towards LLM-driven Dialogue State Tracking (2023.emnlp-main)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. |
| Approach: | They present a framework for a domain-slot instruction tuning method that allows LDST to achieve performance on par with ChatGPT. |
| Outcome: | The proposed framework performs better in zero-shot and few-shot settings than previous SOTA methods. |
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Sentiment Analysis (MSA) is effective when using rich information from multiple sources, but the potential sentiment-irrelevant information across modalities may hinder the performance from being further improved. |
| Approach: | They propose an Adaptive Language-guided Multimodal Transformer (ALMT) that learns an irrelevance/conflict-suppressing representation from visual and audio features under guidance of language features at different scales. |
| Outcome: | The proposed model achieves state-of-the-art on several popular datasets and an abundance of ablation shows the effectiveness of the proposed model. |
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion (2023.emnlp-main)
Copied to clipboard
Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, Alessandro Suglia
| Challenge: | Embodied MultiModal Agent (EMMA) is a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text generation. |
| Approach: | They propose an Embodied MultiModal Agent (EMMA) that uses a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text. |
| Outcome: | The proposed model performs on par with similar models on several VL benchmarks and sets a new state-of-the-art success rate on the Dialog-guided Task Completion (DTC) benchmark. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new framework for structured prediction is developed for natural language processing . a systematic approach to structured prediction requires exhaustive pair-wise comparisons of tokens . |
| Approach: | They propose a method that models the relationship between pairs of tokens in a string . they use a parallel method that predicts real numbers for each token in . |
| Outcome: | The proposed method doubles the speed of graph-based dependency parsers and brings 10-times speed-up over graph-driven dependency parses. |
GEMINI: Controlling The Sentence-Level Summary Style in Abstractive Text Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that mimic human summarization techniques are difficult to imitate. |
| Approach: | They propose an adaptive model that integrates a rewriter and a generator to mimic the sentence rewriting and abstracting techniques. |
| Outcome: | The proposed model outperforms baselines on WikiHow and on other datasets. |
Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies. |
| Approach: | They propose a new decoding method that augments the contrastive search framework with context-aware regularization terms to promote tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. |
| Outcome: | The proposed method improves faithfulness across various language models while maintaining output diversity comparable to well-performing decoding algorithms. |
Analyzing Norm Violations in Live-Stream Chat (2023.emnlp-main)
Copied to clipboard
Jihyung Moon, Dong-Ho Lee, Hyundong Cho, Woojeong Jin, Chan Park, Minwoo Kim, Jonathan May, Jay Pujara, Sungjoon Park
| Challenge: | Existing methods for detecting toxic language and norm violations are limited to live-streaming platforms . existing methods are less effective when applied to live streaming platforms based on a limited time frame . |
| Approach: | They propose to use contextual information to automatically moderate toxic content on live streaming platforms. |
| Outcome: | The proposed model improves on live-streaming platforms by 35%. |
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted severe limitations of contrastive learning models in their ability to perform compositional reasoning over objects, attributes, and relations. |
| Approach: | They propose a graph decomposition framework and negative mining techniques to improve attribute binding and relation understanding of scene graphs. |
| Outcome: | The proposed approach improves attribute binding, relation understanding, generalization, and productivity on multiple benchmarks. |
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms (2023.emnlp-main)
Copied to clipboard
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu
| Challenge: | NormLens is a visual-grounded framework for understanding commonsense norms . state-of-the-art models are not well-aligned with human annotation, we show . |
| Approach: | They propose a visual-grounded framework to study commonsense norms by NormLens . they find that models are not well-aligned with human annotation . |
| Outcome: | The proposed model judgments and explanations are not well-aligned with human annotations. |
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus (2023.emnlp-main)
Copied to clipboard
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, Luoyi Fu
| Challenge: | Existing methods for detecting hallucinations in LLMs rely on external knowledge for reference retrieval or require sampling multiple responses for consistency verification. |
| Approach: | They propose a reference-free, uncertainty-based method for detecting hallucinations in Large Language Models that imitates human focus in factuality checking from three aspects: focus on the most informative keywords; focus on unreliable tokens in historical context; focus of token properties such as token type and token frequency. |
| Outcome: | The proposed method achieves state-of-the-art performance across all evaluation metrics and eliminates the need for additional information. |
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing factuality evaluation models are not robust, especially with respect to entity and relation errors in new domains. |
| Approach: | They propose a new approach to factuality evaluation that is generalizable across domains . they propose entities-specific facts, facts extracted from external knowledge bases and facts constructed compositionally through knowledge base walks. |
| Outcome: | The proposed model achieves state-of-the-art on two in-domain news summarization benchmarks and on three out-of domain scientific literature datasets. |
Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern NLP models are often trained over large untrustworthy datasets, raising the potential for a malicious adversary to compromise model behaviour. |
| Approach: | They propose to mitigate spurious correlations between textual triggers and classification labels by combining them with insertion-based attacks. |
| Outcome: | The proposed defence significantly reduces attack success rates across backdoor attacks and provides a near-perfect defence against insertion-based attacks. |
Symbol tuning improves in-context learning in language models (2023.emnlp-main)
Copied to clipboard
Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, Quoc Le
| Challenge: | Language models are sensitive to the way that prompts are given, indicating that they are not reasoning in a robust manner. |
| Approach: | They propose to fine tune language models on in-context input-label pairs where natural language labels are replaced with arbitrary symbols. |
| Outcome: | The proposed model is much stronger at reasoning tasks and more robust to underspecified prompts than the standard model. |
The neural dynamics of word recognition and integration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a computational model of word recognition, listeners combine expectations about upcoming content with incremental sensory evidence. |
| Approach: | They fit this model to scalp EEG signals recorded as subjects passively listened to a fictional story and found that words require more than 150 ms of input to be recognized. |
| Outcome: | The proposed model formalizes this perceptual process in Bayesian decision theory and reveals distinct neural processing of words depending on whether or not they can be quickly recognized. |
Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | In open-domain question answering, users often ask ambiguous questions (AQs) . one approach is to identify all possible interpretations of the AQ and generate a long-form answer addressing them all. |
| Approach: | They propose a framework that generates a long-form answer addressing all possible interpretations of an ambiguous question. |
| Outcome: | The proposed framework outperforms baselines on ASQA in a few-shot setup across metrics while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambigo-ROUGE. |
Incorporating Worker Perspectives into MTurk Annotation Practices for NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to data collection for natural language processing on Amazon Mechanical Turk (MTurk) are susceptible to issues regarding workers’ rights and poor response quality without considering the perspectives of MTurq workers. |
| Approach: | They conducted a critical literature review and a survey of MTurk workers to address open questions regarding fair payment, worker privacy, data quality, and considering worker incentives. |
| Outcome: | The findings suggest that future studies may better account for MTurk workers’ experiences in order to respect workers' rights and improve response quality. |
Predict the Future from the Past? On the Temporal Data Distribution Shift in Financial Sentiment Classifications (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for financial sentiment analysis use random splits of a dataset into training and testing to ensure there is no distribution shift between training and deployment. |
| Approach: | They propose a method that combines out-of-distribution detection with time series modeling for temporal financial sentiment analysis. |
| Outcome: | The proposed method improves the model’s ability to adapt to evolving temporal shifts in a volatile financial market. |
Look-back Decoding for Open-Ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decode open-ended text have addressed degeneration problems in large-scale language models (LLMs) |
| Approach: | They propose an improved decoding algorithm that leverages the Kullback–Leibler divergence to track the distribution distance between current and historical decoding steps. |
| Outcome: | The proposed algorithm outperforms existing methods in document continuation and story generation. |
Large Language Models Can Self-Improve (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have excellent performance in various tasks, but fine-tuning requires extensive supervision. |
| Approach: | They propose to use a pre-trained Large Language Model to generate rationale-augmented answers for unlabeled questions and fine-tune the LLM using those self-generated solutions as target outputs. |
| Outcome: | The proposed approach improves the general reasoning ability of a 540B-parameter LLM without any ground truth label. |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |
Structural generalization in COGS: Supertagging is (almost) all you need (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that neural networks fail to generalize on out-of-distribution examples. |
| Approach: | They extend a neural graph-based parsing framework to address compositional generalization limitations . they introduce a supertagging step with valency constraints and reduce the graph prediction problem . |
| Outcome: | The proposed approach improves results on COGS datasets that require structural generalization. |
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023.emnlp-main)
Copied to clipboard
| Challenge: | et al., 2022) argue that the current models for drug discovery lack the ability to integrate molecules, proteins, and natural language. |
| Approach: | They propose a framework that integrates biological knowledge with chemical knowledge and natural language associations. |
| Outcome: | The proposed framework shows superior performance across a wide range of tasks. |
Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | XLMs can support cross-lingual transfer learning with little to no additional training data. |
| Approach: | They describe a mechanism for cross-lingual transfer learning by measuring the properties of the initial token embedding layer. |
| Outcome: | The proposed model can be used to support cross-lingual transfer learning . the initial token embedding layer is expressive and interpretable . |
Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study defines a conversation target from the system side to proactively steer conversations toward predefined targets or accomplish specific system-side goals. |
| Approach: | They propose a dataset curation framework that automatically curations a large-scale personalized dialogue dataset using a role-playing approach. |
| Outcome: | The proposed dataset is of high quality and could contribute to exploring personalized target-oriented dialogue. |
SeqXGPT: Sentence-Level AI-Generated Text Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for sentence-level AIGT detection are weak . large language models (LLMs) can generate human-like content . |
| Approach: | They propose a sentence-level AIGT detection challenge using LLMs as log probability lists . they propose 'check' GPT' method that uses log probability list features to detect AIGT . |
| Outcome: | The proposed method surpasses baseline methods in sentence- and document-level detection challenges. |
QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)
Copied to clipboard
Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan
| Challenge: | Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights. |
| Approach: | They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary. |
| Outcome: | The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input. |
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating insightful explanations with limited annotations are limited. |
| Approach: | They propose a method that iteratively computes visual features, an answer, and an explanation to improve the explanation quality step by step until the answer converges. |
| Outcome: | The proposed method outperforms previous methods while utilizing 5% of the human-annotated explanations across 10 metrics, showing up to 4.2 and 1.3 increases in BLEU-1 score on the VCR and VQA-X datasets. |
‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science Journalism (2023.emnlp-main)
Copied to clipboard
| Challenge: | Science journalism is the production of journalistic content covering scientific topics that are not covered in the scientific literature. |
| Approach: | They propose to use a dataset to generate a scientific paper's tuples, a summary snippet and a novel technical framework to integrate a paper' s discourse structure with its metadata to guide generation. |
| Outcome: | The proposed system outperforms baseline methods in elaborating a content plan meaningful for the target audience, simplifying the information selected, and producing a coherent final report in a layman’s style. |
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied Instruction Following has shown an impressive success rate when the environment has been seen in training, but when deployed in an unseen environment, it tends to struggle when deployed with an unsightly environment. |
| Approach: | They propose to explicitly align the agent’s hidden states with the instructions via contrastive learning to bridge the semantic gap between high-level language instructions and the agent's low-level action space. |
| Outcome: | The proposed meta-actions achieve a 4.5% success rate in unseen environments compared to a strong multi-modal Transformer baseline . |
Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output. |
| Approach: | They propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. |
| Outcome: | The proposed approach generates high-quality sentences resembling human output. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Clinical Contradiction Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Detecting contradictions in text is difficult since it requires clinical expertise. |
| Approach: | They propose to use a medical ontology to build a seed of potential medical contradictions in medical abstracts by distant supervision. |
| Outcome: | The proposed method weakly supervises state-of-the-art deep learning models and shows significant improvements across multiple medical contradiction datasets. |
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the advances of language models, they still produce text that contains trivial commonsense errors. |
| Approach: | They propose a general-purpose commonsense statement verification model that learns to estimate the plausibility of declarative statements based on commonsensical knowledge. |
| Outcome: | The proposed model outperforms existing models that can be repurposed for commonsense verification, even including GPT-3.5/ChatGPT/GPT-4. |
Text-Transport: Toward Learning Causal Effects of Natural Language (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for causal inference require strong assumptions about the data, meaning the data from which one *can* estimate valid causal effects is not representative of the actual target domain of interest. |
| Approach: | They propose a method for estimation of causal effects from natural language under any text distribution using the notion of distribution shift. |
| Outcome: | The proposed method can be used to estimate causal effects from natural language under any text distribution. |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
Unveiling the Implicit Toxicity in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, but LLMs can generate diverse implicit toxic output that are difficult to detect via simply zero-shot prompting. |
| Approach: | They propose a reinforcement learning based attacking method to induce the implicit toxic outputs in large language models by fine-tuning toxicity classifiers. |
| Outcome: | The proposed method generates implicit toxic outputs that are difficult to detect via zero-shot prompting on five widely-adopted toxicity classifiers. |
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in scale have enabled large language models to perform NLP tasks zero-shot . however, it is not known whether ChatGPT can serve as a generalist model that can perform many NLP jobs zero- shot. |
| Approach: | They empirically evaluate ChatGPT's zero-shot learning ability on 20 popular NLP datasets . they find it performs well on many tasks favoring reasoning abilities . |
| Outcome: | The proposed model can perform many NLP tasks zero-shot without adaptation on downstream data. |
Length is a Curse and a Blessing for Document-level Semantics (2023.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models. |
| Approach: | They propose a document-based contrastive learning framework that is length-agnostic self-reference based on document length. |
| Outcome: | The proposed framework achieves state-of-the-art on the standard information retrieval benchmark. |
ALCUNA: Large Language Models Meet New Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks do not adequately measure large-scale language models’ capabilities when faced with new knowledge. |
| Approach: | They propose a benchmark called ALCUNA to evaluate LLMs' ability to handle new knowledge by altering existing entity attributes and relationships. |
| Outcome: | The proposed approach generates new knowledge by altering existing entity attributes and relationships, resulting in artificial entities distinct from real-world entities. |
Location-Aware Visual Question Generation with Lightweight Models (2023.emnlp-main)
Copied to clipboard
Nicholas Suwono, Justin Chen, Tun Hung, Ting-Hao Huang, I-Bin Liao, Yung-Hui Li, Lun-Wei Ku, Shao-Hua Sun
| Challenge: | a novel task aims to generate engaging questions from location-aware information . a lightweight model can be used to generate such questions . |
| Approach: | They propose a task to generate engaging questions from location-aware data . they represent location-based information with surrounding images and a GPS coordinate . |
| Outcome: | The proposed method outperforms baselines regarding human evaluation and evaluation metrics. |
MemeCap: A Dataset for Captioning and Interpreting Memes (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new dataset aims to understand meme captioning tasks using visual metaphors . vision and language models are proving to be effective in image captioning and visual question answering tasks . |
| Approach: | They present a dataset that contains 6.3K memes and 6.3k meme captions . they show that vision and language models still struggle with visual metaphors despite their advanced capabilities . |
| Outcome: | The proposed dataset contains 6.3K memes along with the title of the post containing the meme, meme captions, literal image caption, and visual metaphors. |
Where to start? Analyzing the potential value of intermediate models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a finetuned model may be better base models than the vanilla pretrained model . this scheme, often referred to as intertraining, is the focus of the present work . |
| Approach: | They propose a scheme to analyze the potential intertraining gain independently for the target dataset and for a base model being considered as a starting point. |
| Outcome: | The proposed model is strong even if training data was not aligned with target dataset. |
Transcending Scaling Laws with 0.1% Extra Compute (2023.emnlp-main)
Copied to clipboard
Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani
| Challenge: | Existing scaling of language models is expensive and requires significant computational costs. |
| Approach: | They propose a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
| Outcome: | The proposed method significantly improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Annotated data plays a critical role in training models and evaluating their performance. |
| Approach: | They propose a paradigm for Human-LLM co-annotation of unstructured texts at scale that utilizes uncertainty to estimate LLMs’ annotation capability. |
| Outcome: | The proposed model outperforms existing models on many text-annotation tasks with up to 21% performance improvement over random baseline. |
Optimizing Retrieval-augmented Reader Models via Token Elimination (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for ODQA use a retrieval-augmented language model . a generative model can cause a significant bottleneck in decoding time . |
| Approach: | They propose to eliminate some of the retrieved information that might not contribute essential information to the answer generation process. |
| Outcome: | The proposed method reduces run-time by up to 62.2% with only 2% reduction in performance and improves performance. |
WSDMS: Debunk Fake News via Weakly Supervised Detection of Misinforming Sentences with Contextualized Social Wisdom (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debunking fake news rely on blending of authentic and fabricated content by creators. |
| Approach: | They propose a model that detects misinformation at sentence-level using social media conversations . they use a bag-level annotation system to train the model . |
| Outcome: | The proposed model outperforms existing state-of-the-art models on three real-world benchmarks and outperformed existing state of the art models in debunking fake news at sentence and article levels. |
Robust Prompt Optimization for Large Language Models Against Distribution Shifts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has explored automatic prompt optimization methods to eliminate manual effort in identifying effective prompts for a given task. |
| Approach: | They propose a framework for prompt optimization that can be generalized to an unlabeled target group. |
| Outcome: | The proposed framework improves on target group and source group while generalizing to unlabeled target group. |
Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have great potential for synthetic data generation. |
| Approach: | They show that large language models can generate useful data even for complex tasks . they use a symmetric task difficulty asymmetry to prompt an LLM to generate plausible input text for a target output structure. |
| Outcome: | The proposed approach outperforms existing models by a substantial margin on closed information extraction tasks with 1.8M data points and 770M parameters. |
Condensing Multilingual Knowledge with Lightweight Language-Specific Modules (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to boost performance in multilingual models but scalability is difficult to manage. |
| Approach: | They propose a method that incorporates language-specific (LS) modules to boost model performance. |
| Outcome: | The proposed method outperforms state-of-the-art methods while outperforming existing methods. |
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Inference is estimated to make up 80 to 90% of ML cloud computing demand . |
| Approach: | They propose to identify bottlenecks in deep learning frameworks that are causing the disparity in model latency as hardware speed increases over time. |
| Outcome: | The proposed models show that the framework tax is increasing as the hardware speed increases over time. |
Evaluating Cross-Domain Text-to-SQL Models and Benchmarks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-SQL benchmarks are used to evaluate progress made in the field . however, matching a model-generated SQL query to a reference SQL query fails due to various reasons. |
| Approach: | They conduct an extensive evaluation of text-to-SQL benchmarks and re-evaluate some of the top-performing models. |
| Outcome: | The results show that a recent model surpasses the gold standard reference queries in the Spider benchmark in human evaluation. |
Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate knowledge graphs are unable to handle non-English textual information. |
| Approach: | They propose a task of automatic Knowledge Graph Completion to bridge the gap between English and non-English textual information. |
| Outcome: | The proposed method bridges the gap between the quantity and quality of textual information between English and non-English languages. |
Memory-Based Invariance Learning for Out-of-Domain Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learning invariant representations rely on the assumption that training and test sets come from the same domain. |
| Approach: | They propose to extend a classification model trained on multiple source domains to an unseen target domain by using key-value memory. |
| Outcome: | The proposed method improves on sentiment analysis and natural language inference tasks. |
Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023.emnlp-main)
Copied to clipboard
| Challenge: | asymmetric outliers in transformer language models are a challenge for post-training quantization . we propose a framework for outlier suppression that can be seamlessly migrated into subsequent modules . |
| Approach: | They propose a framework for post-training quantization that includes the channel-wise shifting and scaling for concentration. |
| Outcome: | The proposed framework can be migrated into subsequent modules while maintaining equivalence. |
Three Stream Based Multi-level Event Contrastive Learning for Text-Video Event Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event extraction ignore motion representations in videos and are misguided by background noise. |
| Approach: | They propose a text-video based multimodal event extraction framework that integrates video appearance features and motion representations with video appearance. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods in the event extraction field. |
Diversify Question Generation with Retrieval-Augmented Style Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing question generation systems focus on the internal knowledge within the textual passage or the semantic word space for diverse content planning. Existing solutions focus on relying on the knowledge of the text and the semantic words, but have not considered the potential of external knowledge for expression diversity. |
| Approach: | They propose a framework for Retrieval-Augmented Style Transfer that utilizes the style of diverse templates for question generation. |
| Outcome: | The proposed framework outperforms baselines on diversity while being comparable in terms of consistency scores. |
Fast and Accurate Factual Inconsistency Detection Over Long Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative AI models exhibit remarkable potential, however, hallucinations across various tasks present a significant challenge, particularly for longer inputs. |
| Approach: | They propose a task-agnostic model that uses large text chunks to condition over long texts and employ a novel algorithm to explain its decisions through relevant source sentence retrieval. |
| Outcome: | The proposed model outperforms existing methods on benchmarks and a new long-form dialogue dataset and surpasses competitive systems in efficiency and model explanation evaluations. |
Interpreting Embedding Spaces by Conceptualization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have a significant drawback: they are incomprehensible to humans. |
| Approach: | They propose a method for understanding embeddings by transforming a latent embeddable space into a comprehensible conceptual space. |
| Outcome: | The proposed method compares the semantics of the original latent embedding space to the semantic of the vectors. |
Knowledge-Augmented Language Model Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent Language Models (LMs) generate factually incorrect answers to queries . authors propose to augment LMs with knowledge retrieved from external source . |
| Approach: | They propose to augment LMs with knowledge retrieved from external sources . they validate the output and knowledge of the knowledge-augmented LM with a separate verifier . |
| Outcome: | The proposed model can generate factually incorrect answers on multiple questions . a verifier detects retrieval errors and can correct them by retrieving new knowledge or generating new text . |
A Generation-based Deductive Method for Math Word Problems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing generation methods suffer from repeated sub-expression generation and deductive methods are restricted to dealing with binary operations. |
| Approach: | They propose a multivariate directed acyclic graph (mDAG) which generates the topological ordering of mDAg by equipping a generation model with a re-encoder to keep the deductive property but avoid the expensive enumeration of deductive methods. |
| Outcome: | The proposed model performs well on the widely used benchmarks and solves multivariate operators on the CMWPA benchmark. |
Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance, but they keep repeating similar mistakes due to their inability to capture relationships among samples. |
| Approach: | They propose a tuning-free rule accumulation framework that guides LLMs in improving their performance by learning from previous mistakes. |
| Outcome: | The proposed framework improves over baselines by a large margin over previous frameworks. |
Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve persona consistency are centered around supervised learning or online reinforcement learning (RL). Existing approaches to improve consistency are expensive and require additional training. |
| Approach: | They propose an offline supervised learning framework to improve persona consistency of dialogue systems by punishing and rewarding specific utterances. |
| Outcome: | The proposed framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot. |
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using mixture-of-memory augmenting to augment language models improves model generalization but with diminishing return. |
| Approach: | They develop a mechanism that augments language models with mixture-of-memory Augmentation (MoMA) they augment strong T5-based retrievers with the option to "plug in" unseen memory at inference time. |
| Outcome: | The proposed model outperforms methods with larger model sizes on the BEIR benchmark and achieves comparable or even better performance than methods relying on target-specific pretraining. |
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models on diverse tasks with instructions. |
| Approach: | They propose a framework to identify informative tasks and then actively tune models on selected tasks. |
| Outcome: | The proposed method outperforms baseline strategies for task selection on NIV2 and Self-Instruct datasets. |
Towards Example-Based NMT with Multi-Levenshtein Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval-augmented machine translation (RAMT) is attracting growing attention . it is assumed to implement some form of domain adaptation . |
| Approach: | They propose a retrieval-augmented version of the Levenshtein Transformer to make it more transparent . they propose to perform training and inference in this model, based on multi-way alignment algorithms and imitation learning. |
| Outcome: | The proposed architecture improves translation performance and improves consistency of translations compared to previous models. |
DUnE: Dataset for Unified Editing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models are susceptible to errors necessitating a comprehensive retraining process. |
| Approach: | They propose to define an edit as any natural language expression that solicits a change in the model’s outputs. |
| Outcome: | The proposed editing benchmarks show that retrieval-augmented language modeling outperforms specialized editing techniques and neither set of approaches has fully solved the generalized editing problem covered by the proposed benchmark. |
“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work treats gender bias as a binary classification task, but a comparative annotation framework can be used to assess the impact of biases. |
| Approach: | They propose to generate a dataset with normative ratings of gender bias in English text with a comparative annotation framework. |
| Outcome: | The first dataset of GPT-generated English text with normative ratings of gender bias is analyzed using Best–Worst Scaling . |
Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Inverted file structure is a common technique for accelerating dense retrieval, but its lossy nature degrades it. |
| Approach: | They propose a hybrid index where embedding clusters and salient terms work collaboratively to accelerate dense retrieval. |
| Outcome: | The proposed method achieves lossless retrieval quality with competitive efficiency across index settings. |
ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (2023.emnlp-main)
Copied to clipboard
| Challenge: | generative large language models (LLMs) are replacing human workers for some tasks . crowdsourcing has several downsides: 1) the workforce is costly, 2) output quality is difficult to achieve, and 3) there are overheads related to the design and organization of the process. |
| Approach: | They investigate whether ChatGPT-created paraphrases are more diverse and robust . they use a crowdsourcing tool to collect training or validation examples . |
| Outcome: | The proposed models are more diverse and robust than the existing models. |
Query-as-context Pre-training for Dense Passage Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve passage retrieval performance by using context-supervised pre-training are weakly correlated. |
| Approach: | They propose to use query-as-context pre-training to train passage-query pairs . they evaluate the pre-trained models on large-scale passage retrieval benchmarks . |
| Outcome: | The proposed technique improves performance on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. |
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (2023.emnlp-main)
Copied to clipboard
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
| Challenge: | Existing datasets for webpages contain only fragments of webpages . generative tasks like page description generation and section summarization are often left unstudied . |
| Approach: | They introduce a Wikipedia Webpage suite that contains 2M pages with all associated image, text, and structure data. |
| Outcome: | The proposed approach performs better than full attention with lower computational complexity. |
Democratizing Reasoning Ability: Tailored Learning from Large Language Model (2023.emnlp-main)
Copied to clipboard
Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Large language models (LLMs) exhibit impressive emergent abilities in natural language processing, but their democratization is hindered due to huge computation requirements and closed-source nature. |
| Approach: | They propose a tailored learning approach to distill the exclusive reasoning ability to smaller LMs to facilitate democratization. |
| Outcome: | The proposed approach enables the democratization of the exclusive reasoning ability by leveraging the black-box model as a reasoning teacher. |
OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on a limited set of predefined aspects, resulting in a lack of realistic open aspect setting. |
| Approach: | They propose a benchmark for multi-document open aspect-based summarization using an annotation protocol. |
| Outcome: | The proposed benchmark satisfies the needs of users in real-world scenarios. |
PEFTDebias : Capturing debiasing information using PEFTs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research suggests that biases acquired during pretraining can propagate to downstream models, resulting in superficial text dependencies and potential implicit bias. |
| Approach: | They propose a parameter-efficient fine-tuning approach to mitigate implicit biases within foundation models by incorporating parameters into the model and freezing them during the fine-uning process. |
| Outcome: | The proposed method reduces biases in foundation models by incorporating parameters and freezing them during fine-tuning. |
Byte Pair Encoding for Symbolic Music (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on reducing the overall sequence length by merging embeddings or combining tokens. |
| Approach: | They propose to convert music into a sequence of discrete tokens using Byte Pair Encoding . they propose to combine tokens with embeddings to improve the results . |
| Outcome: | The proposed method reduces the sequence length while increasing the vocabulary size. |
Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP have led to the use of pre-trained Transformer models for transfer learning tasks becoming the most common way to solve target tasks. |
| Approach: | They propose a 3-phase technique to adjust a base model for a classification task by adapting the model’s signal to the data distribution and a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets. |
| Outcome: | The proposed method is compared with other methods and compares it with other approaches. |
Self-Influence Guided Data Reweighting for Language Model Pre-training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) pre-trained with selfsupervision on large text data are the default starting point for developing models for various downstream tasks. |
| Approach: | They propose a method for jointly reweighting samples by leveraging self-influence scores as an indicator of sample importance and pre-training. |
| Outcome: | The proposed method promotes novelty and stability for model pre-training. |
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation. |
| Approach: | They propose to use a multi-head model to learn from disagreements in an active learning setting to identify annotators with a single head. |
| Outcome: | The proposed model outperforms a single-head model in terms of uncertainty estimation and prediction while saving 70% of the annotation budget. |
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency are limited in their effectiveness. |
| Approach: | They propose a method for generating synthetic data by annotating diverse model-generated summaries using a LLM. |
| Outcome: | The proposed method outperforms state-of-the-art models and the LLM teacher on TRUE benchmarks. |
VivesDebate-Speech: A Corpus of Spoken Argumentation to Leverage Audio Features for Argument Mining (2023.emnlp-main)
Copied to clipboard
| Challenge: | a corpus of spoken argumentation is used to leverage audio features for argument mining tasks . a vast majority of arguments-based natural language processing resources only take text features into account . |
| Approach: | They describe a corpus of spoken argumentation created to leverage audio features for argument mining tasks. |
| Outcome: | The proposed corpus of spoken argumentation improves when integrating audio features into the argument mining pipeline. |
Tagging-Assisted Generation Model with Encoder and Decoder Supervision for Aspect Sentiment Triplet Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in the ASTE task have been driven by Natural Language Generation-based approaches, but most NLG methods overlook the supervision of the encoder-decoder hidden representations and fail to fully utilize the semantic information provided by the labels. |
| Approach: | They propose a tagging-assisted generation model with encoder and decoder supervision that enhances the supervision of the encoder-decoder through multiple-perspective tabbing assistance and label semantic representations. |
| Outcome: | The proposed model enhances the supervision of the encoder and decoder through multiple-perspective tagging assistance and label semantic representations. |
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language model probing is often used to test specific capabilities of models, but results are limited when benchmarks are small and lack statistical power. |
| Approach: | They extend existing NEG-136 and ROLE-88 benchmarks to 750 sentence pairs and create an extended negation dataset using template-based generation. |
| Outcome: | The proposed datasets show that models perform better on the extended datasets than on the smaller benchmarks. |
Norm of Word Embedding Encodes Information Gain (2023.emnlp-main)
Copied to clipboard
| Challenge: | Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? |
| Approach: | They propose to use the squared norm of static word embedding to encode the information gain conveyed by the word. |
| Outcome: | The proposed model can encode the information gain of a word in a language model or neural network. |
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) show powerful reasoning abilities on text-based tasks, but their reasoning capability on structured data such as tables has not been systematically explored. |
| Approach: | They first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis and then construct a complex reasoning QA dataset over tabular dataset. |
| Outcome: | The proposed method is able to solve table reasoning tasks without handcrafted demonstrations. |
Promoting Topic Coherence and Inter-Document Consorts in Multi-Document Summarization via Simplicial Complex and Sheaf Graph (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate summaries from multiple sources often lack accuracy and accuracy due to the length of tokens used in encoding. |
| Approach: | They propose a novel encoder-decoder model that uses pre-trained BART to analyze linguistic nuances, simplicial complex layer to apprehend inherent properties that transcend pairwise associations and sheaf graph attention to effectively capture heterophilic properties. |
| Outcome: | The proposed model achieves consistent performance improvement across all evaluation metrics (syntactical, semantical and faithfulness). |
MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a knowledge cutoff and are costly to finetune repeatedly. |
| Approach: | They introduce a language evaluation suite that incorporates diverse tokens and prompt settings to simulate real-world complexity. |
| Outcome: | The proposed evaluation suite incorporates diverse tokens and prompt settings to simulate real-world complexity. |
Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023.emnlp-main)
Copied to clipboard
| Challenge: | Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. |
| Approach: | They propose to fine-tune large language models to simulate how previous students would have responded to unseen items to generate high-quality parallel tests. |
| Outcome: | The proposed test forms are designed to be content-equivalent and produce identical individual scores as the original test form. |
Counter Turing Test (CT2): AI-Generated Text Detection is Not as Easy as You May Think - Introducing AI Detectability Index (ADI) (2023.emnlp-main)
Copied to clipboard
Megha Chakraborty, S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Shreya Gautam, Tanay Kumar, Krish Sharma, Niyar Barman, Chandan Gupta, Vinija Jain, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | a number of issues have arisen regarding the risk and consequences of AI-generated text detection. |
| Approach: | They propose a counter-turing test to evaluate the robustness of existing AGTD methods . they propose ADI, a quantifiable spectrum to assess detectability of LLMs . |
| Outcome: | The proposed method evaluates the robustness of existing AGTD methods . it shows that larger LLMs tend to have lower ADI, indicating they are less detectable . |
Revisiting the Optimality of Word Lengths (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing theories of communication cost are based on the length of an utterance, but they are not based upon the frequency of the utterant. |
| Approach: | They propose a method to minimize CCH's cost by comparing language lengths to surprisal's expectation and variance-to-mean ratios. |
| Outcome: | The proposed method does not minimize CCH’s cost, but rather a lower bound, which we term CCH-lower. |
Document-level Relationship Extraction by Bidirectional Constraints of Beta Rules (2023.emnlp-main)
Copied to clipboard
| Challenge: | Document-level Relation Extraction (DocRE) aims to extract relations among entity pairs in documents. |
| Approach: | They propose a logic constraint framework that uses bidirectional constraints to model rules by beta contribtion and reconstruct rule consistency loss by bidirectional constraint. |
| Outcome: | The proposed framework outperforms existing models in relation extraction performance and logical consistency. |
Instructed Language Models with Retrievers Are Powerful Entity Linkers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative approaches powered by large language models have demonstrated emergent abilities in tasks that require complex reasoning abilities. |
| Approach: | They propose a sequence-to-sequence training objective with instruction-tuning that enables casual language models to perform entity linking over knowledge bases. |
| Outcome: | The proposed framework outperforms existing approaches with +6.8 F1 points gain on average and huge advantage in training data efficiency and compute consumption. |
Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in vision-language learning have significantly advanced Human-Computer Interactions (HCI). |
| Approach: | They propose a method to align the semantic spaces between speech and text by incorporating two modules to align semantic spaces. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on AVOS benchmarks. |
PROSE: A Pronoun Omission Solution for Chinese-English Spoken Language Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pro-drop (‘pronoun-dropping’) language requires NMT systems to recover omitted pronouns, but this task lacks sufficient datasets for benchmarking . |
| Approach: | They propose a benchmarking method that leverages the semantic embedding of dropped pronouns to augment training pairs to alleviate the negative impact introduced by pro-drop . |
| Outcome: | The proposed method outperforms existing methods regarding omitted pronoun retrieval and overall translation quality on four Chinese-English translation corpora. |
A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why? (2023.emnlp-main)
Copied to clipboard
| Challenge: | a systematic framework to analyze the evolution of research topics in a scientific field is crucial for keeping abreast of its continuous advancement. |
| Approach: | They propose a framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. |
| Outcome: | The proposed framework uncovers evolutionary trends and causes for a wide range of NLP topics. |
Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work neither proves that pre-trained models successfully learn the injected factual knowledge nor proves there is a causal relation between injected knowledge and downstream performance improvements. |
| Approach: | They propose a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm. |
| Outcome: | The proposed framework shows that factual knowledge injection is successful but correctness of injected knowledge only has limited effect on the models’ downstream performance. |
Syntactic Substitutability as Unsupervised Dependency Syntax (2023.emnlp-main)
Copied to clipboard
| Challenge: | Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. |
| Approach: | They propose a method to induce syntactic dependencies theory-agnostically by substituting words from the same category for words at either end of a dependency. |
| Outcome: | The proposed method achieves 79.5% recall on long-distance subject-verb agreement constructions compared to 8.9% using a previous method. |
MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with knowledge bases or gazetteers and unlabeled corpus. |
| Approach: | They propose a noise-robust prototype network named MProto for a DS-NER task . they propose an optimal transport algorithm to mitigate the noise from incomplete labeling . |
| Outcome: | The proposed network achieves state-of-the-art on several DS-NER benchmarks. |
The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions (2023.emnlp-main)
Copied to clipboard
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Recent advances in large language models (LLMs) have produced models that exhibit remarkable performance across a variety of NLP tasks. |
| Approach: | They analyze a large-scale collection of user-GPT conversations to identify a significant gap between academic research in NLP and the needs of real-world NLP applications. |
| Outcome: | The proposed model outperforms existing models in a large-scale collection of user-GPT conversations and identifies a significant gap between the tasks that users frequently request from LLMs and the tasks commonly studied in academic research. |
Learning the Visualness of Text Using Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual text evokes an image in a person’s mind, while non-visual text fails to do so. |
| Approach: | They propose a method to automatically detect visualness in text to enable text-to-image retrieval and generation models to augment text with relevant images. |
| Outcome: | The proposed method performs better than several baseline models and heuristics for the task. |
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values (2023.emnlp-main)
Copied to clipboard
| Challenge: | Incorporating human feedback into Large Language Models is a welcome development, but it introduces new biases and challenges. |
| Approach: | They propose to survey 95 articles that use human feedback to steer, guide or tailor the behaviours of large language models. |
| Outcome: | The proposed approaches are based on 95 articles primarily from the ACL and arXiv repositories and highlight five unresolved conceptual and practical challenges. |
TempTabQA: Temporal Question Answering for Semi-Structured Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Semi-structured data often include temporal information about entities, either implicitly or explicitly. |
| Approach: | They present a dataset that includes 11,454 question-answer pairs from Wikipedia Infobox tables spanning more than 90 distinct domains. |
| Outcome: | The proposed dataset can be used as a benchmark to improve models for temporal reasoning on semi-structured tables. |
Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results prove the superiority of our proposed method on challenging classification tasks. |
| Approach: | They propose a task-level thinking step that eliminates bias introduced by demonstrations . they propose 'progressive revision framework' which can improve the thinking steps by correcting hard demonstrations. |
| Outcome: | The proposed method achieves best performance on three kinds of classification tasks in zero-shot and few-shot settings. |
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023.emnlp-main)
Copied to clipboard
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, Weizhu Chen
| Challenge: | RepoCoder is a repository-level code completion framework that utilizes the useful information scattered in files. |
| Approach: | They propose a repository-level code completion framework called RepoCoder . it integrates a similarity-based retriever and a pre-trained code language model . they propose 'repoBench' benchmark to validate the framework's effectiveness . |
| Outcome: | The proposed framework outperforms the vanilla retrieval-augmented code completion approach in the real-world. |
Influence Scores at Scale for Efficient Language Data Sampling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that ML models can be fine-tuned on as much data as possible without degradation in performance metrics. |
| Approach: | They evaluate the applicability of influence scores in language classification tasks by random sampling and stress-testing one of the scores. |
| Outcome: | The proposed model can be fine-tuned on 50% of the original data without degradation in performance metrics. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
Learning Retrieval Augmentation for Personalized Dialogue Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Personalized dialogue generation is a popular approach for conversational AI applications . however, persona profiles may not provide comprehensive descriptions of the persona . |
| Approach: | They propose a method that leverages persona profiles and dialogue context to generate personalized dialogues by leveraging personas and persona profile. |
| Outcome: | The proposed method outperforms baselines on the CONVAI2 dataset . it is expected to generate personalized dialogues based on persona profiles and dialogue context . |
The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive Remediations (2023.emnlp-main)
Copied to clipboard
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | Recent advances in Large Language Models have generated widespread acclaim, but hallucination has also emerged as a by-product. |
| Approach: | They propose a fine-grained discourse on profiling hallucination based on its degree, orientation, and category . they categorize hallucines into six types: acronym ambiguity, generated golem, virtual voice, geographic erratum, time wrap . |
| Outcome: | The proposed method categorizes hallucination into six types based on their degree, orientation, and category . |
NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural document rerankers require dedicated hardware for serving, which is costly and often not feasible. |
| Approach: | They propose a method that captures 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10-6% of . the model architecture is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. |
| Outcome: | The proposed model captures 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function. |
Analyzing Modular Approaches for Visual Question Decomposition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modular neural networks without additional training have been shown to surpass end-to-end neural networks on challenging vision–language tasks. |
| Approach: | They propose to use BLIP-2-based modular neural networks without additional training to build programs and a number of skill-specific, task-oriented modules to execute them. |
| Outcome: | The proposed methods outperform end-to-end neural networks on vision language tasks and retain performance when they use task-agnostic selections. |
Improving Summarization with Human Edits (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown the promise of learning with human feedback paradigms to produce human-determined high-quality text. |
| Approach: | They propose a novel technique to use both human-edited and model-generated data together in the training loop. |
| Outcome: | The proposed technique outperforms the conventional RLHF method (designed for human preferences) when applied to human-edit data. |
Did You Mean...? Confidence-based Trade-offs in Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | a calibrated model can help balance common trade-offs in task-oriented parsing. |
| Approach: | They propose a model which rephrases low-confidence inputs to improve usability and safety. |
| Outcome: | The proposed system reduces the number of incorrect low-confidence programs executed, but at a cost to usability. |
The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks. |
| Approach: | They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding. |
| Outcome: | The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models. |
Understanding the Effect of Model Compression on Social Bias in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text, leading to representational harm. |
| Approach: | They propose to use quantization and knowledge distillation to reduce the computational burden of LLMs to mitigate the effects of inappropriate social biases learned during pretraining. |
| Outcome: | The proposed methods reduce the computational burden of large language models by reducing their size and complexity. |
BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have impressive capabilities on a wide range of tasks, such as question answering and the generation of coherent text and code. |
| Approach: | They propose a framework for automatic evaluation of large language models on open-ended planning problems and a dataset of biology protocols with corresponding pseudocode representations. |
| Outcome: | The proposed framework evaluates an LLM on a dataset of biology protocols with corresponding pseudocode representations. |
Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for zero-shot CoT are limited to a single language, making it difficult to generalize to other languages and hindering global development. |
| Approach: | They introduce cross-lingual prompting (CLP) to improve zero-shot CoT reasoning across languages. |
| Outcome: | The proposed method outperforms existing prompting methods on several benchmarks. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
Boosting Summarization with Normalizing Flows and Aggressive Training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that FlowSUM improves the quality of generated summaries with minimal impact on inference time. |
| Approach: | They propose a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. |
| Outcome: | The proposed model improves the quality of generated summaries and reduces inference time. |
Indicative Summarization of Long Discussions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we generate indicative summaries instead of informative summary for long discussions. |
| Approach: | They propose an unsupervised approach to generating indicative summaries using large language models using large-scale language models. |
| Outcome: | The proposed method clusters argument sentences, generates abstractive summaries, and classifies the generated cluster labels into argumentation frames. |
A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for building multimodal open-domain dialogue agents based on large datasets are limited in real-world settings . |
| Approach: | They propose a new learning strategy called vision-language warm-up tasks for multimodal dialogue models that relies solely on learning from target data. |
| Outcome: | The proposed learning strategy achieves comparable and in some cases superior performance compared to existing state-of-the-art models on various evaluation metrics. |
Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based models perform cross-attention over input pairs, leading to computational cost. |
| Approach: | They propose a lightweight cross-attention mechanism that performs query encoding only once while modeling the query-candidate interaction in parallel. |
| Outcome: | The proposed model speeds up sentence pairing by over 113x while achieving comparable performance as the more expensive models. |
Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models (LLMs) have been used to prompt different reasoning thoughts, such as Chain of Thought and Program of Though. |
| Approach: | They propose a framework that prompts large language models with diverse reasoning thoughts by iterating between different prompting methods. |
| Outcome: | The proposed framework is able to generate multiple reasoning thoughts in 10 popular math reasoning datasets and is orthogonal to recent work that makes improvements on single reasoning methods and can generalise to logical reasoning domain. |
GLEN: General-Purpose Event Detection for Thousands of Types (2023.emnlp-main)
Copied to clipboard
| Challenge: | ACE 2005 2 is the first large-scale event extraction dataset with 205K event mentions and 3,465 different types. |
| Approach: | They propose to use the DWD Overlay to map PropBank rolesets to a large distantlysupervised training dataset with partial labels to make event extraction more accessible. |
| Outcome: | The proposed model performs better than baselines including InstructGPT and ACE 2005 2 despite being 18 years old . key limitations of ACE include its small event ontology of 33 types, small dataset size of around 600 documents and restricted domain (with a significant portion concentrated on military conflicts). |
Hierarchical Pretraining on Multimodal Electronic Health Records (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pretraining models on EHR data are too specific, limiting their transferability. |
| Approach: | They propose a general, unified pretraining framework for hierarchically multimodal EHR data that can be used to train models on a large dataset before fine-tuning it on 'upstream' tasks. |
| Outcome: | The proposed model performs on eight downstream tasks spanning three levels and compares with baselines on 18 different tasks. |
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination of text lacking grounding in input data is a problem in neural data-to-text generation. |
| Approach: | They propose to combine probabilistic output of a generator language model with the output of an “text critic” classifier which guides the generation by assessing the match between the input data and the generated text. |
| Outcome: | The proposed method improves on the WebNLG and OpenDialKG benchmarks. |
Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models require associated image with input sentence, which is difficult to satisfy at inference. |
| Approach: | They propose to use synthetic and authentic images to generate translations using text-to-image generation models. |
| Outcome: | The proposed model achieves state-of-the-art performance on En-De and En-Fr datasets while remaining independent of authentic images during inference. |
DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that pretrained language models memorize and regurgitate a significant portion of training data, including atypical data points that appear only once in the training data. |
| Approach: | They propose a method to locate and erase risky neurons in order to eliminate the impact of privacy data in the model in batches. |
| Outcome: | The proposed method eliminates the impact of privacy data in the model in batches without affecting the model's performance. |
Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques (2023.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing techniques that target sentence representations are being investigated in multilingual models . a growing interest in addressing bias detection and mitigation in NLP due to their societal implications. |
| Approach: | They examine the transferability of debiasing techniques across different languages within multilingual models by using a dataset from CrowS-Pairs. |
| Outcome: | The proposed techniques reduce bias in English, French, German, and Dutch by 13% . the authors also show that the techniques with additional pretraining exhibit enhanced cross-lingual effectiveness for the languages included in the analyses . |
Can Language Models Laugh at YouTube Short-form Videos? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that focus on verbal cues and focus on short-form funny videos focus on focusing on verbs and visual cue. |
| Approach: | They curate a user-generated dataset of 10K multimodal funny videos from YouTube and annotate each video with timestamps and explanations for funny moments. |
| Outcome: | The proposed dataset improves the ability of large language models to understand humor. |
Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learning on Knowledge Graphs (KGs) are not critical for learning on KGs. |
| Approach: | They propose an alternative approach to represent entities by composing entity-corresponding codewords matched from predefined small-scale codebooks. |
| Outcome: | The proposed approach achieves similar results to existing methods. |
Exploring All-In-One Knowledge Distillation Framework for Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods only obtain one lightweight student each time . this could be resource-intensive and resulting in multiple students not being optimally utilized . |
| Approach: | They propose a knowledge distillation framework which generates multiple satisfactory students at once. |
| Outcome: | The proposed framework generates multiple satisfactory students at once. |
HistAlign: Improving Context Dependency in Language Generation by Aligning with History (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) can generate hallucinations and incoherent outputs due to their weak context dependency. |
| Approach: | They propose a training approach to ensure good cache alignment so that the model receives useful signals from the history. |
| Outcome: | The proposed approach improves text coherence and faithfulness on diverse language generation tasks. |
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters. |
| Approach: | They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set. |
| Outcome: | The proposed method improves performance in all cases, while using a domain expert 23x smaller. |
Image Manipulation via Multi-Hop Instructions - A New Dataset and Weakly-Supervised Neuro-Symbolic Approach (2023.emnlp-main)
Copied to clipboard
Harman Singh, Poorva Garg, Mohit Gupta, Kevin Shah, Ashish Goswami, Satyam Modi, Arnab Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla
| Challenge: | Recent studies have shown that neuro-symbolic models lack interpretability and are not robust to noise. |
| Approach: | They propose to extend Neuro Symbolic Concept Learning (NSCL) which has been quite effective for the task of Visual Question Answering (VQA) they create a new dataset for the image manipulation task and demonstrate that NeuroSIM is highly competitive with or beats SOTA baselines that make use of supervised data for manipulation. |
| Outcome: | The proposed system performs complex multi-hop reasoning over multi-object scenes and only requires weak supervision in the form of annotated data for VQA. |
Generative Spoken Language Model based on continuous word-sized audio tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-based language models outperform character-based models, but speech inputs are 20ms or 40ms-long discrete units. |
| Approach: | They propose a generative language model based on word-size continuous audio tokens . they replace lookup table for lexical types with a Lexical Embedding function . |
| Outcome: | The proposed model is five times more memory efficient than discrete unit GSLMs and is phonetically and semantically interpretable. |
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study validates the effectiveness of chat language models by fine-tuning instruction data. |
| Approach: | They propose to use a large-scale dataset of instructional conversations to fine-tune a conversational model on instruction data. |
| Outcome: | The proposed model outperforms open-source models in key metrics including scale, average length, diversity, coherence, etc. |
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. |
| Approach: | They propose two approaches to contextualise visual entities in a multimodal setup by using verbalised scene graphs and masked relation prediction. |
| Outcome: | The proposed models can learn better representations from weakly-supervised relations data. |
Unsupervised Grammatical Error Correction Rivaling Supervised Methods (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current state-of-the-art grammatical error correction systems rely on labeled data . current systems require manual correction and require a large quantity of labeles . |
| Approach: | They propose an unsupervised method to build a grammatical error correction system using a fixer and a critic. |
| Outcome: | The proposed system outperforms previous unsupervised systems on English and Chinese GEC. |
S2abEL: A Dataset for Entity Linking from Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking (EL) is a longstanding problem in natural language processing and information extraction. |
| Approach: | They propose a neural baseline method for EL on scientific tables containing many out-of-knowledge-base mentions and a method that significantly outperforms a generic table EL method. |
| Outcome: | The proposed method significantly outperforms state-of-the-art generic table EL method on scientific tables with many out-of knowledge-base mentions. |
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2023.emnlp-main)
Copied to clipboard
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li
| Challenge: | Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. |
| Approach: | They propose a runnable evaluation system consisting of 73 API tools and an annotation system for 314 tool-use dialogues with 753 API calls. |
| Outcome: | The proposed benchmark assesses the effectiveness of existing LLMs by analyzing 314 tool-use dialogues with 753 API calls. |
Language and Mental Health: Measures of Emotion Dynamics from Text as Linguistic Biosocial Markers (2023.emnlp-main)
Copied to clipboard
| Challenge: | valence variability was significantly lower in the control group compared to ADHD, depression, bipolar disorder, MDD, PTSD, and OCD but not PPD. |
| Approach: | They study the relationship between tweet emotion dynamics and mental health disorders by using a user-disclosed diagnosis. |
| Outcome: | The results show that the measures varied by the user's self-disclosed diagnosis. |
Lion: Adversarial Distillation of Proprietary Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from a proprietary large language model to an open-source LLM have been criticized for not incorporating any feedback to boost student model's proficiency iteratively. |
| Approach: | They propose a novel adversarial distillation framework that integrates feedback to identify challenging instructions where the student model’s performance falls short. |
| Outcome: | The proposed framework surpasses existing state-of-the-art LLMs by 55.4% in challenging zero-shot reasoning benchmarks and 16.7% on AGIEval. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. |
| Approach: | They propose a methodology to study the soundness of Social-IQ by applying simple perturbations to a dataset of multiple choice questions on videos of complex social interactions. |
| Outcome: | The proposed method reduces biases in the original dataset and improves performance. |
Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence that LLMs hallucinate is growing, but there is no evidence to support this claim. |
| Approach: | They propose a method that constrains LLMs to produce output that satisfies evidential closure. |
| Outcome: | The proposed procedure yields faithful output by rejecting output that is not synonymous with claims for which the LLM has evidence. |
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | snippets are not meant to be read outside their original document. |
| Approach: | They propose a framework that decomposes the task into three stages: question generation, question answering, and rewriting. |
| Outcome: | The proposed framework decomposes the task into three stages: question generation, question answering, and rewriting. |
SLOG: A Structural Generalization Benchmark for Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization benchmarks focus on lexical generalisation, the interpretation of novel lexicals in syntactic structures familiar from training. |
| Approach: | They propose a semantic parsing dataset that extends COGS with 17 structural generalization cases to evaluate how well models generalize to new complex linguistic expressions. |
| Outcome: | The proposed model generalization accuracy is far below the near-perfect accuracy of existing models on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
Can LLMs Facilitate Interpretation of Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to uncover knowledge encoded within pre-trained language models are limited in terms of scalability and scope of interpretation. |
| Approach: | They propose to use a large language model, ChatGPT, as an annotation tool . they demonstrate that ChatGPt produces accurate and semantically richer annotations . |
| Outcome: | The proposed method produces accurate and semantically richer annotations compared to human annotations. |
Enhancing Low-resource Fine-grained Named Entity Recognition by Leveraging Coarse-grained Datasets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) often suffers from insufficient labeled data when the number of annotations exceeds several tens of labels. |
| Approach: | They propose a model with a fine-to- coarse mapping matrix to leverage hierarchical structure explicitly. |
| Outcome: | The proposed model outperforms both K-shot learning and supervised learning methods when dealing with a small number of fine-grained annotations. |
Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) require vast datasets for pretraining, making it difficult to train LLMs from scratch for lowresource languages. |
| Approach: | They propose to transform a language of the GLUE benchmark and then fine tune a pretrained model on that dataset. |
| Outcome: | The proposed models recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. |
Non-Autoregressive Math Word Problem Solver with Unified Tree Structure (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing MWP solvers do not handle variants that can be derived via mathematical manipulation. |
| Approach: | They propose a non-autoregressive solver to present a solution expression and decode it from a given problem description. |
| Outcome: | The proposed solver is able to decode multiple expression variants and correct them . it is based on a unified tree structure and is available on Math23K and MAWPS. |
Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Singing Voice Synthesis (SVS) synthesizes pleasing vocals based on music scores and lyrics . current acoustic models ignore the significance of local modeling within the sequence and the hard-to-synthesize parts in the predicted mel-spectrogram . |
| Approach: | They propose a method to enhance local modeling in the acoustic model by focusing on phoneme tokens located before and after the phoneme. |
| Outcome: | The proposed method improves local modeling in the acoustic model by focusing on the hard-to-synthesize parts of the predicted mel-spectrogram. |
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems (2023.emnlp-main)
Copied to clipboard
Navita Goyal, Eleftheria Briakou, Amanda Liu, Connor Baumler, Claire Bonial, Jeffrey Micher, Clare Voss, Marine Carpuat, Hal Daumé III
| Challenge: | Existing NLP systems can only access the retrieved context to determine the answer, resulting in a knowledge gap between the information that is required to answer the question and the information available to assess the model’s correctness. |
| Approach: | They ask whether adding relevant background helps mitigate users’ over-reliance on predictions. |
| Outcome: | The proposed approach reduces over-reliance on model predictions even in the absence of sufficient information to assess their correctness. |
GROOViST: A Metric for Grounding Objects in Visual Storytelling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Several evaluation metrics for visual storytelling do not consider images at all . authors propose a novel evaluation tool that accounts for cross-modal dependencies and temporal misalignments . |
| Approach: | They propose a visual storytelling evaluation tool that evaluates visual grounding . they use cross-modal dependencies, temporal misalignments and human intuitions . |
| Outcome: | The proposed evaluation tool accounts for cross-modal dependencies, temporal misalignments and human intuitions on visual grounding. |
VIBE: Topic-Driven Temporal Adaptation for Twitter Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language features are evolving in real-world social media, resulting in deteriorating performance of text classification. |
| Approach: | They propose a model that allows models to adapt to shifted data via latent topic evolution . they use two information bottleneck regularizers to distinguish past and future topics . |
| Outcome: | The proposed model outperforms state-of-the-art models on Twitter on three tasks with 3% of data. |
TOD-Flow: Modeling the Structure of Task-Oriented Dialogues (2023.emnlp-main)
Copied to clipboard
Sungryull Sohn, Yiwei Lyu, Anthony Liu, Lajanugen Logeswaran, Dong-Ki Kim, Dongsub Shim, Honglak Lee
| Challenge: | Recent advances in task-oriented dialogue systems have limitations regarding transparency and controllability. |
| Approach: | They propose to infer the TOD-flow graph from dialog data annotated with dialog acts and integrate it with any dialogue model to improve its prediction performance, transparency, and controllability. |
| Outcome: | The proposed approach improves dialog act classification and response generation performance in the MultiWOZ and SGD benchmarks. |
TopWORDS-Poetry: Simultaneous Text Segmentation and Word Discovery for Classical Chinese Poetry via Bayesian Inference (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental studies confirm that TopWORDS-Poetry can successfully segment poetry words without pre-given vocabulary or training corpus. |
| Approach: | They propose an unsupervised method that can achieve reliable text segmentation and word discovery for classical Chinese poetry simultaneously without pre-given vocabulary or training corpus. |
| Outcome: | Experimental results show that TopWORDS-Poetry can segment poetry lines into meaningful words with high quality without pre-given vocabulary or training corpus. |
Knowledge Rumination for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models lack the capacity to handle knowledge-intensive tasks alone. |
| Approach: | They propose a new paradigm to help pre-trained language models utilize latent knowledge without retrieving it from external corpus. |
| Outcome: | The proposed paradigm can be applied to pre-trained language models without retrieving external knowledge from the corpus. |
Struct-XLM: A Structure Discovery Multilingual Language Model for Enhancing Cross-lingual Transfer through Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods require syntactic labels that are difficult to obtain and of poor quality for low-resource languages. |
| Approach: | They propose a syntactic alignment model that leverages reinforcement learning to discover universal syntaktic structures for cross-lingual PLM alignment. |
| Outcome: | The proposed model improves cross-lingual representation alignment on the XTREME benchmark. |
AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. |
| Approach: | They propose a domain-specialization approach that decouples SEPT from DAPT by training a SEPT adapter on a pre-trained PLM. |
| Outcome: | The proposed approach matches or surpasses the performance of full SEPT on DAPT-ed PLMs while significantly reducing training costs. |
Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for CQA use pre-collected human-human conversations . previous methods use model-predicted dialogue history instead of ground truth . |
| Approach: | They propose an automatic evaluation approach that uses the model's dialogue history to evaluate models. |
| Outcome: | The proposed method improves on existing models and their evaluations on QuAC and CoQA. |
TCFLE-8: a Corpus of Learner Written Productions for French as a Foreign Language and its Application to Automated Essay Scoring (2023.emnlp-main)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) aims to automatically assess the quality of essays. |
| Approach: | They propose to use a corpus of 6.5k essays collected in the context of the Test de Connaissance du Français (TCF) certification exam to foster the development of AES for French. |
| Outcome: | The proposed system can assess the quality of essays in a language certification exam using a corpus of 6.5k essays collected in the TCFLE-8 exam. |
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional human evaluation methods for text simplification often relies on individual, shallow sentence-level ratings, easily affected by the annotator's preference or bias. |
| Approach: | They propose an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. |
| Outcome: | The proposed framework is able to predict sentence- and word-level quality simultaneously and report promising results. |
Confidence-based Ensembling of Perspective-aware Models (2023.emnlp-main)
Copied to clipboard
Silvia Casola, Soda Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, Cristina Bosco
| Challenge: | Human label variability has been a topic of research in the field of NLP recently . Exploiting disagreements in annotations has been shown to offer advantages for accurate modelling and fairer evaluation. |
| Approach: | They propose a highly perspectivist model that exploits disagreements in annotations to capture the subjectivity encoded in the annotation process. |
| Outcome: | The proposed model is validated on irony and hate speech detection scenarios in in-domain and cross-domain settings. |
ToViLaG: Your Visual-Language Generative Model is Also An Evildoer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale Visual-Language Generative Models (VLGMs) generate toxic content, e.g., offensive text and pornography images, raising significant ethical risks. |
| Approach: | They propose a bottleneck-based detoxification method to reduce toxicity while maintaining comparable generation quality. |
| Outcome: | The proposed method could reduce toxicity while maintaining comparable generation quality. |
GPT-RE: In-context Learning for Relation Extraction using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to in-context learning (ICL) are lacking in relation extraction (RE) . emergence of large language models (LLMs) such as GPT-3 represents a significant advancement in natural language processing. |
| Approach: | They propose to incorporate task-aware representations into demonstration retrieval and enrich the demonstrations with gold label-induced reasoning logic. |
| Outcome: | The proposed model achieves SOTA and competitive performances on the Semeval and SciERC datasets. |
Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current research on developing computational models of social norms has focused on American society. |
| Approach: | They propose to leverage a Chinese Q&A platform and a socialchiemistry dataset as proxies for contrasting cultural axes and align social situations cross-culturally. |
| Outcome: | The proposed model can reason across cultures using a Chinese Q&A platform and the existing socialChemistry dataset. |
INFORM : Information eNtropy based multi-step reasoning FOR large language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance with dedicated Chain-of-Thought (CoT) prompts. |
| Approach: | They propose a new method by introducing information entropy as a criteria on for CoT prompt selection. |
| Outcome: | The proposed model outperforms existing models on seven reasoning benchmarks using two language models. |
Adaptive Gating in Mixture-of-Experts based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models employ a fixed gating network where each token is computed by the same number of experts. |
| Approach: | They propose a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. |
| Outcome: | The proposed model reduces training time and inference quality while maintaining sparsity while maintaining inference accuracy. |
On the Automatic Generation and Simplification of Children’s Stories (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have made it possible to generate children's educational texts with appropriate lexical and readability levels. |
| Approach: | They first examine the ability of several popular LLMs to generate stories with properly adjusted lexical and readability levels. |
| Outcome: | The proposed models can generalize to the domain of children's stories and create an efficient pipeline for their automatic generation. |
When Do Decompositions Help for Machine Reading? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on decompositions of complex questions has focused on multi-step reasoning . but, in machine reading, it is unclear when decomposing is helpful . |
| Approach: | They conduct experiments on decompositions in machine reading to unify recent work . they find that decomposing complex questions can be helpful in zero or limited-data settings . |
| Outcome: | The proposed model can learn decompositions implicitly even with limited data, the study shows . the results are consistent with previous work on decomposing complex questions . |
The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been shown to possess impressive capabilities, but they are not problem-free. |
| Approach: | They explore the behavior of large language models when presented with (un)answerable queries. |
| Outcome: | The proposed models encode the answerability of an input query, the authors show . they also show that the first decoded token is a strong indicator . |
Identifying Informational Sources in News Articles (2023.emnlp-main)
Copied to clipboard
| Challenge: | Identifying sources of information in news articles is relevant to many tasks in NLP, including misinformation detection and argumentation. |
| Approach: | They propose a task to study compositionality of sources in news articles to understand how they are chosen to complement each other. |
| Outcome: | The proposed dataset can be used to train high-performing models for information detection and source attribution. |
Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel retrofitting method to induce emotion aspects into pre-trained language models is proposed . the models are computationally less expensive and open, but do not capture affective aspects of human communication well. |
| Approach: | They propose a retrofitting method to induce emotion aspects into pre-trained language models . they retrofit text fragments exhibiting similar emotions into pretrained networks . |
| Outcome: | The proposed method produces emotion-aware text representations for sentiment analysis and sarcasm detection tasks. |
Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing PLMs are infeasible for processing long documents due to computational costs and incomprehensive document understanding. |
| Approach: | They propose a retrieval model that models local semantics and global context semantics in a tightly-coupled manner. |
| Outcome: | The proposed model overcomes three core challenges of long document retrieval: substantial computational cost, incomprehensive document understanding, and scarce annotations. |
Revisiting De-Identification of Electronic Medical Records: Evaluation of Within- and Cross-Hospital Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | De-identification is a natural language processing task to detect and remove the protected health information (PHI) from electronic medical records (EMRs). |
| Approach: | They propose a de-identification dataset comprising EMRs from three hospitals in China and use it to establish a benchmark for evaluating both within- and cross-hospital generalization. |
| Outcome: | The proposed model with almost perfect within-hospital performance struggles when transferred across hospitals. |
Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent attempts at prompt decomposition toward solving complex, multi-step reasoning problems depend on the ability of the LLM to simultaneously decompose and solve the problem. |
| Approach: | They propose a decomposition generator that decomposes complex problems into subproblems that require fewer reasoning steps. |
| Outcome: | The proposed method can produce competitive or even better performance compared to its larger successor, GPT-4. |
Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual pretrained models can recall factual knowledge without additional fine-tuning. |
| Approach: | They propose two parameter-free language representation projection modules to transfer factual knowledge between English and non-English languages. |
| Outcome: | The proposed modules improve factual knowledge retrieval accuracy and transferability across diverse non-English languages. |
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract grammatical knowledge is key to linguistic generalization in humans . strong evidence for grammatikal abstraction in humans comes from structural priming . |
| Approach: | They compare human models of crosslingual structural priming to human models . they find evidence for abstract monolingual and crosslingual grammatical representations . |
| Outcome: | The results show that grammatical representations in multilingual models are similar to humans . the strongest evidence for grammatikal abstraction in humans comes from structural priming . |
ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph (2023.emnlp-main)
Copied to clipboard
| Challenge: | Question Answering over Knowledge Graph (KGQA) aims to find answer entities for natural language questions based on knowledge graphs. |
| Approach: | They propose a subgraph-aware self-attention mechanism to imitate the graph neural network (GNN) based module to perform multi-hop reasoning on KG. |
| Outcome: | The proposed method surpasses state-of-the-art models by a large margin even with fewer updated parameters and less training data. |
Deep Natural Language Feature Learning for Interpretable Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a small transformer language model, we can break down a complex task into a set of intermediary easier sub-tasks. |
| Approach: | They propose a method to break down a main task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
| Outcome: | The proposed method breaks down a complex task into a set of easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
ROBBIE: Robust Bias Evaluation of Large Generative Language Models (2023.emnlp-main)
Copied to clipboard
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Smith
| Challenge: | generative large language models (LLMs) are becoming more performant and prevalent . we need tools to measure and improve their fairness, authors say . |
| Approach: | They propose to compare 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
| Outcome: | The proposed model can be tested on more datasets to better characterize and mitigate biases . the study compared 6 prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
Enhancing Task-oriented Dialogue Systems with Generative Post-processing Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes a method to optimize pipelined dialogue systems by fine-tuning modules directly. |
| Approach: | They propose a new post-processing component for natural language generation (NLG) they use dialogue act contribution to evaluate contribution of GenPPN-generated utterances . |
| Outcome: | The proposed method improves the performance of task-oriented dialogue systems by modifying arbitrary modules including non-differentiable ones. |
Adapting Language Models to Compress Contexts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based language models have a finite context window and expensive computational cost of processing long text documents. |
| Approach: | They propose to adapt pre-trained LMs into AutoCompressors to compress text into summary vectors . authors propose to use summary vector to speed up inference over long contexts based on a finite context window . |
| Outcome: | The proposed model can compress long contexts into summary vectors, which are accessible as soft prompts. |
Selective Labeling: How to Radically Lower Data-Labeling Costs for Document Extraction Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a key bottleneck in developing automatic extraction models for visually rich documents is the cost of acquiring labeled documents. |
| Approach: | They propose selective labeling to provide "yes/no" labels for candidate extractions predicted by a model trained on partially labeled documents. |
| Outcome: | The proposed method reduces the cost of acquiring labeled data by 10 with a negligible loss in accuracy. |
TRAVEL: Tag-Aware Conversational FAQ Retrieval via Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods aim to fully utilize the dynamic conversation context to enhance the semantic association between the user query and FAQ questions, but they are limited by noise and e.g., users may click questions they don't like, leading to inaccurate semantics modeling. |
| Approach: | They propose to introduce tags of FAQ questions to reduce noise in the conversation context and integrate them into a reinforcement learning framework to minimize the negative impact of irrelevant information. |
| Outcome: | The proposed method can eliminate irrelevant information and minimize negative impact of irrelevant information in the dynamic conversation context. |
Continual Dialogue State Tracking via Example-Guided Question Answering (2023.emnlp-main)
Copied to clipboard
Hyundong Cho, Andrea Madotto, Zhaojiang Lin, Khyathi Chandu, Satwik Kottur, Jing Xu, Jonathan May, Chinnadhurai Sankar
| Challenge: | Dialogue systems are frequently updated to accommodate new services, but naively updating them by continually training with data for new services causes catastrophic forgetting. |
| Approach: | They propose to reformulate dialogue state tracking (DST) as a bundle of example-guided question answering tasks to minimize the task shift between services. |
| Outcome: | The proposed model achieves state-of-the-art performance on DST continual learning metrics without relying on any complex regularization or parameter expansion methods. |
Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media (2023.emnlp-main)
Copied to clipboard
| Challenge: | Claim span identification (CSI) is an important step in fact-checking pipelines . despite its importance to journalists and fact-seekers, it remains a understudied problem . |
| Approach: | They propose to use social media claims to identify text segments that contain a check-worthy claim or assertion in a social media post. |
| Outcome: | The proposed dataset outperforms other cross-lingual transfer methods on multiple languages. |
COVID-19 Vaccine Misinformation in Middle Income Countries (2023.emnlp-main)
Copied to clipboard
| Challenge: | a multilingual dataset of COVID-19 vaccine misinformation is available from Brazil, Indonesia, and Nigeria. |
| Approach: | They propose to use a multilingual dataset of COVID-19 vaccine misinformation from Brazil, Indonesia, and Nigeria to assess their relevance to vaccines and the presence of misinformation. |
| Outcome: | The proposed models improve from 2.7 to 15.9 percentage points in macro F1-score compared to baseline models. |
Contrastive Learning of Sentence Embeddings from Scratch (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings with unlabeled data are limited due to copyright restrictions, data distribution issues, and messy formats. |
| Approach: | They propose a contrastive learning framework that trains sentence embeddings with synthetic data. |
| Outcome: | The proposed framework produces positive and negative annotations given unlabeled sentences and generates sentences along with their corresponding annotations from scratch. |
A Rose by Any Other Name would not Smell as Sweet: Social Bias in Names Mistranslation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using demographics, we hypothesize that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated name. |
| Approach: | They propose a translation evaluation procedure based on round-trip translation of names that are demographically aligned and analyze the effect of name demographics on translation quality using generalized linear mixed effects models. |
| Outcome: | The proposed evaluation procedure is based on round-trip translation of names from a dataset of names that are demographically aligned and shows that the ability of translation systems to translate female-associated names is significantly lower than male-associated name. |
Investigating Efficiently Extending Transformers for Long Input Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained Transformer models have proven capable at tackling natural language tasks, but handling long sequence inputs still poses a significant challenge. |
| Approach: | They propose an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. |
| Outcome: | The proposed model achieves strong performance on long input summarization tasks comparable with much larger models. |
CS2W: A Chinese Spoken-to-Written Style Conversion Dataset with Multiple Conversion Types (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets focus on a single type of spoken style, such as disfluencies. |
| Approach: | They propose a Chinese Spoken-to-Written style conversion dataset with 7,237 spoken sentences extracted from transcribed conversational texts. |
| Outcome: | The proposed dataset covers four major conversion problems corresponding to the majority of spoken styles. |
Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to deal with resource scarcity have not been developed to deal effectively with the problem. |
| Approach: | They propose to use a set of tools to harness data from one or more high-resource "source" languages to compensate for a shortage of data in low-resourced "target" languages. |
| Outcome: | The proposed technique can be easily adapted to unseen languages, extending the range of the proposed technique and translation-based transfer more broadly. |
A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent instruction fine-tuned models can solve multiple NLP tasks when prompted to do so, but current research focuses on standard performance benchmarks, leaving compelling fairness and ethical considerations behind. |
| Approach: | They propose a method to mitigate gender bias in machine translation by using a corpus of machine translations from the WinoMT corpus. |
| Outcome: | The proposed model can solve multiple NLP tasks when prompted, but it lacks fairness and ethical considerations. |
DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing text mining models are fine-tuned by fine-timing a large pre-trained language model (PLM) in downstream tasks. |
| Approach: | They propose a semi-supervised learning framework for fine-tuning a cohort of small student models generated from a large pre-trained language model using knowledge distillation. |
| Outcome: | The proposed framework outperforms baseline models on semi-supervised text classification and extractive summarization tasks while maintaining comparable performance. |
Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for instruction tuning do not include associating instructions with existing datasets. |
| Approach: | They propose a dynamic growth paradigm for the automatic curation of instruction-tuning data . they use existing datasets to automatically construct instruction-uning datasets . |
| Outcome: | The proposed model reduces the API cost for generating instructions and provides high-quality data. |
Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of event processing do not understand the essentiality of step events towards a goal event. |
| Approach: | They propose to deconstruct a goal event into a discrete representation of finer-grained (step) events, which are not equally important to the goal. |
| Outcome: | The proposed model can understand the essentiality of different step events towards a goal event. |
Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handwritten mathematical expression recognition are limited by CFGs and pre-generated triplet data. |
| Approach: | They propose an architecture that integrates recognition and language features to output corrected sequences while optimizing with a string decoder recognition model. |
| Outcome: | The proposed architecture outperforms state-of-the-art methods on CROHME datasets. |
Vicinal Risk Minimization for Few-Shot Cross-lingual Transfer in Abusive Language Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for few-shot cross-lingual transfer learning are limited in target languages due to the scarcity of resources. |
| Approach: | They propose a method which interpolates pairs of instances based on the angle of their representations and propose augmentation methods to enhance few-shot cross-lingual abusive language detection. |
| Outcome: | The proposed method improves few-shot cross-lingual abusive language detection in seven languages typologically distinct from English and three different domains. |
SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Empirical studies show that supervised learning is extremely effective in in-domain datasets and models trained on SuperDialseg can achieve good generalization ability on out-of-domain data. |
| Approach: | They propose a supervised definition of dialogue segmentation points using document-grounded dialogues and a large-scale supervised dataset called SuperDialseg. |
| Outcome: | The proposed model can achieve good generalization ability on out-of-domain data. |
ATFormer: A Learned Performance Model with Transfer Learning Across Devices for Deep Learning Tensor Programs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Compilation-based methods with performance models have poor measurement accuracy and transferability between platforms. |
| Approach: | They propose a compiler that automatically generates tensors and automatically tunes them for different hardware platforms. |
| Outcome: | The proposed model reduces inference time and costs on modern DNN benchmarks. |
mRedditSum: A Multimodal Abstractive Summarization Dataset of Reddit Threads with Images (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization datasets do not cover multimodal discussions, multiple modalities, or both . mRedditSum consists of 3,033 discussion threads and images with human-written summaries. |
| Approach: | They propose a multimodal discussion summarization dataset that annotates 3,033 discussion threads with a human-written summary. |
| Outcome: | The proposed method outperforms existing models and serves as competitive baseline for future work. |
Sparse Low-rank Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained large language models in a parameter-efficient manner are gaining traction within the research community. |
| Approach: | They propose a method of low-rank adaptation that enables dynamic adjustments to the intrinsic rank during the adaptation process. |
| Outcome: | The proposed approach outperforms the current method with a fixed and unalterable intrinsic rank and a low-rank adaptation process. |
Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generating images with Text-to-Image models often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image. |
| Approach: | They compile a dataset of iterative interactions of human users with Midjourney and analyze the dynamics of the user prompts along these iterations. |
| Outcome: | The proposed model produces better images for a specific language style than other models. |
ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision (2023.emnlp-main)
Copied to clipboard
| Challenge: | A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs). |
| Approach: | They propose an algorithm which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LF. |
| Outcome: | The proposed algorithm denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LF. |
The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting relies on the initial decisions, causing errors in early steps to accumulate and impact the final answers. |
| Approach: | They propose a divide-and-conquer style algorithm that leverages large language models to raise and answer sub-questions until collecting enough information to tackle the original one. |
| Outcome: | The proposed algorithm is more robust to errors and errors than CoT prompting and Tree-of-Thought prompting methods. |
Ideology Takes Multiple Looks: A High-Quality Dataset for Multifaceted Ideology Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for the ID task only label a text as ideologically left- or right-leaning as a whole, regardless whether the text containing one or more different issues. |
| Approach: | They construct an ideological schema for a multifaceted ideology detection task using MITweet and an English Twitter dataset. |
| Outcome: | The proposed task uses a MITweet dataset with 12,594 English Twitter posts, each annotated with a Relevance and an Ideology label for all twelve facets. |
Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models (2023.emnlp-main)
Copied to clipboard
Pierre Colombo, Victor Pellegrain, Malik Boudiaf, Myriam Tami, Victor Storchan, Ismail Ayed, Pablo Piantanida
| Challenge: | Proprietary and closed APIs are impacting the practical applications of natural language processing. |
| Approach: | They propose a scenario where a pre-trained model is served through a gated API . they propose 'transductive inference' that leverages statistics of unlabelled data . |
| Outcome: | The proposed model performs a few-shot classification task with unlabelled data using a gated API . the proposed model can be used to perform the task with a handful of classes . |
MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)
Copied to clipboard
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation. |
| Approach: | They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field. |
| Outcome: | The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models. |
Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Mis- and disinformation online are a major source of harms of different kinds . out-of-context information is where different pieces of information are falsely associated . past studies have attempted to defend against OOC mis- and deinformation through external evidence, but they disregard the role of different pieces with different stances. |
| Approach: | They propose a stance extraction network that can extract stances of different pieces of evidence in a single framework. |
| Outcome: | The proposed model outperforms the state-of-the-art models on a public large-scale dataset with a performance gain of 3.2% in accuracy. |
Video-Helpful Multimodal Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal machine translation datasets contain images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity. |
| Approach: | They propose an MMT dataset that contains ambiguous subtitles and a video-helpful evaluation set. |
| Outcome: | The proposed model performs significantly better than existing models on ambiguous subtitles dataset . it is based on a training set and video-helpful evaluation set . |
Large Language Models are Temporal and Causal Reasoners for Video Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. |
| Approach: | They propose a framework that exploits linguistic shortcuts and mitigates 'linguistic bias' by flipping the source pair and target label to understand their complex relationships. |
| Outcome: | The proposed framework outperforms both LLMs-based and non-LLMs- based models on five challenging VideoQA benchmarks. |
Uncertainty Guided Global Memory Improves Multi-Hop Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformers are used to solve multi-hop question answering tasks that require reasoning over multiple parts of a long document. |
| Approach: | They propose a method that collects relevant information over the entire document and then combines it with local context to solve a multi-hop question answering task. |
| Outcome: | The proposed method improves on three MHQA datasets compared to the baseline model. |
Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for question generation over knowledge bases rely on annotated data for fine-tuning . emergence of Large Language Models (LLMs) has shown impressive generalization ability in few-shot tasks. |
| Approach: | They propose to use a logical form to generate a question in a reasoning problem . they propose to extend the prompting method into a method that can generate questions in logical forms . |
| Outcome: | The proposed method outperforms baselines on three public KBQG datasets. |
TrojanSQL: SQL Injection against Natural Language Interface to Database (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text-to-SQL systems have not investigated its security aspects . however, how to implement such attacks remains an open question. |
| Approach: | They propose a backdoor-based SQL injection framework for text-to-SQL systems that uses boolean-based injection and union-based injecting techniques to exploit SQL injection vulnerabilities. |
| Outcome: | The proposed framework can produce harmful SQL statements invalidating user queries or compromise sensitive information about the database. |
Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses a risk to personal privacy if exposed. |
| Approach: | They propose a novel unlearning approach that utilizes an efficient reinforcement learning feedback loop via proximal policy optimization to incentivize the LLMs to learn a paraphrasing policy to unlearn the pre-training data. |
| Outcome: | The proposed approach surpasses strong baselines and state-of-the-art methods in terms of its ability to generalize and strike a balance between privacy and LLM performance. |
MingOfficial: A Ming Official Career Dataset and a Historical Context-Aware Representation Learning Framework (2023.emnlp-main)
Copied to clipboard
You-Jun Chen, Hsin-Yi Hsieh, Yu Lin, Yingtao Tian, Bert Chan, Yu-Sin Liu, Yi-Hsuan Lin, Richard Tsai
| Challenge: | In Chinese studies, understanding the nuanced traits of historical figures can be challenging due to the need for domain expertise, specialist knowledge, and context-specific insights. |
| Approach: | They propose a large-scale multi-modal dataset for Chinese officials from the Ming Dynasty that integrates structured and text data to enable investigation of social structures. |
| Outcome: | The proposed dataset could enable exploratory analysis of official identities and significantly boost performance in tasks such as identifying nuance identities from 24.6% to 98.2% F1 score in hold-out test set. |
DPP-TTS: Diversifying prosodic features of speech via determinantal point processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep generative models have succeeded in synthesizing human-like speech. |
| Approach: | They propose a text-to-speech model with a prosody diversifying module that considers perceptual diversity in each sample and among multiple samples. |
| Outcome: | The proposed model generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time. |
Meta-Learning Online Adaptation of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode impressively broad world knowledge in their parameters, but the knowledge in static models becomes stale. |
| Approach: | They propose to use a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning to maximize the out-of-date model’s ability to answer questions about a document. |
| Outcome: | The proposed approach improves the model's ability to answer questions on streams of thousands of documents compared with fine-tuning and baseline heuristics. |
Self-Detoxifying Language Models via Toxification Reversal (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate toxic content in pretrained language models are resource-intensive and require additional components. |
| Approach: | They propose a method that enables the PLM itself to achieve "self-detoxification" they identify the toxification direction from the normal generation process to the one prompted with the negative prefix and then steer the generation to the reverse direction by manipulating the information movement within the attention layers. |
| Outcome: | The proposed method can achieve comparable performance with state-of-the-art methods without any fine-tuning or extra components. |
Interactive Text Generation (2023.emnlp-main)
Copied to clipboard
Felix Faltings, Michel Galley, Kianté Brantley, Baolin Peng, Weixin Cai, Yizhe Zhang, Jianfeng Gao, Bill Dolan
| Challenge: | Advances in generative modeling have made it possible to automatically generate high-quality texts, code, and images, but they can be unsatisfactory in many respects. |
| Approach: | They propose a task that allows training generation models interactively without the costs of involving real users. |
| Outcome: | The proposed model trains with Imitation Learning without the cost of involving real users and is superior to non-interactive models. |
Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a method for knowledge transfer from one model to another . recent studies suggest it is based on label smoothing, but it is not . |
| Approach: | They propose to compare the predictive confidences of models trained with knowledge distillation . they propose to use a method that is similar to label smoothing to train models . |
| Outcome: | Experiments on four text classification tasks show that knowledge distillation and label smoothing drive model confidence in opposite directions. |
Analyzing Cognitive Plausibility of Subword Tokenization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of subword tokenization focus on engineering criteria such as compression rate . a recent study evaluated subwords for their cognitive plausibility in languages with limited vocabulary size . |
| Approach: | They propose a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. |
| Outcome: | The proposed tokenization algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes than previous evaluations. |
POE: Process of Elimination for Multiple Choice Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current language models perform well on multiple choice reasoning tasks, but the options are not treated equally. |
| Approach: | They propose a two-step scoring method that scores options and masks them to make the final prediction from the remaining options. |
| Outcome: | The proposed method is especially performant on logical reasoning tasks. |
NeuSTIP: A Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal Knowledge Graphs (KGs) are factual information repositories where a fact is associated with a time interval. |
| Approach: | They propose a temporal NS model for knowledge graph completion that performs link prediction and time interval prediction in a TKG. |
| Outcome: | The proposed model shows competitive performance on link prediction and time prediction. |
Standardizing Distress Analysis: Emotion-Driven Distress Identification and Cause Extraction (DICE) in Multimodal Online Posts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for identifying hate speech have been limited to analyzing textual content. |
| Approach: | They propose a method for distress identification and cause extraction from social media posts using emotional information. |
| Outcome: | The proposed method improves F1 and ROS scores by 1.95% and 3% relative to the best-performing baseline. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the robustness of LLMs with few-shot prompting techniques are limited. |
| Approach: | They propose to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. |
| Outcome: | The proposed model is more sensitive to certain perturbations such as replacing words with synonyms and more robust to few-shot prompting methods. |
Can Large Language Models Capture Dissenting Human Voices? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. |
| Approach: | They evaluate the performance and alignment of large language models with humans using Monte Carlo Estimation and Log Probability Estimationic methods to estimate the multinomial distribution. |
| Outcome: | The proposed models fail to capture human disagreement distribution and inference and human alignment performance plunge even further on data samples with high disagreement levels raising concerns about their natural language understanding ability and representativeness to a larger human population. |
DecoMT: Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows the power of few-shot prompting with large language models for tasks like machine translation, summarization, and question answering. |
| Approach: | They propose a few-shot prompting approach that decomposes the translation process into word chunks. |
| Outcome: | The proposed approach outperforms established few-shot prompting models with 8 chrF++ scores across languages. |
Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are limited to low-data regimes and require learning different modules to adapt to diverse tasks. |
| Approach: | They propose a framework for parameter-efficient fine-tuning that trains modules per task . they use an instance-dense retriever and a prototypical hypernetwork to generate conditional modules . |
| Outcome: | The proposed framework outperforms existing methods on multi-task learning and few-shot transfer learning. |
Towards Building More Robust NER datasets: An Empirical Study on NER Dataset Bias from a Dataset Difficulty View (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models rely on superficial entity patterns for predictions, without considering evidence from the context. |
| Approach: | They propose to de-bias NER datasets by altering entity-context distribution . they also validate the feasibility of the proposed de-bianking techniques . |
| Outcome: | The proposed methods can be applied to different models and improve existing models. |
GradSim: Gradient-Based Language Grouping for Effective Multilingual Training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that not all languages positively influence each other . multilingual training can help in those cases by sharing knowledge across languages . |
| Approach: | They propose a gradient similarity-based language grouping method for multilingual training that is better correlated with cross-lingual model performance. |
| Outcome: | The proposed method leads to the largest performance gains on a multilingual dataset and is better correlated with cross-lingual model performance. |
Discovering Universal Geometry in Embeddings with ICA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on achieving sparse embeddings or acquiring semantic axes, but this study focuses on the intrinsic independence present within embeddables. |
| Approach: | They propose to use independent component analysis to extract independent semantic components from pre-trained embeddings by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis. |
| Outcome: | The proposed method reveals that embeddings can be expressed as a composition of a few interpretable axes and that these axe axe are consistent across languages, algorithms, and modalities. |
Toward a Critical Toponymy Framework for Named Entity Recognition: A Case Study of Airbnb in New York City (2023.emnlp-main)
Copied to clipboard
| Challenge: | Critical toponymy studies the dynamics of power, capital, and resistance through place names and the sites to which they refer. |
| Approach: | They propose a model that measures how cultural and economic capital shape the ways in which people refer to places through an annotated dataset of Airbnb listings in New York City. |
| Outcome: | The proposed model can identify important discourse categories integral to the characterization of place. |
Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge selection methods are costly to learn and difficult to interpret when errors arise in the generated responses. |
| Approach: | They propose a generator-agnostic knowledge selection method to select context-related knowledge among different knowledge structures and variable knowledge requirements. |
| Outcome: | The proposed method can select knowledge accurately in advance and reduce learning, adjustment, and interpretation burden of later models. |
Merging Generated and Retrieved Knowledge for Open-Domain QA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain question answering systems often have retrieval modules but retrieving passages from external knowledge sources is known to suffer from insufficient knowledge coverage. |
| Approach: | They propose a Compatibility-Oriented knowledge Merging framework to leverage both sources of information by matching LLM-generated passages with retrieved counterparts into compatible pairs. |
| Outcome: | The proposed framework outperforms baselines on three out of four tested open-domain QA benchmarks. |
Best of Both Worlds: Towards Improving Temporal Knowledge Base Question Answering via Targeted Fact Extraction (2023.emnlp-main)
Copied to clipboard
Nithish Kannen, Udit Sharma, Sumit Neelam, Dinesh Khandelwal, Shajith Ikbal, Hima Karanam, L Subramaniam
| Challenge: | Temporal question answering (QA) is a complex task that requires reasoning over facts asserting time intervals of events. |
| Approach: | They propose a temporal fact extraction technique that helps QA when it fails to retrieve temporal facts from the KB. |
| Outcome: | The proposed technique can extract temporal facts that failed to get retrieved from the KB without additional training cost. |
Text Fact Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing text style transfer models struggle with text fact transfer due to their inability to preserve the specificity and phrasing of the source text and tendency to hallucinate errors. |
| Approach: | They propose a task that seeks to transfer factual content between topics without changing its style. |
| Outcome: | The proposed framework can transfer factual content without sacrificing style without changing the style of the source text. |
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing diffusion models have limitations in modeling discrete data, e.g., languages . we present a novel diffusion model for language modeling inspired by linguistic features in languages based on iterative denoising . |
| Approach: | They propose a method that iteratively denoises and adds corruptions to the textual data through soft-masking to better noise it. |
| Outcome: | The proposed model achieves better generation quality and lower training cost than current models with better performance. |
Mirages. On Anthropomorphism in Dialogue Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Automated dialogue systems are anthropomorphised by developers and personified by users. |
| Approach: | They propose to examine linguistic factors that contribute to the anthropomorphism of dialogue systems and the harms that can arise thereof. |
| Outcome: | The proposed systems are anthropomorphised and personified by users . linguistic factors can also be used to reinforce gender stereotypes and conceptions of acceptable language. |
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past work has found that query–probe disagreements occur in situations where either probes or queries are uncertain. |
| Approach: | They propose to identify three different classes of disagreement, which they term confabulation, deception, and heterogeneity. |
| Outcome: | The proposed methods are more accurate than LM outputs and better on uncertain answers than queries and probes. |
KEBAP: Korean Error Explainable Benchmark Dataset for ASR and Post-processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional evaluation metrics for automatic speech recognition systems produce a singular aggregate score, which is insufficient for understanding specific system vulnerabilities. |
| Approach: | They propose to introduce the Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP) this method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
| Outcome: | The proposed method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
Adaptive Policy with Wait-k Model for Simultaneous Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous machine translation require a robust read/write policy . a standalone multi-path wait-k model performs competitively with adaptive policies . |
| Approach: | They propose a more flexible approach by decoupling the adaptive policy model from the translation model. |
| Outcome: | The proposed approach outperforms baseline approaches in translation tasks. |
Cross-Document Event Coreference Resolution on Discourse Structure (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that our proposed model outperforms several baselines and achieves the competitive performance with the start-of-the-art baselines. |
| Approach: | They propose to use discourse rhetorical structure constructor to construct tree structures to represent documents and a multi-layer perceptron to capture similarities of event mention pairs. |
| Outcome: | The proposed model outperforms baselines and achieves competitive performance with the start-of-the-art baselines. |
Post-hoc Utterance Refining Method by Entity Mining for Faithful Knowledge Grounded Conversations (2023.emnlp-main)
Copied to clipboard
Yoonna Jang, Suhyune Son, Jeongwoo Lee, Junyoung Son, Yuna Hur, Jungwoo Lim, Hyeonseok Moon, Kisu Yang, Heuiseok Lim
| Challenge: | Despite advances in language generation, models suffer from hallucinations that are either untrue or unfaithful to a given source. |
| Approach: | They propose a method to refine hallucinated utterances based on source knowledge . REM implicitly uses key entities in the knowledge to refine the utterant . |
| Outcome: | The proposed method reduces entity hallucination in the generated utterance and improves the quality of the model. |
Can We Edit Factual Knowledge by In-Context Learning? (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-context knowledge editing (IKE) is a new paradigm for NLP research that can be applied to large language models with tens or hundreds of parameters. |
| Approach: | They propose to use in-context knowledge editing (IKE) without gradient updating to edit factual knowledge without a gradient update. |
| Outcome: | The proposed method achieves a competitive success rate compared to gradient-based methods on GPT-J but with fewer side effects. |
EDIS: Entity-Driven Image Search over Multimodal Web Content (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing image retrieval methods require large datasets and a large candidate set. |
| Approach: | They propose a news-domain dataset for cross-modal image search with 1 million web images . they propose combining multimodal image-text pairs with a million candidates . |
| Outcome: | The proposed dataset challenges state-of-the-art methods with dense entities and the large-scale candidate set. |
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multi-query attention (MQA) can lead to quality degradation and training instability . it may not be feasible to train separate models optimized for quality and inference. |
| Approach: | They propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original training compute. |
| Outcome: | The proposed model achieves comparable quality to multi-head attention with comparable speed. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |
BiasX: “Thinking Slow” in Toxic Content Moderation with Explanations of Implied Social Biases (2023.emnlp-main)
Copied to clipboard
| Challenge: | Toxicity annotators and content moderators often default to mental shortcuts when making decisions, leading to subtle toxicity being missed and seemingly harmless content being over-detected. |
| Approach: | They propose a framework that provides AI-generated explanations of statements’ implied social biases to enhance content moderation setups. |
| Outcome: | The proposed framework significantly improves content moderation setups by enabling users to think more thoroughly about their decisions. |
Text encoders bottleneck compositionality in contrastive vision-language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal models are often unable to reason about simple spatial relations or attribute attachments. |
| Approach: | They first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture . then train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL model. |
| Outcome: | The proposed model can reconstruct captions from single-vector text representations produced by several models on a broader range of scenes compared to previous models. |
Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition (2023.emnlp-main)
Copied to clipboard
Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, Jordan Boyd-Graber
| Challenge: | Large Language Models are increasingly being deployed in interactive contexts that involve direct user engagement. |
| Approach: | They run a global prompt hacking competition to encourage research on prompt hacks . they elicit 600K+ adversarial prompts against three state-of-the-art LLMs based on a dataset . |
| Outcome: | The results of the competition show that current LLMs can be manipulated via prompt hacking . the competition elicits 600K+ adversarial prompts against three state-of-the-art LLM models . |
MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) based sparse architectures are prone to overfitting on low-resource language translation. |
| Approach: | They propose a modularized MNMT framework that flexibly assembles dense and MoE-based sparse modules to achieve the best of both worlds. |
| Outcome: | The proposed framework outperforms existing models on low-resource language translation and zero-shot translation on benchmark datasets. |
Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on active object grounding from an egocentric perspective is focusing on localizing and tracking active objects that undergo major state change as a result of human actions/interactions to the environment without being told exactly what/where to ground. |
| Approach: | They propose to use a narrated egocentric video dataset to localize and track active objects that undergo major state change as a result of human actions/interactions to the environment without being told exactly what/where to ground. |
| Outcome: | The proposed framework leads to 54% improvement in standard metrics on the TREK-150-OPE-Det localization + tracking task, and >7% improvement in all standard metrics. |
Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines (2023.emnlp-main)
Copied to clipboard
| Challenge: | Parallelism is a common stylistic tool in rhetorical structures, but it is rarely investigated in the field of natural language processing. |
| Approach: | They propose a task of rhetorical parallelism detection to investigate its structure and meaning . they use a Latin and adapted Chinese dataset to define parallelise and define it using a family of metrics . |
| Outcome: | The proposed method achieves F1 scores on Latin and Chinese datasets. |
Prompting is not a substitute for probability measurements in large language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompting is a dominant method for evaluating the linguistic knowledge of large language models (LLMs). |
| Approach: | They compare metalinguistic prompting and direct probability measurements as ways of measuring LLMs’ linguistic knowledge. |
| Outcome: | The results show that the results relying on metalinguistic prompts cannot be taken as conclusive evidence that an LLM lacks a particular linguistic generalization. |
Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques . data labeling is notoriously time-consuming and expensive, hindering the development of sizable labeled datasets . |
| Approach: | They propose to use active learning to reduce labeling costs by minimizing label complexity . they find PEFT adapter modules have significant potential in low-resource settings . |
| Outcome: | The proposed model outperforms FFT in low-resource settings and shows that it yields more stable representations of early and middle layers than FFT. |
Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Common NLP models are trained on data crawled from the internet, and it is difficult to audit at scale. |
| Approach: | They propose three strategies to prevent data contamination by encrypting test data and preventing it from being released on the internet. |
| Outcome: | The proposed strategies can make a difference in preventing data contamination. |
CoLT5: Faster Long-Range Transformers with Conditional Computation (2023.emnlp-main)
Copied to clipboard
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
| Challenge: | Many natural language processing tasks require long inputs, but processing long documents with a Transformer model is expensive due to quadratic attention complexity and applying feedforward and attention projection layers to every input token. |
| Approach: | They propose a long-input Transformer model that builds on the intuition that some tokens are more important than others and uses conditional computation to devote more computation to important tokens. |
| Outcome: | The proposed model achieves stronger performance than LongT5 with faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. |
DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to task-oriented conversation system DST use hand-crafted templates and additional slot information to fine-tune and prompt large pre-trained language models and elicit slot values from the dialogue context. |
| Approach: | They propose a generalizable in-context tuning approach that retrieves highly relevant training examples for a given dialogue to fine-tune the model without any hand-crafted templates. |
| Outcome: | Experiments with the MultiWOZ benchmark datasets show that DiSTRICT outperforms existing approaches in various zero-shot and few-shot settings using a much smaller model. |
Cross-Cultural Analysis of Human Values, Morals, and Biases in Folk Tales (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on folk tales focus on European tales, ignoring large swaths of the world's diverse cultures. |
| Approach: | They compile a corpus of over 1,900 folk tales originating from 27 diverse cultures across six continents and employ lexicon-based correlation analyses to examine human values, morals, and gender biases. |
| Outcome: | The results show that folk tales are influenced by cultural norms and cultural values and are well-known for their morals and values. |
Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using APEL, non-programmers can annotate natural language utterances with complex programs that represent their meaning. |
| Approach: | They introduce a framework in which non-programmers select among candidate programs generated by a seed semantic parser. |
| Outcome: | The proposed framework achieves the same annotation accuracy as the original expert annotators (75%) and exposes many subtle errors in the original annotations. |
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023.emnlp-main)
Copied to clipboard
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, Roger Levy
| Challenge: | Logical reasoning is an important task for artificial intelligence, says a new study . many prompting-based strategies to enable large language models fail in subtle and unpredictable ways. |
| Approach: | They propose to reformulate logical reasoning tasks by leveraging large language models . they use a modular neurosymbolic programming approach to translate premises and conclusions from natural language to logic . |
| Outcome: | The proposed approach outperforms open-source models on FOLIO and ProofWriter while showing distinct failure modes. |
Non-autoregressive Streaming Transformer for Simultaneous Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous machine translation models are trained to strike a balance between latency and translation quality. |
| Approach: | They propose a non-autoregressive streaming Transformer which generates blank tokens and decodes repetitive tokens to adjust its READ/WRITE strategy flexibly. |
| Outcome: | The proposed model outperforms previous strong autoregressive models on various benchmarks on siMT. |
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks. |
| Approach: | They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection . |
| Outcome: | The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters. |
RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to identify semantic relations between entities are time-consuming and labor-intensive. |
| Approach: | They propose a relation-aware prototype learning method for document-level relation extraction (FSDLRE) they propose RAPL, which judiciously leverages relation descriptions and real NOTA instances as guidance . |
| Outcome: | The proposed method outperforms state-of-the-art approaches by 2.61% F1 . it generates task-specific NOTA prototypes and refines relation prototypes . |
GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models do not utilize valuable geospatial information in large databases, e.g., OpenStreetMap. |
| Approach: | They propose a geospatially grounded language model that connects linguistic and geospheric contexts. |
| Outcome: | The proposed model bridges the gap between natural language processing and geospatial sciences. |
Cross-Modal Conceptualization in Bottleneck Models (2023.emnlp-main)
Copied to clipboard
Danis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov, Vladimir Ivanov, Alexey Kornaev, Ivan Titov
| Challenge: | Existing models that use text descriptions to predict labels are limited in their interpretations. |
| Approach: | They propose to use text descriptions to guide the induction of concepts in CBMs . they propose to employ a more moderate assumption and instead use text to guide induction . |
| Outcome: | The proposed model adopts a more moderate assumption and uses text descriptions to guide the induction of concepts. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
DREAM: Deployment of Recombination and Ensembles in Argument Mining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to Argument Mining (AM) take a holistic view of the overall pipeline. |
| Approach: | They propose a framework that allows for the (automated) combination of AM components instead of all-new solutions. |
| Outcome: | The proposed framework outperforms the best single systems in terms of accuracy measured by an AM benchmark. |
MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments (2023.emnlp-main)
Copied to clipboard
| Challenge: | In the context of the Indian judiciary, there is an additional complexity - Indian legal case judgments are mostly written in complex English due to historical reasons, but a significant portion of India's population lacks a strong command of the English language. |
| Approach: | They propose to summarize Indian legal case judgments in English and Hindi by combining the summaries of 3,122 case judgment from Indian courts into one dataset. |
| Outcome: | The proposed dataset compares the summarization methods with other datasets and shows that the proposed approaches perform better than previous approaches. |
Query Rewriting in Retrieval-Augmented Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on adapting either the retriever or the reader, but this approach is more focused on adaptation of the query itself. |
| Approach: | They propose a new framework for retrieval-augmented Large Language Models . they propose rewrite-retrieve-read instead of retrieve-then-read . |
| Outcome: | The proposed framework improves performance on downstream tasks, open-domain QA and multiple-choice QA. |
PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work often tackles the problem of text classification when there is a limited amount of training data. |
| Approach: | They propose a method to generate more helpful augmented data by utilizing the LLM's ability to follow instructions and perform few-shot classifications. |
| Outcome: | The proposed method generates more helpful examples near class boundaries, but generating borderline examples increases the risk of false positives in the dataset. |
COHESENTIA: A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts (2023.emnlp-main)
Copied to clipboard
| Challenge: | linguistics has been used to assess the coherence of generated texts . a benchmark of coherency scores is developed to measure the quality of generated text . |
| Approach: | They propose a benchmark to assess coherence of automatically generated texts . they use global and incremental methods to score sentences for coherency . |
| Outcome: | The proposed benchmark measures human-perceived coherence of automatically generated texts . it uses global and incremental scoring, and shows that the models are unsatisfactory . |
QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics poorly approximate parser quality, says a new study . questions under discussion is a linguistic framework that views discourse as asking questions and answering them . |
| Approach: | They propose a framework for automatic evaluation of QUD parsing . they use a dataset of fine-grained evaluation of 2,190 QUD questions . |
| Outcome: | The proposed framework shows that satisfying constraints of QUD is still challenging for modern LLMs. |
PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are too large to be fine-tuned with budget constraints and some are only accessible via APIs. |
| Approach: | They propose a pluggable Reward-Driven Contextual Adapter that integrates large language models as generators and trains them to refine the retrieved information. |
| Outcome: | The proposed method improves ReQA performance on three datasets by up to 20% compared to existing methods. |
Exploring Chain of Thought Style Prompting for Text-to-SQL (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks. |
| Approach: | They propose a new chain of thought prompting method that enhances LLMs’ reasoning ability through chain of thinking prompting, including the original chain-of-thought prompting and least-to-most prompting. |
| Outcome: | The proposed method brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gain, versus the least-to-most prompting. |
Efficient Algorithms for Recognizing Weighted Tree-Adjoining Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | a class of tree-adjoining languages can be characterized by various two-level formalisms controlled by semiring-weighted CFGs and PDAs. |
| Approach: | They propose semiring-weighted versions of controllable CFGs and PDAs . they also introduce a WPDA normal form that is analogous to Chomsky's normal form for CFG . |
| Outcome: | The proposed algorithms are more time-efficient than the previous ones for LIG, PAA, and EPDA. |
Harnessing Black-Box Control to Boost Commonsense in LM’s Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen remarkable progress in massively Pre-Trained Language Models such as GPT-3 . however, their generated outputs lack commonsense at times . |
| Approach: | They propose a framework that steers a frozen Pre-Trained Language Model towards more commonsense generation by training an auxiliary model. |
| Outcome: | The proposed framework produces plausible outputs that incorporate concepts in a meaningful way. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
Representative Demonstration Selection for In-Context Learning with Two-Stage Determinantal Point Process (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods tend to select different demonstrations for each test instance, which is time-consuming and poses limitations in practical scenarios. |
| Approach: | They propose to select a representative subset of in-context demonstrations that can prompt different test instances in a specific task. |
| Outcome: | The proposed method can be used to generate representative in-context demonstrations. |
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are useful interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. |
| Approach: | They evaluate the effectiveness of up-scaling and augmenting the LM with a passage retrieval database to reduce inconsistency. |
| Outcome: | The proposed models reduce inconsistency but retrieval augmentation is more efficient. |
ViPE: Visualise Pretty-much Everything (2023.emnlp-main)
Copied to clipboard
| Challenge: | Figure and non-literal expressions are deeply integrated in human communication . text-to-image models like Stable Diffusion struggle to depict non-figural expression . |
| Approach: | They propose a series of lightweight and robust language models that can be used to visualise non-literal expressions. |
| Outcome: | The proposed language models are more robust than existing models and can generate high-quality images. |
Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Document-level Relation Extraction (DocRE) is a task that aims to extract relations from a long context. |
| Approach: | They propose an automated annotation method that integrates an LLM and a natural language inference module to generate relation triples. |
| Outcome: | The proposed method can extract relations from document-level relation datasets with minimal human effort. |
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines how epistemic markers of certainty, uncertainty, or evidentiality affect LMs' attitudes toward knowledge . accuracies of expressions of high certainty and low certainty are associated with questions . prior work focused on learning the mapping between internal probabilities of a model and an ordinal output . |
| Approach: | They develop a typology of epistemic markers and inject 50 markers into questions . they find that LMs are highly sensitive to epistemical markers in prompts . |
| Outcome: | The proposed model epistemology study shows that LMs are sensitive to epistemic markers in prompts . expressions of high certainty result in 7% decrease in accuracy, while factive verbs hurt performance . |
Elaborative Simplification as Implicit Questions Under Discussion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Automated text simplification is often thought of as a monolingual translation task . this view fails to account for elaborative simplification, where new information is added into the simplified text. |
| Approach: | They propose to view elaborative simplification through the lens of the Question Under Discussion framework . they propose to model 1.3K elongations accompanied by implicit QUDs to investigate what writers elaborate upon . |
| Outcome: | The proposed framework provides a robust way to investigate what writers elaborate upon, how they elaborate, and how elaborations fit into the discourse context. |
EntSUMv2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity-centric summarization is a form of controllable summarizing that aims to generate a summary for a specific entity given a document. |
| Approach: | They propose to use a more abstract version of the original entity-centric ENTSUM summarization dataset to generate a shorter annotated summary for downstream users. |
| Outcome: | The proposed method is more abstract and uses supervised fine-tuning and large-scale instruction tuning to provide more specific and useful summaries for downstream users. |
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks. |
| Approach: | They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search. |
| Outcome: | The proposed model outperforms existing models by over 2 points absolute. |
A Diachronic Perspective on User Trust in AI under Uncertainty (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern NLP systems are rarely calibrated and are often confidently incorrect about their predictions, which violates users’ mental model and erodes their trust. |
| Approach: | They propose to use a mental model to bet on the correctness of an NLP system and to study how trust is rebuilt as a function of time after these events. |
| Outcome: | The proposed model shows that even a few highly inaccurate confidence estimation instances damage users’ trust in the system and performance, which does not easily recover over time. |
CT-GAT: Cross-Task Generative Adversarial Attack based on Transferability (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural network models are vulnerable to adversarial examples, and current methods based on adversarially transferable models rely on substitute models, which can be impractical and costly in real-world scenarios due to the unavailability of training data and the victim model’s structural details. |
| Approach: | They propose a novel approach that directly constructs adversarial examples by extracting transferable features across various tasks. |
| Outcome: | The proposed approach achieves superior attack performance with small cost on ten datasets and demonstrates that it is a novel approach. |
Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent supervised neural models have greatly promoted the development of topic segmentation, but the deeper relationship between coherence and topic segmenting is underexplored. |
| Approach: | They propose to use topic-aware Sentence Structure Prediction and Contrastive Semantic Similarity Learning to capture coherence from logical structure and semantic similarity perspectives to further improve topic segmentation performance. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on WIKI-727K and achieves an average relative reduction of 4.3% on Pk on WikiSection. |
Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents (2023.emnlp-main)
Copied to clipboard
Hyungjoo Chae, Yongho Song, Kai Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, Jinyoung Yeo
| Challenge: | a human-like chatbot requires commonsense reasoning to comprehend and respond to information . however, identifying and aggregating key evidence within a single hop is a challenge . a knowledge distillation framework is proposed that leverages LLMs as unreliable teachers . |
| Approach: | They propose a framework that leverages large language models as unreliable teachers to facilitate multi-hop reasoning over a dialogue context. |
| Outcome: | The proposed framework leverages LLMs as unreliable teachers and selectively distills consistent and helpful rationales via alignment filters. |
Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives (2023.emnlp-main)
Copied to clipboard
| Challenge: | 'information value' quantifies the predictability of an utterance relative to a set of plausible alternatives. |
| Approach: | They propose a method to obtain interpretable estimates of information value using neural text generators and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. |
| Outcome: | The proposed method is able to obtain interpretable estimates of information value using neural text generators and exploits their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. |
Generating Commonsense Counterfactuals for Stable Relation Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction struggle to identify causal terms under the invariant entity constraint. |
| Approach: | They propose a framework to generate commonsense counterfactuals for stable relation extraction by using a knowledge base WordNet and a constituency parser. |
| Outcome: | The proposed framework significantly enhances the stability of relation extraction models. |
C-STS: Conditional Semantic Textual Similarity (2023.emnlp-main)
Copied to clipboard
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, Karthik Narasimhan
| Challenge: | Semantic textual similarity (STS) is a cornerstone task in natural language processing, but it is inherently ambiguous. |
| Approach: | They propose a task called conditional STS which measures similarity conditioned on an aspect elucidated in natural language. |
| Outcome: | The proposed task reduces subjectivity and ambiguity and enables fine-grained similarity evaluation using diverse conditions. |
Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence of demographic bias in SA systems is limited to a handful of languages, and it is costly to create supervised training data in a new language. |
| Approach: | They use counterfactual evaluation to test whether gender or racial biases are imported when using cross-lingual transfer . r&r is much more prevalent than gender biase . |
| Outcome: | The proposed model is compared with monolingual systems in five languages and shows that it is biased more than monolingual ones. |
Rumor Detection on Social Media with Crowd Intelligence and ChatGPT-Assisted Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on rumor detection challenges the expressive power of text encoding sequences, and insufficient mining of semantic structural information. |
| Approach: | They propose a Crowd Intelligence-based semantic feature learning module to capture textual content’s sequential and hierarchical features and a knowledge-based structural mining module that leverages ChatGPT for knowledge enhancement. |
| Outcome: | The proposed system achieves performance improvement in rumor detection tasks validating the effectiveness and rationality of using large language models as auxiliary tools. |
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual illusions are a phenomenon that is often seen in human perception but are not always faithful to the physical world. |
| Approach: | They build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusion in state-of-the-art VLMs. |
| Outcome: | The proposed dataset reveals that larger models are closer to human perception and more susceptible to visual illusions. |
Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study (2023.emnlp-main)
Copied to clipboard
| Challenge: | a network of doppelganger websites (impersonating genuine news sites) was discovered in 2022 . a novel dataset enables studies of disinformation networks and the training of NLP tools for disinformation detection. |
| Approach: | They analyze two hitherto unstudied sites sharing state-backed disinformation . they perform cross-site topic clustering and perform linguistic and temporal analysis . |
| Outcome: | The proposed dataset includes 14,053 articles, annotated with each language version, and additional metadata such as links and images. |
Controllable Contrastive Generation for Multilingual Biomedical Entity Linking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual biomedical entity linking (MBEL) aims to map language-specific mentions in biomedically text to standardized concepts in a multilingual knowledge base (KB). |
| Approach: | They propose a prompt-based controllable contrastive generation framework for MBEL which summarizes multidimensional information of the UMLS concept mentioned in biomedical text into a natural sentence following a predefined template. |
| Outcome: | The proposed framework matches against UMLS concepts in as many languages and types as possible, thus facilitating cross-information disambiguation. |
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts (2023.emnlp-main)
Copied to clipboard
Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, Steven Hoi
| Challenge: | Recent studies suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. |
| Approach: | They propose a method that dynamically generates router parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. |
| Outcome: | Experiments on a wide range of tasks show that the proposed method performs better than existing methods. |
MediaHG: Rethinking Eye-catchy Features in Social Media Headline Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Creating a good headline on social media platforms requires a disentanglement-based model to balance the content and contextual features. |
| Approach: | They propose a disentanglement-based headline generation model which can balance the content and contextual features by incorporating contrastive learning and auxiliary multi-tasking to choose the best domain-suitable headline. |
| Outcome: | The proposed model can balance content and contextual features, while allowing bloggers to obtain more site traffic and profits while readers can have easier access to topics of interest. |
Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can answer many questions correctly, but can also hallucinate and give wrong answers. |
| Approach: | They propose a question-answering benchmark for Wikidata that uses SPARQL to ground large language models. |
| Outcome: | The proposed method outperforms the state-of-the-art for QALD-7 by 3.6% in F1 score. |
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs cannot generalize to domain-specific parsing tasks in a zero-shot setting. |
| Approach: | They propose a task-oriented parsing method that decomposes parse problem into abstractive and extractive question-answering problems. |
| Outcome: | The proposed method decomposes a parsing problem into abstractive and extractive question-answering (QA) problems. |
Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on using synthetic data for grammatical error correction . lack of annotated training data hinders progress in the field . |
| Approach: | They propose auxiliary tasks that exploit alignment between original and corrected sentences . they propose a sequence-to-sequence problem and perform multi-task training . |
| Outcome: | The proposed auxiliary tasks outperform the best models with a BART-based model on 11B parameters. |
The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language-and-vision models have impressive performance in downstream tasks, but it remains unclear whether this improves understanding of image-text interaction. |
| Approach: | They propose to use BLA to evaluate multimodal models on basic linguistic constructions that even preschool children can typically master. |
| Outcome: | The proposed model improves basic language skills in a zero-shot learning setting. |
RainProof: An Umbrella to Shield Text Generator from Out-Of-Distribution Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is a widely covered topic in classification tasks, but most methods rely on hidden features output by the encoder. |
| Approach: | They propose to leverage soft-probabilities in a black-box framework to detect OOD . they propose to use a more operational evaluation setting to enable OOD detection . |
| Outcome: | The proposed framework can access soft-predictions but not the internal states of the model. |
KEPL: Knowledge Enhanced Prompt Learning for Chinese Hypernym-Hyponym Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on hypernym-hyponym (“is-a”) relations is mostly in the English language. |
| Approach: | They propose a Knowledge Enhanced Prompt Learning method for Chinese hypernym-hyponym relation extraction using Hearst-like patterns as the prior knowledge. |
| Outcome: | The proposed method is able to extract hypernym-hyponym relations from Chinese unstructured texts using Hearst-like patterns and embed patterns and text simultaneously. |
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings (2023.emnlp-main)
Copied to clipboard
Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang
| Challenge: | Prior studies diagnose the anisotropy problem in sentence embeddings from pre-trained language models without fine-tuning. |
| Approach: | They propose an unsupervised method that weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. |
| Outcome: | Empirical evaluations show that the proposed method can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks. |
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on pairwise matching, ignoring robustness . current models exhibit frustrating degradation, with a maximum drop of 23.43 F1 score . |
| Approach: | They propose a benchmark that simulates the evaluation of open information extraction models in the real world . they perform experiments on typical models published in the last decade and a representative large language model . |
| Outcome: | The proposed model is rated robust on a knowledge-invariant clique with different syntactic and expressive forms. |
Why Should This Article Be Deleted? Transparent Stance Detection in Multilingual Wikipedia Editor Discussions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, only 20% of the English comments explicitly mention content moderation policies, but as few as 2% of the German and Turkish comments. |
| Approach: | They propose to use a multilingual dataset to predict stances with existing content moderation policies and to use them to explain moderation decisions. |
| Outcome: | The proposed model predicts stances and corresponding reasons with high accuracy, adding transparency to the decision-making process. |
Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to speed up inference latency have shown performance degradation caused by a state copying mechanism or numerous exit paths. |
| Approach: | They propose a framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. |
| Outcome: | The proposed framework outperforms existing frameworks on extensive generation tasks. |
End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions (2023.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end task-oriented dialogue (EToD) can generate responses in an end-to end fashion without modular training, which attracts escalating popularity. |
| Approach: | They present a systematic review of EToD and propose a unified perspective to summarize existing approaches and recent trends. |
| Outcome: | The proposed approaches can generate responses in an end-to-end fashion without modular training, which attracts escalating popularity. |
Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern systems for multi-hop question answering (QA) break questions into a sequence of reasoning steps, termed chain-of-thought (CoT) Often, multiple chains are sampled and aggregated, but the intermediate steps themselves are discarded. |
| Approach: | They propose a method which prompts large language models to meta-reason over multiple chains of thought rather than aggregate their answers. |
| Outcome: | The proposed approach outperforms baselines on 7 multi-hop QA datasets. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
Multi-level Contrastive Learning for Script-based Character Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scripts are written text for plays, movies, or broadcasts. |
| Approach: | They propose a multi-level contrastive learning framework to capture characters’ global information in a fine-grained manner. |
| Outcome: | The proposed framework improves on three character understanding sub-tasks by a considerable margin. |
CHEF in the Language Kitchen: A Generative Data Augmentation Leveraging Korean Morpheme Ingredients (2023.emnlp-main)
Copied to clipboard
| Challenge: | Korean morphological variations present unique opportunities and challenges in natural language processing (NLP), necessitating an advanced understanding of morpheme-based sentence construction. |
| Approach: | They propose a method to replicate morphological transformations inherent in Korean sentences based on lexical and functional morphemes through generative data augmentation. |
| Outcome: | The proposed method improves performance in Korean multiple classification datasets without incurring external data usage. |
Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for analyzing argumentative debates are insufficient to understand complex tasks. |
| Approach: | They propose a hybrid method to automatically predict the winning stance in argumentative debates using arguments from argumentation theory and semantics. |
| Outcome: | The proposed method is based on an unexplored new instance of the automatic analysis of natural language arguments. |
Transfer-Free Data-Efficient Multilingual Slot Labeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Slot labeling (SL) is a key component of task-oriented dialogue systems . extending the system to any new language-domain-task configuration requires expensive data annotation . |
| Approach: | They propose a two-stage slot labeling approach which transforms sentence encoders into effective slot labels. |
| Outcome: | The proposed approach is especially effective for the most challenging transfer-free few-shot setups. |
Towards Interpretable Mental Health Analysis with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models lack adequate evaluations and prompting strategies for explainability. |
| Approach: | They evaluate the mental health analysis and emotional reasoning ability of large language models (LLMs) using 11 datasets across 5 tasks. |
| Outcome: | The proposed model shows strong in-context learning ability but still has a significant gap with advanced task-specific methods. |
Learning to Rank Generation with Pairwise Partial Rewards (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for conditional text generation suffer from large action space and delayed reward, as the reward can be computed only after an entire sequence is generated. |
| Approach: | They propose a method that provides partial rewards for intermediate actions taken on partial sequences to prioritize actions that lead to the generation of more desirable sequences. |
| Outcome: | The proposed method overcomes the limitations of the prevalent supervised maximum likelihood estimation approach. |
GreedyCAS: Unsupervised Scientific Abstract Segmentation with Normalized Mutual Information (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstracts of scientific papers typically contain premises and conclusions, but in non-structured abstracts the concluding information is not marked. |
| Approach: | They propose to use Normalized Mutual Information (NMI) to optimize the NMI score between two segments by assuming that conclusions are strongly semantically linked with preceding premises. |
| Outcome: | The proposed approach outperforms baseline methods on structured abstracts and on non-structured abstracts. |
Spoiler Detection as Semantic Text Matching (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on spoiler detection shows promising results in safeguarding viewers from general spoilers, but it fails to address the issue of users abstaining from show-related content during their watch. |
| Approach: | They propose to use semantic text matching to assign an episode number to a spoiler given a specific TV show and a dataset to evaluate its performance. |
| Outcome: | The proposed dataset can be used to evaluate the performance of the proposed model and to compare it with other datasets. |
Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied task completion requires an agent to predict environment actions to complete tasks based on natural language instructions and egocentric visual observations. |
| Approach: | They propose a method to generate human-human dialogues and use them as training data for plan prediction. |
| Outcome: | The proposed model outperforms language-only models but falls short of oracle plans. |
GEM: Gestalt Enhanced Markup Language Model for Web Understanding via Render Tree (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use plain HTMLs do not include crucial visual information in the rendered web. |
| Approach: | They propose a Gestalt Enhanced Markup Language Model for hosting visual information without visual input. |
| Outcome: | The proposed model can handle multiple downstream tasks without visual input. |
Abstractive Open Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing OpenIE datasets and metrics are ill-suited for this task. |
| Approach: | They propose a new open-domain task that extends OpenIE to include inferred relations . they propose metric to evaluate the effectiveness of open-source OpenIE . |
| Outcome: | The proposed model can extract inferred relations from the extracted relation tuples. |
CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on detecting explicit hate speech has focused on indirect or coded language. |
| Approach: | They propose a context synergized neural network that integrates user- and conversational-contexts for detecting implicit hate speech in online conversations. |
| Outcome: | The proposed framework outperforms baselines on 6 hate speech datasets and shows that it is highly efficient. |
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating the performance of Grammatical Error Correction systems is a challenging task due to its subjectivity. |
| Approach: | They propose a method to evaluate GEC systems in multi-reference evaluation setting . they use consistent edit boundaries to eliminate bias caused by inconsistent edit boundaries . |
| Outcome: | The proposed evaluation metric eliminates bias caused by inconsistent edit boundaries on six English reference sets. |
Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods (2023.emnlp-main)
Copied to clipboard
| Challenge: | Feature attribution scores are used to explain the prediction of a text classifier to users by highlighting a k number of tokens. |
| Approach: | They propose to determine the number of optimal k tokens that should be displayed from sequential properties of attribution scores. |
| Outcome: | The proposed method is dynamic across sentences, method-agnostic, and deals with sentence length bias. |
SentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for online sentiment analysis rely on pre-existing datasets. |
| Approach: | They propose a co-training framework specifically designed for efficient sentiment analysis within dynamic data streams. |
| Outcome: | The proposed framework surpasses existing methods in terms of accuracy and computational efficiency. |
HyperNetwork-based Decoupling to Improve Model Generalization for Few-Shot Relation Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies cannot generalize well to unseen relations using Prototypical Networks . current approaches are dependent on large amount of labeled data and cannot deal with unseense relations well. |
| Approach: | They propose a HyperNetwork-based Decoupling approach to improve FSRE generalization . they propose FSre models with an encoder, network generator and refined classifiers . |
| Outcome: | The proposed method improves the generalization of few-shot relation extraction models. |
Solving Hard Analogy Questions with Relation Embedding Chains (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to model how concepts are related are incomplete and noisy. |
| Approach: | They propose to model relations as paths but associate their edges with relation embeddings. |
| Outcome: | The proposed representations are useful for solving hard analogy questions. |
Modeling Empathic Similarity in Personal Narratives (2023.emnlp-main)
Copied to clipboard
| Challenge: | EmpathicStories is a dataset of 1,500 personal stories annotated with empathic similarity features and 2,000 pairs of stories annnotated by empathism. |
| Approach: | They propose a task to identify similarity in personal stories based on empathic resonance . they use a dataset of 1,500 personal stories annotated with empathism features . |
| Outcome: | The proposed model outperforms semantic similarity models on correlation and retrieval metrics. |
Tree Prompting: Efficient Task Adaptation without Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are the main interface for applying them to new tasks, but their large size makes them difficult to fine-tune with gradients for specific downstream tasks. |
| Approach: | They propose to use training data to form a decision tree based on prompt-LM calls, with each prompt determined by the outcomes of previous calls. |
| Outcome: | The proposed method improves accuracy over competing methods and is competitive with fine-tuning. |
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the promising potential of chat models, they are only accessible through restricted APIs, creating barriers for new research and progress in the field. |
| Approach: | They propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself. |
| Outcome: | The proposed pipeline generates a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself, simulating both user and AI responses. |
Empathy Intent Drives Empathy Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Empathy is essential in human social interaction. |
| Approach: | They propose to annotate healthy empathy detection datasets IEMPATHIZE and TwittEmp with 8 empathy intent labels and perform joint training for the two tasks. |
| Outcome: | The proposed framework outperforms baselines on the two datasets. |
Adaptive End-to-End Metric Learning for Zero-Shot Cross-Domain Slot Filling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research on slot filling has witnessed considerable improvement with considerable data and label shifts. |
| Approach: | They propose an adaptive end-to-end metric learning scheme for zero-shot slot filling that uses context-aware soft label representations and slot-level contrastive representation learning to mitigate the data and label shift problems. |
| Outcome: | The proposed approach outperforms existing methods on public benchmarks and shows that it is simple, efficient and generalizable. |
BasahaCorpus: An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current research on automatic readability assessment (ARA) has focused on improving the performance of models in high-resource languages such as English. |
| Approach: | They propose a hierarchical cross-lingual modeling approach that takes advantage of a language’s placement in the family tree to increase the amount of available training data. |
| Outcome: | The proposed model improves the performance of models in high-resource languages such as English and Hiligaynon, minasbate, Karay-a, and Rinconada. |
ReTAG: Reasoning Aware Table to Analytic Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Table to text models generate descriptive summaries that repeat information contained within a table in sentences. |
| Approach: | They propose a table-aware table-to-text model that uses vector-quantization to infuse different types of analytical reasoning into the output. |
| Outcome: | The proposed model achieves 2.2%, 2.9% improvement on PARENT metric over state-of-the-art models. |
Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. |
| Approach: | They propose a COmpreheNsive kNowledge Evaluation framework to evaluate generated knowledge from six important perspectives . they conduct extensive empirical analysis of generated knowledge on two widely studied knowledge-intensive tasks . |
| Outcome: | The proposed framework evaluates generated knowledge from six important perspectives on two knowledge-intensive tasks. |
Compressing Context to Enhance Inference Efficiency of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across various tasks. |
| Approach: | They propose a method that prunes redundancies in the input context to make the input more compact. |
| Outcome: | The proposed method reduces memory and inference time while maintaining comparable performance compared to full context. |
MoT: Memory-of-Thought Enables ChatGPT to Self-Improve (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive abilities on various tasks. however, fundamentally improving them depends on high-quality datasets or computationally expensive fine-tuning. |
| Approach: | They propose a framework to let Large Language Models self-improve without annotated datasets or parameter updates. |
| Outcome: | The proposed framework can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference. |
4 and 7-bit Labeling for Projective and Non-Projective Dependency Trees (2023.emnlp-main)
Copied to clipboard
| Challenge: | encodings that map trees into sequences of one discrete label per word have been proposed for constituency parsing and dependency parse. |
| Approach: | They propose an encoding that can represent any projective dependency tree as a sequence of 4-bit labels, one per word. |
| Outcome: | The proposed encoding achieves substantial accuracy gains over the previously best-performing sequence labeling encoders. |
Can You Follow Me? Testing Situational Understanding for ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have identified certain SU limitations in non-chatbot Large Language models, but the extent and causes of these limitations are not well understood. |
| Approach: | They propose a synthetic environment for SU testing in chat-oriented models . they test models' ability to track and enumerate environment states . |
| Outcome: | The proposed environment allows for controlled and systematic testing of SU in chat-oriented models, and to better understand underlying causes for performance patterns. |
Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4 (2023.emnlp-main)
Copied to clipboard
Kellin Pelrine, Anne Imouza, Camille Thibault, Meilina Reksoprodjo, Caleb Gupta, Joel Christoph, Jean-François Godbout, Reihaneh Rabbany
| Challenge: | Misinformation is a critical societal challenge, and current approaches have yet to produce an effective solution. |
| Approach: | They propose to focus on generalization, uncertainty and how to leverage large language models . they propose techniques to handle uncertainty that can detect impossible examples and strongly improve outcomes . |
| Outcome: | The proposed tools outperform previous methods in multiple settings and languages. |
Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on grammatical error correction (GEC) in morphologically rich languages have been limited due to data scarcity and language complexity. |
| Approach: | They propose to use Arabic GEC to improve performance across three datasets . they define Arabic grammatical error detection task as auxiliary input . |
| Outcome: | The proposed models achieve SOTA results on two Arabic GEC shared task datasets and establish a strong benchmark on a recently created dataset. |
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) are prone to generate hallucinations . a recent study shows that LLMs are able to generate content that conflicts with the source or cannot be verified by factual knowledge. |
| Approach: | They propose a framework to evaluate the performance of large language models (LLMs) they propose to use a sample of generated and human-annotated hallucinated samples to evaluate their performance . |
| Outcome: | The proposed framework generates and annotates hallucinated samples from ChatGPT . the results show that existing LLMs face great challenges in recognizing hallucines . |
Enabling Large Language Models to Generate Text with Citations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work relies on commercial search engines and human evaluation, making it difficult to reproduce and compare different modeling approaches. |
| Approach: | They propose a new generation paradigm that requires large language models to provide citations to one or a few text passages for any statement they generate. |
| Outcome: | The proposed model improves factual correctness and verifiability of large language models by providing citations to a set of questions and retrieval corpora and generating answers with citation. |
Revisiting Machine Translation for Cross-lingual Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in cross-lingual learning has pivoted around multilingual models, which are typically pretrained on unlabeled corpora in multiple languages using some form of language modeling objective. |
| Approach: | They propose to use a stronger machine translation system to mitigat mismatch between training on original text and running inference on machine translated text. |
| Outcome: | The proposed approach is highly task dependent and calls into question the dominance of multilingual models for cross-lingual classification. |
Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for augmenting language models with human scanpaths have been developed, but the potential of synthetic gaze data across NLP tasks remains unexplored. |
| Approach: | They propose to combine synthetic scanpath generation with a scanpath-augmented language model, eliminating the need for human gaze data. |
| Outcome: | The proposed model outperforms the underlying language model and achieves comparable performance to a language model augmented with real human gaze data. |
Counting the Bugs in ChatGPT’s Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model (2023.emnlp-main)
Copied to clipboard
Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, David Mortensen
| Challenge: | Existing studies on large language models (LLMs) ignore the remarkable ability of humans to generalize and focus only on English. |
| Approach: | They conduct the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages. |
| Outcome: | The proposed model massively underperforms purpose-built systems, particularly in English. |
Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated their capability with few-shot inference . however, in-domain demonstrations are not always available in real scenarios . |
| Approach: | They propose unsupervised domain adaptation problem to adapt language models from source domain to target domain without any target labels. |
| Outcome: | The proposed model performs better than baseline models on Sentiment Analysis and Named Entity Recognition tasks. |
Understanding the Inner-workings of Language Models Through Representation Dissimilarity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dissimilarity measures measure the extent to which two model’s internal representations differ . they can identify and locate generalization properties of models that are invisible via in-distribution test set performance. |
| Approach: | They propose to use representation dissimilarity measures to measure the extent to which two model’s internal representations differ. |
| Outcome: | The proposed dissimilarity measures can identify and locate generalization properties of models that are invisible via in-distribution test set performance and new evaluations of how language model features vary as width and depth are increased. |
Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |
Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to task-oriented dialogue systems integrate knowledge retrieval and response generation, which poses scalability challenges when dealing with extensive knowledge bases. |
| Approach: | They propose a retriever-generator architecture that harnesses a retrieval and a generator to generate system responses by using feedback from the generator as pseudo-labels. |
| Outcome: | The proposed architecture shows superior performance on three benchmark datasets. |
Construction Artifacts in Metaphor Identification Datasets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing metaphor identification datasets can be gamed by completely ignoring the potential metaphorical expression or the context in which it occurs. |
| Approach: | They show that existing metaphor identification datasets can be gamed by fully ignoring the potential metaphorical expression or the context in which it occurs. |
| Outcome: | The proposed system can be gamed by fully ignoring the potential metaphorical expression or the context in which it occurs. |
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to enhance Language Models fail to address diverse error types . generic feedback is a bottleneck for addressing diverse errors in reasoning chains . |
| Approach: | They propose an iterative refinement framework that integrates multiple feedback modules . they propose to address errors in reasoning chains by integrating frozen LMs with external tools . |
| Outcome: | The proposed framework improves performance in Mathematical Reasoning and Logical Entailment by 20% and 18% respectively. |
Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for automatic Brain CT reports are limited by coarse-grained supervision and coupled cross-modal alignment. |
| Approach: | They propose a pathological Graph-driven cross-modal alignment model that learns fine-grained visual cues and aligns them with textual words. |
| Outcome: | The proposed model can improve the automatic generation of Brain CT reports and contribute to improved cranial disease diagnosis. |
Enhancing Structured Evidence Extraction for Fact Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain fact verification requires extracting and integrating both structured and unstructured evidence to verify a claim. |
| Approach: | They propose a method to enhance the extraction of structured evidence by leveraging the row and column semantics of tables. |
| Outcome: | The proposed method achieves evidence recall of 60.01% on the test set, higher than the previous state-of-the-art method. |
Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Keyphrase generation is a longstanding task in NLP with widespread applications. |
| Approach: | They propose a likelihood-based decode-select algorithm for seq2seq PLMs that improves greedy search by an average of 4.7% semantic F1 across five datasets. |
| Outcome: | The proposed algorithm improves greedy search by an average of 4.7% semantic F1 across five datasets. |
A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of entity linking systems often lack detailed error analysis or a closer look at the results. |
| Approach: | They evaluate existing entity linking systems and propose two new benchmarks . they characterize their strengths and weaknesses and report on reproducibility aspects . |
| Outcome: | The evaluations of existing system have strong biases and artifacts . they characterize their strengths and weaknesses and report on reproducibility aspects . |
A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research focuses on isolated dimensions of discourse coherence . Existing discourse cohesion analyses focus on isolated aspects of discourse . |
| Approach: | They introduce a Chinese Essay Discourse Coherence Corpus (CEDCC) which integrates coherence grading, topical continuity, and discourse relations. |
| Outcome: | The proposed dataset captures the subtleties of real-world texts and stimulates progress in Chinese discourse coherence analysis. |
SKD-NER: Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones. |
| Approach: | They propose a model that leverages knowledge distillation to retain memory and employs reinforcement learning strategies to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting. |
| Outcome: | The proposed model outperforms state-of-the-art models on two benchmark datasets showing that it significantly improves the performance of the CL-NER task. |
Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on maintaining old knowledge while paying little attention to knowledge transfer across tasks. |
| Approach: | They propose to train a model on a sequence of generation tasks to learn new generation patterns while avoiding the forgetting of previous knowledge. |
| Outcome: | The proposed model outperforms existing methods in different settings. |
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have questioned assumptions of majority vote aggregated labels. |
| Approach: | They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. |
| Outcome: | The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators. |
Lazy-k Decoding: Constrained Decoding for Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Specifically, we combine probabilistic models with constrained decoding approaches in structured prediction tasks. |
| Approach: | They propose a constrained decoding method called Lazy-k to combine probabilistic models with constrained methods in structured prediction. |
| Outcome: | The proposed method allows for more flexibility between decoding time and accuracy. |
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns. |
| Approach: | They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model. |
| Outcome: | The proposed model outperforms standard distillation with only one third of the data. |
Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal reasoning is a vital component of human communication and understanding, yet remains an underexplored area within the context of Large Language Models (LLMs). |
| Approach: | They propose to use 3 prompting strategies to evaluate 8 different LLMs across 6 datasets and 2 Code Generation LMs to perform the analysis. |
| Outcome: | The proposed models perform better on NLP tasks than the standard models on the same dataset. |
Comparing Styles across Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Communication practices vary across cultures. Inherent differences in how people think and behave influence cultural norms. |
| Approach: | They propose a framework to extract stylistic differences from multilingual language models (LMs) they use a multilingual lexica to consolidate feature importances into comparable lexical categories . |
| Outcome: | The proposed framework generates comprehensive style lexica in any language and consolidates feature importances from LMs into comparable lexical categories. |
Event Causality Extraction via Implicit Cause-Effect Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not exploited the interactions between the cause and effect event that could provide crucial clues for causality reasoning. |
| Approach: | They propose an Implicit Cause-Effect interaction framework which captures the implicit intra- and inter-event interactions by incorporating the privileged information for reasoning. |
| Outcome: | The proposed framework captures the implicit intra- and inter-event interactions by incorporating the privileged information (ground truth event types and arguments) for reasoning. |
Evaluation of African American Language Bias in Natural Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that large language generation models disadvantaging African American Language (AAL) can be biased for certain language varieties, but there is little research on the impact of these biases on other languages. |
| Approach: | They evaluate how well LLMs understand African American Language (AAL) in comparison to white Mainstream English (WME) using a dataset of AAL texts from a variety of regions and contexts, they find dialectal bias in six pre-trained LLM. |
| Outcome: | The proposed models understand African American language in comparison to white mainstream English (WME) the proposed models have performance gaps on two tasks that are not matched by the model. |
A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems (2023.emnlp-main)
Copied to clipboard
Songbo Hu, Han Zhou, Moy Yuan, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Anna Korhonen, Ivan Vulić
| Challenge: | Existing systems trained for Arabic or Turkish using annotated data fully parallel to English ToD data still exhibit diminished ToD task performance. |
| Approach: | They define new quantitative measures of absolute and relative equivalence in system performance, capturing disparities across languages and within individual languages. |
| Outcome: | The proposed measures capture disparities across languages and within individual languages. |
Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. |
| Approach: | They propose to use a protein language model to train on multiple sequence alignments to train a model on phonological reconstruction. |
| Outcome: | The proposed model outperforms existing models on cognate reflex prediction task. |
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
Weakly Supervised Semantic Parsing with Execution-based Spurious Program Filtering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train a semantic parser from weak supervision focus on exploiting similarities between examples based on domain-specific knowledge. |
| Approach: | They propose a domain-agnostic filtering mechanism based on program execution results to identify and filter out programs with significantly different semantics from the other programs. |
| Outcome: | The proposed method improves the performance of existing weakly-supervised parsers by incorporating a majority vote on the program search results. |
Taxonomy Expansion for Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
Karthikeyan K, Yogarshi Vyas, Jie Ma, Giovanni Paolini, Neha John, Shuai Wang, Yassine Benajiba, Vittorio Castelli, Dan Roth, Miguel Ballesteros
| Challenge: | Training a Named Entity Recognition model involves fixing a taxonomy of entity types . however, requirements evolve and a model may need to recognize additional entity types. |
| Approach: | They propose a method that uses only partially annotated datasets to train a model to recognize additional entity types. |
| Outcome: | The proposed approach performs better with partially annotated datasets than other approaches . the gap between the proposed approach and other approaches is large in additional datasets . |
Rather a Nurse than a Physician - Contrastive Explanations under Investigation (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study suggests that contrastive explanations are closer to how humans explain a decision than non-contrastive explanations. |
| Approach: | They analyze four English text-classification datasets to determine whether humans explain in contrast to alternatives. |
| Outcome: | The proposed explanations are closer to how humans explain a decision than non-contrastive explanations. |
EtiCor: Corpus for Analyzing LLMs for Etiquettes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Etiquettes define a system of rules and conventions that regulate social and professional behavior. |
| Approach: | They propose an Etiquettes Corpus with etiquettes from five different regions . they propose a task of Etiquette Sensitivity to evaluate LLMs for etiques . |
| Outcome: | The proposed corpus provides a test bed for evaluating LLMs for knowledge and understanding of region-specific etiquettes. |
An Investigation of LLMs’ Inefficacy in Understanding Converse Relations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Models (LLMs) follow the data distribution of pre-training data. |
| Approach: | They propose a benchmark ConvRe focusing on converse relations which contains 17 relations and 1240 triples extracted from popular knowledge graph completion datasets. |
| Outcome: | The proposed benchmark focuses on converse relations, which contains 17 relations and 1240 triples extracted from popular knowledge graph completion datasets. |
Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning (DL) based APR models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. |
| Approach: | They propose a meta-learning framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. |
| Outcome: | The proposed framework learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resourced bugs. |
ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot cross-lingual transfer have focused on training with adapters of a single source and testing either with the target LA or LA of another related language. |
| Approach: | They propose to leverage LAs of multiple (linguistically or geographically related) source languages for more effective cross-lingual transfer instead of just one source LA . they extend their novel neural architecture, ZGUL, to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language . |
| Outcome: | Extensive experimentation across four language groups, covering 15 unseen target languages, shows improvements of up to 3.2 average F1 points over baselines on POS tagging and NER tasks. |
Log-FGAER: Logic-Guided Fine-Grained Address Entity Recognition from Multi-Turn Spoken Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing name entity recognition methods combine pre-trained language models with supervised models such as BiLSTM/LSTM-CRF to perform poorly in a spoken dialogue context. |
| Approach: | They propose a logic-guided fine-grained address recognition method that softly applies the logic rule to improve the accuracy of FGAER. |
| Outcome: | The proposed method improves fine-grained address entity recognition from multi-turn spoken dialogues. |
Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition, Relation Extraction, Semantic Role Labeling are examples of sequence labeling problems that require finetuning to the target format. |
| Approach: | They propose a dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples. |
| Outcome: | The proposed approach improves performance in low-resource settings and in extreme low-level settings. |
On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on LMs as formal languages, but they do not consider language membership. |
| Approach: | They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine . |
| Outcome: | The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings . |
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
Benchmarking and Improving Text-to-SQL Generation under Ambiguity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing decoding algorithms treat SQL queries as a string and produce unhelpful token-level diversity in the top-k. |
| Approach: | They propose a benchmarking algorithm that generates all SQLs in top-k ranked outputs . they use plan-based template generation and constrained infilling to bridge this gap . |
| Outcome: | The proposed algorithm is 2.5 times more effective than state-of-the-art models at generating all candidate SQLs in the top-k ranked outputs. |
Non-autoregressive Text Editing with Copy-aware Latent Alignments (2023.emnlp-main)
Copied to clipboard
| Challenge: | Seq2Edit approaches still face several challenges such as inflexibility in generation and difficulty in generalizing to other languages. |
| Approach: | They propose a non-autoregressive text editing method that models the edit process with latent CTC alignments and introduces the copy operation into the edit space. |
| Outcome: | The proposed method outperforms existing Seq2Edit models and achieves similar or even better results than Seq1Edit with over 4 speedup. |
Translating away Translationese without Parallel Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Translated texts exhibit systematic linguistic differences compared to original texts in the same language, referred to as translationese . studies show translationeses have effects on various cross-lingual natural language processing tasks . |
| Approach: | They propose a translation-based style transfer approach that learns from monolingual data . they combine a self-supervised approach with an unsupervised approach . |
| Outcome: | The proposed method reduces translationese to a level of a random classifier after style transfer while preserving the content and fluency in the target original style. |
Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Optimal policy planning is a difficult task, authors say . many goal-oriented conversations require subjective strategies, they say - a problem in goal-orientated settings . |
| Approach: | They propose an approach to perform goal-oriented dialogue policy planning without model training. |
| Outcome: | The proposed approach performs goal-oriented dialogue policy planning without model training. |
UniMath: A Foundational and Multimodal Mathematical Reasoner (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for interpreting and processing diverse mathematical modalities are limited . existing systems are limited in interpreting complex mathematical tasks and implementing them in a multimodal manner. |
| Approach: | They propose a multimodal mathematical reasoning system that utilizes a fine-tuned T5 model augmented with a variational autoencoder (VAE)-based image tokenizer. |
| Outcome: | The proposed model achieves state-of-the-art performance on SVAMP, GeoQA, and TableMWP datasets and is generalized on two additional datasets. |
CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing legal-oriented PLMs rely on replacing general domain training data with legal data or extending the input length to fit the long-length characteristic of legal data. |
| Approach: | They propose a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases. |
| Outcome: | The proposed model outperforms existing general domain pre-training models and legal-specific pre-trainers on multiple benchmarks. |
HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies (2023.emnlp-main)
Copied to clipboard
| Challenge: | A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. |
| Approach: | They propose a cooperative game that is played between the code-generating LLM "Solver" and the "Oracle" it is based on natural language schemas and ensures the security of the underlying data. |
| Outcome: | The proposed game shows that LLMs are ineffective at generalizing and performing on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. |
Causal Document-Grounded Dialogue Pre-training (2023.emnlp-main)
Copied to clipboard
Yingxiu Zhao, Bowen Yu, Bowen Li, Haiyang Yu, Jinyang Li, Chao Wang, Fei Huang, Yongbin Li, Nevin Zhang
| Challenge: | Existing methods for document-grounded dialogue (DocGD) rely on general pre-trained language models without a tailored pre-training approach that explicitly captures causal relationships. |
| Approach: | They propose a causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora and a perturbation-based strategy to capture causality. |
| Outcome: | The proposed strategy yields significant and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings. |
Accented Speech Recognition With Accent-specific Codebooks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Degradation in performance across underrepresented accents is a severe deterrent to inclusive adoption of ASR. |
| Approach: | They propose an approach to adapt speech accents to unseen accents by using cross-attention with a trainable set of codebooks. |
| Outcome: | The proposed approach yields significant performance gains on the seen English accents and unseen accents on the Mozilla Common Voice dataset. |
Linking Surface Facts to Large-Scale Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) methods extract facts in the form of triples . ambiguity of these triples hinders their downstream usage . |
| Approach: | They propose a benchmark that measures fact linking performance on a granular triple slot level . they propose to use a system that can detect out-of-KG entities and predicates . |
| Outcome: | The proposed benchmark can measure fact linking performance on a granular triple slot level while also measuring if a system can recognize that a surface form has no match in the existing KG. |
Sentiment Analysis on Streaming User Reviews via Dual-Channel Dynamic Graph Neural Network (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for sentiment analysis on user reviews neglect their time-varying characteristics. |
| Approach: | They propose a dual-channel framework that models temporal user and product dynamics for sentiment analysis. |
| Outcome: | The proposed framework is superior to existing methods on five real-world datasets. |
DUMB: A Benchmark for Smart Evaluation of Dutch Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Dutch monolingual models under perform and suggest training larger models with other architectures and pre-training objectives. |
| Approach: | They propose a Dutch Model Benchmark that compares performance of language models to a strong baseline that can be referred to in the future even when assessing different sets of language model. |
| Outcome: | The proposed benchmark compares the performance of 14 pre-trained language models to a strong baseline . the results suggest training larger models with other architectures and pre-training objectives . |
OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding (2023.emnlp-main)
Copied to clipboard
Zhan Shi, Guoyin Wang, Ke Bai, Jiwei Li, Xiang Li, Qingjun Cui, Belinda Zeng, Trishul Chilimbi, Xiaodan Zhu
| Challenge: | Recent studies show that contrastive learning is effective in sentence representation learning . but, the surface structure bias is a problem in the current model . |
| Approach: | They propose to combine a sentence with a sub-semantic sentence to investigate the surface structure bias. |
| Outcome: | The proposed model achieves state-of-the-art on standard semantic textual similarity tasks using different pre-trained backbones. |
End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation (2023.emnlp-main)
Copied to clipboard
Juan Pablo Zuluaga-Gomez, Zhaocheng Huang, Xing Niu, Rohit Paturi, Sundararajan Srinivasan, Prashant Mathur, Brian Thompson, Marcello Federico
| Challenge: | Conventional speech-to-text translation systems are trained on single-speaker utterances, but they may not be applicable to real-life scenarios where the audio contains conversations by multiple speakers. |
| Approach: | They propose a speaker-turn-aware conversational speech translation model that integrates automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. |
| Outcome: | The proposed model outperforms the reference systems on the multi-speaker condition while attaining comparable performance on the single-speakspeaker conditions. |
A Fine-Grained Taxonomy of Replies to Hate Speech (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new corpus of responses to hate speech is developed to counter hate speech . authors work with real, user-generated hate speech and all the replies it elicits . counterspeech refers to a "direct response that counters hate speech" |
| Approach: | They propose a taxonomy of responses to hate speech and a new corpus to analyze responses . they find that responses to user-generated hate speech are more effective than replies generated by a third party . |
| Outcome: | The proposed taxonomy of responses to hate speech and a new corpus provide insights into content real users reply with and which replies are empirically most effective. |
JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to semi-supervised text classification suffer from pseudo-label bias and error accumulation. |
| Approach: | They propose a pseudo-labeling approach to semi-supervised text classification that unifies ideas from semi-semi-supervised learning and the task of learning with noise. |
| Outcome: | The proposed approach achieves a significant improvement on benchmark datasets even in the extremely-scarce-label setting. |
Simple Temporal Adaptation to Changing Label Sets: Hashtag Prediction via Dense KNN (2023.emnlp-main)
Copied to clipboard
Niloofar Mireshghallah, Nikolai Vogler, Junxian He, Omar Florez, Ahmed El-Kishky, Taylor Berg-Kirkpatrick
| Challenge: | Existing methods to adapt to temporal change of user-generated social media data are stale without retraining. |
| Approach: | They propose a non-parametric dense retrieval technique to adapt to temporal change . they use a Twitter dataset to study temporal distribution shift in tweet-hashtag prediction . |
| Outcome: | The proposed method improves over the best static parametric baseline on a year-long Twitter dataset while avoiding costly re-training. |
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4 (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that open AI models memorize a wide collection of copyrighted materials . however, these models also present a challenge for establishing the validity of results . |
| Approach: | They propose to use a name cloze membership inference query to infer books that are known to ChatGPT and GPT-4. |
| Outcome: | The proposed model performs better on memorized books than on non-memorized books for downstream tasks. |
A Study on Accessing Linguistic Information in Pre-Trained Language Models by Using Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to access linguistic information in pre-trained multilingual language models are difficult to use. |
| Approach: | They propose prompting and formulate linguistic tasks to test the LM's access to explicit grammar principles and find out what type of information can be obtained . |
| Outcome: | The proposed method can provide access to linguistic features in pre-trained models, but some are harder to capture . |
CiteBench: A Benchmark for Scientific Citation Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on citation text generation are based upon widely diverging task definitions, making it hard to study this task systematically. |
| Approach: | They propose a benchmark for citation text generation that unifies multiple datasets and enables standardized evaluation of citation texts across task designs and domains. |
| Outcome: | The proposed benchmark examines the performance of multiple strong baselines and enables standardized evaluation of citation text generation models across task designs and domains. |
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance in various language tasks, but are prone to spurious correlations and illusory information. |
| Approach: | They propose to use pre-trained language models to justify decisions with formalized, coherent reasoning chains. |
| Outcome: | The proposed strategies improve coherence of rationalizations yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP). |
A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to summarize video content have only considered video and image data, and the trend towards multimodal video summarization is changing. |
| Approach: | They propose a multimodal video summarization task setting and a dataset to train and evaluate the task. |
| Outcome: | The proposed task is useful as a practical application and presents a highly challenging problem worthy of study. |
Copyright Violations and Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the extent to which language models can memorize training data . a fair use exemption to copyright laws allows for limited use of copyrighted material . |
| Approach: | They examine the extent to which language models can redistribute copyrighted text . they use a range of popular books and coding problems to study copyright violations . |
| Outcome: | This study examines the extent to which language models can redistribute copyrighted text . it shows that language models may memorize entire chunks of training data . |
Effects of sub-word segmentation on performance of transformer language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models are a fundamental task in natural language processing, but few studies focus on the effect of sub-word segmentation on the performance of models. |
| Approach: | They compare GPT and BERT models trained with statistical segmentation algorithm BPE to unsupervised morphological segmentation algorithms Morfessor and StateMorph. |
| Outcome: | The proposed model trains for several languages and compares them with two unsupervised morphological segmentation algorithms. |
Symbolic Planning and Code Generation for Grounded Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models excel at processing and generating text and code, but lack a grounded task-oriented dialogue system that can handle grounding. |
| Approach: | They propose a modular and interpretable grounded dialogue system that integrates a reader and planner to convert partner utterances into executable code and a symbolic planner to determine the next appropriate response. |
| Outcome: | The proposed system outperforms the existing state-of-the-art on a one-common dialogue task and improves task success in human evaluations from 56% to 69% in the most challenging setting. |
Universal Self-Adaptive Prompting (2023.emnlp-main)
Copied to clipboard
| Challenge: | a hallmark of modern large language models is their impressive general zero-shot and few-shot abilities . however, zero- shot performances are weaker due to the lack of guidance and the difficulty of applying existing automatic prompt design methods in general tasks. |
| Approach: | They propose an automatic prompt design approach specifically tailored for zero-shot learning that categorizes a possible NLP task into one of three possible task types and then uses a selector to select the most suitable queries and zero- shot model-generated responses as pseudo-demonstrations. |
| Outcome: | The proposed approach is able to generalize ICL to zero-shot learning tasks while also allowing for a more efficient and efficient prompt design. |
Somali Information Retrieval Corpus: Bridging the Gap between Query Translation and Dedicated Language Resources (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on the Somali language information retrieval relies on query translation . lack of digital resources is key obstacle to advancing language technologies . |
| Approach: | They develop an annotated corpus for Somali information retrieval using query expansion technique. |
| Outcome: | The proposed corpus comprises 2335 documents collected from well-known online sites . it can be used for text classification-related tasks and question-answering research purposes. |
Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT (2023.emnlp-main)
Copied to clipboard
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu
| Challenge: | Large language models (LLMs) are capable of performing tasks but are likely to be misused. |
| Approach: | They propose a zero-shot black-box method to detect LLM-generated texts . they revise the text to be detected using the ChatGPT model . |
| Outcome: | The proposed method can detect LLM-generated texts with a zero-shot black-box model . it is based on intuition that the model will make fewer revisions to LLMs than to human-written texts . |
Faithful Model Evaluation for Model-Based Metrics (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not consider variance change due to metric model errors, which can lead to wrong conclusions. |
| Approach: | They establish the mathematical foundation of significance testing for model-based metrics . they show that metric errors can change the conclusions in certain experiments . |
| Outcome: | The proposed method can be used to derive accurate conclusions using model evaluations. |
Content- and Topology-Aware Representation Learning for Scientific Multi-Literature (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on learning textual information at sentence- or document-level, but ignore inter-document connections. |
| Approach: | They propose a model that extends representation learning to the multi-document level . it integrates latent semantic and rich relatedness information from topological networks . |
| Outcome: | The proposed model learns latent semantic information from content and rich relatedness information from topological networks. |
Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have found that higher quality language models provide more powerful predictors of human reading behavior, but empirical support for the QP hypothesis is mixed. |
| Approach: | They propose to test the quality–power hypothesis by using surprisal language models to test their ability to predict eye tracking data. |
| Outcome: | The proposed model is based on a set of language models with a 'quality-power' hypothesis. |
Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity and Relation Extraction (ERE) is an important task in information extraction. |
| Approach: | They propose a hypergraph neural network for ERE built upon the PL-marker . they use a pruner mechanism to transfer the burden of entity identification to the joint module . |
| Outcome: | The proposed model improves on three widely used benchmarks on ERE task . it uses a pruner mechanism to transfer the burden of entity identification to the joint module . |
Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past decade. |
| Approach: | They propose a benchmark dataset for evaluating the problem solving abilities of large language models (LLMs) they curate 515 challenging problems from the highly competitive IIT JEE-Advanced exam. |
| Outcome: | The proposed model performs better on open-source and proprietary models than the current model, but with techniques like self-consistency, self-refinement and chain-of-thought prompting. |
StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structure (2023.emnlp-main)
Copied to clipboard
| Challenge: | Structured Autoencoder framework StrAE enables effective learning of multi-level representations through strict adherence to explicit structure. |
| Approach: | They propose a Structured Autoencoder framework that strictly adheres to explicit structure and uses a contrastive objective over tree-structured representations. |
| Outcome: | The proposed framework outperforms baselines that don’t involve explicit hierarchical compositions and is comparable to models given informative structure. |
WiCE: Real-World Entailment for Claims in Wikipedia (2023.emnlp-main)
Copied to clipboard
| Challenge: | Textual entailment models are increasingly used in fact-checking, presupposition verification in question answering, or summary evaluation. |
| Approach: | They propose a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia that provides en-tailment judgments over sub-sentence units of the claim and a minimal subset of evidence sentences that support each subclaim. |
| Outcome: | The proposed dataset improves on multiple datasets at test time and shows that real claims involve verification and retrieval problems that existing models fail to address. |
Natural Disaster Tweets Classification Using Multimodal Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms are used for expressing opinions or conveying information. |
| Approach: | They propose a hierarchical system that can integrate multimodal data and perform sequential hierarchic classification. |
| Outcome: | The proposed system can find the damage and its severity along with classify the data into humanitarian categories. |
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)
Copied to clipboard
| Challenge: | Perception of toxicity evolves over time and differs between geographies and cultural backgrounds. |
| Approach: | They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results. |
| Outcome: | The Perspective API has been updated to reflect the changes in toxicity scores. |
RoBoCoP: A Comprehensive ROmance BOrrowing COgnate Package and Benchmark for Multilingual Cognate Identification (2023.emnlp-main)
Copied to clipboard
Liviu Dinu, Ana Uban, Alina Cristea, Anca Dinu, Ioan-Bogdan Iordache, Simona Georgescu, Laurentiu Zoicas
| Challenge: | Existing databases for romance cognates are scattered, incomplete, noisy, or have uncertain availability. |
| Approach: | They propose to use etymological information to identify Romance cognates and borrowings from dictionaries to identify their ethymology. |
| Outcome: | The proposed method achieves 94% accuracy on two pairs of Romance languages. |
Instructive Dialogue Summarization with Query Aggregations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional dialogue summarization methods generate summaries without considering user’s specific interests. |
| Approach: | They propose a three-step approach to synthesize high-quality query-based summarization triples by training a unified model on three summarizing datasets with multi-purpose instructive triples. |
| Outcome: | The proposed model outperforms state-of-the-art models and even models with larger sizes on four datasets including dialogue summarization and dialogue reading comprehension. |
Semantic matching for text classification with complex class descriptions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text classification support zero-shot learning but not both . Existing approaches do not support zero or few-shot, and are insufficient for complex classes . |
| Approach: | They propose a method which rapidly adapts from seen classes to new/unseen ones . they use labels and complex class descriptions to perform zero- and few-shot learning . |
| Outcome: | The proposed method beats baselines on complex class descriptions by 22.48% . it also improves zero-shot learning by 4.29% . |
MADNet: Maximizing Addressee Deduction Expectation for Multi-Party Conversation Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-party conversations rely on addressee labels and can only be applied to an ideal setting where addresses are missing. |
| Approach: | They propose a method that maximizes addressee deduction expectation in heterogeneous graph neural networks for MPC generation. |
| Outcome: | The proposed method outperforms baseline models on Ubuntu IRC channel benchmarks on the task of MPC generation under a common and challenging setting where addressee labels are missing. |
GLEN: Generative Retrieval via Lexical Index Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document retrieval bypass auxiliary index structures and can be optimized through end-to-end learning. |
| Approach: | They propose a method to generate a relevant document's identifier using an index learning strategy. |
| Outcome: | The proposed method achieves state-of-the-art or competitive performance on benchmark datasets. |
Turn-Level Active Learning for Dialogue State Tracking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to annotate dialogues require supervised training, which requires human workers to manually annotates dialogues. |
| Approach: | They propose a turn-level active learning framework to actively select dialogue turns to annotate . their approach can achieve comparable performance to traditional training approaches . |
| Outcome: | The proposed model achieves comparable performance to existing training approaches with significantly less annotated data. |
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal dialogue systems are limited by the scale and quality of available datasets or the coarse concept of visual knowledge. |
| Approach: | They propose to explicitly split visual knowledge into finer granularity and turn-level . they propose a framework to add visual representation into vanilla dialogue models . |
| Outcome: | The proposed framework outperforms state-of-the-art methods on automatic and human evaluations. |
Modeling Conceptual Attribute Likeness and Domain Inconsistency for Metaphor Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Metaphor detection aims to distinguish between metaphorical and literal expressions in text. |
| Approach: | They propose an attribute likeness and domain inconsistency learning framework for wordpair metaphor detection based on conceptual metaphor theory . they model attribute likeity with an attribute siamese network and devise a domain contrastive learning strategy to learn semantic inconsistentness of concepts in source and target domains . |
| Outcome: | The proposed framework outperforms existing word-pair and token-level methods on four datasets. |
Referring Image Segmentation via Joint Mask Contextual Embedding Learning and Progressive Alignment Network (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for referring image segmentation may encounter limitations in maintaining focus on relevant information during specific stages and rectifying errors propagated from early stages. |
| Approach: | They propose a network that integrates a Learnable Contextual Embedding module and a Progressive Alignment Network to enhance the cascade framework. |
| Outcome: | The proposed network achieves state-of-the-art results on three commonly used benchmarks. |
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023.emnlp-main)
Copied to clipboard
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro
| Challenge: | a recent study shows that retrieval-augmented LMs can improve text generation quality and accuracy. |
| Approach: | They propose a model that reproduces RETRO parameters while retrieving a text corpus . they find RETRO outperforms GPT on text generation with less repetition . |
| Outcome: | The proposed model outperforms standard retrieval-augmented GPT and retrieval augmented GTP on text generation and accuracy tasks. |
SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. |
| Approach: | They present a dataset of 1.2K expert-verified scientific claims that require compositional reasoning for verification. |
| Outcome: | The proposed model outperforms existing models in table-based pretraining models and large language models. |
Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous Speech Translation (SimulST) is a task focused on ensuring high-quality translation of speech in low-latency situations. |
| Approach: | They propose a token-level cross-modal alignment method to improve the translation of text to audio . they use audio transcription pairs to pre-train the encoder and a random wait-k-tokens strategy to optimize the task. |
| Outcome: | The proposed method achieves better trade-off between translation quality and latency. |
SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples (2023.emnlp-main)
Copied to clipboard
| Challenge: | Manually collecting challenging negative examples can be costly and domain-specific. |
| Approach: | They propose a method that synthesizes negative examples from only positive ones using a mask infilling model. |
| Outcome: | The proposed method closes 69.6% of the performance gap on a dataset where half of the evaluation examples are unanswerable compared to a model trained on SQuAD 2.0. |
Enhancing Code-Switching for Cross-lingual SLU: A Unified View of Semantic and Grammatical Coherence (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models rely on annotated training data, limiting their scalability to low-resource languages. |
| Approach: | They propose a method termed SoGo for zero-shot cross-lingual SLU that uses keywords as substitution options to extract keywords and a token-level alignment strategy to ensure grammatical coherence. |
| Outcome: | The proposed method improves zero-shot cross-lingual SLU across nine languages on MultiATIS++. |
Task-Agnostic Low-Rank Adapters for Unseen English Dialects (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study found that LLMs are trained on corpora disproportionally weighted in favor of Standard American English . prior work on dialect struggle with generalizing to evolving and emerging dialects in a scalable manner. |
| Approach: | They propose a method that leverages linguistic knowledge to enable resource-efficient adaptation . their method disentangles dialect-specific and cross-dialectal information . |
| Outcome: | a new method improves generalization to unseen dialects in a task-agnostic fashion . it achieves the best or most competitive performance across 5 dialects . |
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning of Large Language Models (LLMs) can incur performance degradation or low training efficiency. |
| Approach: | They propose a prompt tuning approach with Adaptive Optimization to enable efficient FL of LLMs. |
| Outcome: | The proposed approach improves performance and efficiency simultaneously and addresses client drift problems on both the device and server sides. |
TheoremQA: A Theorem-driven Question Answering Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. |
| Approach: | They propose to use theorem-driven question-answering dataset to evaluate AI models' ability to apply theoretic concepts to solving challenging science problems. |
| Outcome: | TheoremQA is curated by domain experts and contains 800 high-quality questions covering 350 theoremics from Math, Physics, EE&CS, and Finance. |
Scalable-DSC: A Structural Template Prompt Approach to Scalable Dialogue State Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to correct wrong slot values in dialogue state tracking are intertwined with specific DST models, limiting their applicability to other DSTs. |
| Approach: | They propose a Scalable Dialogue State Correction model that corrects wrong slot values in predicted dialogue states by using a structural template prompt. |
| Outcome: | The proposed model achieves state-of-the-art results on MultiWOZ 2.0-2.4. |
Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can perform a wide variety of language tasks when presented in English. |
| Approach: | They propose a method to evaluate the multilingual capabilities of large language models using a prompt back-translation method to find out how LLMs acquire their multilingual abilities. |
| Outcome: | The proposed method shows that large language models can transfer learned knowledge across different languages, but struggle to provide accurate results in translation-variant tasks. |
M3Seg: A Maximum-Minimum Mutual Information Paradigm for Unsupervised Topic Segmentation in ASR Transcripts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic segmentation aims to split automatic speech recognition transcriptions into segments that are bounded by thematic meanings. |
| Approach: | They propose a Maximum-Minimum Mutual information paradigm for linear topic segmentation without using any parallel data. |
| Outcome: | The proposed paradigm outperforms the state-of-the-art methods by a significant margin. |
Empirical Study of Zero-Shot NER with ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been a key component of natural language processing (NLP) . |
| Approach: | They propose to decompose the NER task into simpler subproblems by labels and propose a syntactic augmentation strategy to stimulate model's intermediate thinking. |
| Outcome: | The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets. |
Automatic Prompt Optimization with “Gradient Descent” and Beam Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort. |
| Approach: | They propose an algorithm that uses numerical gradient descent to automatically improve prompts by rewriting vague task descriptions into more precise annotation instructions. |
| Outcome: | The proposed algorithm outperforms previous methods and improves performance on three benchmark NLP tasks and the novel problem of LLM jailbreak detection. |
Active Retrieval Augmented Generation (2023.emnlp-main)
Copied to clipboard
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig
| Challenge: | Generative language models (LMs) have a tendency to hallucinate and create inaccurate output. |
| Approach: | They propose a method which iteratively uses a prediction of the upcoming sentence to anticipate future content. |
| Outcome: | The proposed method achieves superior or competitive performance on all tasks . iteratively uses a prediction of the upcoming sentence to anticipate future content . |
GD-COMET: A Geo-Diverse Commonsense Inference Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | GD-COMET is a geo-diverse version of the COMET commonsense inference model . it captures and generates culturally nuanced commonsensense knowledge . lack of cultural awareness may lead to models perpetuating stereotypes and reinforcing societal inequalities for users from non-Western countries. |
| Approach: | They propose a geo-diverse version of COMET commonsense reasoning model that generates inferences pertaining to a broad range of cultures. |
| Outcome: | The proposed model generates inferences pertaining to a broad range of cultures and is culturally nuanced. |
Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation (2023.emnlp-main)
Copied to clipboard
Chenxu Yang, Zheng Lin, Lanrui Wang, Chong Tian, Liang Pang, Jiangnan Li, Qirong Ho, Yanan Cao, Weiping Wang
| Challenge: | Existing knowledge-grounded dialogue generation models struggle with dull and repetitive outputs, a problem commonly termed as text degeneration. |
| Approach: | They propose a framework that allows the model to "cheat" the objective by duplicating knowledge segments in a superficial pattern matching based on overlap. |
| Outcome: | The proposed framework can be applied to a WoW dataset and shows that it works across models and decoding strategies. |
Enhancing Biomedical Lay Summarisation with External Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to lay summarisation are reliant on the source article, which is unlikely to include all the information necessary for a lay audience. |
| Approach: | They augment existing biomedical lay summarisation dataset with article-specific knowledge graphs that contain detailed information on relevant biomedically related concepts. |
| Outcome: | The proposed methods improve readability and explanation of technical concepts by integrating graph-based domain knowledge within lay summarisation models. |
A Diffusion Weighted Graph Framework for New Intent Discovery (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn from unlabeled data generate noisy supervisory signals . current methods only rely on semantic similarities to generate supervisory signal . |
| Approach: | They propose a weighted DWGF framework to capture semantic similarities and structure relationships in data. |
| Outcome: | The proposed method outperforms state-of-the-art models on evaluation metrics across multiple benchmark datasets. |
A Self-enhancement Multitask Framework for Unsupervised Aspect Category Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on learning embedding spaces for seed words and sentences to establish similarities between sentences and aspects. |
| Approach: | They propose a framework that enhances the quality of initial seed words and selects high-quality sentences instead of using the entire dataset. |
| Outcome: | The proposed framework surpasses strong baselines on standard datasets and improves on the noise resolution task. |
DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting has improved the reasoning capabilities of Large Language Models (LLMs) but it is ineffective or detrimental to the performance on reasoning tasks in Smaller Language Model (SLMs) with less than 10 billion parameters. |
| Approach: | They propose a Dialogue-guided Chain-of-Thought method to improve the reasoning capabilities of Large Language Models (LLMs) by generating intermediate reasoning steps in a dialogue format to guide the model to the final answer. |
| Outcome: | The proposed method can achieve significant performance gains over state-of-the-art competitors on four arithmetic reasoning datasets. |
Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages. |
| Approach: | They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N ||right) neurons to represent an LM . |
| Outcome: | The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models. |
Revisiting Source Context in Nearest Neighbor Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research does not explicitly consider the source context when retrieving similar examples . |
| Approach: | They propose a method to improve neural machine translation via source context enhancement by integrating a source-aware distance calibration module. |
| Outcome: | The proposed approach can be integrated with representative kNN-MT baselines and achieve significant performance improvements. |
Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems require large number of accurate annotations, such as image-level labels and location-level labeling. |
| Approach: | They propose a joint anaphora resolution and object localization dataset targeting visual-linguistic ambiguity. |
| Outcome: | The proposed framework improves visual-linguistic alignment and object localization with one joint model compared to a strong single-task baseline. |
Background Summarization of Event Timelines (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generating concise summaries of news events is a challenging task for newcomers to a news story. |
| Approach: | They propose a task of background news summarization that complements each timeline update with a background summary of relevant preceding events. |
| Outcome: | The proposed system performs well on a question-answering-based evaluation metric, Background Utility Score (BUS). |
Superlim: A Swedish Language Understanding Evaluation Benchmark (2023.emnlp-main)
Copied to clipboard
Aleksandrs Berdicevskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey Öhman, Yvonne Adesam, Lars Borin, Dana Dannélls, Markus Forsberg, Tim Isbister, Anna Lindahl, Martin Malmsten, Faton Rekathati, Magnus Sahlgren, Elena Volodina, Love Börjeson, Simon Hengchen, Nina Tahmasebi
| Challenge: | In this paper, we present a multi-task benchmark for Swedish language models . we address methodological challenges, such as mitigating the Anglocentric bias when creating datasets for a less-resourced language . |
| Approach: | They propose a multi-task NLP benchmark for Swedish language models . they propose to use superlim to evaluate Swedish language model performance . |
| Outcome: | The proposed benchmark does not approach ceiling performance on any of the tasks, suggesting it is difficult to implement. |
Reasoning with Language Model is Planning with World Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable reasoning capabilities, particularly with Chain-of-Thought-style prompts. |
| Approach: | They propose a framework that repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) |
| Outcome: | The proposed framework repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) it achieves optimum balance between exploration and exploitation, while achieving high-reward reasoning paths efficiently. |
LLM-enhanced Self-training for Cross-domain Constituency Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to self-training rely on limited and potentially low-quality raw corpora. |
| Approach: | They propose to enhance self-training with the large language model to generate domain-specific raw corpora iteratively and introduce grammar rules that guide the LLM in generating raw corporeals and establish criteria for selecting pseudo instances. |
| Outcome: | The proposed method outperforms traditional methods regardless of the large language model's performance. |
Continual Named Entity Recognition without Catastrophic Forgetting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (CNER) is a burgeoning area of research . a new paradigm has ushered NER into a non-entity type at the current step t . |
| Approach: | They propose a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on ten CNER settings using three datasets. |
DSI++: Updating Transformer Memory with New Documents (2023.emnlp-main)
Copied to clipboard
Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler
| Challenge: | Differentiable Search Indices (DSIs) encode a corpus of documents and use the same model to map queries directly to relevant document identifiers. |
| Approach: | They propose a continual learning challenge for Differentiable Search Indices (DSIs) they propose to continuously index new documents while answering queries related to previously and newly indexed documents. |
| Outcome: | The proposed model stably memorizes more documents and improves the average Hits@10 by +21.1% over baselines. |
Editing Common Sense in Transformers (2023.emnlp-main)
Copied to clipboard
Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, Niket Tandon
| Challenge: | Currently, the performance of transformer-based model editing methods is limited to statements about encyclopedic knowledge with a single correct answer. |
| Approach: | They propose to improve MEMIT's model editing algorithm by varying edit tokens and improving the layer selection strategy to improve commonsense knowledge. |
| Outcome: | The MEMIT editing algorithm outperforms baseline models on PEP3k and 20Q datasets while fine-tuning baselines shows significant trade-offs. |
Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Controllable text generation (CTG) aims to generate text with desired attributes, but current methods lack high levels of controllability. |
| Approach: | They propose a lightweight decoding framework that reconstructs attribute distributions to balance the weights between attribute words and non-attribute words to generate more fluent text. |
| Outcome: | The proposed framework achieves state-of-the-art control performance on multiple CTG tasks. |
Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | 'context mixing' is a feature of Transformers that is used to build up representations of acoustic and linguistic structure in speech models. |
| Approach: | They propose to use a French spelling quirk to probe context mixing in speech models to find out how to translate spoken words into written equivalents. |
| Outcome: | The proposed model incorporates cues to identify correct transcription, whereas encoder-decoder models relegate task to decoder modules. |
Retrieval-Generation Alignment for End-to-End Task-Oriented Dialogue System (2023.emnlp-main)
Copied to clipboard
| Challenge: | generative models struggle to distinguish subtle differences among retrieved knowledge records, resulting in suboptimal quality of generated responses. |
| Approach: | They propose to use maximum marginal likelihood to train a perceptive retriever by utilizing signals from response generation for supervision. |
| Outcome: | The proposed approach improves on three task-oriented dialogue datasets using T5 and ChatGPT as the backbone models. |
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge. |
| Approach: | They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause. |
| Outcome: | The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to LFQA are limited to information-seeking questions . however, users' questions in the real-world may often mislead language models (LMs) to output misinformation. |
| Approach: | They propose a unified approach capable of handling any type of information-seeking question. |
| Outcome: | The proposed approach can handle any type of information-seeking question. |
Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using the counterfactual memorisation metric, we find that when training neural networks, models will memorise some inputs but not others. |
| Approach: | They use the counterfactual memorisation metric to build a resource that places 5M NMT datapoints on a memorisations-generalisation map and describe how the datapoint’s surface-level characteristics and a models’ per-datum training signals are predictive of memorising in NMT. |
| Outcome: | The proposed model places 5M NMT datapoints on a memorisation-generalisation map and shows how their surface-level characteristics and models’ per-datum training signals are predictive of memorising in NMT. |
DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4 (2023.emnlp-main)
Copied to clipboard
| Challenge: | Human preference judgments are important in large language models to produce outputs that align with human values. |
| Approach: | They conduct an in-depth examination of pairwise human judgments released by OpenAI . they find that most favored factors vary across tasks and genres . |
| Outcome: | The proposed model reveals that most favored factors vary across tasks and genres . the findings have implications on the construction of balanced datasets in human preference evaluations - crucial step in shaping behavior of future LLMs. |
Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations. |
| Approach: | They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase . |
| Outcome: | The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks. |
QA-NatVer: Question Answering for Natural Logic-based Fact Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim and its evidence via set-theoretic operators. |
| Approach: | They propose to use question answering to predict natural logic operators using generalization capabilities of instruction-tuned language models. |
| Outcome: | The proposed approach outperforms the best baseline on a Danish verification dataset by 4.3 accuracy points. |
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices. |
| Approach: | They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time. |
| Outcome: | The proposed method eliminates the impact of SFC in the majority of instances. |
Generating Data for Symbolic Language with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) bring performance and complexity, but they incur a large computational cost in practice. |
| Approach: | They propose a task-based model which uses large language models to generate symbolic language data by an informative prompt and agreement-based verifier. |
| Outcome: | The proposed model can generate symbolic language data with a few human demonstrations and saves a considerable amount of inference effort. |
IDTraffickers: An Authorship Attribution Dataset to link and connect Potential Human-Trafficking Operations on Text Escort Advertisements (2023.emnlp-main)
Copied to clipboard
| Challenge: | a significant number of human trafficking cases are associated with online advertisements . identification of HT vendors is challenging for law enforcement agencies . |
| Approach: | IDTraffickers uses 87,595 text ads and 5,244 vendor labels to link HT vendors . a macro-F1 score is achieved in a closed-set classification environment . |
| Outcome: | IDTraffickers is a dataset that enables verification and identification of HT vendors . the model achieves a macro-F1 score in a closed-set classification environment . |
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained machine learning models perpetuate and even amplify existing biases in data . this can result in unfair outcomes that ultimately impact user experience . |
| Approach: | They quantify bias amplification in pretraining and after fine-tuning on vision-and-language models. |
| Outcome: | The results show that pretrained models can perpetuate and even amplify biases in data without compromising performance. |
Improving Dialogue Discourse Parsing via Reply-to Structures of Addressee Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn dialogue discourse parsing with related tasks require additional annotation, thus limiting their generality. |
| Approach: | They propose a multitasking framework that integrates dialogue discourse parsing with addressee recognition to reflect relation-based structure of dialogue. |
| Outcome: | The proposed framework outperforms baselines on the Molweni and STAC datasets. |
Improving Language Models’ Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have exploited data augmentation or implemented specialised loss functions to alleviate the inconsistent behaviour issue, but they consume expensive training resources and can only handle a certain consistency type. |
| Approach: | They propose a method that allows PLMs to capture accurate meaning by learning precise interrelationships between concepts from word-definition pairs in a dictionary. |
| Outcome: | The proposed method can improve multiple types of consistency and integrate pre-trained knowledge with PLMs’ pre-training knowledge. |
DALE: Generative Data Augmentation for Low-Resource Legal NLP (2023.emnlp-main)
Copied to clipboard
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, Dinesh Manocha
| Challenge: | DALE addresses the challenges existing frameworks pose in generating effective data augmentations of legal documents. |
| Approach: | They propose a generative Data Augmentation framework for low-resource legal NLP that exploits domain-specific language characteristics of templated legal documents to mask collocated spans of text. |
| Outcome: | The proposed framework outperforms baseline frameworks on 13 datasets and 4 low-resource settings. |
FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |
trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback (2023.emnlp-main)
Copied to clipboard
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, Louis Castricato
| Challenge: | Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures. |
| Approach: | They propose an open-source framework for reinforcement learning from human feedback . it allows for offline fine-tuning of large language models . |
| Outcome: | The framework can be used to fine-tune models up to and exceeding 70 billion parameters. |
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have grammatical knowledge but fail to interpret negation . a recent study shows that LLMs struggle with negative sentences . |
| Approach: | They propose to use a dataset to grasp LLMs' generalization and inference capability . they also fine-tuned models to assess whether the understanding of negation can be trained . |
| Outcome: | The proposed model is able to generalize and infer negation in 400,000 sentences . but it is suboptimal when it comes to negation, a key step in natural language processing . |
MT2: Towards a Multi-Task Machine Translation Model with Translation-Specific In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sentence-level translation, document-level and terminology constrained translations are important in machine translation. |
| Approach: | They propose a multi-task machine translation model that integrates translation memory sentences . they propose 'in-context learning' paradigm that allows translation-specific context learning . |
| Outcome: | The proposed model improves translation memory, document-level translation, and document-constrained translation tasks. |
CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models achieve F1-scores comparable to or exceed noise level in CoNLL-03 . current models have significant annotation errors, incompleteness, and inconsistencies in the data . |
| Approach: | They propose to add a layer of entity linking annotation to the CoNLL-03 corpus to correct 7.0% of all labels. |
| Outcome: | The proposed approach corrects 7.0% of all labels in the English CoNLL-03 dataset. |
Disentangling Transformer Language Models as Superposed Topic Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic Modelling is an established area of research where the quality of a given topic is measured using coherence metrics. |
| Approach: | They propose a weight-based approach to search and disentangle decoder-only TLM by a Wikipedia corpus. |
| Outcome: | The proposed approach can be applied to GPT-2 models and LLaMA models. |
Conversational Semantic Parsing using Dynamic Context Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on conversational semantic parsing has focused on answering questions in isolation . whereas existing work on KBQA is focused on resolving questions in the context of natural language questions . |
| Approach: | They propose to model conversational semantic parsing over general purpose knowledge graphs with millions of entities and thousands of relation-types by exploiting its underlying structure and encoding it with a graph neural network. |
| Outcome: | The proposed model is better at processing discourse information and longer interactions . it is better than static models at handling ellipsis and coreference, the authors show . |
Not all quantifiers are equal: Probing Transformer-based language models’ understanding of generalised quantifiers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent popularity of generalised quantifiers and role in linguistics and logic raises the question of how they affect transformer-based language models (TLMs) |
| Approach: | They propose to use textual entailment to assess the ability of TLMs to learn the meanings of generalised quantifiers by using a textual model-checking problem defined in a purely logical sense. |
| Outcome: | The proposed method allows the automatic construction of datasets with respect to which we can assess the ability of TLMs to learn the meanings of generalised quantifiers. |
Structure-aware Knowledge Graph-to-text Generation with Planning Selection and Similarity Distinction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate knowledge graph-to-text (KG-to) text rely on pre-trained language models to bridge the gap between the different structures of the input KG and the target text. |
| Approach: | They propose a method that integrates graph structure-aware modules with pre-trained language models to capture the intricate topology information present in the KG. |
| Outcome: | The proposed model captures the topology information present in the knowledge graph and distinguishes similar input KGs through contrastive learning techniques. |
SOUL: Towards Sentiment and Opinion Understanding of Language (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sentiment analysis models often fail to capture the broader complexities of sentiment analysis. |
| Approach: | They propose a task to evaluate sentiment understanding through two subtasks . they annotate a new dataset comprising 15,028 statements from 3,638 reviews . |
| Outcome: | The proposed task evaluates sentiment understanding through two subtasks . it is a challenging task for both small and large language models, with performance gaps of up to 27% . |
Regulation and NLP (RegNLP): Taming Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | polarization in AI safety and ethics debates are swaying political agendas on AI regulation and governance . regulation studies are rich source of knowledge on how to systematically deal with risk and uncertainty . |
| Approach: | They argue that NLP research can benefit from proximity to regulatory studies . they argue that regulation studies should focus on linking scientific knowledge to regulatory processes . |
| Outcome: | The proposed research space should focus on linking scientific knowledge to regulatory processes based on systematic methodologies. |
MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing medical datasets require high quality domain-specific datasets. |
| Approach: | They propose a multi-level, multi-task, and multi-domain medical benchmark to facilitate the development of language models for healthcare. |
| Outcome: | The proposed model provides granular potential usage and supports a wide range of tasks. |
Seeing through the mess: evolutionary dynamics of lexical polysemy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of lexical polysemy suggest multiple senses emerge for words . lexically, multiple sense of a word is the rule rather than the exception, authors say . |
| Approach: | They propose a mathematical model of the evolution of lexical meaning to investigate polysemy . they find conditions under which a sense of a word diversifies itself into multiple senses . |
| Outcome: | The proposed model predicts that diversification is promoted by low frequency and high discriminability . it also shows that the model is robust to a wide range of language variables . |
Are Embedded Potatoes Still Vegetables? On the Limitations of WordNet Embeddings for Lexical Semantics (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge Base Embedding (KBE) models are widely used to encode structured information from knowledge bases, including WordNet, but the evaluation task is often focused on link prediction, ignoring their semantic capabilities. |
| Approach: | They propose to evaluate the performance of Knowledge Base Embedding (KBE) models of WordNet on link prediction and their ability to encode semantic information. |
| Outcome: | The proposed model performs poorly on two semantic tasks and two downstream tasks. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
Event-Location Tracking in Narratives: A Case Study on Holocaust Testimonies (2023.emnlp-main)
Copied to clipboard
| Challenge: | a primary goal of narrative analysis is to represent essential dimensions of stories in a schematic manner. |
| Approach: | They propose a task to extract the sequence of locations where the narrative is set through its progression. |
| Outcome: | The proposed task is based on the test case of Holocaust survivor testimonies . it shows that models that are aware of the larger context can generate more accurate locations chains. |
Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing dialog inpainting methods generate ConvQA datasets with low contextual relevance due to insufficient learning of question-answer alignment. |
| Approach: | They propose a dialog inpainting method that generates ConvQA datasets from documents . they propose re-ranking tasks and a framework that generate contextually relevant questions . |
| Outcome: | The proposed framework generates ConvQA datasets with high contextual relevance from textual sources. |
Learning to Predict Task Transferability via Soft Prompt (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that fine-tuning pretrained language models on helpful intermediate tasks yields further gains. |
| Approach: | They propose to train an affinity scoring function to predict transferability between tasks by conditioning on task embeddings. |
| Outcome: | The proposed method efficiently identifies beneficial tasks for transfer learning. |
Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multistep question answering have shown promise in generating multistep solutions, but they lack robustness. |
| Approach: | They propose a framework that trains a model to robustly answer multistep questions by generating and answering sub-questions. |
| Outcome: | The proposed framework outperforms neuro-symbolic methods on a DROP contrast set and GPT-3.5 on QA adversarial sets. |
Mirror: A Universal Framework for Various Information Extraction Tasks (2023.emnlp-main)
Copied to clipboard
Tong Zhu, Junfei Ren, Zijian Yu, Mengsong Wu, Guoliang Zhang, Xiaoye Qu, Wenliang Chen, Zhefeng Wang, Baoxing Huai, Min Zhang
| Challenge: | Recent studies often formulate IE tasks as a triplet extraction problem, but this paradigm does not support multi-span and n-ary extraction, leading to weak versatility. |
| Approach: | They propose a multi-span cyclic graph extraction problem and a non-autoregressive graph decoding algorithm to extract all spans in a single step. |
| Outcome: | The proposed model outperforms or reaches competitive performance with SOTA systems under few-shot and zero-shot settings and it is compatible with 57 datasets. |
“Mistakes Help Us Grow”: Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms (2023.emnlp-main)
Copied to clipboard
| Challenge: | GMSL has been shown to significantly reduce disparities in academic achievement and enhance students’ learning outcomes. |
| Approach: | They develop a coaching tool to reframe unsupportive utterances to GMSL using large language models. |
| Outcome: | The proposed model outperforms the GMSL-trained teachers in fostering a growth mindset and promoting challenge-seeking behavior. |
Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable proficiency in many tasks, but much about their inner workings remains unclear. |
| Approach: | They propose a test suite to measure the resilience of Large Language Models (LLMs) when subjected to extensive character-level permutations. |
| Outcome: | The proposed test suite measures the resilience of large language models to scrambled inputs. |
Detecting and Mitigating Hallucinations in Multilingual Summarisation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing faithfulness metrics for abstractive summarisation models focus on English . metric mFACT is best suited to detect hallucinations in cross-lingual transfer . |
| Approach: | They propose a method to evaluate the faithfulness of non-English summaries by translation-based transfer from multiple English faithfulness metrics. |
| Outcome: | The proposed method reduces hallucinations in cross-lingual transfer by weighing the loss of each training example by its faithfulness score. |
Exploring Linguistic Probes for Morphological Inflection (2023.emnlp-main)
Copied to clipboard
| Challenge: | morphological inflection models typically employ language-independent data splitting algorithms. |
| Approach: | They propose language-specific probes to test aspects of morphological generalization . they use three morphology-distinct languages to test their generalization abilities . |
| Outcome: | The proposed language-specific probes are used to test morphological generalization abilities on three distinct languages. |
AMR Parsing with Causal Hierarchical Attention and Pointers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic representation of natural language sentences. |
| Approach: | They propose a new target form of AMR parsing and a model which integrates structural localities into the Transformer decoder. |
| Outcome: | The proposed model outperforms baseline models on four out of five benchmarks in the setting of no additional data. |
FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Score (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting out-of-distribution instances are empirical . state-of the-art methods for OOD detection are suboptimal since they only estimate in-distance density pout(x). |
| Approach: | They propose a method that measures the “OOD-ness” of a test case x through the likelihood ratio between out-distribution mathcal Pout and in-division mathcal Pin. |
| Outcome: | The proposed method improves existing methods on popular benchmarks and establishes a new SOTA on popular NLP benchmarks. |
Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks (2023.emnlp-main)
Copied to clipboard
Haoqi Zheng, Qihuang Zhong, Liang Ding, Zhiliang Tian, Xin Niu, Changjian Wang, Dongsheng Li, Dacheng Tao
| Challenge: | Text classification tasks often encounter few-shot scenarios with limited labeled data, and addressing data scarcity is crucial. |
| Approach: | They propose a self-evolution learning (SE) based mixup approach for data augmentation in text classification which generates more adaptive and model-friendly pseudo samples for the model training. |
| Outcome: | The proposed approach can generate more adaptive and model-friendly pseudo samples for the model training. |
IC3: Image Captioning by Committee Consensus (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, image captioning models are trained to generate a single “best’ (most like a reference) image caption. |
| Approach: | They propose a method to generate a single caption that captures high-level details from several annotator viewpoints. |
| Outcome: | The proposed method outperforms baseline SOTA models and improves the performance of automated recall systems by up to 84%. |
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing fact-checking approaches require access to external databases or external databases . a lack of external databases can undermine trust in large language models. |
| Approach: | They propose a sampling-based approach to fact-check black-box models without external databases. |
| Outcome: | The proposed approach can be used to fact-check black-box models without external databases . it can detect non-factual and factual sentences and rank passages in terms of factuality . |
Fair Without Leveling Down: A New Intersectional Fairness Definition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to capture intersectional group fairness lack significant unfairness at intersection levels. |
| Approach: | They propose a new definition of intersectional fairness that combines absolute and relative performance across sensitive groups. |
| Outcome: | The proposed definition does not improve on a simple baseline. |
Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction fine-tuned (IFT) models are gaining traction in industrial NLP to unlock task-specific performance gains and strengthen model alignment with industry requirements. |
| Approach: | They propose to use instruction fine-tuned (IFT) models to enhance the zero-shot capabilities of Large Language Models (LLMs) they also propose to leverage IFT models to analyze the trade-offs that emerge in industrial settings. |
| Outcome: | The proposed model is well adapted to new evaluation metric requirements, and offers practical insights for real-world LLM deployment. |
CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Cascaded approach is the most popular choice for speech translation, but lacks robustness when dealing with noisy inputs. |
| Approach: | They propose a cascaded approach that uses an automatic speech recognition model and a machine translation model to translate speech in one language to text in another language. |
| Outcome: | The proposed approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text. |
M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work mainly utilizes image information to improve the performance of MABSA task. |
| Approach: | They propose a multimodal Aspect-based Sentiment Analysis task that uses image information to improve model performance. |
| Outcome: | The proposed framework outperforms state-of-the-art work on three sub-tasks of MABSA. |
Detection of Multiple Mental Disorders from Social Media with Two-Stream Psychiatric Experts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing mental disease detection methods are not backed by domain knowledge and thus fail to produce interpretable results. |
| Approach: | They propose a framework that can learn the shared clues of all diseases while also capturing the specificity of each single disease. |
| Outcome: | Experiments on the detection of 7 diseases show that the proposed model can boost detection performance by more than 10%, especially in relatively rare classes. |
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture semantic and syntactic information, but no study has examined how information loss in input token characters affects their performance. |
| Approach: | They address this gap by pre-training language models using small subsets of token characters. |
| Outcome: | The proposed model retains 90% and 77% of the full-token model in standard NLU benchmarks and probing tasks even under extreme settings. |
Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to unsupervised Chinese word segmentation require multiple inferences to perform word segmenting. |
| Approach: | They propose a method that integrates the segmentation signal from an unsupervised language model to a pre-trained BERT classifier under a pseudo-labeling framework. |
| Outcome: | The proposed method achieves state-of-the-art performance on the eight UCWS tasks while significantly reducing training time compared to previous approaches. |
EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown that large language models are superior to conventional methods in various tasks. |
| Approach: | They propose a data-independent quantization algorithm that leaves outliers in the weight and quantization ranges . they find the algorithm runs over 10 times faster than the data-dependent methods . |
| Outcome: | The proposed method runs over 10 times faster than the data-dependent methods. |
Polar Ducks and Where to Find Them: Enhancing Entity Linking with Duck Typing and Polar Box Embeddings (2023.emnlp-main)
Copied to clipboard
Mattia Atzeni, Mikhail Plekhanov, Frederic Dreyer, Nora Kassner, Simone Merello, Louis Martin, Nicola Cancedda
| Challenge: | Entity linking methods based on dense retrieval are often not efficient in large-scale applications as they are sensitive to the structure of the embedding space. |
| Approach: | They propose a method to infuse structural information into the space of entity representations by using prior knowledge of entity types. |
| Outcome: | The proposed method outperforms other type-aware approaches and matches generative models with 18 times more parameters. |
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu
| Challenge: | Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement. |
| Approach: | They propose a method that involves tuning a small set of soft prompts for pre-trained language models. |
| Outcome: | The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark. |
What’s “up” with vision-language models? Investigating their struggle with spatial reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has re-surfaced a concern that has long plagued vision-language models: poor performance on simple tasks like attribute attachment, counting, etc. |
| Approach: | They evaluate 18 vision-language models and find they perform poorly on VQAv2 . they find that popular vision-linguistic pretraining corpora lack reliable data for learning spatial relationships . |
| Outcome: | The new models are compared with existing datasets on what'sup and visual-language models . they achieve 56% accuracy on the new benchmarks compared to 99% for humans . |
IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using manual data analysis, dataset refinement approaches are often unable to cover all the potential biased features. |
| Approach: | They propose an iterative bias-aware dataset refinement framework which debiases NLU models without predefining biased features. |
| Outcome: | The proposed framework outperforms existing methods and is compatible with model-centric methods. |
Learning Preference Model for LLMs via Automatic Preference Data Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing training methods for large language models rely on human-annotated data. |
| Approach: | They propose to learn the preference model for LLMs via automatic preference data generation (AutoPM) using HHH-guided preference data, they show reliability and potential . |
| Outcome: | The proposed approach enables LLMs to learn human preferences and align with human values. |
Multilingual k-Nearest-Neighbor Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | knearest-neighbor machine translation has shown remarkable improvements in machine translation quality by creating a datastore of cached examples. |
| Approach: | They combine k-nearest-neighbor machine translation with a datastore of cached examples to achieve significant improvements in machine translation quality. |
| Outcome: | The proposed approach improves translation quality by 5.3x in multilingual datastores, while limiting performance to low-resource languages. |
Understanding Computational Models of Semantic Change: New Insights from the Speech Community (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using type-level and token-level word embeddings, we obtain semantic change estimates from type-based models and empirical linguistic properties. |
| Approach: | They analyze 40 target words with type-level and token-level word embeddings, empirical linguistic properties, and speaker-provided acceptability ratings and qualitative remarks. |
| Outcome: | The proposed models are able to describe the sociolinguistic issue of contact-induced semantic shifts in Quebec English and are validated by qualitative interviews with 15 speakers from Montreal. |
Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing attempts to generalize VQA focus on unimodal aspects, overlooking enhancements in multimodal aspects. |
| Approach: | They propose to decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC) they prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC. |
| Outcome: | The proposed model improves multimodal predictions by emphasizing causal reasoning factors. |
StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs. |
| Approach: | They propose an iterative Reading-then-Reasoning framework to solve question answering tasks based on structured data. |
| Outcome: | The proposed framework improves the reasoning ability of large language models over structured data under the few-shot and zero-shot settings. |
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research examines simple classification tasks, but ability of LMs to classify on complex tasks is less well understood. |
| Approach: | They analyze a Supreme Court opinion annotated by a team of domain experts . they find generative models perform poorly when given instructions equal to human annotators . |
| Outcome: | The proposed model performs poorly when given instructions equal to instructions given to human annotations . strongest results derive from fine-tuning models on the annotated dataset . |
Model-tuning Via Prompts Makes NLP Models Adversarially Robust (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models are typically adapted to downstream tasks by appending a randomly initialized multilayer perceptron to their topmost representation layer and fine-tuning the entire model on a downstream task. |
| Approach: | They propose to append a multilayer perceptron to a CLS token and fine-tune the entire model on a downstream task. |
| Outcome: | The proposed model-tuning via prompts outperforms adversarial training-based state-of-art defenses by 3.5% and improves against adversarials by 8% over standard methods. |
Learning Co-Speech Gesture for Multimodal Aphasia Type Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Aphasia is a language disorder caused by brain damage affecting speech functions . a detailed diagnosis of aphasia type is imperative for effective treatment . but, little attention has been paid to developing methods to detect different types of sphasis . |
| Approach: | They propose a multimodal graph neural network for aphasia type detection using co-speech gestures and corresponding speech and gesture patterns. |
| Outcome: | The proposed model outperforms existing methods in F1 and 84.2% of cases. |
STINMatch: Semi-Supervised Semantic-Topological Iteration Network for Financial Risk Detection via News Label Diffusion (2023.emnlp-main)
Copied to clipboard
Xurui Li, Yue Qin, Rui Zhu, Tianqianjin Lin, Yongming Fan, Yangyang Kang, Kaisong Song, Fubang Zhao, Changlong Sun, Haixu Tang, Xiaozhong Liu
| Challenge: | Commercial news provides rich semantics and timely information for automated financial risk detection. |
| Approach: | They propose a semi-supervised Semantic-Topological Iteration Network, STINMatch, along with a news-enterprise knowledge graph to endorse the risk detection enhancement. |
| Outcome: | The proposed model outperforms existing models in terms of generalization and semantics and annotation. |
Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | toxicity detection models focus on marginalized groups, but they obscure harms faced by intersectional subgroups. |
| Approach: | They use outlier detection to identify text about people with demographic attributes distant from the "norm" they find model performance is worse for demographic outliers than non-outliers . |
| Outcome: | The proposed model performance is worse for outliers than non-outliers, the authors say . their analysis also shows that outlier analysis can identify harms faced by intersectional groups . |
Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field. |
| Approach: | They propose a framework for testing category-level perceptual grounding in multi-modal language models by using separate neural networks to generate and interpret descriptions of visual categories. |
| Outcome: | The proposed framework compares prototype- and interpretation-based representations with the performance of the generation model and the interpretation model, which is an indicator of perceptual grounding. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |
ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) have boosted interest on dialogue agents . however, research on these tasks is limited by the insufficiency of public datasets . stance detection and debate summarization are key for engaging argumentative dialogues - but are not available for non-English languages. |
| Approach: | They propose to use ORCHID to benchmark stance detection and debate summarization in Chinese debates. |
| Outcome: | The proposed task is based on 1,218 real-world debates conducted in Chinese on 476 unique topics. |
On the Benefits of Learning to Route in Mixture-of-Experts Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Mixture-of-Expert (MoE) models allow us to scale up model sizes while keeping the amount of compute time fixed. |
| Approach: | They propose to use a router to route inputs to experts in a layer to scale up model sizes while keeping the amount of compute time fixed. |
| Outcome: | The proposed model scales up with the help of a router that routes input tokens to experts in a layer and shows that it is more efficient than a non-trainable router. |
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, Ankur Parikh
| Challenge: | evaluating the quality of generated text is a difficult problem for large language models. |
| Approach: | They propose a dataset for multilingual, multifaceted summarization evaluation. |
| Outcome: | The proposed dataset can be used to train multilingual summarization systems . it shows that the dataset performs well on the out-of-domain meta-evaluation benchmarks TRUE and mFACE . |
Query2doc: Query Expansion with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for sparse and dense retrieval have limited success on popular datasets. |
| Approach: | They propose a query expansion approach that generates pseudo-documents by few-shot prompting large language models and then expands the query with generated pseudo-docs. |
| Outcome: | The proposed method boosts the performance of BM25 on ad-hoc IR datasets by 3% to 15% without any model fine-tuning. |
We Need to Talk About Reproducibility in NLP Model Comparison (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that standard splits produce low reproducible and unreliable conclusions . reproducibility of empirical experimental conclusions is a problem in NLP domain . |
| Approach: | They propose to transform the reproducibility of a model comparison into a probabilistic function . they propose to use a regularized corpus splitting strategy to estimate the model's performance . |
| Outcome: | The proposed estimator achieves a high SNR and significantly increases reproducibility. |
Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing data for instruction-tuning are inadequate for a wide range of tasks, limiting the scope for nuanced comprehension and interactions within these domains. |
| Approach: | They propose to use Large Language Models to explore a multitude of variations or possibilities to improve instruction-tuning data by active exploration. |
| Outcome: | The proposed approach improves domain-specific instruction coverage and shows significant improvements over baselines. |
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
InterFair: Debiasing with Natural Language Feedback for Fair Interpretable Predictions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing methods in NLP models focus on isolating information related to a sensitive attribute (e.g., gender or race) but instead argue that a favorable debiaser should use sensitive information ‘fairly,’ with explanations, rather than blindly eliminating it. |
| Approach: | They propose that a favorable debiasing method should use sensitive information ‘fairly,’ with explanations, rather than blindly eliminating it. |
| Outcome: | The proposed approach reduces bias in explanations while maintaining the same prediction accuracy. |
Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring via Constructing the Optimal Subgraph of Demonstrations and Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models as chatbots can cause hallucinations and lack of empathy, authors report . a dimension-agnostic scoring method is proposed to improve the performance of chatbot performance . |
| Approach: | They propose a dimension-agnostic scoring method that leverages in-context learning . they propose to automatically generate prompts and then request the LLM multiple times . |
| Outcome: | The proposed method outperforms baselines on five datasets. |
Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scaling analysis is a technique that assigns a political actor a score on a predefined scale based on 'typically long' text. |
| Approach: | They propose to use label aggregation and long-input-Transformer-based models to automatically scale political-party manifestos. |
| Outcome: | The proposed models can scale political platforms on a predefined scale based on 'left-right' scales and work robustly across domains and languages. |
ART: rule bAsed futuRe-inference deducTion (2023.emnlp-main)
Copied to clipboard
Mengze Li, Tianqi Zhao, Bai Jionghao, Baoyi He, Jiaxu Miao, Wei Ji, Zheqi Lv, Zhou Zhao, Shengyu Zhang, Wenqiao Zhang, Fei Wu
| Challenge: | Existing studies focus on language-based premises and deduce valid conclusions from visual observations. |
| Approach: | They propose a rule-based deductive reasoning task that uses video to deduce the correct future event . they use commonsense knowledge to annotate video and a strong baseline to conduct reasoning . |
| Outcome: | Empirical studies validate the rationality of ARTNet in deductive reasoning upon visual observations . ART is a method that rigorously follows a set of explicit constraints to deduce valid conclusions from empirical facts . |
EpiK-Eval: Evaluation for Language Models as Epistemic Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Developing systems that can reason through language understanding has been a cornerstone in natural language processing research. |
| Approach: | They propose a question-answering benchmark to evaluate LLMs' ability to combine knowledge from different training documents within their parameter space. |
| Outcome: | The proposed benchmark aims to evaluate LLMs' ability to combine knowledge from different training documents within their parameter space. |
From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work in explainable COC has been limited to annotations by a single expert. |
| Approach: | They construct a two-level task-independent taxonomy from a dataset obtained from two experts in the domain of international human rights law . they find disagreements stem from underspecification of the legal context . |
| Outcome: | The proposed dataset is the first in legal NLP that focuses on human label variation. |
On Bilingual Lexicon Induction with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to induction bilingual lexicons still require cross-lingual word representations . a recent study shows that few-shot prompting with in-context examples from nearest neighbours achieves the best performance . |
| Approach: | They examine whether it is possible to prompt and fine-tune multilingual LLMs for BLI . they experiment with 18 open-source text-to-text mLLMs of different sizes . |
| Outcome: | The proposed approach is compared with existing approaches on two standard BLI benchmarks covering a range of typologically diverse languages. |
Statistical Depth for Ranking and Characterizing Transformer-Based Text Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generalized transformer-based text embedding models have produced state of the art performance results on a variety of tasks such as natural language inference (NLI) |
| Approach: | They propose a statistical depth to measure distributions of transformer-based text embeddings and an associated rank sum test to characterize distributions in synthetic and human-generated corpora. |
| Outcome: | The proposed method improves performance over baseline methods on six text classification tasks. |
CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning is an effective way of aligning large language models with private instruction data. |
| Approach: | They propose a training-free strategy to derive improved emulators from LLMs by using Offsite-Tuning (OFT) they propose CRaSh, which transfers transformer blocks between centralized LLM and downstream emulators . |
| Outcome: | The proposed technique boosts performance of large language models with billions of parameters. |
From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues (2023.emnlp-main)
Copied to clipboard
| Challenge: | Understanding emotions during conversation is a fundamental aspect of human communication. |
| Approach: | They propose an approach that integrates commonsense information with dialogue context to facilitate a deeper understanding of emotions. |
| Outcome: | The proposed approach improves ERC for code-mixed conversations by integrating commonsense with dialogue context. |
Large Language Models are biased to overestimate profoundness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have been suggested to approach AI . however, it is still unclear whether LLMs possess similar reasoning abilities to humans . |
| Approach: | They evaluate GPT-4 and other LLMs in judging the profoundness of mundane statements . they find a significant correlation between the LLM and humans . |
| Outcome: | The proposed model overestimates the profoundness of nonsensical statements . the model overstates the profound nature of non-senior statements, the study finds . |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
DIVE: Towards Descriptive and Diverse Visual Commonsense Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Towards human-level visual understanding, visual commonsense generation has been introduced . but current research on visual commonense generation ignores an important human cognitive ability . |
| Approach: | They propose a visual commonsense generation framework to improve inferences by visual common sense generation. |
| Outcome: | The proposed framework outperforms state-of-the-art models in descriptiveness and diversity . human evaluations confirm that the framework aligns closely with human judgments on descriptiveness . |
Towards Conceptualization of “Fair Explanation”: Disparate Impacts of anti-Asian Hate Speech Explanations on Content Moderators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work at the intersection of AI explainability and fairness has focused on how explanations can improve human-plus-AI task performance . |
| Approach: | They propose to characterize what constitutes an explanation that is itself "fair" they use not just accuracy and label time, but psychological impact of explanations on different groups . |
| Outcome: | The proposed method is based on content moderation of potential hate speech and its differential impact on Asian vs. non-Asian proxy moderators across explanation approaches. |
Bridging Background Knowledge Gaps in Translation with Automatic Explicitation (2023.emnlp-main)
Copied to clipboard
| Challenge: | despite its potential to help users, NLP research on explicitation is limited because of the lack of adequate evaluation methods. |
| Approach: | They propose automatic methods to generate explicitations from a Wikipedia dataset . they use both intrinsic and extrinsic evaluation to evaluate the system's effectiveness . |
| Outcome: | The proposed system bridges the gap between the source speaker and the target audience . it is effective based on intrinsic and extrinsic evaluation, the authors show . |
A Quality-based Syntactic Template Retriever for Syntactically-Controlled Paraphrase Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing syntactically-controlled paraphrase generation models perform well with human-annotated or well-chosen syntaktic templates. |
| Approach: | They propose a quality-based Syntactic Template Retriever to retrieve templates based on the quality of the to-be-generated paraphrases. |
| Outcome: | The proposed algorithm can generate high-quality paraphrases without sacrificing quality. |
Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a shared vocabulary is common practice in multilingual machine translation . however, when words overlap is small, e.g., using different writing systems, knowledge transfer is inhibited . |
| Approach: | They propose a re-parameterized method for building word embeddings using word equivalence classes and graph networks to fuse word embeds across languages. |
| Outcome: | The proposed method achieves evident BLEU improvements on high- and low-resource MNMT scenarios. |
Quantifying the redundancy between prosody and text (2023.emnlp-main)
Copied to clipboard
Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, Tamar Regev
| Challenge: | Existing studies suggest partial redundancy between prosody and linguistic information. |
| Approach: | They use large language models to estimate how much information is redundant between prosody and the words themselves. |
| Outcome: | The proposed model can predict prosodic features across prosodic features, including intensity, duration, pauses, and pitch contours. |
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts in natural language processing (NLP) commonsense reasoning research have produced a number of new datasets and benchmarks. |
| Approach: | They propose a manually-curated, multi-task benchmark that evaluates models' ability to apply commonsense reasoning in the context of six real-world NLP tasks. |
| Outcome: | The proposed benchmark evaluates the ability of models to apply commonsense reasoning in the context of six real-world NLP tasks. |
A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing annotated training datasets hinder development of supervised learning models for multimedia content . lack of annotating benchmarks hinders development of models with satisfactory performance . a recent study shows that large language models have zero-shot performance in multimedia understanding . |
| Approach: | They propose to verbalize long videos to generate their descriptions in natural language . they then perform video-understanding tasks on the generated story as opposed to the original video . |
| Outcome: | The proposed method achieves better results than baselines for video understanding. |
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a promising capability for large language models (LLMs) but its underlying mechanism remains unexplored. |
| Approach: | They propose a demonstration compression technique to expedite inference and an analysis framework for diagnosing ICL errors in GPT2-XL. |
| Outcome: | The proposed method improves ICL performance and expedites inference. |
Prompting Scientific Names for Zero-Shot Species Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | We use visionlanguage models (VLMs) to recognize images of common objects in a zero-shot fashion, but it is underexplored how to use CLIP for zero- shot species recognition of highly specialized concepts. |
| Approach: | They propose a method to translate scientific names to common English names and use them in prompts to improve their performance. |
| Outcome: | The proposed method performs poorly for species recognition with prompts that use scientific names, e.g., “a photo of Lepus Timidus” (which is a scientific name in Latin) and additionally use them in the prompts. |
Active Learning for Natural Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to NLG are limited by the lack of annotated data. |
| Approach: | They propose to use active learning to reduce the cost of manual annotation to improve annotation efficiency by selecting the most informative examples to label. |
| Outcome: | The proposed approach surpasses baseline of random example selection in some cases but not in others. |
Re3Dial: Retrieve, Reorganize and Rescale Conversations for Long-Turn Open-Domain Dialogue Pre-training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-training models lack long-turn dialogue sessions due to the scarcity of long-term sessions. |
| Approach: | They propose a framework that can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
| Outcome: | The proposed framework can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup (2023.emnlp-main)
Copied to clipboard
| Challenge: | Disfluency detection models focus on individual utterances, but discontinuities in spoken transcripts occur across multiple turns. |
| Approach: | They propose a multi-turn "cleanup task" to detect discontinuities in spoken conversations . they leverage two modeling approaches for experimental evaluation as benchmarks . |
| Outcome: | The proposed task detects "discontinuities" in spoken conversations that can be removed . the results are compared with existing methods and are expected to be validated in the future . |
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov
| Challenge: | Language models have evolved from being research prototypes to commercialized products offered as web APIs. |
| Approach: | They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages. |
| Outcome: | The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results. |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |
MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark (2023.emnlp-main)
Copied to clipboard
Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, Maria Bielikova
| Challenge: | MULTITuDE benchmarks lack authentic and machine-generated text in languages other than English . defining characteristic of new generation of LLMs is increased quality of text . |
| Approach: | They propose a benchmarking dataset for multilingual machine-generated text detection that compares detectors with authentic and machine-generated texts in 11 languages. |
| Outcome: | The proposed dataset compares detectors with zero-shot and fine-tuned detectors in 11 languages. |
Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quantisation methods mainly focus on 8-bit LLMs . a lack of scaling offsets in the quantisation process limits the use of LLM inference. |
| Approach: | They propose to use block quantisations to reduce scaling offsets in Large language models . they find that the block quantizations reduce scaling only from an arithmetic perspective . |
| Outcome: | The proposed methods reduce scaling offsets solely from an arithmetic perspective without additional treatments in the computational path. |
Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition (2023.emnlp-main)
Copied to clipboard
Srijith Radhakrishnan, Chao-Han Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez-Cabrero, Jesper Tegnér
| Challenge: | Existing methods for generative error correction in automatic speech recognition (ASR) use a two-pass reranking paradigm to generate n-best hypotheses. |
| Approach: | They propose a cross-modal fusion technique for generative error correction in automatic speech recognition. |
| Outcome: | The proposed technique shows a 37.66% improvement in word error rate relative to the n-best Oracle. |
Reducing Sequence Length by Predicting Edit Spans with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. |
| Approach: | They propose to predict edit spans for local sequence transduction tasks by predicting edit span with a position of the source text and corrected tokens. |
| Outcome: | The proposed method reduces the length of the target sequence and the computational cost for inference by as small as 21%. |
Instruct and Extract: Instruction Tuning for On-Demand Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models with instruction-following capabilities are not suitable for long-tail ad hoc extraction use cases for non-expert users. |
| Approach: | They propose a task that follows instructions to extract the desired content from the associated text and present it in a structured tabular format. |
| Outcome: | The proposed paradigm outperforms existing open-source models of similar size in terms of information extraction. |
Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for large language models (LLMs) are inadequate for conversational recommender systems. |
| Approach: | They propose an evaluation approach based on LLMs that harnesses LLM-based user simulators to evaluate ChatGPT's performance. |
| Outcome: | The proposed evaluation approach can simulate various system-user interaction scenarios. |
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on whether the reasoning chain leads to the correct conclusion, but this view may confound reasoning quality with other spurious shortcuts to predict the answer. |
| Approach: | They propose a framework that evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, respectively. |
| Outcome: | The proposed framework evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, which is helpful towards deriving the generated answer. |
Expand, Highlight, Generate: RL-driven Document Generation for Passage Reranking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies use large language models to generate training data for ranking models. |
| Approach: | They propose a pipeline that generates synthetic documents from queries using large language models . they propose RL-based reinforcement learning to optimize the pipeline . |
| Outcome: | The proposed pipeline outperforms existing state-of-the-art methods in generating synthetic documents more effectively. |
Transformer-based Live Update Generation for Soccer Matches from Microblog Posts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems to generate sports updates from tweets are not able to handle vast amounts of diverse tweets, and this remains a challenge for future studies. |
| Approach: | They propose to generate live updates for soccer matches from tweets using a large pre-trained language model and incorporate a classifier to control the number of updates and a mechanism to reduce redundancy of duplicate and similar updates. |
| Outcome: | The proposed system can generate live updates for soccer matches from tweets and achieve high performance by considering preceding updates. |
Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP, however, data quality might have already become the bottleneck to unlock further gains. |
| Approach: | They propose a general method for improving model performance in the presence of noisy training data based on self-influence and bandit curriculum learning. |
| Outcome: | The proposed method improves model performance in machine translation, question answering and text classification, building up on approaches to self-influence calculation and automated curriculum learning. |
Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews (2023.emnlp-main)
Copied to clipboard
| Challenge: | Medical systematic reviews are time-consuming and often generate inaccurate outputs . authors: a model that generates scientific-sounding outputs can be unusable at best . |
| Approach: | They conduct interviews with systematic review experts to characterize perceived utility and risks of LLMs in medical evidence reviews. |
| Outcome: | a new study characterizes perceived utility and risks of medical evidence reviews . experts say they can assist in the writing process by drafting summaries, distilling information . authors say they expect the model to be more accurate and more reliable . |
PromptST: Abstract Prompt Learning for End-to-End Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that PromptST can improve speech-to-text translation by capturing richer linguistic knowledge. |
| Approach: | They propose a plug-in prompt-enhanced S2T model that captures richer linguistic knowledge . they use a 10GB linguistic probing benchmark to investigate the fusion of speech and text features . |
| Outcome: | The proposed model can improve on a strong baseline by capturing richer linguistic knowledge. |
Text Rendering Strategies for Pixel Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to rendering text use a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks due to redundancy in the input representations. |
| Approach: | They propose four approaches to rendering text in a PIXEL model using character bigrams and patch frequency biases. |
| Outcome: | The proposed models perform better on sentence-level tasks without compromising performance on token-level or multilingual tasks. |
APoLLo : Unified Adapter and Prompt Learning for Vision Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | APoLLo improves generalization capabilities of vision-language pretrained models . despite being largely successful in terms of generalization, these models are difficult to fine-tune for few-shot learning-based downstream tasks. |
| Approach: | They propose a multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. |
| Outcome: | The proposed approach improves generalization capabilities of vision-language pretrained models . it achieves a relative gain of 6.03% over MaPLe on 10 diverse datasets . |
SAMRank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT-2 (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for keyphrase extraction use contextualized embeddings to capture semantic relevance between words, sentences, and documents. |
| Approach: | They propose an unsupervised keyphrase extraction approach that uses only a self-attention map in a pre-trained language model to determine the importance of phrases. |
| Outcome: | The proposed approach outperforms embedding-based models on three keyphrase extraction datasets. |
Contrastive Learning for Inference in Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models show remarkable advances in inference tasks, but their performance in inductive reasoning is far behind deductive reasoning. |
| Approach: | They propose to use negative samples to analyze inferences based on the semantic information gap between dialogue contexts and desired inference. |
| Outcome: | The proposed model improves inference generation by feeding negative samples to the models. |
Editing Large Language Models: Problems, Methods, and Opportunities (2023.emnlp-main)
Copied to clipboard
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in model editing for LLMs have created challenges and opportunities for the community. |
| Approach: | They propose to alter the behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
| Outcome: | The proposed method alters behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
MarkQA: A large scale KBQA dataset with numerical reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing KBQA datasets are insufficient for numerical reasoning . existing KBqa datasets lack multi-hop reasoning and numerical reasoning. |
| Approach: | They propose a task that necessitates the ability to perform multi-hop reasoning and numerical reasoning. |
| Outcome: | The proposed task necessitates the ability to perform multi-hop reasoning and numerical reasoning. |
Comparing Biases and the Impact of Multilingual Training across Multiple Languages (2023.emnlp-main)
Copied to clipboard
Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, Dan Roth
| Challenge: | Currently, studies on bias and fairness in natural language processing focus on a single language and/or across few attributes (e.g. gender, race). However, biases can manifest differently across languages for individual attributes. |
| Approach: | They adapt existing sentiment bias templates in English to Italian, Chinese, Hebrew, and Spanish for race, religion, nationality, and gender. |
| Outcome: | The proposed model favors groups that are dominant in each language's culture, indicating bias amplification, after multilingual finetuning. |
HutCRS: Hierarchical User-Interest Tracking for Conversational Recommender System (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing CRSs assume that users like all attributes of the target item and dislike those unrelated to it, which can introduce bias in attribute-level feedback and impede the system’s ability to accurately identify the target items. |
| Approach: | They propose a framework that allows users to explicitly acquire user preferences through natural language conversations by providing explicit answers (yes/no) for each attribute they require. |
| Outcome: | The proposed framework portrays the conversation as a hierarchical interest tree that consists of two stages. |
Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT (2023.emnlp-main)
Copied to clipboard
Xiaoshuai Song, Keqing He, Pei Wang, Guanting Dong, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, Weiran Xu
| Challenge: | Existing methods to fine-tune discriminative models address these challenges by focusing on in-domain intents. |
| Approach: | They evaluate ChatGPT on OOD intent discovery and generalized intent discovery tasks . they outline the strengths and weaknesses of ChatGPt and outline their results . |
| Outcome: | The proposed task aims to extend a closed intent classifier to open-world intent sets. |
The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the rise of the prompting paradigm with the scaling breakthrough of very large language models, understanding the mechanism of model fine-tuning remains an important endeavor. |
| Approach: | They analyze the masked language modeling pretraining objective function from the perspective of the Distributional Hypothesis and examine whether the distributional property leads to better sample efficiency and better generalization capability of pretrained models. |
| Outcome: | The proposed model pretraining objective function improves sample efficiency and generalization capability but does not explain the generalization ability of natural language models. |
Simple and Effective Input Reformulations for Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Foundation language models learn from their finetuning input context in different ways. |
| Approach: | They propose three different data efficient techniques to improve translation performance . they reformulate inputs during finetuning for challenging translation tasks . |
| Outcome: | The proposed techniques show significant improvements on the Flores200 translation benchmark. |
Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for faithfulness of response are not aligned with human judgments. |
| Approach: | They propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. |
| Outcome: | The proposed metric improves on BEGIN benchmarks and shows that it generates more faithful responses than standard decoding techniques. |
The ACL OCL Corpus: Advancing Open Science in Computational Linguistics (2023.emnlp-main)
Copied to clipboard
| Challenge: | ACL OCL is a scholarly corpus derived from the ACL Anthology . it provides metadata, PDF files, citation graphs and additional structured full texts . |
| Approach: | They present ACL OCL, a scholarly corpus derived from the ACL Anthology . it integrates metadata, PDF files, citation graphs and additional structured full texts . they highlight how it applies to observe trends in computational linguistics . |
| Outcome: | The ACL OCL spans seven decades and contains 73,285 papers . the scholarly corpus is based on the ACL Anthology and is available from HuggingFace . |
Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have demonstrated the ability of neural language models to learn linguistic properties without direct supervision. |
| Approach: | They propose to use an artificial corpus generated by a PCFG to control the gender distribution in training data and determine under which conditions a model correctly captures gender information. |
| Outcome: | The proposed approach allows to control the gender distribution in training data and determine under which conditions a model correctly captures gender information or appears gender-biased. |
Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for named entity recognition have limited range when applied to long documents . lack of supervision for such a task means one has to settle for unsupervised approaches. |
| Approach: | They propose to train a neural context retrieval model based on an instruction-tuned large language model. |
| Outcome: | The proposed method outperforms baselines on an English literary dataset . pre-trained transformer-based models can perform named entity recognition (NER) with great accuracy, but limited range when applied to long documents such as whole novels. |
Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting (2023.emnlp-main)
Copied to clipboard
Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, Jilin Chen
| Challenge: | Existing studies on diversity in large language models focus on the understudied class of fairness and inclusion concern in LLMs. |
| Approach: | They propose a technique to measure diversity in generated responses along people and culture axes by collective-critique and self-voting. |
| Outcome: | The proposed approach outperforms baseline methods and human evaluations with human and automated evaluations. |
Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have exhibited remarkable capabilities in text generation tasks, but their utilization carries inherent risks, including plagiarism and the dissemination of fake news. |
| Approach: | They propose to use a dataset to construct an AI-generated student essay that employs a range of text perturbation methods to evade detection. |
| Outcome: | The proposed methods evade detection and maintain quality of the generated essays while avoiding plagiarism and fake news. |
Contextual Interaction for Argument Post Quality Assessment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for assessing the quality of natural language arguments are limited . existing methods focus on evaluating individual argument posts, but they often fail to distinguish between arguments with a narrow quality gap. |
| Approach: | They propose to use supervised contrastive learning to model arguments' quality . large language models with in-context examples harness the power of LLMs . |
| Outcome: | The proposed approach outperforms state-of-the-art models on a publicly available dataset . it shows that the LLMs with in-context examples are more effective than baseline models . |
Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification (2023.emnlp-main)
Copied to clipboard
Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, Vittorio Castelli
| Challenge: | Existing methods for IC training do not provide sufficient examples for each intent . a novel pre-training method is proposed to provide a better understanding of intents . |
| Approach: | They propose a method that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks. |
| Outcome: | The proposed method achieves 5.4% and 4.0% higher accuracy than the current state-of-the-art method on four IC datasets. |
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored using large language models to generate synthetic datasets . however, the effectiveness of the LLM-generated synthetic data is inconsistent across different classification tasks. |
| Approach: | They propose to use large language models to generate synthetic datasets to better understand factors that moderate the effectiveness of LLM-generated synthetic data. |
| Outcome: | The results show that subjectivity is negatively associated with the performance of the model trained on synthetic data. |
GazeVQA: A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the use of exocentric and egocentric videos in video question answering are focusing on eye-gaze information. |
| Approach: | They propose a task-oriented VQA dataset that captures eye-gaze information . they propose assisting models that ground the perceptual input into semantic information based on three different answer types . |
| Outcome: | The proposed model can ground the perceptual input into semantic information while reducing ambiguities. |
People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past work has shown that counterfactually augmented data (CADs) can improve models' performance on out-of-domain tests. |
| Approach: | They use Polyjuice, ChatGPT, and Flan-T5 to automatically generate CADs . they find that CAD generates a model that flips the original label with minimal changes . |
| Outcome: | The proposed model improves model robustness on out-of-domain test sets and individual data points. |
Unraveling Feature Extraction Mechanisms in Neural Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks have become indispensable across a variety of natural language processing tasks. |
| Approach: | They propose a theoretical approach based on Neural Tangent Kernels to investigate neural networks' internal mechanisms. |
| Outcome: | The proposed approach can be applied to analyze language modeling tasks . it shows that the choice of activation function can affect feature extraction . |
CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that dense retrieval models are better at obtaining query-informed representations. |
| Approach: | They propose a dual-encoder approach that computes latent representations of query and document independently, but inference replaces the real query with a generated one. |
| Outcome: | The proposed approach outperforms previous dense retrieval models on in-domain and out-of-domain datasets. |
Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a small number of gallery data points are frequently retrieved, resulting in a decline in retrieval performance. |
| Approach: | They propose a framework that leverages both gallery and query data to address hubness . they propose dual inverted softmax and dual dynamic inverted hardmax methods to normalize similarity . |
| Outcome: | The proposed framework reduces the occurrence of hubs during inference while improving similarity between non-hubs and queries. |
E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Empathy is a desirable human trait that improves the emotional perceptivity in emotion-bonding social activities. |
| Approach: | They propose a framework that integrates emotion correlation learning, utilization, and supervising. |
| Outcome: | The proposed framework improves empathetic perception and expression on a humanized dialogue dataset. |
What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on decontextualised concept embeddings from language models has focused on capturing taxonomic structure in concepts. |
| Approach: | They propose a strategy for identifying what different concepts have in common with others and representing them in terms of their properties. |
| Outcome: | The proposed approach improves the performance of state-of-the-art models for a multi-label classification problem. |
ALDi: Quantifying the Arabic Level of Dialectness of Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on Dialect Identification (DI) on the sentence level has focused on binary tasks, whereas ALDi treats the task as binary. |
| Approach: | They propose a dataset which contains 127,835 sentences manually labeled with their level of dialectness. |
| Outcome: | The proposed model can identify dialectness on a range of other corpora, providing a more nuanced picture than traditional DI systems. |
3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding (2023.emnlp-main)
Copied to clipboard
| Challenge: | 3D visual grounding aims to localize the desired objects in a 3D point cloud by a free-form language description. |
| Approach: | They propose a relation-aware framework which captures relative spatial relationships between objects and enhances object attributes. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three benchmarks . it captures relative spatial relationships between objects and enhances object attributes . |
Goal-Driven Explainable Clustering via Language Descriptions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing formulations neither consider the users’ goals nor explain clusters’ meanings. |
| Approach: | They propose a task formulation that represents both the goal and the explanations as free-form language descriptions. |
| Outcome: | The proposed method produces more accurate and goal-related explanations than previous methods. |
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large-scale pretrained language models store factual knowledge, but large variations are observed across languages. |
| Approach: | They propose a ranking-based consistency metric to evaluate cross-lingual consistency of factual knowledge in multilingual PLMs. |
| Outcome: | The proposed metric evaluates cross-lingual consistency of factual knowledge across languages independently from accuracy. |
Learning from Mistakes via Cooperative Study Assistant for Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated their potential to refine their generation based on their own feedback, but the feedback from LLM itself is often inaccurate, thereby limiting its benefits. |
| Approach: | They propose a framework with an auxiliary agent to assist the main LLM in learning from mistakes through interactive cooperation. |
| Outcome: | The proposed framework can significantly boost large language models by an accuracy margin of up to 6.6 on BBH and 12.6 on BBQ. |
Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Among the minority groups under-represented in AI, data from low-income households are often overlooked in data collection and model evaluation. |
| Approach: | They evaluate the performance of a vision-language model on a geo-diverse dataset . they highlight insights that can help mitigate these issues and propose actionable steps for economic-level inclusive AI development. |
| Outcome: | The proposed model performs lower for the poorer groups than the wealthier groups across topics and countries. |
Conceptor-Aided Debiasing of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models reflect inherent social biases of their training corpus. |
| Approach: | They propose two methods to identify and remove the bias subspace in pre-trained large language models such as BERT and GPT by applying conceptors to a conceptor NOT operation. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) debiasing while maintaining LLMs’ performance on the GLUE benchmark. |
AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation parsers have improved in recent years, but not solved. |
| Approach: | They propose an evaluation suite that evaluates AMR parsers on a range of phenomena . they find that current parser outputs are far from satisfactory . |
| Outcome: | The proposed evaluation suite reveals the abilities and shortcomings of current parsers. |
Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | auxiliary tasks are highly consistent with end-to-end speech translation (ST) but their effectiveness has not been thoroughly studied. |
| Approach: | They propose an improved multi-task learning approach for the ST task that bridges the modal gap by mitigating the difference in length and representation. |
| Outcome: | The proposed approach achieves state-of-the-art on the MuST-C dataset with 20.8% of training time required by the current SOTA method. |
AD-NLP: A Benchmark for Anomaly Detection in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for Anomaly Detection in text have shown strong empirical results on ad-hoc anomaly setups that are usually made by downsampling some classes of a labeled dataset. |
| Approach: | They propose a unified benchmark for detecting various types of anomalies . they evaluate two strong shallow baselines and two current state-of-the-art neural approaches . |
| Outcome: | The proposed benchmarks provide insights into the knowledge the neural models are learning when performing the task. |
Enhancing the Ranking Context of Dense Retrieval through Reciprocal Nearest Neighbors (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sparse annotation poses persistent challenges to training dense retrieval models . despite potential future endeavors to extend annotation, issue of false negatives persists . |
| Approach: | They propose a method that smooths out the annotation of unlabeled relevant documents . they use reciprocal nearest neighbors to estimate relevance and rerank candidates . |
| Outcome: | The proposed method reduces the issue of false negatives in contrastive learning by reducing sparsity. |
Cross-Lingual Cross-Target Stance Detection with Dual Knowledge Distillation Framework (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on stance detection were conducted mainly in English due to the low-resource problem in most non-English languages. |
| Approach: | They propose to use a cross-lingual teacher and a teacher to transfer knowledge from source to target language to bridge the discrepancy between languages. |
| Outcome: | The proposed framework bridges the discrepancy between languages and generalizes the knowledge to unseen targets in target language. |
PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs (2023.emnlp-main)
Copied to clipboard
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Chuan He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu
| Challenge: | PRESTO dataset contains 550K contextual multilingual conversations between humans and virtual assistants. |
| Approach: | They propose to use a dataset of 550K contextual multilingual conversations between humans and virtual assistants to study some of the more challenging aspects of parsing realistic conversations. |
| Outcome: | The dataset contains 550K contextual conversations between humans and virtual assistants. |
An Iteratively Parallel Generation Method with the Pre-Filling Strategy for Document-level Event Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate event roles require a given generation order . parallel methods suffer from inadequate training and manifest zero accuracies on some event roles. |
| Approach: | They propose an iteratively parallel generation method with the Pre-Filling strategy to generate event roles in parallel to avoid order selection. |
| Outcome: | The proposed method outperforms other entity-enhanced models and achieves state-of-the-art performance on two public datasets. |
CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from demographics in social science experiments and public opinion surveys. |
| Approach: | They propose a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic. |
| Outcome: | The proposed framework measures open-ended LLM simulations’ susceptibility to caricature, defined via two criteria: individuation and exaggeration. |
Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach (2023.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating conversational information retrieval systems requires a significant amount of human labor for annotation. |
| Approach: | They propose to use human annotation to calibrate evaluation results to eliminate evaluation biases. |
| Outcome: | The proposed method consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results. |
BERTie Bott’s Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing corpora, WordNet, and dependency parsing are used to build a semantic role labeling system. |
| Approach: | They use existing corpora, WordNet, and dependency parsing to build a Galician dataset for training semantic role labeling systems. |
| Outcome: | The proposed model outperforms the 2009 CoNLL Shared Task by 0.83 on Spanish datasets. |
Program Translation via Code Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Software version migration and program translation are costly parts of the lifecycle of large codebases. |
| Approach: | They propose a model that captures semantic and structural equivalence of code in a language agnostic intermediate representation. |
| Outcome: | The proposed model achieves state-of-the-art performance on CodeXGLUE and TransCoder GeeksForGeeks translation benchmarks. |
FaMeSumm: Investigating and Improving Faithfulness of Medical Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization models produce unfaithful outputs for medical text summarizing . a framework to improve faithfulness is proposed to improve medical text summary accuracy . |
| Approach: | They propose a framework to improve faithfulness by fine-tuning pre-trained language models based on medical knowledge. |
| Outcome: | The proposed framework improves faithfulness on medical summarization tasks. |
Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing grammar-constrained decoding methods are limited to specific tasks . a grammar constraint is used to control the generation of LMs, but it is limited to a few tasks a task is not performed. |
| Approach: | They propose grammar-constrained decoding to control the generation of large language models . they demonstrate that grammars can describe the output space for a wider range of tasks . |
| Outcome: | The proposed grammars outperform unconstrained models on information extraction, entity disambiguation, and constituency parsing. |
Systematic word meta-sense extension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Many words in the lexicon are polysemous in that the same word form can express multiple distinct yet related senses. |
| Approach: | They propose a task to extend word meaning to denote new semantic domains that bear regular semantic relations with existing senses. |
| Outcome: | The proposed method improves language models' ability to extend word meaning on multiple benchmarks of figurative language understanding. |
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics are conflated and can mislead models, resulting in downstream harms. |
| Approach: | They propose a framework for conceptualizing and evaluating the reliability and validity of evaluation metrics based on empirical data. |
| Outcome: | The proposed framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. |
Revisiting the Knowledge Injection Frameworks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Injecting unaligned knowledge tuple into large language models achieves comparable (and sometimes better) results than aligned knowledge. |
| Approach: | They propose a technique to inject random knowledge into large language models to improve performance. |
| Outcome: | The proposed technique overcomes the sanity problem and pushes the performance limit. |
We Are What We Repeatedly Do: Inducing and Deploying Habitual Schemas in Persona-Based Responses (2023.emnlp-main)
Copied to clipboard
| Challenge: | a variety of personas can be elicited from large language models, but they are opaque and unpredictable. |
| Approach: | They propose an approach to dialogue generation that retrieves relevant schemas to condition a large language model to generate persona-based responses. |
| Outcome: | The proposed method captures habitual knowledge and generates persona-based responses from a large language model. |
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work evaluates faithfulness using models trained on related tasks or in-domain synthetic data. |
| Approach: | They propose to do zero-shot faithfulness evaluation with a foundation language model. |
| Outcome: | The proposed model outperforms ChatGPT on faithfulness and inconsistency detection with 24x fewer parameters and is competitive with existing models. |
TaskWeb: Selecting Better Source Tasks for Multi-task NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in NLP has shown that knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. |
| Approach: | They propose a method to quantify task relationships via pairwise task transfer and build smaller training sets that improve zero-shot performances across 11 different target tasks. |
| Outcome: | The proposed method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. |
Improving Bias Mitigation through Bias Experts in Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to mitigate the detrimental effect of bias on the network include debiasing methods that down-weight the biased examples identified by an auxiliary model, which is trained with explicit bias labels. |
| Approach: | They propose a framework that introduces binary classifiers between the auxiliary model and main model, coined bias experts, to reduce the detrimental effect of bias on the network. |
| Outcome: | The proposed approach outperforms the state-of-the-art on various datasets while achieving high performance on in-distribution data. |
Semi-supervised multimodal coreference resolution in image narrations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a semi-supervised approach is used to resolve multimodal coreferences and narrative grounding in a multimodal context. |
| Approach: | They propose a semi-supervised approach that utilizes image-narration pairs to resolve coreferences and narrative grounding in a multimodal context. |
| Outcome: | The proposed approach outperforms baselines quantitatively and qualitatively for coreference resolution and narrative grounding in a multimodal context. |
A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. |
| Approach: | They conduct a comprehensive study on 39 pretrained MLMs to examine their model factors and their social biases. |
| Outcome: | The proposed model factors influence social biases learned by an MLM and their downstream task performance. |
Argument-based Detection and Classification of Fallacies in Political Debates (2023.emnlp-main)
Copied to clipboard
| Challenge: | Fallacies are arguments that employ faulty reasoning, causing inaccurate conclusions and invalid inferences . ad hominem fallacy is one of the most common fallacy labels used in political debates despite its use in many scenarios . |
| Approach: | They extend the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments by incorporating the most recent Trump-Biden debate. |
| Outcome: | The proposed method extends the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments . |
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter. |
| Approach: | They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I. |
| Outcome: | The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers . |
SpEL: Structured Prediction for Entity Linking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking is a key component of structured data creation by linking spans of text to an ontology or knowledge source. |
| Approach: | They propose to use structured prediction for entity linking to classify each input token as an entity and aggregate the token predictions. |
| Outcome: | The proposed system outperforms the state-of-the-art on the commonly used AIDA benchmark dataset for entity linking to Wikipedia. |
Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It’s Best to Relate Perspectives! (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to subjectivity in natural language processing are subjective . authors argue that disagreement should not be regarded as a problem . |
| Approach: | They propose to account for subjective perspectives of individuals and objective concepts that build a common ground between annotators. |
| Outcome: | The proposed architectures increase the averaged annotator-individual F1-scores up to 43% over a majority-label model. |
Explicit Planning Helps Language Models in Logical Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that use pre-trained large language models to perform multi-step logical reasoning have been unable to perform this task. |
| Approach: | They propose a system that uses language models to perform multi-step logical reasoning and incorporates explicit planning into the inference procedure. |
| Outcome: | The proposed system outperforms other competing methods on multiple datasets and significantly outperformed chain-of-thought prompting on the PrOntoQA dataset. |
clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents (2023.emnlp-main)
Copied to clipboard
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, David Schlangen
| Challenge: | Recent work suggests large language models can be understood as (simulators of) such agents. |
| Approach: | They propose a method for systematic evaluation of "Situated Language Understanding Agents" they propose implementing a framework for implementing rules to be played in "self-play" |
| Outcome: | The proposed model can be evaluated in game-like settings, the authors show . they show that the model can follow game-play instructions and perform better than existing models . |
Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences (2023.emnlp-main)
Copied to clipboard
| Challenge: | a common strategy to explain NLP predictions is to highlight salient tokens in their inputs. |
| Approach: | They propose a technique to generate contrastive phrasal highlights that explain the predictions of a semantic divergence model via phrase alignment guided erasure. |
| Outcome: | The proposed techniques match human rationales of cross-lingual semantic differences better than popular post-hoc saliency techniques and help people detect fine-grained meaning differences in human translations and critical machine translation errors. |
Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | LLaMA achieves highest scores in both multilingual and English-only evaluations . foundation models inherit inaccuracies from training corpora, which are propagated downstream to the models that are fine-tuned from them. |
| Approach: | They propose to produce a 20-language factual association dataset and evaluate 5 models in a multilingual test and benchmark a diverse set of 24 models in an English-only test. |
| Outcome: | The proposed model achieves highest scores in multilingual and English-only evaluations . however, it has significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. |
Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fewshot text classification require substantial computing power and data. |
| Approach: | They propose an efficient method to add task and label information to a sentence transformer model by contrastive learning and a triplet loss to enforce training instances to be closest to their own textual semantic label information. |
| Outcome: | The proposed method achieves strong performance in data-sparse scenarios compared to existing methods across SST-5, Emotion detection, and AG News data even with just two examples per class. |
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (2023.emnlp-main)
Copied to clipboard
Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Sultan, Christopher Potts
| Challenge: | Existing methods for information retrieval tasks require large labeled datasets for fine-tuning, but they can experience significant drops in accuracy due to distribution shifts from the training to the target domain. |
| Approach: | They propose a method for using large language models to generate large numbers of synthetic queries cheaply using an expensive LLM. |
| Outcome: | The proposed method boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods. |
TATA: Stance Detection via Topic-Agnostic and Topic-Aware Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on topic-specific stance classifiers that fail to generalize to unseen topics. |
| Approach: | They propose to use contrastive learning and an unlabeled dataset to train topic-agnostic/TAG and topic-aligned/TAW embeddings for use in downstream stance detection. |
| Outcome: | The proposed model achieves state-of-the-art performance across several public stance detection datasets (0.771 F1-score on the Zero-shot VAST dataset). |
Data Similarity is Not Enough to Explain Language Model Performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models perform well on many but not all downstream tasks. |
| Approach: | They compare large language models with downstream benchmarks to determine whether similarity measures correlate with model performance. |
| Outcome: | The results show that similarity measures are not correlated with accuracy or each other in other benchmarks. |
Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot quantization methods are based on overfitting problem in adversarial learning process, leading to sub-optimal performance. |
| Approach: | They propose a zero-shot sharpness-aware quantization framework for the quantization of various PLMs by optimizing a minimax problem. |
| Outcome: | The proposed framework can achieve significant performance gains on discriminative and generative PLMs. |
Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)
Copied to clipboard
Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi
| Challenge: | Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples. |
| Approach: | They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses. |
| Outcome: | The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance. |
An “Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives” (2023.emnlp-main)
Copied to clipboard
| Challenge: | Mental health conversational agents (a.k.a. chatbots) are widely studied for their potential to offer accessible support to those experiencing mental health challenges. |
| Approach: | They review 534 papers on building mental health-related conversational agents . they recommend a few recommendations to bridge the disciplinary divide . |
| Outcome: | The systematic review reveals 136 key papers on building mental health-related conversational agents with diverse characteristics of modeling and experimental design techniques. |
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks of social language are lacking for large language models. |
| Approach: | They propose a new benchmark that measures how well large language models understand social language by grouping 58 tasks into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
| Outcome: | The proposed model performs well at 58 tasks that are divided into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
Interventional Rationalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for rationalization use spurious correlations in data to compose rationales and make predictions. |
| Approach: | They propose a method to discover the causal rationales by using a structural causal model. |
| Outcome: | The proposed method is based on the causal theory and validates on three real-world datasets. |
Don’t Take This Out of Context!: On the Need for Contextual Models and Evaluations for Stylistic Rewriting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing stylistic text rewriting methods ignore the context of the text, causing generic, incoherent, and generic outputs. |
| Approach: | They propose a contextual evaluation metric that integrates preceding context into stylistic text rewriting. |
| Outcome: | The proposed metric integrates the preceding textual context into rewriting and evaluation stages . human preferences are better reflected by the proposed criterio and other metrics . |
Axiomatic Preference Modeling for Longform Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have helped bridge the "alignment gap" between the responses of raw pretrained language models and responses that resonate more closely with human preferences. |
| Approach: | They propose to use a axiomatic framework to generate a rich variety of preference signals to uphold these signals. |
| Outcome: | The proposed model outperforms GPT-4 and ChatGPT in preference scoring. |
Countering Misinformation via Emotional Response Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms (SMPs) are one of the most effective ways to spread misinformation by engaging in constructive dialogue with users who spread – often in good faith – misleading messages. |
| Approach: | They propose to use social correction to engage in constructive dialogue with users who spread misleading messages. |
| Outcome: | The proposed dataset shows that it improves on previous studies on claim-response pairs and the author-reviewer pipeline. |
Seq2seq is All You Need for Coreference Resolution (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on coreference resolution suggests task-specific models are necessary . a recent line of work that take an alternative approach leveraging advances in seq2seq-based models is needed . |
| Approach: | They propose a pretrained seq2seq transformer to map an input document to a tagged sequence encoding the coreference annotation. |
| Outcome: | The proposed model outperforms or matches the best coreference systems on an array of datasets. |
Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to control speaker-related gender inflections in ST involve dedicated model retraining on gender-labeled data. |
| Approach: | They propose to use a gender-based inference-time solution to control speaker-related gender inflections in ST by replacing the implicitly learned internal language model with gender-specific external LMs. |
| Outcome: | The proposed approach outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. |
StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding (2023.emnlp-main)
Copied to clipboard
Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, Zheng Zhang
| Challenge: | Analogy-making between narratives is crucial for human reasoning . despite its importance, there has been limited research on story analogies . |
| Approach: | They construct a large-scale story-level analogy corpus with 24K story pairs . they find that the tasks are incredibly difficult for large language models such as ChatGPT . |
| Outcome: | The proposed corpus contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. |
Beyond Detection: A Defend-and-Summarize Strategy for Robust and Interpretable Rumor Analysis on Social Media (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing detection models for rumors detection are poor interpretability and lack the textual content to detect rumors. |
| Approach: | They propose a framework that analyzes the textual content and propagation paths of rumors on social media and provides multi-perspective prediction explanations. |
| Outcome: | The proposed framework defends against malicious attacks and provides prediction explanations on three public datasets. |
Crystal: Introspective Reasoners Reinforced with Self-Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-augmented reasoning methods fail to capture the *introspective* nature of knowledge required in commonsense reasoning. |
| Approach: | They propose a method to develop an introspective commonsense reasoner that introspects for knowledge statements related to the given question and makes an informed prediction. |
| Outcome: | The proposed method outperforms standard supervised finetuning and chain-of-thought distilled methods and enhances the transparency of the commonsense reasoning process. |
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for speech generation are not efficient due to low information density of speech data. |
| Approach: | They propose a method to integrate discrete diffusion models into speech generation tasks . they propose to apply diffusion forward process while employing diffusion backward process . |
| Outcome: | The proposed model achieves comparable results to the auto-regressive baselines with significantly fewer decoding steps (50 steps). |
BioFEG: Generate Latent Features for Biomedical Entity Linking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to biomedical entity linking suffer from multiple types of errors due to the rarity of many biomedically relevant entities in real-world scenarios. |
| Approach: | They propose a latent feature generation framework to generate latent semantic features for unseen entities to capture fine-grained coherence information of unseened entities. |
| Outcome: | The proposed framework is superior to existing models on two benchmark datasets. |
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models (2023.emnlp-main)
Copied to clipboard
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang, Chuanyang Zheng, Xiaodan Liang, Ming Zhang, Qun Liu
| Challenge: | Automated theorem proving (ATP) benchmarks focus on symbolic inference but rarely involve understanding complex number combination reasoning. |
| Approach: | They propose a benchmark that requires a model to reduce a trigonometric expression with step-by-step proof and evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
| Outcome: | The proposed benchmark evaluates a generative LM’s reasoning ability on formulas and ability to manipulate, group, and factor number terms. |
Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors (2023.emnlp-main)
Copied to clipboard
| Challenge: | a major challenge in the practical use of Machine Translation (MT) is that users lack information on translation quality to make informed decisions about how to rely on outputs. |
| Approach: | They evaluate quality estimation feedback in vivo with a human study in a medical setting. |
| Outcome: | The proposed method improves appropriate reliance on MT, but backtranslation helps detect harmful errors. |
Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive (2023.emnlp-main)
Copied to clipboard
Tharindu Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri, Christopher Homan, Ashiqur KhudaBukhsh
| Challenge: | a paper examines how machine and human moderators disagree on offensive speech . offensive speech detection is a key component of content moderation . |
| Approach: | They propose a large-scale noise audit and a vicarious offense dataset to investigate disagreement on social web political discourse. |
| Outcome: | The proposed dataset reveals that moderation outcomes vary wildly across different machine moderators. |
Generating Summaries with Controllable Readability Levels (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current text generation approaches focus on a specific readability level, resulting in texts that are not customized to readers’ proficiency levels. |
| Approach: | They propose to generate summaries with fine-grained control over their readability by using instruction-based readability control, reinforcement learning and lookahead to estimate readability of upcoming decoding steps. |
| Outcome: | The generated summaries with different readability levels were compared with previous methods that focus on a specific readability level (e.g., lay summarization) and a lookahead approach significantly improved readability control on news summarizing. |
mAggretriever: A Simple yet Effective Approach to Zero-Shot Multilingual Dense Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | MLIR requires human annotations in multiple languages, making training labor-intensive. |
| Approach: | They propose a multilingual information retrieval model that leverages pre-trained multilingual transformers for dense retrieval. |
| Outcome: | Empirical results show that mAggretriever outperforms state-of-the-art models fine-tuned on English training data. |
CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text. |
| Approach: | They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language. |
| Outcome: | The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models. |
CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs (2023.emnlp-main)
Copied to clipboard
Taha Aksu, Devamanyu Hazarika, Shikib Mehri, Seokhwan Kim, Dilek Hakkani-Tur, Yang Liu, Mahdi Namazifar
| Challenge: | Instruction-based multitasking has played a critical role in the success of large language models (LLMs) when exposed to complex instructions with multiple constraints, they lag against state-of-the-art models like ChatGPT. |
| Approach: | They propose a framework that unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without manual effort. |
| Outcome: | The proposed framework unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without manual effort. |
VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on the concept of vulnerability at the European Court of Human Rights (ECtHR) has focused on classification and analysis of textual data. |
| Approach: | They propose to use an expert-annotated multi-label dataset to assess vulnerability in court cases. |
| Outcome: | The proposed model performs poorly on out-of-domain data and shows that it is robust. |
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023.emnlp-main)
Copied to clipboard
Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, Nanyun Peng
| Challenge: | despite its importance, there are few datasets that cover multimodal counterfactual reasoning . a dataset focusing on this area is limited because of its limited coverage over synthetic environments . |
| Approach: | They develop a video question answering dataset that provides questions on multimodal reasoning . they ask questions about counterfactual hypotheses over visual events . |
| Outcome: | The proposed dataset shows a significant performance gap between models and humans . it provides questions that span physical, social, and temporal dimensions . |
From Parse-Execute to Parse-Execute-Refine: Improving Semantic Parser for Complex Question Answering over Knowledge Base (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for parsing knowledge-base questions into executable logical forms have not been successful on complex KBQA. |
| Approach: | They propose a new semantic parser called KoPL to model the reasoning processes . they propose 'parse-execute-refine' paradigm to unlock reasoning ability . |
| Outcome: | The proposed parser performs better than the state-of-the-art on complex KBQA . the proposed parsed-execute-refine paradigm can model complex reasoning steps . |
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models generate text that is problematic or lacks a desired attribute. |
| Approach: | They propose a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text with certain properties. |
| Outcome: | The proposed procedure outperforms prior weighted decoding methods and matches state-of-the-art techniques that require additional training. |
CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation. (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that focus on company relations and business entities are lacking in relation classification. |
| Approach: | They introduce a few-shot relation classification dataset for company relations and business entities . they use a dataset that includes 4,708 instances of 12 relation types . |
| Outcome: | The proposed dataset includes 4,708 instances of 12 relation types with corresponding textual evidence extracted from company Wikipedia pages. |
Models See Hallucinations: Evaluating the Factuality in Video Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics show little correlation with human factuality annotation. |
| Approach: | They propose a weakly-supervised, model-based factuality metric FactVC which outperforms previous metrics on factual evaluation of video captioning. |
| Outcome: | The proposed model outperforms previous metrics on factuality evaluation of video captioning. |
Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors (2023.emnlp-main)
Copied to clipboard
| Challenge: | proposed method combines back transcription with fine-grained technique for categorizing speech recognition errors . proposed method relies on the use of synthesized speech in place of audio recording . |
| Approach: | They propose a method for investigating the impact of speech recognition errors on NLU models . they use a back transcription procedure and a fine-grained technique for categorizing errors . |
| Outcome: | The proposed method relies on synthesized speech in place of audio recording to evaluate the model. |
Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conceptual spaces are constructed from a set of quality dimensions, which are usually learned from human judgements, which means that applications of conceptual spaces are limited to narrow domains. |
| Approach: | They propose to use Large Language Models to learn perceptually grounded representations by comparing them to larger models of the BERT family. |
| Outcome: | The proposed models outperform the largest model, despite being 2 to 3 orders of magnitude smaller. |
Can Language Models Understand Physical Concepts? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models do not understand basic physical concepts in the human world. |
| Approach: | They propose a method to transfer embodied knowledge from visual models to LMs . they use visual concepts and embodies concepts learned from interaction with the world . |
| Outcome: | The proposed method achieves comparable performance with scaling up parameters of LMs 134. |
SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning is a method of pre-trained models that optimizes the prompt to adapt to downstream tasks. |
| Approach: | They propose a framework that learns to select the proper prompt layers by inserting a probabilistic gate at each intermediate layer. |
| Outcome: | The proposed framework can perform better than the state-of-the-art prompt tuning frameworks on ten benchmark datasets. |
Once Upon a Time in Graph: Relative-Time Pretraining for Complex Temporal Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on strengthening the knowledge-time association between text and time-stamps, but this is insufficient for downstream tasks. |
| Approach: | They propose a model that explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. |
| Outcome: | The proposed model outperforms baseline T5 on multiple temporal question answering datasets . it is especially good at modeling long-range complex temporal dependencies, the authors say . |
Expository Text Generation: Imitate, Retrieve, Paraphrase (2023.emnlp-main)
Copied to clipboard
| Challenge: | Expository documents are vital resources for conveying complex information to readers. |
| Approach: | They propose a task to generate an accurate and stylistically consistent expository text by intelligently searching a knowledge source. |
| Outcome: | The proposed framework overcomes the limitations of retrieval-augmented models and produces factual and organized expository texts that accurately inform readers. |
Large-scale similarity search with Optimal Transport (2023.emnlp-main)
Copied to clipboard
| Challenge: | Word mover's distance (WMD) is a powerful tool for comparing probability distributions in NLP. |
| Approach: | They propose a waterstein distance approximation that uses the L1 embedding method to find the k-nearest neighbors. |
| Outcome: | The proposed approximation performs comparable to the vanilla Wasserstein distance and can be computed three orders of magnitude faster than the vanilla waterstein distance. |
Enhancing Textbooks with Visuals from the Web for Improved Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Textbooks lack visuals that support student learning, but many lack them . e-textbooks lack such visuals, and many lack these visuals . |
| Approach: | They propose to use vision-language models to automatically enhance textbooks with images from the web. |
| Outcome: | The proposed model improves textbooks with images from the web while allowing for better pedagogical value. |
Continual Event Extraction with Semantic Confusion Rectification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on continual event extraction to extract incessantly emerging information . the semantic confusion on event types stems from the annotations of the same text being updated over time . |
| Approach: | They propose a continual event extraction model with semantic confusion rectification to reduce semantic confusion. |
| Outcome: | The proposed model outperforms state-of-the-art models and is proficient in imbalanced datasets. |
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. |
| Approach: | They propose to use several LLMs to ensemble translation hypotheses . they use instruction tuning, quality-based reranking, and minimum Bayes risk (MBR) decoding to improve translation quality. |
| Outcome: | The proposed method improves translation quality and instruction tuning improves the quality of the output. |
FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing passive mental health monitoring systems use alternative features such as activity, app usage, and location via smartphones due to data privacy concerns. |
| Approach: | They propose a mobile mental health monitoring system that utilizes continuous speech and keyboard input in a privacy-preserving way via federated learning. |
| Outcome: | The proposed system achieves 0.15 AUROC improvement and 8.21% MAE reduction in self-reported depression, stress, anxiety, and mood from 46 participants. |
Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models (2023.emnlp-main)
Copied to clipboard
Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, Yoonsik Kim, Sangdoo Yun, Taeho Kil, Bado Lee, Seunghyun Park
| Challenge: | Recent advances in Large Language Models (LLMs) have stimulated a surge of research aimed at extending their applications to the visual domain. |
| Approach: | They propose a novel neural architecture to enhance language-image understanding capability of LLMs by capturing intricate details that are often overlooked in existing methods. |
| Outcome: | The proposed model can achieve better comprehension of language information in visual contexts within images. |
Continual Learning for Multilingual Neural Machine Translation via Dual Importance-based Model Division (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on preventing catastrophic forgetting by making compromises between the original and new language pairs, leading to sub-optimal performance on both translation tasks. |
| Approach: | They propose a dual importance-based model division method to divide the model parameters into two parts and separate the translation of the original and new tasks. |
| Outcome: | The proposed method outperforms strong baselines under different incremental translation scenarios. |
SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. |
| Approach: | They propose a method to deal with dropout noise and a dimension-wise contrastive learning objective to address feature corruption. |
| Outcome: | The proposed method achieves 1.8 points compared to the strong baseline SimCSE and 1.4 points for DiffCSE. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics conflate simplicity with correlated attributes such as fluency or meaning preservation. |
| Approach: | They propose a new learning evaluation metric that focuses on simplicity outperforming most existing metrics in terms of correlation with human judgements. |
| Outcome: | The proposed metric outperforms most existing metrics in terms of correlation with human judgements. |
Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration (2023.emnlp-main)
Copied to clipboard
Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, Kun Kuang
| Challenge: | Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task. |
| Approach: | They propose a framework that leverages the strength of both LLMs and domain-specific models in the context of precedents. |
| Outcome: | The proposed framework leverages the strength of both LLM and domain models in the context of precedents. |
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)
Copied to clipboard
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
| Challenge: | Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly. |
| Approach: | They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source. |
| Outcome: | The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source. |
Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. |
| Approach: | They propose to combine existing chain-of-thought datasets into a unified format that can be used to train and evaluate open-source calculator-using models. |
| Outcome: | The proposed model doubles the accuracy of generating correct results compared to baseline models. |
CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting is popular in reasoning tasks, but its application to Large Language Models (LLMs) in Natural Language Understanding (NLU) is under-explored. |
| Approach: | They propose a Coarse-to-Fine Chain-of-Thought approach that breaks down NLU tasks into multiple reasoning steps where LLMs can learn to acquire essential concepts. |
| Outcome: | The proposed approach is effective in assisting the LLMs adapt to multi-grained NLU tasks under zero-shot and few-shot multi-domain settings. |
When Language Models Fall in Love: Animacy Processing in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Animacy is not always expressed directly in language, but it manifests indirectly in English . atypically animate entities are easier to remember and prioritized in visual processing . |
| Approach: | They find that LMs behave much like humans when presented with entities whose animacy is typical. |
| Outcome: | The proposed model can learn about animacy even when presented with atypically animate entities. |
Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on relation representation learning focus on contrastive learning strategies, but these studies overlook important aspects. |
| Approach: | They propose to use within-sentence pairs augmentation and cross-sentent pairs extraction to increase diversity of positive pairs and strengthen the discriminative power of contrastive learning. |
| Outcome: | The proposed task increases diversity of positive pairs and strengthens discriminative power . it overcomes limitations of traditional Relation Extraction tasks, which require manual annotations . |
Paraphrase Types for Generation and Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to paraphrase generation and detection ignore the intricate linguistic properties of language. |
| Approach: | They propose two tasks to consider specific linguistic perturbations at particular text positions. |
| Outcome: | The proposed tasks address the shortcoming of ignoring the linguistic properties of language. |
Target-to-Source Augmentation for Aspect Sentiment Triplet Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Aspect Sentiment Triplet Extraction (ASTE) is an important task in sentiment analysis, but data scarcity limits performance of existing methods. |
| Approach: | They propose a target-to-source augmentation approach to alleviate the issue of data scarcity in Aspect Sentiment Triplet Extraction (ASTE) they use fluency and alignment discriminators to provide feedback and use this feedback to optimize the generator. |
| Outcome: | The proposed approach significantly improves the performance of existing methods. |
PAC-tuning: Fine-tuning Pre-trained Language Models with PAC-driven Perturbed Gradient Descent (2023.emnlp-main)
Copied to clipboard
| Challenge: | PAC-tuning is a two-stage fine-tune method for pretrained language models . PAC training minimizes the PACBayes generalization bound to learn proper parameter distribution . |
| Approach: | They propose a two-stage fine-tuning method to minimize the PAC-Bayes generalization bound . they use PAC to inject noise with variance learned in the first stage into the model parameters . |
| Outcome: | The proposed method outperforms baseline methods on 5 GLUE benchmark tasks. |
Emergence of Abstract State Representations in Embodied Sequence Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sequence modeling is a simple yet versatile task that can be applied to more complex decision-making domains. |
| Approach: | They build a sequence modeling Transformer which takes a language instruction, actions, and environmental observations as inputs and then trains a model to reconstruct environmental layouts. |
| Outcome: | The proposed model can reconstruct environmental layouts from the inputs of the model and language instructions play a role in the reconstruction accuracy. |
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Toeplitz Neural Networks outperform commonly used Transformer-based models while benefiting from log-linear space-time complexities. |
| Approach: | They propose to convert TNNs to SSMs during inference to combine strengths of TNN and SSM approaches. |
| Outcome: | The proposed method outperforms most Transformer-based models while retaining the advantage of constant inference complexity. |
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on identifying where factual knowledge is encoded in the network, but little is known about how it is extracted from the model parameters during inference. |
| Approach: | They examine how factual associations are stored and retrieved internally in LMs . they use attention edges to identify critical points where information propagates to the prediction . |
| Outcome: | The proposed model aggregates information about subject and relation to predict the correct attribute . the model “queries” the enriched subject to extract the attribute based on the proposed model . |
StereoMap: Quantifying the Awareness of Human-like Stereotypes in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been observed to encode harmful associations present in the training data. |
| Approach: | They propose a framework to map LLMs' perceptions of how demographic groups have been viewed by society using the dimensions of Warmth and Competence. |
| Outcome: | The proposed framework maps LLMs’ perceptions of social groups using the dimensions of Warmth and Competence. |
Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be expensive to train, deploy, and use for specific natural language generation tasks. |
| Approach: | They propose a method to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations using a semantic similarity metric. |
| Outcome: | The proposed method leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labeled data over standard KD approach given the same size of training data. |
Human Raters Cannot Distinguish English Translations from Original English Texts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on translationese has identified common hallmarks of translationeses, but human accuracy of identifying translated text is understudied. |
| Approach: | They perform an evaluation of English original/translated texts to examine whether raters can classify texts as being original or translated English and the features that lead rater to judge text as being translated. |
| Outcome: | The results provide critical insight into work in translation studies and context for assessments of translationese classifiers. |
Impressions: Visual Semiotics and Aesthetic Impact Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning and conditional generation models struggle to simulate plausible human responses to images. |
| Approach: | They propose a dataset to investigate the semiotics of images and how visual features and design choices can elicit specific emotions, thoughts and beliefs. |
| Outcome: | The proposed dataset improves existing models for image captioning and conditional generation. |
DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn compact cluster representations from coarsely labeled data are noisy and degrade the quality of learning. |
| Approach: | They propose a framework that encodes semantic structures of data into the embedding space . they retrieve k-nearest neighbors of a query as positive keys to capture similarities . |
| Outcome: | The proposed framework can retrieve more accurate neighbors and outperform state-of-the-art models by a large margin. |
Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | ProAttack is a novel and efficient method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Approach: | They propose a method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Outcome: | The proposed method achieves state-of-the-art performance on several NLP tasks, particularly in few-shot settings. |
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation (2023.emnlp-main)
Copied to clipboard
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, Qi Zhang
| Challenge: | Large Language Models (LLMs) have impressive capabilities but need for task-specific prompt engineering can hinder their generalization. |
| Approach: | They propose a lightweight and versatile retriever that automatically retrieves prompts for a given zero-shot task input. |
| Outcome: | The proposed model is universally applicable across tasks and models . it mitigates hallucination problem in chatGPT, and it improves even the strongest LLMs. |
KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Task-oriented dialogs (TOD) require a model to generate a response that optimizes for task-related metrics. |
| Approach: | They propose a faster generation procedure that samples from independent next-word distributions and introduce a fine-grained reward function to help the model focus on learning key information in a dialog. |
| Outcome: | The proposed algorithm achieves state-of-the-art performance on an offline task with 15% training time reduction compared to a standard RL algorithm using auto-regressive generation. |
Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models based on English datasets do not provide adequate data for evaluating their capabilities beyond English. |
| Approach: | They propose a multi-task language understanding benchmark for Indonesian culture and languages . it measures language proficiency, reasoning abilities and real-world knowledge . |
| Outcome: | The proposed model passes the primary school level in Indonesia, while other models perform at lower levels. |
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving the correctness of output from large language models generate a constant number of samples per question, but Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%. |
| Approach: | They propose a model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. |
| Outcome: | The proposed technique reduces sample budget by 7.9 times with an average accuracy drop of less than 0.1%. |
Bridging Information-Theoretic and Geometric Compression in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current language models (LMs) encode training data into finitely many variables that allow generalization to infinitely many grammatical utterances. |
| Approach: | They propose to analyze compression in language models from geometric and information-theoretic perspectives. |
| Outcome: | The proposed model can model human language in a relatively small dimension. |
Pre-training Language Models for Comparative Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions. |
| Approach: | They propose a framework to pre-train language models for enhancing comparative reasoning abilities . they collect scalable data for text-based entity comparison . |
| Outcome: | The proposed framework significantly improves comparative reasoning abilities under low-resource conditions on downstream tasks. |
Improved Pseudo Data for Machine Translation Quality Estimation with Constrained Beam Search (2023.emnlp-main)
Copied to clipboard
Xiang Geng, Yu Zhang, Zhejian Lai, Shuaijie She, Wei Zou, Shimin Tao, Hao Yang, Jiajun Chen, Shujian Huang
| Challenge: | evaluating the quality of machine translation outputs becomes increasingly essential with the rapid development of machine language (MT). |
| Approach: | They propose to generate pseudo data using the MT model with constrained beam search (CBSQE) they propose to preserve the reference parts with high MT probabilities as correct translations . |
| Outcome: | The proposed model outperforms strong baselines in both supervised and unsupervised settings. |
Text Embeddings Reveal (Almost) As Much As Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a vector database of dense text embeddings stores only the text data, not the original text . a multi-step method that iteratively corrects and re-embeds text can recover 92% of 32-token text inputs exactly. |
| Approach: | They propose a method that iteratively corrects and re-embeds text to recover 92% of 32-token text inputs exactly. |
| Outcome: | The proposed method recovers 92% of 32-token text inputs exactly. |
AutoTrial: Prompting Language Models for Clinical Trial Design (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) are a popular tool for creating coherent and human-like documents for clinical trials. |
| Approach: | They propose to generate clinical eligibility criteria using language models by a hybrid of discrete and neural prompting and scalable knowledge incorporation via in-context learning. |
| Outcome: | The proposed method generates high-quality criteria texts fluent and coherent with high accuracy against the GPT-3.5 baselines. |
Faster Minimum Bayes Risk Decoding with Confidence-based Pruning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding is a decision rule for conditional sequence generation tasks. |
| Approach: | They propose an algorithm which grows the number of samples used to estimate utility . it prunes hypotheses that are unlikely to have the highest utility based on bootstrap sampling . |
| Outcome: | The proposed method outperforms beam search in conditional language generation and neural machine translation tasks while being statistically indistinguishable from other proposed methods. |
Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end generative retrieval models produce document identifiers in response to a query . however, this approach has two challenges: an overemphasis on top-1 results at the expense of overall ranking quality. |
| Approach: | They propose a generative retrieval model with reinforcement learning from relevance feedback to align token-level docid generation with document-level relevance estimation. |
| Outcome: | The proposed model aligns token-level docid generation with document-level relevance estimation. |
Multi-Source Probing for Open-Domain Conversational Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of open-domain dialogue comprehension have limited conversational understanding and response generation. |
| Approach: | They propose a multi-source probing method to probe dialogue comprehension abilities of open-domain dialogue models. |
| Outcome: | The proposed method aggregates features from multiple sources to accomplish diverse task goals and conducts downstream tasks in a generative manner consistent with dialogue model pre-training to leverage model capabilities. |
Hallucination Mitigation in Natural Language Generation from Large-Scale Open-Domain Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Graph-to-text models trained on small-scale datasets or datasets with limited variety of graph shapes are not adequate for more realistic large-scale, open-domain settings. |
| Approach: | They propose a novel approach that, given a graph-sentence pair in GraphNarrative, trims the sentence to eliminate portions that are not present in the corresponding graph. |
| Outcome: | The proposed model can be trained on existing datasets and is available on github. |
Multi-Source Multi-Type Knowledge Exploration and Exploitation for Dialogue Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on identifying specific types of dialogue knowledge and utilizing corresponding datasets for training, but lack generalization capabilities and computational resources. |
| Approach: | They propose a framework that explores multi-source multi-type knowledge from LLMs by leveraging diverse datasets and exploits it for response generation. |
| Outcome: | The proposed framework exploits multi-source multi-type knowledge from LLMs to generate coherent, informative, and fluent responses. |
Focus Your Attention (with Adaptive IIR Filters) (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to deal with long-range data processing are implicit convolutions and regularized parameterization. |
| Approach: | They propose a new layer where dynamic (i.e., input-dependent) IIR filters are used to process the input sequence prior to applying conventional attention. |
| Outcome: | The proposed layer performs on-par with state-of-the-art networks with a fraction of their parameters and time complexity that is sub-quadratic with input size. |
Identifying Statements Crucial for Awareness of Interpretive Nonsense to Prevent Communication Breakdowns (2023.emnlp-main)
Copied to clipboard
| Challenge: | During remote conversations, communication breakdowns occur when a listener misses certain statements. |
| Approach: | They propose to create a dialogue by omitting two consecutive statements from the original dialogue and then generate text to make the following statement more specific. |
| Outcome: | The proposed method shows that it can be used to identify SCAINs in a dialogue dataset. |
Multilingual Large Language Models Are Not (Yet) Code-Switchers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual Large Language Models are not specifically trained with objectives for managing code-switching scenarios. |
| Approach: | They propose to use multilingual Large Language Models to perform sentiment analysis, machine translation, summarization and word-level language identification to compare their performance to fine-tuned models of much smaller scales. |
| Outcome: | The proposed models show that they underperform in comparison to fine-tuned models of much smaller scales. |
Reinforced Target-driven Conversational Promotion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing conversational recommendation methods focus on acquiring user preferences while ignoring strategic planning for nudging users towards accepting a designated item. |
| Approach: | They propose a Reinforced Target-driven Conversational Promotion framework that integrates short-term and long-term planning via a balanced gating mechanism. |
| Outcome: | The proposed model outperforms state-of-the-art models on automatic metrics and human evaluation. |
Identification of Multimodal Stance Towards Frames of Communication (2023.emnlp-main)
Copied to clipboard
| Challenge: | Until now, determining whether an author is in favor of, against or has no stance towards a frame was performed only when processing texts. |
| Approach: | They propose to use a dataset to infer stance towards 113 different frames of communication in multimodal documents. |
| Outcome: | The proposed model improved the quality of identifying multimedia stance by 20% compared to previous methods, which only performed when processing texts. |
Unsupervised Sounding Pixel Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sounding source localization is a challenging task due to the difficulty of cross-modal alignment. |
| Approach: | They propose an unsupervised method which enables pixel-level sounding source localization in unsupervised paradigm. |
| Outcome: | The proposed method achieves pixel-level sounding source localization without annotations. |
LM vs LM: Detecting Factual Errors via Cross Examination (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) generate inconsistent, non-attributable or factually incorrect text, which hinders their usability. |
| Approach: | They propose a factuality evaluation framework for LMs that is based on cross-examination to detect inconsistencies between LM and examiner. |
| Outcome: | The proposed framework outperforms existing methods and baselines on factual claims on four benchmarks. |
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. |
| Approach: | They argue that LLMs only parrot statistical patterns in training data and that language learning in LLM cannot inform human language learning. |
| Outcome: | The proposed model can generate grammatically correct, fluent text without requiring human intervention. |
PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text classification use label names of target classes as the only supervision. |
| Approach: | They propose a method that uses keyword-based keyword matching to generate pseudo labels . they propose 'pieclass' module that iteratively trains classifiers and updates pseudo labels. |
| Outcome: | The proposed method achieves better performance than existing strong baselines on seven benchmark datasets and similar performance to fully-supervised classifiers on sentiment classification tasks. |
MeaeQ: Mount Model Extraction Attacks with Efficient Queries (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on limited-query budget settings and adopt random sampling or active learning-based sampling strategies on publicly available, unannotated data sources. |
| Approach: | They propose a model extraction attack with efficient Queries that uses a zero-shot sequence inference classifier to filter task-relevant data from a public text corpus instead of a problem domain-specific dataset. |
| Outcome: | The proposed method achieves higher similarity to the victim model than baselines while requiring fewer queries. |
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models with less than 100B parameters perform poorly on chain-of-thought reasoning . we aim to equip smaller LMs with the step-by-step reasoning capability . |
| Approach: | They propose to equip smaller LMs with the step-by-step reasoning capability by tuning with CoT rationales. |
| Outcome: | The proposed dataset outperforms large LMs on 4 domain-specific tasks even with demonstrations . |
Explaining Interactions Between Text Spans (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing highlight-based explanations focus on identifying individual important features or interactions only between adjacent tokens or tuples of tokens. |
| Approach: | They propose a multi-annotator dataset of human span interaction explanations for NLU and FC. |
| Outcome: | The proposed method compares human reasoning processes to those of a fine-tuned large language model. |
Predictive Chemistry Augmented with Text Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | TextReact is a new method to augment predictive chemistry with text descriptions retrieved from the literature. |
| Approach: | They propose a method that directly augments predictive chemistry with texts retrieved from the literature. |
| Outcome: | The proposed method outperforms existing models trained on molecular data. |
System Combination via Quality Estimation for Grammatical Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quality estimation models are not good enough to distinguish good corrections from bad ones, resulting in low F0.5 scores when used for system combination. |
| Approach: | They propose a new quality estimation model that gives a better estimate of the quality of a corrected sentence. |
| Outcome: | The proposed model outperforms the state-of-the-art on the CoNLL-2014 and BEA-2019 test sets, and achieves the highest F0.5 scores published to date. |
Rethinking Negative Pairs in Code Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative learning is a key component in fine-tuning code search models . however, negative samples of InfoNCE may deteriorate its representation learning . |
| Approach: | They propose a loss function that inserts weight terms into InfoNCE to improve contrastive learning. |
| Outcome: | The proposed loss function is a special case of Soft-InfoNCE, the authors show . it is more accurate than other loss functions, and it is faster than other models. |
Question Answering as Programming for Solving Time-Sensitive Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models (LLMs) have shown remarkable intelligence in question answering. |
| Approach: | They propose to reframe the Question Answering task as Programming to overcome this limitation by leveraging LLMs' superior ability in understanding both natural language and programming language. |
| Outcome: | The proposed approach improves on time-sensitive question answering datasets by 14.5% over baselines. |
Joint Geometrical and Statistical Domain Adaptation for Cross-domain Code Vulnerability Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to detect code vulnerability are limited by labeled training data on target domains. |
| Approach: | They propose a cross-domain code vulnerability detection framework called MNCRI . they propose mutual nearest neighbor contrastive learning to align the source and target domains . |
| Outcome: | The proposed framework outperforms state-of-the-art methods in cross-domain code vulnerability detection tasks. |
Revisiting Sparse Retrieval for Few-shot Entity Linking (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking (EL) aims to link ambiguous mentions to their corresponding entities in a knowledge base. |
| Approach: | They propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. |
| Outcome: | The proposed method outperforms state-of-the-art models on the ZESHEL dataset by a significant margin. |
Controlling Pre-trained Language Models for Grade-Specific Text Simplification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to text simplification control output complexity at corpus level disregarding complexity of individual inputs and considering only one level of output complexity. |
| Approach: | They propose a method that predicts edit operations required for a specific grade level . they say this approach improves the quality of the simplified outputs over corpus-level heuristics . |
| Outcome: | The proposed method improves the readability of simplified outputs over corpus-level search-based heuristics. |
CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension. |
| Approach: | They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images . |
| Outcome: | The proposed method improves performance 37.94% on referring expression comprehension task. |
“Are Your Explanations Reliable?” Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attack (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on text datasets demonstrate that XAIFooler significantly outperforms all baselines by large margins in its ability to manipulate LIME’s explanations with high semantic preservability. |
| Approach: | They propose to use LIME to establish a baseline and then propose an algorithm to perturb text inputs and manipulate explanations. |
| Outcome: | The proposed algorithm outperforms baselines on text datasets and achieves high semantic preservability. |
CQE: A Comprehensive Quantity Extractor (2023.emnlp-main)
Copied to clipboard
| Challenge: | Quantities are essential in documents to describe factual information. |
| Approach: | They propose a comprehensive quantity extraction framework that detects combinations of values and units, the behavior of a quantity and the concept a quantity is associated with. |
| Outcome: | The proposed framework outperforms existing methods and is the first to detect concepts associated with identified quantities. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |
A Unified View of Evaluation Metrics for Structured Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | a framework that unifies evaluation metrics for structured prediction tasks is presented . metric design decisions are motivated by specific characteristics of tasks, and we suggest modifications to existing metrics to meet those motivations. |
| Approach: | They propose a framework that unifies a variety of evaluation metrics for different structured prediction tasks. |
| Outcome: | The proposed framework can be used to create new metrics based on the output structure of a number of tasks. |
A Deeper (Autoregressive) Approach to Non-Convergent Discourse Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks for dialogic discourse parsing are not suitable for contentious discussions . authors propose a model for non-convergent discourse paring that does not require label collocation . |
| Approach: | They propose a multi-label scheme for contentious dialog parsing that uses multiple labels . they propose combining embeddings of the utterance, context and the labels through GRN layers . |
| Outcome: | The proposed model achieves comparable results with SOTA without label collocation and without training a unique architecture/model for each label. |
We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other) |
| Approach: | They quantify the degree of influence between 23 fields of study and NLP on each other . they find that cross-field engagement of NLP has declined from 0.58 in 1980 to 0.31 in 2022 . |
| Outcome: | The proposed Citation Field Diversity Index (CFDI) has declined from 0.58 in 1980 to 0.31 in 2022, the authors show . |
Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing variants of Kendall's tau have weaknesses in handling of ties, and in some situations can even be gamed. |
| Approach: | They propose to use Kendall's tau to meta-evaluate machine translation evaluation metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties and a tie calibration procedure that automatically introduces ties into metric scores. |
| Outcome: | The proposed model gives metrics credit for correctly predicting ties and introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. |
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (2023.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
| Challenge: | a dataset of 1.5 million conversations distilled from everyday spoken situations is limited in scale due to its associated costs. |
| Approach: | They propose to make SODA a publicly available, million-scale high-quality social dialogue dataset . they contextualize social commonsense knowledge from a knowledge graph to distill broad spectrum of social interactions . |
| Outcome: | The proposed dataset is the first publicly available, million-scale high-quality social dialogue dataset. |
Multi-view Contrastive Learning for Entity Typing over Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to knowledge graph entity typing ignore the way types can be clustered together. |
| Approach: | They propose a method that effectively encodes coarse-grained knowledge from clusters into entity and type embeddings. |
| Outcome: | The proposed method encodes coarse-grained knowledge from clusters into entity and type embeddings. |
MailEx: Email Event and Argument Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on email event extraction only covers one specific aspect of email information and cannot connect with other relevant tasks. |
| Approach: | They propose a new taxonomy for performing event extraction from conversational email threads. |
| Outcome: | The proposed taxonomy covers 10 event types and 76 arguments in the email domain. |
Optimized Tokenization for Transcribed Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | transcribed-like data is often used to correct recurring errors, but training with synthetic data is difficult. |
| Approach: | They propose to use synthetic transcribed-like data to train error correction models . they show that synthetic data outperforms the common approach of random perturbations . |
| Outcome: | The proposed method outperforms the common method using random perturbations in transcribed data and language-specific adjustments to the vocabulary of a BPE tokenizer. |
Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved great success in question answering, but their robustness is insufficient to support their practical applications. |
| Approach: | They propose a method which regularizes the model's output and an efficient side block to reduce its inference time. |
| Outcome: | The proposed method achieves comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8 to 4.4 speedup compared to previous methods. |
Generative Adversarial Training with Perturbed Token Detection for Model Robustness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial training methods use discrete tokens to deceive models . current approaches use embeddings, whereas actual text-based training uses discrete text tokens. |
| Approach: | They propose a framework that integrates gradient-based learning, adversarial example generation and perturbed token detection to enhance adversariarial robustness. |
| Outcome: | The proposed framework surpasses the state-of-the-art results of ChatGPT by 10% in average accuracy. |
Multi-Task Knowledge Distillation with Embedding Constraints for Scholarly Keyphrase Boundary Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scholarly keyphrase boundary classification is the task of identifying highly summative phrases from scientific papers and classifying them into a set of predefined classes. |
| Approach: | They propose a constraint which enforces the teachers and student similarity in the embedding space. |
| Outcome: | The proposed constraint outperforms previous studies and strong baselines on three datasets of scientific documents. |
Set Learning for Generative Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to employ sequence-to-sequence models to solve IE tasks have been focused on a single problem: structured objects are an unordered set, resulting in a potential order bias. |
| Approach: | They propose a sequence-to-sequence (Seq2Sequen) model that considers multiple permutations of structured objects to optimize set probability approximately. |
| Outcome: | The proposed model improves existing frameworks on vast tasks and datasets. |
Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual word sense disambiguation (VWSD) is a challenging task involving multiple candidates . context given for an ambiguous word is minimal, most often limited to a single word . |
| Approach: | They propose to use large language models to enhance given phrases and resolve ambiguity related to the target word. |
| Outcome: | The proposed frameworks improve the image representation of ambiguous words among candidates and achieve competitive ranking results. |
Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. |
| Approach: | They propose to use RL to train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. |
| Outcome: | The proposed system fails to learn the value of compromise in a negotiation, which can lead to no agreements, and ultimately hurt the model's overall performance. |
Doolittle: Benchmarks and Corpora for Academic Writing Formalization (2023.emnlp-main)
Copied to clipboard
Shizhe Diao, Yongyu Lei, Liangming Pan, Tianqing Fang, Wangchunshu Zhou, Sedrick Keh, Min-Yen Kan, Tong Zhang
| Challenge: | Existing methods of language refinement focus on narrow, specific linguistic features within isolated sentences, such as grammatical errors and improper word use. |
| Approach: | They propose a task to improve the overall quality of academic writing at paragraph level by integrating automatic feedback into the training process. |
| Outcome: | The proposed task improves the overall quality of formal academic writing at the paragraph level. |
Token Prediction as Implicit Classification to Identify LLM-Generated Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach for identifying large language models (LLMs) involved in text generation is proposed . instead of adding an additional classification layer, we reframe the classification task as a next-token prediction task . |
| Approach: | They propose a novel approach for identifying large language models involved in text generation . instead of adding an additional classification layer, they reframe the task as a next-token prediction task . |
| Outcome: | The proposed method performs exceptionally well in the text classification task . it can distinguish distinctive writing styles among various LLMs even without an explicit classifier. |
On Evaluation of Bangla Word Analogies (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing word embeddings in Bangla struggle to perform well on low-resource data sets. |
| Approach: | They propose to use a benchmark dataset of Bangla word analogies to evaluate the quality of existing Bangla embeddings. |
| Outcome: | The proposed evaluation set includes 16,678 unique word analogies in Bangla and a translated and curated version of the original Mikolov dataset (10,594 samples) . |
Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to meeting summarization are limited due to noise, lengthy transcripts, and scattered salient information. |
| Approach: | They propose a two-step framework for meeting summarization that leverages a self-supervised paradigm to reconstruct transcripts and a relative positional bucketing algorithm to equip models to generate the summary. |
| Outcome: | The proposed method significantly reduces memory consumption and processing time on two meeting summarization datasets. |
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)
Copied to clipboard
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R . |
| Approach: | They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. |
| Outcome: | The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks. |
Character-LLM: A Trainable Agent for Role-Playing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to simulate human behaviors . a recent study suggests that LLMs can be more effective at generating human behavior . |
| Approach: | They propose to use large language models to train agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API. |
| Outcome: | The proposed model trains agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API. |
Natural Language Decompositions of Implicit Content Enable Better Text Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new method for the analysis of text takes implicitly communicated content into account . authors: human interpretation of each individual utterance is intractable . |
| Approach: | They propose a method that takes explicit communication into account when people interpret text . they use a large language model to generate propositions that are inferentially related to the text analyzed . |
| Outcome: | The proposed method proves useful in multiple problems that involve interpretation of utterances . it uncovers high-level narratives in public commentary, which are often not expressed in surface forms . |
A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document classification focus on local layout, sidelining holistic comprehension of content and organisation. |
| Approach: | They propose a framework for Table of Contents extraction that uses hierarchical structure to extract text from ESG annual reports. |
| Outcome: | The proposed framework outperforms the state-of-the-art with a fraction of running time. |
Semantic Space Grounded Weighted Decoding for Multi-Attribute Controllable Dialogue Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Controlling chatbot utterance generation with multiple attributes is a useful but under-studied problem. |
| Approach: | They propose a framework that possesses strong controllability with a weighted decoding paradigm and improves generation quality with an attribute semantics space. |
| Outcome: | The proposed framework achieves high control accuracy with simultaneous control of 3 aspects while producing interesting and sensible responses even in an out-of-distribution robustness test. |
How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual language models can learn generalisations useful for other languages . yet, it remains unclear to what extent and under which conditions these models benefit from multilingual data and cross-lingual sharing. |
| Approach: | They propose a training data attribution method to retrieve training samples from multilingual data that are most influential for test predictions in a given language. |
| Outcome: | The proposed method exploits the ability to learn generalisations useful for other languages on zero-shot cross-lingual transfer for many languages. |
COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation (2023.emnlp-main)
Copied to clipboard
Nan Wang, Qifan Wang, Yi-Chia Wang, Maziar Sanjabi, Jingzhou Liu, Hamed Firooz, Hongning Wang, Shaoliang Nie
| Challenge: | Personalized text generation (PTG) is a key component of our digital lives but can inadvertently associate different levels of linguistic quality with users’ protected attributes. |
| Approach: | They propose a framework to achieve measure-specific counterfactual fairness in explanation generation by focusing on one of the most studied settings: generating natural language explanations for recommendations. |
| Outcome: | The proposed framework achieves measure-specific counterfactual fairness in explanation generation. |
NameGuess: Column Name Expansion for Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Tabular data is used for storing and organizing information in web and enterprise applications. |
| Approach: | They propose a task to expand column names as a natural language generation problem by conditioning on table content and column header names to improve auto-regressive models. |
| Outcome: | The proposed task improves auto-regressive models on table content and column header names to match human performance. |
BLESS: Benchmarking Large Language Models on Sentence Simplification (2023.emnlp-main)
Copied to clipboard
Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, Matthew Shardlow
| Challenge: | BLESS is a performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). |
| Approach: | They present a performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). |
| Outcome: | The proposed benchmarks show that the most recent state-of-the-art LLMs perform better on the task of text simplification (TS). |
To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. |
| Approach: | They conduct interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity to identify cyclical patterns in the field and new shifts without historical parallel . they conclude by discussing shared visions, concerns, and hopes for the future of NLP . |
| Outcome: | The authors identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. |
PALS: Personalized Active Learning for Subjective Tasks in NLP (2023.emnlp-main)
Copied to clipboard
Kamil Kanclerz, Konrad Karanowski, Julita Bielaniewicz, Marcin Gruza, Piotr Miłkowski, Jan Kocon, Przemyslaw Kazienko
| Challenge: | Personalized active learning techniques can be used to learn subjective NLP problems . to acquire training data, texts are often randomly assigned to users for annotation . |
| Approach: | They propose to apply an active learning paradigm to a personalized context to learn preferences . they validated their techniques on a Wiki discussion text labeled with aggression and toxicity . |
| Outcome: | The proposed methods outperform random selection and random selection by 30% on three datasets. |
ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation (2023.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models have limited performance in structural knowledge extraction, such as relations between objects. |
| Approach: | They propose to leverage the inherent structure of programming language to depict visual structural information in a well-organized structured format. |
| Outcome: | The proposed framework improves visual structural knowledge extraction on visual structure prediction tasks. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to augment training data with counterfactuals fail to handle multi-hop fact verification due to their incapability to preserve complex logical relationships. |
| Approach: | They propose to augment training data with counterfactuals that alter causal features of the original data by preserving logical relationships. |
| Outcome: | The proposed method outperforms the baselines and can generate linguistically diverse counterfactuals without disrupting their logical relationships. |
An Exploration of Left-Corner Transformations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how left-corner transformations remove left recursion from context-free grammars . a common use case is to remove left-recursed grammars, which is necessary for top-down parsing . |
| Approach: | They generalize left-corner transformations to support semiring-weighted production rules . they also provide finer-grained control over which left corners may be moved . |
| Outcome: | The proposed transformation removes left recursion from context-free grammars . it also provides finer-grained control over which left corners may be moved . |
Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need (2023.emnlp-main)
Copied to clipboard
| Challenge: | a scientific claim verification requires thorough examination and assessment to ascertain its validity . attention architectures and pre-trained language models fail to establish a comprehensive chain of causal inference . |
| Approach: | They propose a qualitative causal structure-based graph neural network model to facilitate causal reasoning across relevant causally-potent factors. |
| Outcome: | The proposed model outperforms state-of-the-art models by incorporating semantic features . the proposed model is based on a qualitative causal structure . |
FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual models have been released, but many of the world's languages are not covered. |
| Approach: | They propose a method that initializes the embedding matrix for a new tokenizer based on information in the source model's embeddable matrix. |
| Outcome: | The proposed method outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). |
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)
Copied to clipboard
| Challenge: | We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. |
| Approach: | They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task. |
| Outcome: | The proposed games can generate runnable games on unseen topics in 28% of cases. |
Skill-Based Few-Shot Selection for In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods based on pre-trained embeddings can be easily biased by surface features that are not important for the target task. |
| Approach: | They propose a skill-based few-shot selection method for in-context learning . it generates skill-specific descriptions for each test case and candidate example . |
| Outcome: | The proposed method significantly outperforms existing methods in five cross-domain semantic parsing datasets and six backbone models. |
MaNtLE: Model-agnostic Natural Language Explainer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research suggests that practitioners prefer examining language explanations that explain sub-groups of examples. |
| Approach: | They propose a model-agnostic natural language explainer that generates faithful explanations of classifier rationale for structured classification tasks. |
| Outcome: | The proposed model-agnostic natural language explainer generates faithful explanations of classifier rationale for structured classification tasks. |
PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods have training instability issues due to large variance of scores . existing prompt tuning algorithms have training stability issues due a slight change of input data . |
| Approach: | They propose an algorithm that smooths the loss landscape of vanilla prompt tuning by perturbation-based regularizers. |
| Outcome: | The proposed method improves the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks. |
Ling-CL: Understanding NLP Models through Linguistic Curricula (2023.emnlp-main)
Copied to clipboard
| Challenge: | linguistic complexity is a key component of learning in NLP, according to a new study . linguistic complex is based on lexical diversity, word sophistication, and readability . |
| Approach: | They employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula. |
| Outcome: | The proposed approach will inform future research in all NLP areas . it uses linguistic metrics (indices) that inform the challenges and reasoning required to address each task . |
Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on word-level predictions and highlighting semantic differences in natural language documents did not focus on semantic differences as the main target. |
| Approach: | They propose to perform a token-level regression task to highlight semantic differences between two documents . they use word alignment and sentence-level contrastive learning to evaluate the approaches . |
| Outcome: | The proposed approach has a robust correlation to gold labels, but all unsupervised approaches leave a margin of improvement. |
Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has investigated causes of poor zero-shot performance, but new study suggests it does not exhibit poor zero shot capability. |
| Approach: | They propose to investigate the presence of significant variations in zero-shot performance . target-side translation quality is most influential factor, with vocabulary overlap impacting zero- shot capabilities . |
| Outcome: | The results show that the target side translation quality is the most influential factor . linguistic properties, such as language family and writing system, play a role . |
SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-Context Learning with Large Language Models (LLMs) has shown great performance on reasoning tasks. |
| Approach: | They propose a method for selecting a set of exemplars that is representative and diverse. |
| Outcome: | The proposed method outperforms existing methods on FinQA and TAT-QA on hybrid questions. |
Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | open-domain chatbots focus on short single-session dialogue, neglecting the potential need for understanding contextual information in multiple consecutive sessions. |
| Approach: | They propose a 1M multi-session dialogue dataset for integrating time intervals and speaker relationships into a long-term conversation setup. |
| Outcome: | The proposed model can generate coherent responses according to time intervals and speaker relationships with high user engagement without contradiction in a long-term conversation setup. |
DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative learning models for vision and language models are gaining popularity . dueT trains only adapters inserted into pre-trained image and text encoders . |
| Approach: | They propose a transfer learning method for vision and language models built by contrastive learning that trains only adapters inserted into the frozen image and text encoders. |
| Outcome: | The proposed method outperforms fine-tuning, and the LoRA-based adapter method in English and Japanese domains. |
Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use separate recommendation and dialogue modules produce inconsistent results . a multi-task learning model is proposed to bridge the gap between recommendation and generated responses . |
| Approach: | They propose a multi-task learning model that integrates knowledge from two teachers and selectively gates between them via Contextualized Knowledge Distillation. |
| Outcome: | The proposed model significantly improves recommendation performance while enhancing fluency and achieves comparable results in terms of diversity. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
MoPe: Model Perturbation based Privacy Attacks on Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that Large Language Models can unintentionally leak sensitive information . a new method to identify with high confidence if a given text is in training data is proposed . |
| Approach: | They propose a method to detect if a given text is in a pre-trained language model . they show that MoPe is more effective than existing loss-based attacks . |
| Outcome: | The proposed method is more effective than loss-based attacks and perturbation-based methods. |
q2d: Turning Questions into Dialogs to Teach Models How to Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent dialog generation models use external search APIs to generate grounded responses. |
| Approach: | They propose an automatic data generation pipeline that generates dialogs from questions . they use a large language model to create conversational versions of question answering datasets . |
| Outcome: | The proposed method improves query generation models on a QReCC dataset. |
Aligning Large Language Models through Synthetic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, alignment learning requires significant human demonstrations and feedback from proprietary LLMs such as ChatGPT. |
| Approach: | They propose a framework that uses synthetic feedback to align large language models to human values without extensive human annotations and proprietary LLMs. |
| Outcome: | The proposed model outperforms open-source models on human-annotated demonstrations in alignment benchmarks. |
You Told Me That Joke Twice: A Systematic Investigation of Transferability and Robustness of Humor Detection Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that there is little research on how models trained on humor datasets generalize and behave in the wild. |
| Approach: | They analyze existing English humor datasets and train RoBERTa-based and Nave Bayes classifiers on them. |
| Outcome: | The proposed models show that they can generalize and behave on humor datasets, but the transferability of the models is poor. |
Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs). |
| Approach: | They propose a method to predict token sequences within visually-rich documents by a simple prediction head. |
| Outcome: | The proposed method can be used to predict token mentions as token sequences within documents. |
Empower Nested Boolean Logic via Self-Supervised Curriculum Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new self-supervised learning method allows language models to generalize to much harder and longer-hop logic. |
| Approach: | They propose a self-supervised learning method to empower language models with boolean logic . they augment training data with nested booles and program training from simpler to harder ones . |
| Outcome: | The proposed method allows language models to generalize to much harder and longer-hop logic, which can hardly be learned through naive training. |
The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis (2023.emnlp-main)
Copied to clipboard
Pranav Venkit, Mukund Srinath, Sanjana Gautam, Saranya Venkatraman, Vipul Gupta, Rebecca Passonneau, Shomir Wilson
| Challenge: | Existing research reveals a notable absence of interdisciplinary endeavors to comprehend the social dimensions of sentiment analysis, encompassing aspects like emotion and fairness. |
| Approach: | They propose an ethics sheet encompassing critical inquiries to guide practitioners in ensuring equitable utilization of SA. |
| Outcome: | The proposed ethics sheet outlines the importance of adopting an interdisciplinary approach to defining sentiment in SA and offers a pragmatic solution for its implementation. |
Poisoning Retrieval Corpora by Injecting Adversarial Passages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrievers have outperformed traditional lexical methods in a range of information retrieval tasks, but to what extent can they be safely deployed in real-world applications? |
| Approach: | They propose a method where a malicious user injects a small number of adversarial passages into a retrieval corpus to maximize similarity with a set of training queries. |
| Outcome: | The proposed attack fools retrieval systems into returning top results for queries not seen by the attacker. |
DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) that focus on Standard American English (SAE) often suffer from performance degradation when applied to other dialects. |
| Approach: | They propose a modular approach to imbue SAE-trained models with multi-dialectal robustness . they propose adapters which handle specific linguistic features to imbibe SAe-taught models . |
| Outcome: | The proposed approach improves performance across multiple dialects and dialects. |
Clustering Pseudo Language Family in Multilingual Translation Models with Fisher Information Matrix (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to cluster languages based on ancestral families can yield suboptimal results due to variations in the datasets employed during the model’s training phase. |
| Approach: | They propose a method that leverages the fisher information matrix to cluster language families anchored on the multilingual translation model's characteristics. |
| Outcome: | The proposed method improves performance over conventional language families in adapting a multilingual translation model to unfamiliar language pairs. |
Unifying Discrete and Continuous Representations for Unsupervised Paraphrase Generation (2023.emnlp-main)
Copied to clipboard
Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Jie Fu, Jian Lan, Mei Li, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, Jiancheng Lv
| Challenge: | Existing unsupervised paraphrase generation methods require large-scale, manually annotated paraphrase datasets, which are labor-intensive to build. |
| Approach: | They propose a self-supervised pseudo-data construction method that generates diverse pseudo-paraphrases in distinct surface structures for a given sentence. |
| Outcome: | The proposed method generates diverse pseudo-paraphrases in distinct surface structures for a given sentence. |
The Benefits of Label-Description Training for Zero-Shot Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data to classify among specific label sets in downstream tasks. |
| Approach: | They propose to use a small finetuning dataset to describe the labels for a task and to use it to further improve zero-shot accuracies. |
| Outcome: | The proposed model is more accurate than zero-shot by 17-19% absolute across topic and sentiment datasets and more robust to choices required for zero- shot classification. |
Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that pixel representations can be finetuned across scripts without vocabulary extensions, adapters, or transliteration. |
| Approach: | They propose to use pixel representations to train multilingual machine translation models . they explore parameter sharing within and across scripts to better understand where they lead to positive transfer . |
| Outcome: | The proposed model improves on two multilingual datasets with different language coverage compared to subword embeddings . the proposed model can be finetuned cross-lingually or to unseen scripts, and is more data-efficient than other alternatives such as vocabulary expansion . |
Finding Authentic Counterhate Arguments: A Case Study with Public Figures (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing attempts to generate fake counterhate arguments for hateful content are limited to hallucinate unsupported arguments. |
| Approach: | They propose a method that assures the authenticity of the counter argument and its specificity to the individual of interest. |
| Outcome: | The proposed method assures the authenticity of the counter argument and its specificity to the individual of interest. |
Can We Edit Multimodal Large Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit multimodal models have been used to incrementally infuse a language model with a new set of facts. |
| Approach: | They construct a benchmark for editing multimodal Large Language Models and establish metrics for evaluation. |
| Outcome: | The proposed benchmarks show that editing multimodal models is not as difficult as editing single-modal models. |
Exploring Discourse Structure in Document-level Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document-level machine translation (DocMT) are under-utilizing the context. |
| Approach: | They propose a paragraph-to-paragraph translation mode that utilizes discourse information . they propose 'speech-based' translation mode which utilizes contextual information based on the context . |
| Outcome: | The proposed method utilizes discourse information and performs better than previous methods. |
ClusterLLM: Large Language Models as a Guide for Text Clustering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on 14 datasets show that ClusterLLM consistently improves clustering quality, at an average cost of $0.6 per dataset. |
| Approach: | They propose a text clustering framework that leverages feedback from an instruction-tuned large language model, such as ChatGPT. |
| Outcome: | Extensive experiments on 14 datasets show that ClusterLLM consistently improves clustering quality, at an average cost of $0.6 per dataset. |
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)
Copied to clipboard
| Challenge: | NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge . |
| Approach: | They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code . |
| Outcome: | The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages. |
Learn and Consolidate: Continual Adaptation for Zero-Shot and Multilingual Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual neural machine translation models perform poorly on language pairs with no parallel corpus. |
| Approach: | They propose a two-stage approach that encourages original models to acquire language-agnostic multilingual representations from new data and preserves the model architecture without introducing parameters. |
| Outcome: | The proposed approach improves performance in translation directions where existing models are weak and mitigates degeneration in the well-performing translation directions, offering flexibility in the real-world scenario. |
e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing mental health workforce is struggling to meet the needs adequately. |
| Approach: | They propose a novel polite interpersonal psychotherapy dialogue system that is annotated at two levels: dialogue-level and utterance-level. |
| Outcome: | The proposed system can address depression, anxiety, schizophrenia and other mental health issues. |
AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages (2023.emnlp-main)
Copied to clipboard
Shamsuddeen Muhammad, Idris Abdulmumin, Abinew Ayele, Nedjma Ousidhoum, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Alipio Jorge, Pavel Brazdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu Lawan, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Messelle, Hailu Balcha, Sisay Chala, Hagos Gebremichael, Bernard Opoku, Stephen Arthur
| Challenge: | Africa has the highest linguistic diversity among all continents. |
| Approach: | They introduce a sentiment analysis benchmark that contains >110,000 tweets in 14 African languages . they describe the data collection methodology, annotation process, and challenges . |
| Outcome: | The proposed dataset contains >110,000 tweets in 14 African languages . the tweets were annotated by native speakers and used in the shared task . |
Quantifying Character Similarity with Vision Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Off-the-shelf string matching methods are widely used to link entities across datasets, but they are not available for many settings. |
| Approach: | They propose to use augmented digital fonts to measure character substitution costs for OCR’ed documents by using vision transformers. |
| Outcome: | The proposed method significantly improves record linkage compared to other string matching methods. |
Syllogistic Reasoning for Legal Judgment Analysis (2023.emnlp-main)
Copied to clipboard
Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, Pengjie Ren
| Challenge: | Legal judgment assistants are developing fast due to impressive progress of large language models. |
| Approach: | They construct and manually correct a syllogistic reasoning dataset for legal judgment analysis using large language models as benchmarks. |
| Outcome: | The proposed dataset contains 11,239 criminal cases covering 4 criminal elements, 80 charges and 124 articles. |
Improving Transformer-based Program Repair Model through False Behavior Diagnosis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on automated program repair models lack investigation of false behavior . a large-scale experiment on 55,562 instances found 86.6% accuracy for false behavior classification . |
| Approach: | They propose a method for diagnosing and treating false behavior in transformer-based program repair models . they propose 'behavior vector' and 'bédisc' that identifies false behaviors . |
| Outcome: | The proposed method exhibited a balanced accuracy of 86.6% for false behavior classification on 55,562 instances using four datasets and three models. |
SUT: Active Defects Probing for Transcompiler Models (2023.emnlp-main)
Copied to clipboard
Mengnan Qi, Yufan Huang, Maoquan Wang, Yongqiang Yao, Zihan Liu, Bin Gu, Colin Clement, Neel Sundaresan
| Challenge: | Existing datasets are often criticized for their lack of granularity, which can mask deficiencies in basic syntactic elements that humans care about. |
| Approach: | They propose a new program translation metrics that address basic syntax errors . they propose BLUE, CodeBLUE and computation accuracy metrics which address these errors based on a highly interpretable evaluation harness. |
| Outcome: | The proposed model passes the unit tests with a 26.15% pass rate compared to previous models . |
KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies indicate that language models generate non-factual information that is not supported by evidence with a high level of confidence. |
| Approach: | They propose a knowledge-constrained decoding method that guides a frozen LLM to generate text aligned with the reference knowledge at each decoding step. |
| Outcome: | The proposed method reduces the risk of misinformation generated by LLMs by reducing training costs and catastrophic forgetting for multi-tasking models. |
CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQL (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text-to-SQL generators require the entire schema to be encoded with the user text. |
| Approach: | They propose a method that uses an LLM to hallucinate a minimal DB schema . they use the hallucinated schema to retrieve a subset of the actual schema based on multiple dense retrievals . |
| Outcome: | The proposed method leads to significantly higher recall than existing methods. |
This Reads Like That: Deep Learning for Interpretable Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this work, we explore the extension of prototypical networks to natural language processing. |
| Approach: | They propose a weighted similarity measure that enhances the similarity computation by focusing on informative dimensions of pre-trained sentence embeddings. |
| Outcome: | The proposed method improves predictive performance on AG News and RT Polarity datasets and the rationale-based recurrent convolutions. |
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs (2023.emnlp-main)
Copied to clipboard
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, Amir Globerson
| Challenge: | Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. |
| Approach: | They propose to integrate structured annotations into visual and textual representations to improve VLMs' understanding of compositional scenes. |
| Outcome: | The proposed method improves VLMs on multiple VL datasets with only a mild degradation in ZS capabilities. |
TLM: Token-Level Masking for Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Structured dropout approaches have been investigated to regularize the multi-head attention mechanism in Transformers. |
| Approach: | They propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting by manipulating the connections between tokens in the multi-head attention via masking. |
| Outcome: | The proposed regularization scheme outperforms attention dropout and DropHead on 18 datasets and can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). |
Addressing NER Annotation Noises with Uncertainty-Guided Tree-Structured CRFs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named entity recognition datasets are notorious for their noisy nature due to annotation errors, inconsistencies, and subjective interpretations. |
| Approach: | They propose a method that considers NER as a constituency tree parsing problem and uses a tree-structured Conditional Random Fields with uncertainty evaluation for integration. |
| Outcome: | The proposed model exhibits superb performance even in extreme scenarios with 90% annotation noise. |
Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus (2023.emnlp-main)
Copied to clipboard
| Challenge: | Societal gender asymmetries and inequalities are perpetuated through language . MT often defaults to masculine representations by making undue binary gender assumptions . |
| Approach: | They propose a benchmark and automated evaluation methods to assess gender-neutral translation from English to Italian. |
| Outcome: | The proposed method is based on a survey on gender-neutral translation. |
Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale (2023.emnlp-main)
Copied to clipboard
Marta Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, Carleigh Wood
| Challenge: | Multilingual HolisticBias dataset includes 20,459 sentences in 50 languages . dataset is intended to uncover demographic imbalances and quantify mitigations . |
| Approach: | They propose a multilingual extension of the HolisticBias dataset . they use 118 demographic descriptors and three patterns to build multilingual sentences . |
| Outcome: | The proposed model improves translation quality when the source input only differs in gender . it also improves when the masculine human reference is used in the model . |
GlobalBench: A Benchmark for Global Progress in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, Graham Neubig
| Challenge: | despite advances in NLP, significant disparities in performance across languages still exist . prior benchmarks focused on a limited number of tasks and languages, but now GlobalBench tracks progress on all languages. |
| Approach: | They propose to use global benchmarks to track progress on all NLP datasets in all languages. |
| Outcome: | a new tool tracks progress on all NLP datasets in all languages and tracks per-speaker utility and equity . globalbench is designed to identify the most under-served languages and reward research efforts . a globalbech is available at https://github.com/neulab/globalbench. |
DetGPT: Detect What You Need via Reasoning (2023.emnlp-main)
Copied to clipboard
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang
| Challenge: | Recent advances in the field of computer vision have enabled more effective and sophisticated interactions between humans and machines. |
| Approach: | They propose a reasoning-based object detection paradigm that leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user’s instructions and the visual scene. |
| Outcome: | The proposed method enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. |
Language Models with Rationality (2023.emnlp-main)
Copied to clipboard
| Challenge: | lack of interpretability is a growing impediment to widespread use of large language models . a new approach to solve this problem is to add a rational layer on top of the LLM . |
| Approach: | They propose to add a rational layer to the large language models to make model beliefs explicit . they also propose to identify and minimize contradictions in the model belief graph . |
| Outcome: | a new approach improves consistency without harming overall answer accuracy . the proposed approach makes model beliefs explicit and resolves inconsistencies . |
Self-Improvement of Non-autoregressive Model via Sequence-Level Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive Transformers (NAT) models generate the entire sequence in parallel, but the multimodality problem limits their performance. |
| Approach: | They propose a method to generate distilled data by the NAT model itself, eliminating the need for additional teacher networks. |
| Outcome: | The proposed method can generate distilled data by the NAT model without teacher networks and adapt to different NAT models without precise adjustments. |
Mitigating Temporal Misalignment by Discarding Outdated Facts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal misalignment is a problem for knowledge-intensive tasks where models must rely on data from the past to make predictions. |
| Approach: | They propose a temporal misalignment task to predict how long a given fact will remain true. |
| Outcome: | The proposed task improves calibration for knowledge-intensive tasks under temporal misalignment by discarding volatile facts. |
Open-world Semi-supervised Generalized Relation Discovery Aligned in a Real-world Setting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to open-world relation extraction assume that all instances of unlabeled data belong to novel classes. |
| Approach: | They propose a method that classifies relations from known and novel classes within unlabeled data. |
| Outcome: | The proposed method outperforms existing methods on Open-world RE benchmarks. |
IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on IE comprehension has focused on detecting idiomaticity, but this fails to account for IEs' non-compositionality. |
| Approach: | They construct a commonsense knowledge graph for figurative interpretations of IEs that can be used to convert PTLMs into knowledge models that encode and infer commonsensical knowledge related to IE use. |
| Outcome: | The proposed model can generalize to IEs unseen during training. |
Bias Neutralization in Non-Parallel Texts: A Cyclic Approach with Auxiliary Guidance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for subjective bias neutralization rely on parallel text training and lose important bias-independent context. |
| Approach: | They propose a bias neutralization model that uses an auxiliary guided cycle consistent GAN to train with a combination of adversarial, cycleconsistency and identity mapping loss. |
| Outcome: | The proposed model significantly improves subjective bias neutralization compared to existing methods. |
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent ubiquity and disruptive impacts of large language models have raised concerns about their potential to be misused. |
| Approach: | They propose a strategy that leverages LLMs' generative and emergent reasoning capabilities to counter human-written and LLM-generated disinformation. |
| Outcome: | The proposed strategy synthesizes authentic and deceptive LLM-generated content through paraphrase-based and perturbation-based prefix-style prompts, respectively. |
SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. |
| Approach: | They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt. |
| Outcome: | The proposed method outperforms baseline methods while preserving memory usage. |
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | lateral thinking tasks require implicit and complex reasoning, relying on human-like commonsense mechanisms. |
| Approach: | They propose a lateral thinking benchmark to test models' ability to exhibit lateral reasoning and defy default commonsense associations. |
| Outcome: | The proposed model exhibits lateral thinking and defies default commonsense associations. |
When are Lemons Purple? The Concept Association Bias of Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval tasks. |
| Approach: | They propose to use "question text" as input for the text encoder of CLIP to make the prediction harder than it should be. |
| Outcome: | The proposed model treats input as a bag of concepts and attempts to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. |
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |
Text Representation Distillation via Information Bottleneck Principle (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have recently shown great success in text representation field, however, the high computational cost and high-dimensional representation of PLMs pose significant challenges for practical applications. |
| Approach: | They propose a Knowledge Distillation method that distills large models into smaller representation models to reduce performance degradation after distillation. |
| Outcome: | Empirical results on two main downstream applications of the proposed method show that it reduces the risk of over-fitting and maximizes the mutual information between the model and the input data. |
Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for distilling large language models into smaller, more efficient student models are based on educational science principles such as knowledge tracing and personalized learning. |
| Approach: | They propose a method for distilling large language models into smaller, more efficient student models that are aligned with educational science principles such as knowledge tracing and personalized learning. |
| Outcome: | The proposed approach outperforms LLMs on three benchmarks while employing significantly fewer parameters. |
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations for theory of mind (ToM) use passive narratives that lack interactivity. |
| Approach: | They propose a benchmark to stress-test ToM within information-asymmetric conversational contexts via question answering. |
| Outcome: | The proposed benchmark is challenging for state-of-the-art language models, which perform significantly worse than humans even with chain-of thought reasoning or fine-tuning. |
Exploring the Boundaries of GPT-4 in Radiology (2023.emnlp-main)
Copied to clipboard
Qianchu Liu, Stephanie Hyland, Shruthi Bannur, Kenza Bouzid, Daniel Castro, Maria Wetscherek, Robert Tinn, Harshita Sharma, Fernando Pérez-García, Anton Schwaighofer, Pranav Rajpurkar, Sameer Khanna, Hoifung Poon, Naoto Usuyama, Anja Thieme, Aditya Nori, Matthew Lungren, Ozan Oktay, Javier Alvarez-Valle
| Challenge: | Recent success of general-domain large language models has changed the natural language processing paradigm towards a unified foundation model across domains and applications. |
| Approach: | They evaluate the performance of GPT-4 on a variety of radiology tasks . they find it outperforms or matches current SOTA radiology models . |
| Outcome: | The proposed model outperforms or matches current SOTA radiology models on a range of tasks. |
A Frustratingly Easy Post-Training Quantization Scheme for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Efficient inference is crucial for hyper-scale AI models, including large language models, as their parameter count continues to increase for enhanced performance. |
| Approach: | They propose a quantization scheme that fully utilizes the Transformer structure used in large language models to minimize the frequency of DRAM access while exploiting the parallelism of operations. |
| Outcome: | The proposed method minimizes the frequency of DRAM access while exploiting the parallelism of operations through a dense matrix format. |
A Comprehensive Evaluation of Biomedical Entity Linking Models (2023.emnlp-main)
Copied to clipboard
David Kartchner, Jennifer Deng, Shubham Lohiya, Tejasri Kopparthi, Prasanth Bathala, Daniel Domingo-Fernández, Cassie Mitchell
| Challenge: | Current methods struggle to correctly link genes and proteins and often have difficulty incorporating context into linking decisions. |
| Approach: | They evaluate nine recent state-of-the-art biomedical entity linking models under a unified framework. |
| Outcome: | The proposed models are compared along axes of accuracy, speed, ease of use, generalization, adaptability and adaptability to new ontologies and datasets. |
Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work suggests that instead of directly countering surface-level reasoning, one should follow an argumentation style inspired by the Jiu-Jitsu “soft” combat system. |
| Approach: | They propose a task of attitude and theme-guided rebuttal generation for peer reviews to enrich existing discourse structure with attitude roots, attitude themes, and canonical reversals. |
| Outcome: | The proposed task is based on an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. |
LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, existing systems cannot accurately identify most of the world's 7000 languages due to lack of data and computational challenges. |
| Approach: | They propose a misprediction-resolution hierarchical model, LIMIT, that reduces error by 55% on a children's stories dataset and by 40% on 'fLORES-200' benchmark. |
| Outcome: | The proposed model reduces error by 55% on the MCS-350 and 40% on the FLORES-200 benchmarks. |
FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern machine learning models require a huge collection of precisely labeled data, which can be labor-intensive and time-consuming. |
| Approach: | They propose a collaborative learning framework that interactively distills and filters the task-specific knowledge from LLMs. |
| Outcome: | The proposed framework improves zero-shot performance on eight benchmark datasets without human supervision. |
API-Assisted Code Generation for Question Answering on Varied Table Structures (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to table question answering have been limited to specific table structures. |
| Approach: | They propose a unified TableQA framework that uses Python as a querying language and few-shot prompting to translate NL questions into Python programs. |
| Outcome: | The proposed framework provides a unified representation for structured tables as multi-index Pandas data frames and uses Python as a powerful querying language to translate NL questions into Python programs. |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
ChatEdit: Towards Multi-turn Interactive Facial Image Editing via Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to interactive facial image editing treat multi-turn editing as a sequence of successive single-turn edits, leading to attribute forgetting and error accumulation. |
| Approach: | They propose a framework for interactive facial image editing through dialogues based on the CelebA-HQ dataset and a benchmark dataset to evaluate this. |
| Outcome: | The proposed framework outperforms existing methods and improves existing ones. |
Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings do not capture the semantic similarity of sentences. |
| Approach: | They propose a framework that integrates compositional sentence operations into the embedding space and optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddables. |
| Outcome: | The proposed framework improves the interpretability of sentence embeddings on four textual generation tasks while maintaining strong performance on traditional semantic similarity tasks. |
Outlier Dimensions Encode Task Specific Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that fine-tuning outlier dimensions is detrimental to the representational quality of embeddings. |
| Approach: | They investigate how fine-tuning impacts outlier dimensions by testing their hypothesis that a single outlier dimension can complete downstream tasks with a minimal error rate. |
| Outcome: | The proposed model can encode crucial task-specific knowledge and the value of a representation in a single outlier dimension drives downstream model decisions. |
Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining (2023.emnlp-main)
Copied to clipboard
Jingcong Liang, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei
| Challenge: | Recent studies have discussed its capability to assist language models for various applications. |
| Approach: | They propose a structure to organize arguments using the **Hi**erarchical **Ar**gumentation **G**raph (Hi-ArG) and propose two approaches to exploit Hi-AarG, including a text-graph multi-modal model GreaseArR and a framework augmented with graph information. |
| Outcome: | The proposed structure supersedes existing language models on two argumentation tasks while incorporating graph information during further training improves vanilla language models. |
Biomedical Named Entity Recognition via Dictionary-based Synonym Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for biomedical named entity recognition require laborious human effort. |
| Approach: | They propose a Synonym Generalization framework that recognizes biomedical concepts using span-based predictions. |
| Outcome: | The proposed framework outperforms dictionary-based approaches on a wide range of benchmarks. |
GNAT: A General Narrative Alignment Tool (2023.emnlp-main)
Copied to clipboard
| Challenge: | Algorithmic sequence alignment is a common operation in many NLP tasks, but it is difficult to recognize similarities between distant versions of narratives such as translations and retellings. |
| Approach: | They propose a general approach to narrative alignment coupling the Smith-Waterman algorithm from bioinformatics with modern text similarity metrics. |
| Outcome: | The proposed approach can be used to identify similarities between two different versions of narratives, and to define rigorous p-values on the significance of any alignment. |
Self-Ensemble of N-best Generation Hypotheses by Lexically Constrained Decoding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have improved generation quality by explicitly reranking N-best candidates. |
| Approach: | They propose a method that ensembles N-best hypotheses to improve natural language generation by combining high-quality fragments of N- best hypothese . they use tokens that should or should not be present in the final output as lexical constraints to improve quality of generation. |
| Outcome: | Empirical results show that the proposed method outperforms strong N-best reranking methods on paraphrase generation, summarisation, and constrained text generation. |
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chart-based data analysis neglect explicit modeling of chart structures. |
| Approach: | They propose a pretrained model for chart comprehension and reasoning that encodes relevant text, data, and visual elements of charts and uses a chart-grounded text decoder for text generation. |
| Outcome: | The proposed model outperforms existing methods that lack explicit modeling of chart structures and lacks explicit modeling. |
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments show that MEO significantly improves computational efficiency . compared to dense networks, sparsely activated networks only employ a few parameters for each input . |
| Approach: | They propose a method that merges multiple experts into one to reduce computation costs . they demonstrate that a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters for each input . |
| Outcome: | The proposed approach reduces the computational cost to that of a single expert by 83.3% compared to 82.6% in vanilla MoE. |
Distance-Based Propagation for Efficient Knowledge Graph Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge graph completion (KGC) aims to predict unseen edges in knowledge graphs (KGs) . a few recent attempts to address this problem sacrifice the performance to gain efficiency. |
| Approach: | They propose a method that aggregates path information to solve this problem by aggregating paths in a fixed window for each source-target pair. |
| Outcome: | The proposed method can cut down on the number of propagated messages by 90% while achieving competitive performance on multiple KG datasets. |
What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate section-wise summaries of contracts can be tedious due to length and complexity of legalese. |
| Approach: | They propose a task of party-specific extractive summarization for legal contracts . they train a pairwise importance ranker and propose incorporating domain-specific notions of importance . |
| Outcome: | The proposed system generates a party-specific contract summary using a dataset of lease agreements and lease agreements. |
Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proficient in natural language processing tasks, but their deployment is limited by extensive parameter sizes and computational demands. |
| Approach: | They propose a method to enhance computational efficiency in large language models by 4-bit weight and 8-bit activation quantization. |
| Outcome: | The proposed techniques significantly boost task accuracies to levels comparable with full-precision models. |
CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current work on understanding assembly code is oriented towards generating function names, which involve numerous abbreviations that make them confusing. |
| Approach: | They propose a control flow graph and pseudo code guided binary code summarization framework to learn the comprehensive binary function execution behavior and logic semantics. |
| Outcome: | The proposed framework improves the efficiency of reverse engineering on 3 different binary optimization levels for 3 different computer architectures. |
Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take advantage of step-by-step reasoning instructions . negation is a core linguistic phenomenon that is difficult to process . |
| Approach: | They examine the step-by-step reasoning ability of large language models with a focus on negation . negation is a core linguistic phenomenon that is difficult to process . |
| Outcome: | The proposed models perform better when using chain-of-thought prompting . the results highlight unique limitations in each LLM family . |
Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) is a technique that guides large language models to decompose complex tasks into multi-step reasoning processes. |
| Approach: | They propose a two-step reasoning framework based on prompt tuning to implement step-by-step thinking for MLMs on NLU tasks. |
| Outcome: | The proposed framework outperforms baselines and achieves state-of-the-art performance on two NLU tasks. |
Large Language Models are Complex Table Parsers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments and results on Complex Table QA datasets, i.e., the open-domain dataset HiTAB and the aviation domain dataset AIT-QA show that our approach significantly outperforms previous work on both datasets. |
| Approach: | They propose to incorporate Generative Pre-trained Transformer 3.5 to address the specific challenges posed by Complex Table QA by reconstructing tables into tuples and using prompt templates to create dialogues. |
| Outcome: | The proposed approach outperforms previous work on complex table parsing datasets and leads to state-of-the-art (SOTA) performance. |
R2H: Building Multimodal Navigation Helpers that Respond to Help Requests (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing dialog-based embodied datasets are not sufficient to develop intelligent navigation-helper agents capable of navigating users in unfamiliar areas. |
| Approach: | They introduce a novel benchmark, Respond to Help Requests, to promote the development of multi-modal navigation helpers capable of responding to requests for help . they also propose two approaches to construct the navigation-helper agent, including fine-tuning a task-oriented multi-mod response generation model that can see and respond, named SeeRee, and employing . a multi-module large language model in a zero-shot manner. |
| Outcome: | The proposed model outperforms the baseline model and the proposed model on two tasks based on human evaluations and automatic benchmarking. |
Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing contextual biasing techniques require additional parameterization . state-of-the-art ASR systems often fail to recognize named entities or critical rare words . |
| Approach: | They propose an algorithm that uses nearest-neighbor matching to predict ASR models . a list of rare entities is indexed in a memory and then stored the best possible match . |
| Outcome: | The proposed algorithm improves the prediction of state-of-the-art ASR models on rare words . it prevents spurious matches by restricting to word-level matches . |
Generative Table Pre-training Empowers Models for Tabular Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to use table pre-training to boost tabular prediction performance remain open . a bachelor's degree earns less than 50K, and a generative LM can be used to unify tasks via one LM. |
| Approach: | They propose a method that leverages table pre-training to empower tabular prediction models. |
| Outcome: | The proposed method outperforms baseline models on 12 datasets and can be easily combined with various backbone models. |
Learning to Describe for Predicting Zero-shot Drug-Drug Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing computational methods for DDI prediction fail to capture interactions for new drugs due to the lack of knowledge. |
| Approach: | They propose a problem setup as zero-shot DDI prediction that deals with the case of new drugs by using textual information from online databases. |
| Outcome: | The proposed method improves on several settings including zero-shot and few-shot DDI prediction and the selected texts are semantically relevant. |
A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge. |
| Approach: | They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs . |
| Outcome: | The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets. |
Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Summarization of poetry is a challenging task as it can be easily lost if only the literal meaning is considered. |
| Approach: | They propose to use poetry as a model to summarize poetry and provide a dataset to evaluate their creative language interpretation capacity. |
| Outcome: | The proposed dataset consisting of 3011 samples and its corresponding summarized interpretation in the English language provides an opportunity to evaluate the creative language interpretation capacity of the proposed models. |
Privacy Implications of Retrieval-Based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of retrieval-based language models shows improved interpretability, factuality, and adaptability compared to parametric counterparts . kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models . |
| Approach: | They present the first study of privacy risks in retrieval-based language models . they aim to strike a balance between utility and privacy in domains where privacy is of concern . |
| Outcome: | The proposed methods improve interpretability, factuality, and adaptability compared to parametric models . the study finds that kNN-LMs are more susceptible to leaking private data than parametric ones . |
IMTLab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems (2023.emnlp-main)
Copied to clipboard
Xu Huang, Zhirui Zhang, Ruize Gao, Yichao Du, Lemao Liu, Guoping Huang, Shuming Shi, Jiajun Chen, Shujian Huang
| Challenge: | Existing systems that use a left-to-right completion paradigm are inefficient and expensive. |
| Approach: | They propose an open-source end-to-end interactive machine translation system platform . they propose to use a prefix-constrained decoding approach to achieve end- to-end evaluation . |
| Outcome: | The proposed system can guarantee high-quality, error-free translations . it uses prefix-constrained decoding and improves on previous systems . |
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023.emnlp-main)
Copied to clipboard
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, Zhaochun Ren
| Challenge: | Existing work utilizes generative LLMs for Information Retrieval (IR) rather than direct passage ranking. |
| Approach: | They investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR and use a test set to verify the model’s ability to rank unknown knowledge. |
| Outcome: | The proposed model outperforms a 3B supervised model on the BEIR benchmark. |
DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization datasets lack natural language variation due to limited data scale or lack of diversity. |
| Approach: | They propose a compositional generalization task to evaluate natural language understanding ability under compositional settings. |
| Outcome: | The proposed method outperforms the plain seq2seq trained version by a large margin . it uses two strong baseline methods and large language models to tackle the task . |
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts. |
| Approach: | They analyze a visual question answering dataset tailored for info-seeking questions . they show that pre-trained visual and language models can use fine-grained knowledge . |
| Outcome: | The proposed dataset elicits models to use fine-grained knowledge learned during pre-training. |
EDeR: Towards Understanding Dependency Relations Between Events (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on event relation extraction focuses on hierarchical, temporal and causal relations but ignores the interdependence between events. |
| Approach: | They propose to use a human-annotated Event Dependency Relation dataset to identify event dependency relations between two events. |
| Outcome: | The proposed dataset integrates existing annotations with the OntoNotes dataset and shows that recognizing such event dependency relations can further benefit critical NLP tasks, including semantic role labelling and co-reference resolution. |
It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies lack diversity in problem types, lexical usage patterns, languages, and intermediate solution forms for the math word problem. |
| Approach: | They propose a new MWP dataset with a wide range of diversity in problem types, lexical usage patterns, languages, and intermediate solutions. |
| Outcome: | The proposed dataset provides an opportunity to evaluate the capability of large language models. |
Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using the TREC Misinformation dataset, we empirically evaluate ChatGPT to show not just its effectiveness but reveal that knowledge passed in the prompt can bias the model to the detriment of answer correctness. |
| Approach: | They empirically evaluate ChatGPT to find out whether a prompt can bias the model to the detriment of answer correctness. |
| Outcome: | The proposed model can be biased to the detriment of answer correctness by using retrieved-then-generate pipelines and how a user phrases their question as well as the question type. |
kNN-LM Does Not Improve Open-ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Interpolation-based retrieval-augmented language models (LMs) are a subtype of retrieval augmented language model that computes the probability of the next token by interpolating between the softmax distribution of the original LM and a token distribution formed by retrieving over an external datastore. |
| Approach: | They propose to interpolate the predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. |
| Outcome: | The proposed methods do not exhibit improvements in open-ended generation quality, as measured by automatic evaluation metrics and human evaluations. |
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models. |
| Approach: | They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states . |
| Outcome: | The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures. |
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023.emnlp-main)
Copied to clipboard
Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun
| Challenge: | Parameter-efficient tuning (PET) methods can drive large pre-trained language models by training only minimal parameters. |
| Approach: | They propose a parameter-efficient tuning method that is compatible with a tunable module and uses a random number generator to optimize fewer table parameters. |
| Outcome: | The proposed method is compatible with a tunable module and tested on 11 NLP tasks. |
STAIR: Learning Sparse Text and Image Representation in Grounded Tokens (2023.emnlp-main)
Copied to clipboard
Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Jose, Alexander Toshev, Yantao Zheng, Jonathon Shlens, Ruoming Pang, Yinfei Yang
| Challenge: | State-of-the-art contrastive learning models like CLIP and ALIGN are less interpretable and suffer from inferior accuracy than dense representations. |
| Approach: | They extend CLIP and ALIGN models to build a sparse semantic representation that is interpretable and easy to integrate with existing retrieval systems. |
| Outcome: | The proposed model outperforms CLIP and ALIGN models on image and text retrieval tasks with a 4.9% and +4.3% improvement on COCO-5k textimage and imagetext retrieval respectively. |
Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting (2023.emnlp-main)
Copied to clipboard
| Challenge: | idioms are common in everyday language, but often pose a challenge to translators because their meanings do not follow from the meanings of their parts. |
| Approach: | They propose to use retrieval-augmented models to increase the accuracy of a strong pretrained machine translation model on idiomatic sentences by up to 13%. |
| Outcome: | The proposed techniques improve the accuracy of a strong pretrained model on idiomatic sentences by up to 13% in absolute accuracy, and holds potential benefits for non-idiomatic phrases. |
CoRec: An Easy Approach for Coordination Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing syntactic parsers are slow and suffer from errors, especially for long and complicated sentences. |
| Approach: | They propose a pipeline model COordination RECognizer with coordinator identifier and conjunct boundary detector. |
| Outcome: | The proposed model improves the yield of state-of-the-art Open IE models by reducing errors and slow processing time. |
A linear time approximation of Wasserstein distance with word embedding selection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Wasserstein distance is a powerful method for measuring the dissimilarity between distributions and is used in natural language processing to measure dissimilarities between documents. |
| Approach: | They propose a method to combine feature selection and tree approximation of Wasserstein distance to handle high-dimensional problems. |
| Outcome: | The proposed method achieves high performance on document classification using word embeddings and word embeds. |
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides in complex reasoning tasks, but their reasoning is often constrained by their intrinsic understanding, lacking external insights. |
| Approach: | They propose a framework that enables cross-model communication during problem-solving. |
| Outcome: | The proposed framework surpasses established baselines in complex reasoning tasks and is cost-effective. |
Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Emotion recognition is a crucial task for human conversation understanding . multimodal data, e.g., language, voice, and facial expressions, add complexity to the task. |
| Approach: | They propose a relational temporal Graph Neural Network with Auxiliary Cross-Modality Interaction framework that captures conversation-level cross-modality interactions and utterance-level temporal dependencies with modality-specific manner for conversation understanding. |
| Outcome: | The proposed framework captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. |
Connecting degree and polarity: An artificial language learning study (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that degree modifiers are related to sentence polarity, but they are not related to the grammatical number of an expression. |
| Approach: | They propose to generalize degree modifiers to their polarity sensitivity in pre-trained language models by applying the Artificial Language Learning experimental paradigm from psycholinguistics to a neural language model. |
| Outcome: | The proposed generalisations are consistent with existing linguistic observations that relate de-gree semantics to polarity sensitivity, including the main one: low degree semantics is associated with preference towards positive polarities. |
Prompting with Pseudo-Code Instructions (2023.emnlp-main)
Copied to clipboard
| Challenge: | ambiguity in natural language can hinder performance of large language models. |
| Approach: | They manually create a dataset of pseudo-code prompts for 132 different classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. |
| Outcome: | The pseudo-code prompts improve the performance of two LLM families, BLOOM and CodeGen. |
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for reasoning about events in narratives do not understand the complexity of the causal relationships of events in the narrative. |
| Approach: | They propose a Causal Reasoning Assessment Benchmark to evaluate causal understanding of events in narratives. |
| Outcome: | The proposed model performs worse when models are derived from complex causal structures than simple linear causal chains. |
NORMSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to understand acceptable behavior have focused on a single culture and manually built datasets from non-conversational settings. |
| Approach: | They propose a framework to automatically extract culture-specific norms from multi-lingual conversations. |
| Outcome: | The proposed framework extracts culture-specific norms from multi-lingual conversations. |
A State-Vector Framework for Dataset Effects (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent DNN-based systems gain linguistic abilities on multiple levels ranging from syntax, semantics, and even some discourse-related abilities. |
| Approach: | They propose a state-vector framework that uses idealized probing test results as the bases of a vector space to quantify the effects of both standalone and interacting datasets. |
| Outcome: | The proposed framework allows to quantify the effects of both standalone and interacting datasets. |
Challenges in Context-Aware Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | despite well-reasoned intuitions, most context-aware neural machine translation models show only modest improvements over sentence-level systems. |
| Approach: | They propose a more realistic setting for document-level translation called paragraph-to-paragraph (PARA2PARA) they collect a dataset of Chinese-English novels to promote future research . |
| Outcome: | The proposed model improves translation quality across document-level metrics and discourse phenomena. |
Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate text in mental health are limiting, but they are effective for many tasks. |
| Approach: | They propose a task-adaptive tokenizer that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. |
| Outcome: | The proposed tokenization approach improves generation performance on psychological question-answering tasks in Chinese and English while using 60% fewer tokens. |
FACTIFY3M: A benchmark for multimodal fact verification with explainability through 5W Question-Answering (2023.emnlp-main)
Copied to clipboard
Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Aditya Pakala, Ishan Paul, Janvita Reddy, Arghya Sarkar, Kinjal Sensharma, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | Disinformation can cause disruption in the share market, panic and anxiety in society, and even death during crises. |
| Approach: | a new dataset is being developed to help combat disinformation . the dataset is a multimodal fake news dataset with 5W question-answering . |
| Outcome: | FACTIFY 3M is the largest dataset and benchmark for multimodal fact verification. |
Building Multi-domain Dialog State Trackers from Single-domain Dialogs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-domain dialog state tracking models require significant manual effort to define domain relations and collect data. |
| Approach: | They propose a divide-and-conquer (DAC) DST paradigm and a multi-domain dialog synthesis framework to build multi- domain DST models from single-domain dialogues. |
| Outcome: | The proposed paradigm makes building multi-domain DST models easier on unseen domain combinations. |
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning can be a data-efficient method for transforming large language models into generalist models, but their performance lags behind specialist models trained exclusively for specific tasks. |
| Approach: | They propose to incorporate broadcoverage generalist instruction tuning into large language models to build a specialist model by incorporating task specificity and skill requirements. |
| Outcome: | The proposed method improves model performance when task coverage is broad and when training data is limited. |
Making Large Language Models Better Data Creators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced the field of NLP significantly, but deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. |
| Approach: | They propose a unified data creation pipeline that requires only a single formatting example. |
| Outcome: | The proposed pipeline can generate data with a single formatting example. |
Hallucination Detection for Generative Large Language Models by Bayesian Sequential Estimation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting hallucinations require large numbers of observations to be retrieved, increasing response times. |
| Approach: | They propose a framework that leverages Bayesian sequential analysis to optimize the trade-off between costs and benefits during the hallucination detection process. |
| Outcome: | The proposed framework surpasses existing methods in efficiency and precision of hallucination detection. |
Guideline Learning for In-Context Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can perform a task by conditioning on task instructions and a few input-output examples without optimizing any parameters. |
| Approach: | They propose a guideline learning framework for In-context IE which reflectively learns and follows guidelines. |
| Outcome: | The proposed framework improves the performance of in-context IE by synthesizing and following guidelines. |
Open Information Extraction via Chunks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing OIE systems split a sentence into tokens and recognize token spans as tuple relations and arguments. |
| Approach: | They propose to split a sentence into tokens and recognize token spans as tuple relations and arguments. |
| Outcome: | The proposed model achieves state-of-the-art on multiple OIE datasets showing that SaC has better properties than sentence as token sequence. |
Rethinking Word-Level Auto-Completion in Computer-Aided Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for word-level auto-completion (WLAC) do not meet the criterion of good auto-completes. |
| Approach: | They propose a measurable criterion to address the question: what kind of words are good auto-completions? they propose an approach to enhance WLAC performance by promoting adherence to the cri-terion. |
| Outcome: | The proposed approach outperforms the top-performing system submitted to the WLAC shared tasks in WMT2022 while using significantly smaller model sizes. |
Automatic Transcription of Handwritten Old Occitan Language (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handwritten text recognition have shown promising results, but low-resource languages often lack resources. |
| Approach: | They propose an HTR approach that leverages the Transformer architecture for recognizing handwritten Old Occitan language. |
| Outcome: | The proposed approach surpasses state-of-the-art models for Old Occitan HTR, including open-source Transformer-based models and commercial applications like Google Cloud Vision. |
CorefPrompt: Prompt-based Event Coreference Resolution by Measuring Event Type and Argument Compatibilities (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event coreference resolution (ECR) do not leverage human-summarized rules to guide the model. |
| Approach: | They propose to transform ECR into a cloze-style MLM task using a prompt-based approach . they introduce two auxiliary prompt tasks, event-type compatibility and argument compatibility . |
| Outcome: | The proposed method performs well in a state-of-the-art (SOTA) benchmark. |
Anaphor Assisted Document-Level Relation Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document-level relation extraction are incomplete and lack anaphor for identifying relations between entities. |
| Approach: | They propose an Anaphor-Assisted (AA) framework for document-level relation extraction . they use a document or sentences as intermediate nodes to model cross-sentence entity interactions . |
| Outcome: | The proposed framework achieves state-of-the-art on the widely-used datasets. |
FinEntity: Entity-level Sentiment Classification for Financial Texts (2023.emnlp-main)
Copied to clipboard
| Challenge: | FinEntity annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. |
| Approach: | They introduce an entity-level sentiment classification dataset called FinEntity that annotates financial entity spans and their sentiment in financial news. |
| Outcome: | The proposed dataset annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. |
All Things Considered: Detecting Partisan Events from News Media with Cross-Article Comparison (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that media influence opinion via the inclusion or omission of partisan events. |
| Approach: | They develop a latent variable-based framework to predict the ideology of news articles by comparing multiple articles on the same story and identifying partisan events whose inclusion or omission reveals ideology. |
| Outcome: | The proposed framework validates the existence of partisan event selection and detects partisan events and article ideology better than baselines. |
Rationale-Enhanced Language Models are Better Continual Relation Learners (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have found that catastrophic forgetting arises from the model’s lack of robustness against future analogous relations. |
| Approach: | They propose a multi-task rationale tuning strategy to help the model learn current relations robustly and conduct contrastive rationale replay to further distinguish analogous relations. |
| Outcome: | The proposed method outperforms the state-of-the-art models on two benchmarks. |
BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have tried to detect and control the spread of such abusive memes on social media platforms. |
| Approach: | They build a Bengali meme dataset to test models for abusive memes . they find that multimodal models that use both textual and visual information outperform unimodal models . |
| Outcome: | The proposed model outperforms unimodal models in a Bengali meme dataset. |
ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Eye movements in reading are a key part of psycholinguistic research, but the lack of eye movement data and its unavailability at application time pose a major challenge for this line of research. |
| Approach: | They propose a novel sequence-to-sequence diffusion model that generates synthetic scanpaths on texts by leveraging pre-trained word representations and jointly embedding both the stimulus text and the fixation sequence. |
| Outcome: | The proposed model outperforms state-of-the-art models in psycholinguistic analysis and is able to exhibit human-like reading behavior. |
From Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing large-scale surveys soliciting opinions on issues can be costly and laborious. |
| Approach: | They propose to use value-injected large language models to inject a target value distribution into large language model (LLM) and have them predict opinions and behaviors of people with similar values. |
| Outcome: | The proposed method significantly outperforms baseline methods on four tasks and the results suggest opinions and behaviors can be better predicted using value-injected LLMs. |
Analyzing Film Adaptation through Narrative Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the book-to-film adaptation process by examining the differences between the two media . novel adaptations often require dropping sections of the source text from the movie script . |
| Approach: | They use a Smith-Waterman local alignment algorithm to quantify text similarity between scenes and book units. |
| Outcome: | The proposed method reveals that novel adaptations often require dropping parts of the source text from the movie script. |
Inverse Scaling Can Become U-Shaped (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scaling up language models has been shown to improve performance on a wide range of downstream tasks, but are there any tasks for which performance gets worse as models scale? |
| Approach: | They evaluate models trained on five times more compute and evaluated them on 280B parameters and 500 zettaFLOPs of training compute. |
| Outcome: | The proposed tasks show that performance decreases as models scale and increases again as models get larger. |
Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Nearest Neighbor Machine Translation (kNN-MT) is a powerful domain adaptation tool . the reasons for its success have not been thoroughly investigated . |
| Approach: | They propose to integrate pre-trained Neural Machine Translation models with token-level retrieval . they propose to implicitly execute gradient descent on the output projection layer of NMT . |
| Outcome: | The proposed approach outperforms model fine-tuning on in-domain tests while achieving better performance on out-of-domain sets. |
Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new method for finding semantic differences in words appears in two corpora, but it requires a variance of word vectors . a word covers more meanings in a corpus, and its mean word vector becomes shorter . |
| Approach: | They propose a method to measure the coverage of meanings of a word in a corpus through the norm of its mean word vector. |
| Outcome: | The proposed methods rival the best-performing system in the SemEval-2020 Task 1 . they are robust for the skew in corpus sizes and capable of detecting infrequent words . |
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have demonstrated impressive molecule understanding ability on 1D text-related tasks, but lack 2D graph perception, a critical ability of human professionals in comprehending molecules’ topological structures. |
| Approach: | They propose to combine a cross-modal projector and a uni-modal adapter to enable an LM to understand both text- and graph-based molecular contents via a Q-Former. |
| Outcome: | The proposed model outperforms the baselines on tasks such as molecule captioning, IUPAC name prediction, and molecule-text retrieval. |
A Training-Free Debiasing Framework with Counterfactual Reasoning for Conversational Emotion Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies in Emotion Recognition in Conversations (ERC) focus on capturing context-sensitive and speaker-sensitive dependencies, ignoring the unintended dataset biases of data. |
| Approach: | They propose a training-free debiasing framework that extracts biases from the model by generating counterfactual utterances and contexts and mitigates them using simple yet empirically robust element-wise subtraction operations. |
| Outcome: | Experiments on three public datasets show that the proposed framework improves generalization ability and fairness across different ERC models. |
Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown striking ability to adapt to target tasks with a few input-output demonstrations. |
| Approach: | They propose a framework which bootstraps LMs’ intrinsic capabilities to perform zero-shot ICL. |
| Outcome: | The proposed framework outperforms baselines on 23 BIG-Bench Hard tasks on average accuracy and head-to-head comparison. |
Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
Taolin Zhang, Ruyao Xu, Chengyu Wang, Zhongjie Duan, Cen Chen, Minghui Qiu, Dawei Cheng, Xiaofeng He, Weining Qian
| Challenge: | Existing methods for pre-training KEPLMs with relational triples are difficult to adapt to close domains due to the lack of sufficient domain graph semantics. |
| Approach: | They propose a Knowledge-enhanced language representation learning framework for various closed domains that captures the implicit graph structure among the entities. |
| Outcome: | The proposed framework outperforms existing methods for pre-training KEPLMs in closed domains significantly. |
ScdNER: Span-Based Consistency-Aware Document-Level Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is an important task for many natural language processing applications. |
| Approach: | They propose to fuse global features of tokens via word-based key-value memory to produce documentlevel encoding for token label prediction. |
| Outcome: | The proposed model can produce consistent and consistent predictions on word level with reduced impact of non-entity sequences and adaptive global feature fusion. |
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retraining from scratch are limited and only work on the recall of edited facts. |
| Approach: | They propose a benchmark method that allows users to ask multi-hop questions to assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. |
| Outcome: | The proposed method outperforms existing models and scales well with LLMs (up to 175B) it is based on a memory-based approach that stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts. |
Stance Detection on Social Media with Background Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies of stance detection focus on learning stance information about specific targets from context, but in real-world scenarios, we usually have a certain understanding of a target when we express our stance on it. |
| Approach: | They propose to take the background knowledge of the target into account for better stance detection by categorizing it into episodic and discourse knowledge categories and a heuristic retrieval algorithm based on the topic to retrieve the Wikipedia documents relevant to the sample. |
| Outcome: | The proposed framework achieves state-of-the-art on four benchmark datasets showing that the proposed framework is able to detect stances in-target and zero-shot scenarios. |
Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models train a visual encoder with weak cross-modal supervision signals, resulting in a limited capacity to capture non-textual features and suboptimal performance. |
| Approach: | They propose a Visually-Asymmetric coNsistenCy Learning approach that enhances the model’s ability to capture fine-grained visual and layout features through the incorporation of color priors. |
| Outcome: | The proposed approach outperforms the strong LayoutLM series baseline on benchmark datasets and provides insights for optimizing model performance. |
NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social norms fundamentally shape interpersonal communication. |
| Approach: | They propose a human-in-the-loop pipeline to synthesize a bilingual dyadic dialogue dataset with turn-by-turn annotations of social norms for Chinese and American cultures. |
| Outcome: | The proposed dataset is high-quality through human evaluation and compares with existing models. |
ClimateBERT-NetZero: Detecting and Assessing Net Zero and Reduction Targets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing tools for extracting information about net zero and emission reduction targets have not been used to assess the vast amounts of information about sustainability commitments made by public and private actors. |
| Approach: | They propose a data set and train and release a natural language classifier to detect whether a text contains a net zero or reduction target. |
| Outcome: | The proposed model can be combined with conventional Q&A models to analyze the ambitions displayed in net zero and reduction targets. |
Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Inefficient transformers have been a challenge for many years, requiring computational costs that scale quadratically with the length of the input sequence. |
| Approach: | They propose a token reduction approach that dynamically routes tokens within layers to ensure that all tokens remain accessible in subsequent layers. |
| Outcome: | The proposed approach achieves up to 25x faster inference time without significant loss in accuracy. |
Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Query-focused Summarization (QfS) is a system that generates summaries from document(s) based on a query. |
| Approach: | They propose a Query-focused Summarization approach that uses a generalization of Reinforcement Learning (RL) for Natural Language Generation and a better semantic similarity reward. |
| Outcome: | The proposed approach improves on the ROUGE-L metric and in a benchmark dataset. |
Fair Text Classification with Wasserstein Independence (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel method for mitigating group fairness in text classification remains an open challenge . a recent paper focuses on the problem of group fair in the domain of natural language processing . |
| Approach: | They propose a method for mitigating group fairness in neural text classification . they use adversarial training to induce Wasserstein independence between representations . |
| Outcome: | The proposed method exhibits a comparable or better fairness-accuracy trade-off compared to existing methods. |
TacoPrompt: A Collaborative Multi-Task Prompt Learning Method for Self-Supervised Taxonomy Completion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for automating taxonomy completion use subtasks to learn subtask results, ignoring the effects of subtask on the final prediction. |
| Approach: | They propose a multi-task automatic taxonomy completion method that attaches emerging concepts to an appropriate pair of hypernym and hyponym in existing taxonomies. |
| Outcome: | The proposed method improves on three datasets and improves inference efficiency. |
An Attribution Method for Siamese Encoders (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the success of Siamese encoder models, little is known about the aspects of inputs they base their decisions on. |
| Approach: | They propose a local attribution method for Siamese encoders by generalizing the principle of integrated gradients to models with multiple inputs. |
| Outcome: | The proposed method can be reduced to a token–token matrix and account for the model’s full computation graph and is guaranteed to converge to the actual prediction. |
Global Voices, Local Biases: Socio-Cultural Prejudices across Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on human biases are heavily skewed towards Western and European languages . despite growing interest in language models, there are several shortcomings in the literature . |
| Approach: | They scale the Word Embedding Association Test to 24 languages and add culturally relevant information for each language. |
| Outcome: | The proposed language models can reflect and often amplify the effects of bias across linguistic, cultural, and societal borders. |
Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogue systems can generate informative responses based on dialogue history and external knowledge source. |
| Approach: | They conduct a thorough experiment to determine the optimal knowledge form, mutual effects between knowl- edge and model selection, and the few-shot performance of knowledge. |
| Outcome: | The proposed method combines knowledge-grounded dialogue with human-generated dialogues to generate informative and meaningful responses. |
Are Compressed Language Models Less Subgroup Robust? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce model size and latency while retaining overall performance are not known about their impact on subgroup robustness. |
| Approach: | They investigate the effects of model compression on subgroup robustness of BERT language models. |
| Outcome: | The proposed compression methods do not worsen the performance on minority subgroups. |
Length Does Matter: Summary Length can Bias Summarization Metrics (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization metrics favor shorter or longer summaries, but evaluations of these metrics are flawed. |
| Approach: | They propose a Bayesian normalization technique that effectively diminishes this bias. |
| Outcome: | The proposed method significantly improves the concordance between human annotators and most metrics in terms of summary coherence. |
NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal Logic (TL) can be used to specify complex high-level specifications for systems in many engineering domains. |
| Approach: | They propose a framework for translation between NL and TL using Large Language Models . they use a dataset to create a model with 23K NL-TL pairs and human annotation . |
| Outcome: | The proposed framework achieves higher accuracy (> 95%) using only 10% training data compared with baseline model. |
Reformulating NLP tasks to Capture Longitudinal Manifestation of Language Disorders in People with Dementia. (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dementia is associated with language disorders which impede communication. |
| Approach: | They propose to use a pre-trained language model to automatically learn linguistic disorder patterns by forcing it to focus on reformulated natural language processing (NLP) tasks and associated linguistic patterns. |
| Outcome: | The proposed communication marker outperforms existing linguistic approaches and shows external validity via significant correlation with clinical markers of behaviour. |
Elevating Code-mixed Text Handling through Auditory Information of Words (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current language models focus on the semantic representation of words and ignore the auditory phonetic features. |
| Approach: | They propose an approach to create language models for handling code-mixed textual data using auditory phonetic features from SOUNDEX using auditorian information. |
| Outcome: | The proposed approach improves robustness against adversarial attacks on code-mixed classification tasks and improves classification results over baselines. |
Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | sarcasm detection depends on content spoken, tonality, facial expressions, context, and personal traits like language proficiency and cognitive capabilities. |
| Approach: | They propose to use synthetic gaze data to improve sarcasm detection in conversational context . they collect gaze features for 20% of data instances and use them to predict gaze features . |
| Outcome: | The proposed model improves performance on a conversational dataset using gaze features . it achieves a gain of 6.6% points on the complete dataset with only predicted gaze features. |
Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn medical vision-language representations by contrasting images with entire reports are not effective. |
| Approach: | They propose a phenotype-driven medical vision-language representation learning framework to bridge the gap between visual and textual modalities for improved text-oriented generation. |
| Outcome: | The proposed framework bridges the gap between visual and textual modalities for improved radiology report generation. |
ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) performance has improved with the advent of denoising Diffusion Probabilistic Models . however, perceived quality of audio depends on content, pitch, rhythm, and energy . |
| Approach: | They propose a visual TTS model with scalable diffusion transformers that complement phoneme sequences with visual information to generate high-perceived audio. |
| Outcome: | The proposed model outperforms existing models regardless of visibility of the scene . it can generate high-perceived audio, opening up new avenues for AR and VR applications . |
Consistency Analysis of ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 have been reported to be more reliable and trustworthy, provided they behave similarly to humans. |
| Approach: | They propose to compare ChatGPT and GPT-4 in terms of logically consistent behaviour and the properties of negation, symmetric, and transitive consistency. |
| Outcome: | The proposed models show that they can be more reliable and trustworthy provided they behave similarly to humans. |
Do Differences in Values Influence Disagreements in Online Discussions? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Disagreement is an important aspect of online discussions since it can drive novel ideas, incentivize evaluation of the proposed ideas, and avoid echo chambers. |
| Approach: | They propose to use human-annotated agreement labels to estimate personal values and to include value information in agreement prediction to improve performance. |
| Outcome: | The proposed models show that dissimilarity of value profiles correlates with disagreement in specific cases and that including value information in agreement prediction improves performance. |
Automated Fact-Checking in Dialogue: Are Specialized Models Needed? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that typical fact-checking models struggle with claims made in conversation. |
| Approach: | They propose to fine-tune models for dialogue on conversational data to improve performance on typical fact-checking. |
| Outcome: | The proposed models perform better on stand-alone claims than state-of-the-art models for dialogue while maintaining their performance on standalone claim. |
A Digital Language Coherence Marker for Monitoring Dementia (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that dementia is associated with thought disorders relating to inability to produce coherent communication. |
| Approach: | They propose to capture language coherence as a human-interpretable digital marker for monitoring cognitive changes in people with dementia. |
| Outcome: | The proposed model shows a significant difference between people with mild cognitive impairment, those with Alzheimer’s Disease and healthy controls and high association with clinical bio-markers. |
Detecting Spoilers in Movie Reviews with External Movie Knowledge and User Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on the textual content of the review, while spoiler detection requires putting the review into the context of facts and knowledge regarding movies. |
| Approach: | They propose a network-based spoiler detection model that takes into account external knowledge about movies and user activities on movie review platforms. |
| Outcome: | The proposed model takes into account external knowledge about movies and user activities on movie review platforms while incorporating user networks. |
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing graph-based methods fail to depict global contextual features and local diverse unimodal features in a dialogue. |
| Approach: | They propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition using a multimodal fusion mechanism and a graph contrastative learning framework. |
| Outcome: | The proposed method improves multimodal emotion recognition on unbalanced and small-scale emotional datasets. |
HyperRank: Hyperbolic Ranking Model for Unsupervised Keyphrase Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing unsupervised keyphrase extraction models overlook latent hierarchical structures when extracting keyphrases. |
| Approach: | They propose a new ranking model that models global and local contexts to estimate the importance of each candidate keyphrase within the hyperbolic space. |
| Outcome: | The proposed model outperforms state-of-the-art models in keyphrase extraction tasks. |
Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a subject-verb agreement in the presence of an attractor noun is a complex behavior . formal linguistic theories postulate the existence of an underlying structure that governs language processing . |
| Approach: | They hypothesize that the attractor-verb agreement may be a factor in grammatical decision-making . they hypothesized that classical models of attraction might suffice to explain this phenomenon . |
| Outcome: | The proposed model improves on humans and artificial neural networks while keeping the length of the sentence equal. |
Federated Meta-Learning for Emotion and Sentiment Aware Multi-modal Complaint Identification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on complaint identification are limited to text. |
| Approach: | They propose a meta-learning-based multi-modal multi-task framework for identifying complaints using emotion recognition and sentiment analysis as auxiliary tasks. |
| Outcome: | The proposed framework outperforms baselines and state-of-the-art approaches in centralized and federated meta-learning settings. |
Semantic Similarity Models for Depression Severity Estimation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Public health systems have limited capacity for case detection and diagnosis due to the widespread use of social media. |
| Approach: | They propose to use social media content to generate semantic rankings for depressive symptoms and severity levels and use them to predict symptoms severity. |
| Outcome: | The proposed pipeline improves on two Reddit-based benchmarks and shows that it is more efficient than state-of-the-art in terms of measuring depression level. |
Hop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on supervision for both answers and rationales, but they have limited capacities in modeling interactions between sentences, let alone reasoning across multiple documents. |
| Approach: | They propose a principled, probabilistic approach for training explainable multi-hop question answering systems without rationale supervision. |
| Outcome: | The proposed method is more accurate at selecting rationales than previous methods while maintaining similar accuracy in predicting answers. |
To Split or Not to Split: Composing Compounds in Contextual Vector Spaces (2023.emnlp-main)
Copied to clipboard
| Challenge: | Contextual word embedding models rely on sub-word tokenization to represent single orthographic words but are often suboptimal in under-resourced contexts. |
| Approach: | They propose to use a masked language modelling task to evaluate the model's performance . they use re-trained tokenizers to pre-split compounds into constituents . |
| Outcome: | The proposed models improve on the masked language modelling task and compositionality prediction by pre-splitting compounds into constituents. |
ToolWriter: Question Specific Tool Synthesis for Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Tabular question answering (TQA) requires joint reasoning of natural language with large amounts of semi-structured data. |
| Approach: | They propose to use query-specific programs to generate query-based tools to simplify large tables and detect when to apply them to transform tables. |
| Outcome: | The proposed tool improves state-of-the-art on two tabular question-answering datasets. |
Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate SQL from natural language are still making many mistakes . a new interaction mechanism allows users to edit a step-by-step explanation of a query to fix errors. |
| Approach: | They propose a mechanism that allows users to edit a step-by-step explanation of a query to fix errors. |
| Outcome: | The proposed approach can achieve better performance than multiple SOTA approaches on multiple datasets and 24 participants. |
CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent proposed methods fail to consider the linguistic structure of texts and lack the ability to handle the low-resource problem. |
| Approach: | They propose a coherence-based contrastive learning model named CoCo to detect MGTs under low-resource scenario. |
| Outcome: | The proposed model outperforms state-of-the-art methods on two datasets and two self-constructed datasets. |
AnyTOD: A Programmable Task-Oriented Dialog System (2023.emnlp-main)
Copied to clipboard
Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, Yonghui Wu
| Challenge: | a neuro-symbolic approach allows zero-shot adaptation to unseen tasks and domains . a neural LM keeps track of events that occur during a conversation and a symbolic program implements dialog policy is executed to recommend actions. |
| Approach: | They propose an end-to-end, zero-shot task-oriented dialog system . it is designed to adapt to unseen tasks or domains without prior training . |
| Outcome: | The proposed system can be programmed to adapt to unseen tasks without training . it reduces data collection and training requirements for enabling new TOD 1 16189 tasks . |
Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization benchmarks overlap in time with pre-training corpora and fine-tuning datasets. |
| Approach: | They propose a temporal generalization benchmark that contains data samples from 2010 to 2022 to understand the temporal ability of abstractive summarization models. |
| Outcome: | The proposed benchmark analyzes data samples from 2010 to 2022 to understand the temporal generalization ability of abstractive summarization models. |
Zero-Shot Multi-Label Topic Inference with Sentence Encoders and LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we focus on Zero-shot approaches for inferring topics from documents where both the document and topics were never seen by a model previously. |
| Approach: | They propose to use Sentence Encoders and Large Language Models to perform a "definition-wild zero-shot topic inference" where users define or provide topics of interest in real-time. |
| Outcome: | The proposed methods outperform ChatGPT-3.5 and PaLM and Sentence-BERT on the definition-wild zero-shot topic inference task on seven datasets. |
TaskDiff: A Similarity Metric for Task-Oriented Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Popularity of chatGPT and Llama 2 has led to a race to build custom task-oriented conversational assistants in enterprise domains like finance and retail. |
| Approach: | They propose a conversational similarity metric that uses different dialogue components to compute similarity. |
| Outcome: | Experiments on a benchmark dataset show that the proposed metric outperforms existing approaches and is more robust than previous approaches. |
Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms are used by half of U.S. adults for everyday news consumption. |
| Approach: | They propose to analyze video headlines and whether annotators believe the headline is representative of the video’s contents. |
| Outcome: | The proposed dataset analyzes video headlines and explains why annotators view a video as misleading. |
Learning From Free-Text Human Feedback – Collect New Datasets Or Extend Existing Ones? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for learning from free-text human feedback are scarce. |
| Approach: | They manually annotate a subset of a popular dialogue dataset with error and user response types using an improved version of the Integrated Error Taxonomy and a newly proposed user response type taxonomies. |
| Outcome: | The proposed dataset provides new insights into dataset composition, error types, user response types, and the relations between them. |
Euphemistic Abuse – A New Dataset and Classification Experiments for Implicitly Abusive Language (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, only explicit abuse can be reliably detected due to the increasing amount of abusive language on the Web. |
| Approach: | They propose a crowdsourced dataset that can detect euphemistic abuse by paraphrasing simple explicit utterances. |
| Outcome: | The proposed classifier augments training data with automatically-generated GPT-3 completions. |
Exploring Distributional Shifts in Large Language Models for Code Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Since the late 2000s, researchers have been reporting poor generalization of statistical learning models to new software systems, such as GitHub Copilot, Amazon CodeWhisperer, Replit, etc. |
| Approach: | They systematically study how three large language models with code capabilities generalize to out-of-domain data. |
| Outcome: | The proposed model outperforms the existing model for code generation on multiple domains at once. |
ATHENA: Mathematical Reasoning with Thought Expansion (2023.emnlp-main)
Copied to clipboard
| Challenge: | a mathematical word problem requires a method that articulates the problem, the lens through which models view human lexical expressions. |
| Approach: | They propose a method to mimic human thought expansion mechanisms to generate thought expansions . they show ATHENA can be used to generate reasonable thoughts by selecting valid pathways . |
| Outcome: | The proposed model is compelling in variant questions even when the informativeness in training examples is restricted. |
A Benchmark for Reasoning with Spatial Prepositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Spatial reasoning is a fundamental building block of human cognition . large language models (LLMs) are not on par with advanced aspects of human cognitive domains . |
| Approach: | They propose a benchmark to assess inferential properties of statements with spatial prepositions . they use prompt engineering to test the performance of two large language models . |
| Outcome: | The proposed benchmark shows that none of the models reaches human performance. |
TIMELINE: Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articles (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing temporal relation extraction models have low inter-annotator agreement due to lack of specificity of annotation guidelines . authors propose a method for annotating all temporal relations, including long-distance ones, which automates the process . |
| Approach: | They propose a new annotation scheme that defines criteria for temporal relations to be annotated . scheme includes events even if they are not expressed as verbs, they argue . |
| Outcome: | The proposed method reduces time and manual effort on the part of annotators. |
Mitigating Over-Generation for Unsupervised Keyphrase Extraction with Heterogeneous Centrality Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing keyphrase extraction models incorrectly determine a keyphrase as a phrase but output other candidates as keyphrases because they contain the same word. |
| Approach: | They propose a new approach that detects both implicit and explicit centrality within a heterogeneous graph as the importance score of each candidate keyphrase. |
| Outcome: | The proposed approach outperforms state-of-the-art keyphrase extraction models on three benchmark datasets. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding (2023.emnlp-main)
Copied to clipboard
Steven Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dmitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, Dan Hendrycks
| Challenge: | Merger Agreement Understanding Dataset (MAUD) is an expert-annotated reading comprehension dataset based on the American Bar Association's 2021 Public Target Deal Points Study. |
| Approach: | They propose a Merger Agreement Understanding Dataset with over 39,000 examples and over 47,000 annotations. |
| Outcome: | The Merger Agreement Understanding Dataset (MAUD) is an expert-annotated reading comprehension dataset based on the American Bar Association's 2021 Public Target Deal Points Study. |
PK-ICR: Persona-Knowledge Interactive Multi-Context Retrieval for Grounded Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Identifying relevant persona or knowledge for conversational systems is difficult, but recent work has shown that it is more realistic to optimize for concrete persona. |
| Approach: | They propose a persona-knowledge dual context retrieval method that utilizes all dialogue contexts simultaneously. |
| Outcome: | The proposed method performs zero-shot top-1 knowledge retrieval and precise persona scoring. |
More Than Spoken Words: Nonverbal Message Extraction and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on extracting NMs from small-scale well-structured corpora such as movie scripts wherein NM is enclosed in parentheses by scriptwriters, which greatly decreases the difficulty of extraction. |
| Approach: | They propose to extract nonverbal messages (NMs) from written text and NMs from spoken text by using a semi-supervised learning algorithm. |
| Outcome: | The extracted NMs can generate more relevant, valid, and factually consistent NM than the purely supervised generator. |
Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Analogical reasoning is a common way to evaluate word embeddings in NLP, but it is also of interest to investigate whether or not it is able to be learned. |
| Approach: | They propose to use proportional analogies to evaluate word embeddings in NLP . they also test whether analogical reasoning is a task in itself that can be learned . |
| Outcome: | The proposed models can learn analogical reasoning even with small amounts of data. |
FAME: Flexible, Scalable Analogy Mappings Engine (2023.emnlp-main)
Copied to clipboard
| Challenge: | Most work on computational analogy relies heavily on complex, manually crafted input. |
| Approach: | They propose to use commonsense representations to identify mappings between entities and use them to interpret their output. |
| Outcome: | The proposed model outperforms human models on large analogy problems and outperfies human predictions. |
A Self-training Framework for Automated Medical Report Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Medical report generation is an important medical artificial intelligence task. |
| Approach: | They propose a framework for medical report generation that exploits unlabeled medical images and a reference-free evaluation metric. |
| Outcome: | The proposed framework performs better than previous fully-supervised models trained on entire training data. |
A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work uses pre-trained language models to reason about plans from text instructions in embodied visual environments. |
| Approach: | They propose to use pre-trained language models to reason about plan sequences from text instructions in embodied visual environments. |
| Outcome: | The proposed approach outperforms previous approaches on the ALFWorld and VirtualHome benchmarks. |
Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that multi-head attention module evolves functional specialization after multi-task training. |
| Approach: | They propose a method to quantify the degree of functional specialization in multi-head attention . they propose 'multi-task training' method to increase functional specialisation and mitigate negative information transfer . |
| Outcome: | The proposed method increases functional specialization and mitigates negative information transfer in multi-task learning without adding any parameters. |
Multilingual Previously Fact-Checked Claim Retrieval (2023.emnlp-main)
Copied to clipboard
Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, Maria Bielikova
| Challenge: | Fact-checkers are often hampered by the sheer amount of online content that needs to be fact-checked. |
| Approach: | They propose a multilingual dataset for previously fact-checked claim retrieval using social media posts and 206k fact- checks in 39 languages written by professional fact- checkers. |
| Outcome: | The proposed method improves on the previously unsupervised method and shows that a multilingual dataset has its complexities and needs to be carefully interpreted. |
ALCAP: Alignment-Augmented Music Captioner (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional approaches to music captioning ignore the intricate interplay between the two . however, a comprehensive understanding of music necessitates the integration of both these elements. |
| Approach: | They propose a method to learn multimodal alignment between audio and lyrics through contrastive learning. |
| Outcome: | The proposed method achieves new state-of-the-art on two music captioning datasets. |
Do Transformers Parse while Predicting the Masked Word? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained language models encode linguistic structures like parse trees while being trained unsupervised. |
| Approach: | They propose to train pre-trained language models to encode linguistic structures like parse trees while unsupervised. |
| Outcome: | The proposed model performs optimally for masked language modeling loss on the English PCFG. |
Composable Text Controls in Latent Space with ODEs (2023.emnlp-main)
Copied to clipboard
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, Zhiting Hu
| Challenge: | Existing approaches to composable text operations often require plug-and-play . a single LM can perform arbitrary text operation composition in the latent space . |
| Approach: | They propose an efficient approach for composable text operations in the latent space of text . they connect pretrained LMs to the laten space and adapt them to the space . |
| Outcome: | The proposed approach improves on existing methods in the latent space of text. |
P5: Plug-and-Play Persona Prompting for Personalized Response Selection (2023.emnlp-main)
Copied to clipboard
| Challenge: | a plug-and-play persona prompting system can be used to generate personalized responses for real applications . a recent study shows that dialog context alone is insufficient for personalized response selection . |
| Approach: | They propose a plug-and-play persona prompting method that can be used in real applications . they show that the method performs well in the zero-shot setting . |
| Outcome: | The proposed method performs well in the zero-shot setting, and can be fine-tuned for even better performance. |
Reader: Model-based language-instructed reinforcement learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of RL are limited and need to be re-trained for every new problem. |
| Approach: | They propose a model-based reinforcement learning approach to tackle the environment Read To Fight Monsters, a grounded policy learning problem. |
| Outcome: | The proposed approach performs better than existing model-free SOTA agents in the read to fight monsters environment and is more sample efficient than existing models. |
Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to streaming speech translation use an offline model with a wait-k policy . however, there is a mismatch problem with an offline inference model trained with complete utterances . |
| Approach: | They propose an offline streaming speech translation model with wait-k policy to support different latency requirements. |
| Outcome: | The proposed model achieves better trade-offs between translation quality and latency than baselines. |
Relation-aware Ensemble Learning for Knowledge Graph Embedding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to explore semantics of knowledge graphs have been proposed to explore these semantics in distinct ways. |
| Approach: | They propose to leverage existing methods in relation-aware manner to learn an ensemble by leveraging existing methods. |
| Outcome: | The proposed method has the same computation cost as general ensemble methods but with much better performance on benchmark datasets. |
GenEx: A Commonsense-aware Unified Generative Framework for Explainable Cyberbullying Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | a significant gap exists in understanding code-mixed languages and the need for explainability in this context. |
| Approach: | They propose to annotate posts with four labels to identify bullies in code-mixed languages . they propose to use a generative framework to reimagine the multitask problem as a text-to-text generation task. |
| Outcome: | The proposed model outperforms baseline models and state-of-the-art models on the BullyExplain dataset. |
Document-Level Machine Translation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing tasks. |
| Approach: | They examine the impact of different prompts on document-level translation quality and discourse phenomena using figures and lines, which are invisible to GPT-4. |
| Outcome: | The proposed models outperform commercial MT systems and advanced document-level MT methods on a number of benchmarks and show potential to become a new paradigm for document- level translation. |
Multilingual Simplification of Medical Texts (2023.emnlp-main)
Copied to clipboard
Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vishnesh Ramanathan, Wei Xu, Byron Wallace, Junyi Jessy Li
| Challenge: | Existing work on medical text simplification has focused on monolingual settings . important findings in medicine are typically presented in technical, jargon-laden language . text simulating models can generate viable simplified texts, but there are outstanding challenges . |
| Approach: | They propose a dataset for medical text simplification in four languages . they evaluate fine-tuned and zero-shot models across these languages based on human assessments and analyses . |
| Outcome: | The proposed dataset evaluates models in English, Spanish, French, and Farsi . it shows that the models can generate viable simplified texts, but there are challenges . |
When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews (2023.emnlp-main)
Copied to clipboard
| Challenge: | despite being widely accepted standard for validating scholarly research, peer-review process has faced criticism. |
| Approach: | They propose a task of automatically identifying contradictions among reviewers on a given article. |
| Outcome: | The proposed model detects contradictory statements from the review pairs and makes it publicly available for further investigations. |
Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation (2023.emnlp-main)
Copied to clipboard
Jiayu Lin, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei
| Challenge: | Existing work describes paragraph-level counter-argument generation task as paragraph-based . however, sentence-level generation can be quite different due to its unique constraints and brevity-focused challenges. |
| Approach: | They propose a benchmark framework for sentence-level counter-argument generation . they use an annotated debate forum dataset to generate high-quality counter-argments . |
| Outcome: | The proposed framework and evaluator are competitive in counter-argument generation tasks. |
JASMINE: Arabic GPT Models for Few-Shot Learning (2023.emnlp-main)
Copied to clipboard
El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Inciarte, Md Tawkat Islam Khondaker
| Challenge: | generative pretraining (GPT) scholarship remains acutely anglocentric, leaving serious gaps in our understanding of the whole class of autoregressive models. |
| Approach: | They propose to use Arabic autoregressive models to evaluate their performance . they use a benchmark to evaluate the models and code for experimenting with them . |
| Outcome: | JASMINE is a suite of powerful Arabic autoregressive models . it shows powerful performance intrinsically and in few-shot learning on a wide range of tasks. |
NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports (2023.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trial reports (CTRs) are indispensable for the development of personalized medicine. |
| Approach: | They propose a resource to help researchers interpret clinical trial reports . they use natural language inference to compute textual entailment . |
| Outcome: | The proposed resource is the first to cover interpretation of full clinical trial reports . it includes tasks to determine inference relation between natural language statements and CTRs . |
Addressing Linguistic Bias through a Contrastive Analysis of Academic Writing in the NLP Domain (2023.emnlp-main)
Copied to clipboard
| Challenge: | a reviewer’s opinion of the nativeness of expression in an academic paper affects the likelihood of it being accepted for publication. |
| Approach: | They conduct a statistical analysis of paper abstracts from the natural language processing domain to identify how authors from different linguistic backgrounds differ in the lexical, morphological, syntactic and cohesive aspects of their writing. |
| Outcome: | The results suggest that there is potential for linguistic bias in the domain of natural language processing. |
RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Grammatical Error Correction (GEC) systems perform well in academic benchmarks, but in practical applications they may not correct errors when users perform irrelevant modifications. |
| Approach: | They propose a benchmark to evaluate the context robustness of Grammatical Error Correction systems. |
| Outcome: | The proposed method improves the accuracy of errors corrected by human annotations. |
Detecting Propaganda Techniques in Code-Switched Social Media Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new study aims to detect propaganda in multiple languages using code-switching . social media platforms have made it easier for anyone to spread information to a wide audience . |
| Approach: | They propose to detect propaganda techniques in code-switched texts using a corpus of 1,030 texts . they propose to model multilinguality directly rather than using translation . |
| Outcome: | The proposed method combines different languages within the same text, presenting a challenge for automatic systems. |
Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in Indonesian (2023.emnlp-main)
Copied to clipboard
| Challenge: | Ambiguity is one of the challenges in natural language processing. |
| Approach: | They propose to resolve structurally ambiguous sentences into unambiguous texts in Indonesian using prosodic information. |
| Outcome: | The proposed system achieves a disambiguation accuracy of 79.6% while the proposed direct system yields an even more impressive disambiguations accuracy of 82%. |
Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Gender bias is a significant issue in machine translation, but most studies focus on debiasing bilingual models without consideration for multilingual systems. |
| Approach: | They propose a method which debiases bilingual models for unambiguous cases where there is a single correct translation. |
| Outcome: | The proposed method improves gender accuracy by a wide margin without hampering translation performance. |
Code-Switching Metrics Using Intonation Units (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures of code-switching (CS) complexity are word-based, meaning any word is equally likely to switch between any two words. |
| Approach: | They adapt two NLP metrics, multilinguality and CS probability, and put forward Intonation Units (IUs) as basic tokens for transcribed bilingual speech. |
| Outcome: | The proposed measures account for prosodic and prosodic constraints on CS in bilingual speech. |