Papers with decoder
AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate (2021.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive neural machine translation models suffer from the multi-modality problem . aligNART leverages full alignment information to explicitly reduce the modality of the target distribution . |
| Approach: | They propose an alignment decomposition method which explicitly reduces the modality of the target distribution. |
| Outcome: | The proposed model outperforms previous models that focus on modality reduction on two translation tasks. |
Compilable Neural Code Generation with Compiler Feedback (2022.findings-acl)
Copied to clipboard
Xin Wang, Yasheng Wang, Yao Wan, Fei Mi, Yitong Li, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, Qun Liu
| Challenge: | Existing deep-learning approaches model code generation as text generation, but few of them account for compilability of the generated programs. |
| Approach: | They propose a three-stage pipeline utilizing compiler feedback for compilable code generation to improve compilability. |
| Outcome: | The proposed pipeline improves compilability of generated programs by combining compiler feedback, language model fine-tuning, and compilable discrimination. |
Learning to Order Graph Elements with Application to Multilingual Surface Realization (D19-63)
Copied to clipboard
| Challenge: | Recent advances in deep learning have shown promises in solving combinatorial optimization problems, such as sorting variable-sized sequences. |
| Approach: | They propose an encoder-decoder framework that learns the representation for each element and predicts the ordering of each local neighborhood of the graph in turn. |
| Outcome: | The proposed framework outperforms previous frameworks on multilingual surface realization tasks while outperforming those below by a large margin. |
Semantics-aware Attention Improves Neural Machine Translation (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing attempts to integrate semantic structures into NMT Transformers have failed . |
| Approach: | They propose two parameter-free methods for injecting semantic information into Transformers, using a Scene-Aware Self-Attention (SASA) head and a Scenario-Award Cross-Action (SACrA) head. |
| Outcome: | The proposed methods improve on the vanilla Transformer and syntax-aware models for four language pairs and show an additional gain when using both semantic and syntactic structures in some language pairs. |
Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (IE) and Information Extraction (IE), both focus on extracting the main information from natural language texts. |
| Approach: | They propose an AMR-guided framework for joint information extraction using a pre-trained AMR parser. |
| Outcome: | The proposed framework achieves state-of-the-art on all IE subtasks. |
Learning to Generate Question by Asking Question: A Primal-Dual Approach with Uncommon Word Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing automatic question generation methods focus on encoding passage and answer to generate question. |
| Approach: | They propose an automatic question generation approach which integrates question generation with its dual problem, question answering, into a unified primal-dual framework. |
| Outcome: | The proposed approach outperforms existing methods on SQuAD and HotpotQA benchmarks. |
Towards Modeling Role-Aware Centrality for Dialogue Summarization (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing methods for dialogue summarization consider roles separately where interactions among different roles are not fully explored. |
| Approach: | They propose a novel role-aware centrality model to capture role interactions by involving role prompts to control what kind of summary to generate. |
| Outcome: | The proposed model achieves state-of-the-art on two public benchmark datasets, CSDS and MC. |
Improving Character-Based Decoding Using Target-Side Morphological Information for Neural Machine Translation (N18-1)
Copied to clipboard
| Challenge: | Morphologically complex words (MCWs) are multi-layer structures consisting of different subunits, each of which carries semantic information and has a specific syntactic role. |
| Approach: | They propose an extension to the state-of-the-art model which works at the character level and boosts the decoder with target-side morphological information. |
| Outcome: | The proposed model improves on the state-of-the-art model and can be extended to include morphologically complex words (MCWs) in three languages. |
Automatic Dialogue Generation with Expressed Emotions (N18-2)
Copied to clipboard
| Challenge: | a growing interest in neural dialogue generation systems is focusing on generating human-like responses based on past utterances . despite efforts, few consider putting restrictions on the response itself . authors present three models that concatenate the desired emotion with the source input . |
| Approach: | They propose three models that concatenate the desired emotion with the source input or push the emotion in the decoder. |
| Outcome: | The proposed model is more efficient than the previous models, but it lacks the emotion vector. |
Unsupervised Opinion Summarization Using Approximate Geodesics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for opinion summarization are limited due to the scarcity of data. |
| Approach: | They propose a system to perform unsupervised extractive opinion summarization using a dictionary-based representation learning model that generates topical representations of texts. |
| Outcome: | The proposed system achieves strong performance on three opinion summarization datasets. |
Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization (2021.acl-long)
Copied to clipboard
| Challenge: | Text style transfer aims to alter the style of a sentence while preserving its content. |
| Approach: | They propose to remove style information at token level and fuse it to style representations using conditional layer normalization. |
| Outcome: | The proposed model outperforms the state-of-the-art models in terms of content preservation and fluency. |
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Encoder-decoder transformer models suffer from high inference latency due to auto-regressive decoding . Typically, the decoder takes up most of the latency because of the auto-decoding - a problem that is not solved by the current model. |
| Approach: | They propose an approach to perform Dynamic Early Exit on Decoder to reduce inference latency by 20%-74% by using a multi-exit encoder-decoder transformer model trained with deep supervision. |
| Outcome: | The proposed model reduces inference latency by 20%-74% with comparable or even higher accuracy compared to baseline models. |
Natural Language Generation by Hierarchical Decoding with Linguistic Patterns (N18-2)
Copied to clipboard
| Challenge: | a common and mostly adopted method is the rule-based (or template-based) method for natural language generation. |
| Approach: | They propose a hierarchical decoding NLG model based on linguistic patterns in different levels. |
| Outcome: | The proposed method outperforms the traditional one with a smaller model size. |
simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to image captioning combine visual and semantic attention to generate a detailed and comprehensive caption. |
| Approach: | They propose a stepwise image-topic merging network that integrates visual and semantic attentions to generate a detailed caption. |
| Outcome: | The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performance. |
Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation (D19-1)
Copied to clipboard
| Challenge: | Current approaches to visual dialog learning involve an end-to-end framework that maps the multi-modal context to a deep vector and in order to decode a natural dialog response. |
| Approach: | They propose a framework that trains a RL policy for image guessing and a seq2seq model to improve dialog quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on a guessWhich task . it can be applied to a wide range of tasks including assisting blind people . |
Natural Answer Generation with Heterogeneous Memory (N18-1)
Copied to clipboard
| Challenge: | Recent work on memory augmented encoder-decoder frameworks has shown promising progress for natural language generation tasks. |
| Approach: | They propose a memory-augmented encoder-decoder framework that takes care of memory contents from different sources to explicitly avoid repetition. |
| Outcome: | The proposed approach can produce readable and meaningful answer sentences while maintaining high coverage for given answer information. |
Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation (P19-2)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) uses sequence to sequence architectures, but requires a huge amount of parallel data. |
| Approach: | They use Elastic Weight Consolidation to regularize weights of two language models . they then fine-tune the model on parallel data to avoid forgetting the original task . |
| Outcome: | The proposed method achieves BLEU scores similar to the previous work, but is slower and requires less training data. |
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities (2023.acl-demo)
Copied to clipboard
Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, Han Guo, Weigang Gou, Taiqiang Wu, Tao Zhu, Wenhang Shi, Chen Chen, Shan Huang, Sihong Chen, Liqun Liu, Feifei Li, Xiaoshuai Chen, Xingwu Sun, Zhanhui Kang, Xiaoyong Du, Linlin Shen, Kimmo Yan
| Challenge: | Several pre-training models of different modalities are showing a rising trend of homogeneity in their model structures. |
| Approach: | They propose a toolkit that supports pre-training models of different modalities. |
| Outcome: | The proposed toolkit can match the performance of the original implementations on text, vision, and audio benchmarks. |
From Partial to Strictly Incremental Constituent Parsing (2024.eacl-short)
Copied to clipboard
| Challenge: | Incremental NLP aims to learn and adapt partial representations as information unfolds, but studies on incremental approaches have focused on non-incremental approaches. |
| Approach: | They build incremental constituent parsers that adhere to a strong definition of incrementality across languages and compare them against non-incremental and partially incremental models. |
| Outcome: | The proposed models are compared against non-incremental and partially incremental models. |
Neural Argument Generation Augmented with Externally Retrieved Evidence (P18-1)
Copied to clipboard
| Challenge: | Existing methods for generating arguments are limited to retrieval-based methods. |
| Approach: | They propose an encoder-decoder-based argument generation model enriched with externally retrieved evidence from Wikipedia. |
| Outcome: | The proposed model generates arguments with more topic-relevant content than current models based on automatic evaluation and human assessments on a large-scale dataset from reddit. |
FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm (D19-55)
Copied to clipboard
| Challenge: | Existing spell checkers for Chinese are based on denoising autoencoder and decoder paradigms that require a small amount of data to be effective. |
| Approach: | They propose a Chinese spell checker based on a new paradigm which consists of a denoising autoencoder and a decoder. |
| Outcome: | The proposed spell checker is faster, more Adaptable to simplified and traditional Chinese texts and has a much simpler structure to be as much Powerful in error detection and correction. |
Exploring Interpretability in Event Extraction: Multitask Learning of a Neural Event Classifier and an Explanation Decoder (2020.acl-srw)
Copied to clipboard
| Challenge: | EE is a key requirement for machine learning in many domains, e.g., legal, medical, finance. |
| Approach: | They propose an interpretable approach for event extraction that jointly trains a classifier and a rule decoder for event processing. |
| Outcome: | The proposed approach can be used for semi-supervised learning and its performance improves when trained on automatically-labeled data generated by a rule-based system. |
Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) often fails in one-to-many translation, e.g., in the translation of multi-word expressions, compounds, and collocations. |
| Approach: | They propose a phrase-based NMT model that generates embeddings of words or phrases. |
| Outcome: | The proposed model performs on par with state-of-the-art phrase-based NMT. |
Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations (D19-3)
Copied to clipboard
| Challenge: | Currently, the main alternatives to deal with sequences are Recurrent Neural Networks (RNN) architectures and the Transformer. |
| Approach: | They propose a web-based tool that visualizes the sentence and token representations of RNNs and Transformer architectures at the sentence level. |
| Outcome: | The proposed visualization tool analyses gender inequalities in contextual word embeddings and the common language representation in a multilingual machine translation system. |
Global Encoding for Abstractive Summarization (P18-2)
Copied to clipboard
| Challenge: | Existing models for abstractive summarization suffer from repetition and semantic irrelevance. |
| Approach: | They propose a global encoding framework which controls the information flow from the encoder to the decoder based on the global information of the source context. |
| Outcome: | The proposed model outperforms baseline models on the LCSTS and English Gigaword and can generate summary of higher quality and reduce repetition. |
PG-GSQL: Pointer-Generator Network with Guide Decoding for Cross-Domain Context-Dependent Text-to-SQL Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text-to-SQL generation depend on interaction history and current utterances. |
| Approach: | They propose an encoder-decoder model based on interaction-level encoder to capture historical information of SQL query and reuse the previous SQL query tokens. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on the SParC benchmark . it achieves 34.0% question matching accuracy and 19.0% interaction matching accuracy . |
RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for dense retrieval are not effective, but there are still challenges. |
| Approach: | They propose a retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE) where the sentence embedding is generated from the encoder’s masked input and the original sentence is recovered based upon the sentence embedded and decoded input via mangled language modeling. |
| Outcome: | The proposed model significantly improves the SOTA performance on a wide range of NLP benchmarks, like BEIR and MS MARCO. |
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | masked language models have been used for natural language processing tasks but few studies have adopted it in the sequence-to-sequence models. |
| Approach: | They propose to combine encoder and decoder to train a masked sequence-to-sequence model . they propose to train the encoder more rigorously by masking the encoded input . |
| Outcome: | The proposed model achieves 27.69/32.24 BLEU scores on English-German/German-English tasks with 5+ times speed up compared with an autoregressive model. |
Checkpoint Reranking: An Approach to Select Better Hypothesis for Neural Machine Translation Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has produced excellent results in the field of machine translation due to generation of high-quality translations for different language pairs. |
| Approach: | They propose a method of re-ranking the outputs of Neural Machine Translation systems by focusing on the decoder's ability to generate distinct tokens and without the use of any language model or data. |
| Outcome: | The proposed method achieves translation improvement up to +0.16 BLEU points over baseline. |
Contextual Parameter Generation for Universal Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multilingual neural machine translation lack language-specific parameterization. |
| Approach: | They propose a modification to existing neural machine translation models that allows for language specific parameterization and domain adaptation. |
| Outcome: | The proposed model surpasses state-of-the-art for both the IWSLT-15 and IWSTL-17 datasets and can perform zero-shot translation. |
Is Encoder-Decoder Redundant for Neural Machine Translation? (2022.aacl-main)
Copied to clipboard
| Challenge: | Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. |
| Approach: | They propose to combine bilingual and multilingual translations to train a language model to do translation. |
| Outcome: | The proposed approach performs on par with the baseline encoder-decoder Transformer . the proposed approach is compared with the translation model in the target language . |
A Family of Pretrained Transformer Language Models for Russian (2024.lrec-main)
Copied to clipboard
Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, Alena Fenogenova
| Challenge: | Developing Transformer language models for the Russian language has received little attention . most of these LMs are developed for English, which imposes substantial constraints on the potential of the language technologies. |
| Approach: | They propose to release 13 Russian Transformer language models that span three languages . they aim to broaden the scope of NLP research directions and develop industrial solutions for the Russian language. |
| Outcome: | The proposed models are based on Russian language datasets and benchmarks. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
CGF: Constrained Generation Framework for Query Rewriting in Conversational AI (2022.emnlp-industry)
Copied to clipboard
Jie Hao, Yang Liu, Xing Fan, Saurabh Gupta, Saleh Soltan, Rakesh Chada, Pradeep Natarajan, Chenlei Guo, Gokhan Tur
| Challenge: | Large-scale conversational AI agents such as Alexa, Siri and Google Assistant help millions of users to perform a lot of tasks. |
| Approach: | They propose a Constrained Generation Framework for query rewriting at global and personalized levels. |
| Outcome: | The proposed framework significantly boosts the query rewriting performance. |
A Bi-Model Based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling (N18-2)
Copied to clipboard
| Challenge: | Intent detection and slot filling are two main tasks for building a spoken language understanding system. |
| Approach: | They propose to use a sequence to sequence model to generate both intent and slot filling tasks together to perform the two tasks jointly. |
| Outcome: | The proposed model achieves 0.5% intent accuracy improvement and 0.9 % slot filling improvement on the ATIS benchmark data. |
CARE: Causality Reasoning for Empathetic Responses by Conditional Graph Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to empathetic response generation only consider causalities between the user’s emotion and the user's experiences and neglect interdependence among causalities and reason them independently. |
| Approach: | They propose to use a conditional variable Graph Auto-Encoder to reason all plausible causalities interdependently and simultaneously given the user’s emotion, dialogue history, and future dialogue content. |
| Outcome: | The proposed method achieves state-of-the-art in a real-world situation. |
UR2N: Unified Retriever and ReraNker (2025.coling-industry)
Copied to clipboard
| Challenge: | XTR-style retrieval on top of trained Mono-T5 reranker is suboptimal for two-stage retrieval, arguing that it is sub-optimal. |
| Approach: | They propose a unified encoder-decoder architecture with a novel training regimen which enables the encoder representation to be used for retrieval and the decoder for re-ranking within a single unified model. |
| Outcome: | The proposed architecture outperforms ColBERT, XTR, and even serves as a superior reranker compared to the Mono-T5 re-ranker. |
Efficient Dialogue State Tracking by Selectively Overwriting Memory (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work in dialogue state tracking (DST) is inefficient in that it predicts dialogue state at every turn from scratch. |
| Approach: | They propose a method that selectively overwrites memory for dialogue state tracking by predicting dialogue state on each memory slot and overwriting it with new values. |
| Outcome: | The proposed model achieves state-of-the-art joint goal accuracy with 51.72% in MultiWOZ 2.0 and 53.01% in MultiWoz 2.1 in an open vocabulary-based DST setting. |
Select and Attend: Towards Controllable Content Selection in Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent neural network models conflate content selection and surface realization into a black-box architecture, resulting in content to be described in text cannot be explicitly controlled. |
| Approach: | They propose to decouple content selection from the decoder to allow finer-grained control over the generation. |
| Outcome: | The proposed model can be trained end-to-end without human annotations and achieves promising results in data-totext and headline generation tasks. |
Efficient Sample-Specific Encoder Perturbations (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing approaches to training encoder-decoder systems often depend on teacher-forcing with the likelihood criteria, e.g. next token prediction of the reference sequence. |
| Approach: | They propose an inference-efficient way to modify the behaviour of an encoder-decoder system according to a specific attribute of interest by using a small proxy network. |
| Outcome: | The proposed framework improves the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. |
Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)
Copied to clipboard
| Challenge: | Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units. |
| Approach: | They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors. |
| Outcome: | The proposed model eliminates the decoder and reduces the search space for splitting points. |
Spoken Language Understanding for Task-oriented Dialogue Systems with Augmented Memory Networks (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research shows promising results by jointly learning of slot filling and intent detection tasks. |
| Approach: | They propose a way to combine slot filling and slot filler learning to achieve state-of-the-art results. |
| Outcome: | The proposed model outperforms existing methods on benchmark datasets and ATIS datasets. |
Memory-Based Semantic Parsing (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for context-dependent semantic parsing focus on parse utterances in isolation . a decoder cannot copy or modify the parser from the previous utterrance . |
| Approach: | They propose to represent contextual information using an external memory by maintaining the cumulative meaning of sequential user utterances. |
| Outcome: | The proposed model can better process context-dependent information without task-specific decoders. |
Latent Part-of-Speech Sequences for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing methods for learning target side syntactic structure are greedy and only allow them to explore a limited portion of the latent space. |
| Approach: | They propose a new latent variable model, LaSyn, that captures the co-dependence between syntax and semantics while allowing for effective inference over the latent space. |
| Outcome: | The proposed model captures the co-dependence between syntax and semantics while allowing for efficient inference over the latent space. |
A Self-supervised Joint Training Framework for Document Reranking (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models have been successfully applied to a wide range of tasks . however, the pretraining tasks were based on the context of documents . |
| Approach: | They propose a self-supervised joint training framework with a method called Masked Query Prediction to establish semantic relations between given queries and positive documents. |
| Outcome: | The proposed framework outperforms existing models on document reranking tasks without further pre-training . it uses a self-supervised method to establish semantic relations between given queries and positive documents. |
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing translation pipelines require additional cascade components to achieve speech-to-speech translation. |
| Approach: | They propose a non-autoregressive generation framework for simultaneous speech translation . it integrates both text-to-text and speech-tospeech tasks into a unified framework . |
| Outcome: | The proposed framework outperforms state-of-the-art models in speech-to-text and speech- to-speech tasks. |
Planning with Learned Entity Prompts for Abstractive Summarization (2021.tacl-1)
Copied to clipboard
| Challenge: | a simple but flexible mechanism is used to ground the generation of abstractive summaries. |
| Approach: | They propose a mechanism to learn an intermediate plan to ground the generation of abstractive summaries. |
| Outcome: | The proposed model outperforms state-of-the-art methods for faithfulness on CNN and BillSum. |
Online Back-Parsing for AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) is a semantic graph representation that abstracts meaning away from a sentence. |
| Approach: | They propose a decoder that back predicts projected AMR graphs on target sentences . their results show superiority over previous state-of-the-art decoded graph Transformer . |
| Outcome: | The proposed model outperforms the state-of-the-art model on two AMR benchmarks. |
Hierarchical Pointer Net Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to parsing are greedy transition-based and globally optimized . however, the decision-making process is based on local information, causing error propagation to subsequent steps. |
| Approach: | They propose hierarchical pointer network parsers and apply them to dependency and sentence-level discourse parsing tasks. |
| Outcome: | The proposed method outperforms existing methods and sets new state-of-the-art methods on benchmark datasets. |
Aspect and Sentiment Aware Abstractive Review Summarization (C18-1)
Copied to clipboard
| Challenge: | Abstractive summarization is a task that generates short and concise summaries of user generated reviews. |
| Approach: | They propose an interactive attention mechanism to learn the representations of context and aspect words within reviews, acted as an encoder. |
| Outcome: | The proposed model achieves impressive results compared to other strong competitors on a real-life dataset. |
Improving Abstractive Dialogue Summarization with Hierarchical Pretraining and Topic Segment (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for meeting summary have limited the ability to deal with long-term dependency. |
| Approach: | They propose a hierarchical transformer encoder-decoder network with multi-task pre-training to capture key sentences at word level and generate them at word-level. |
| Outcome: | The proposed model is superior to the previous methods in meeting summary datasets AMI and ICSI. |
Extract and Attend: Improving Entity Translation in Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve entity translation in Neural machine translation still suffer from inaccurate translation of entities due to the lack of entity training instances. |
| Approach: | They propose an extract-and-tend approach to enhance entity translation in NMT by extracting entities from a dictionary and attending to them with a prefix. |
| Outcome: | Experiments on En-Zh and En-Ru show that the proposed approach improves translation accuracy and translation quality. |
Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to syntactically controlled paraphrase generation require annotated paraphrase pairs for training and are costly to extend to new domains. |
| Approach: | They propose to leverage Abstract Meaning Representations (AMR) to improve the performance of unsupervised syntactically controlled paraphrase generation. |
| Outcome: | The proposed model generates more accurate syntactically controlled paraphrases, both quantitatively and qualitatively, compared to the existing unsupervised approaches. |
StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing automatic story evaluation methods place a premium on story lexical level coherence, deviating from human preference. |
| Approach: | They propose a novel Story Evaluation method that mimics human preference when judging a story . the model is based on a well-annotated dataset and a longformer-encoder-decoder . |
| Outcome: | The proposed method is applicable to machine-generated and human-written stories. |
Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention (N18-1)
Copied to clipboard
| Challenge: | Neural machine translation models learn to map from source language sentences to target language sentences via continuous-space intermediate representations. |
| Approach: | They propose an encoder with character attention which augments the (sub)word-level representation with character-level information and a decoder with multiple attentions that enable the representations from different levels of granularity to control the translation cooperatively. |
| Outcome: | The proposed model outperforms the standard word-based model, subword-based models, and strong character-based ones on translation tasks. |
Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for numerical reasoning are not flexible enough to handle diverse expressions. |
| Approach: | They propose a Relational Graph enhanced Hybrid table-text Numerical reasoning model with Tree decoder which captures relationship between numerical value, table schema, and text information on the encoder side. |
| Outcome: | The proposed model outperforms the baseline model and achieves state-of-the-art results on the publicly available tabletext hybrid QA benchmark. |
Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction (2021.acl-short)
Copied to clipboard
| Challenge: | Recent work focuses on question answering based on machine reading comprehension . current approaches treat QA as extracting a consecutive piece of text to a given question. |
| Approach: | They propose a generative QA model that incorporates an extractive mechanism into a model. |
| Outcome: | The proposed model improves quality and semantic accuracy over baseline models. |
Exemplar Encoder-Decoder for Neural Conversation Generation (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to generate conversational systems suffer from lack of diversity in responses and generation of short, repetitive and uninteresting responses. |
| Approach: | They propose a novel conversation model that uses similar examples from training data to generate responses. |
| Outcome: | The proposed model outperforms state-of-the-art sequence to sequence learning on several evaluation metrics on two large data sets. |
Fast and Accurate Reordering with ITG Transition RNN (C18-1)
Copied to clipboard
| Challenge: | Attention-based sequence-to-sequence neural networks learn to jointly align and translate. |
| Approach: | They propose to use a reordering RNN that shares the input encoder with the decoder to decouple re-ordering from translation. |
| Outcome: | The proposed model can achieve superior reordering accuracy without feature engineering and is 2.5x faster in decoding. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
Self-Attentive Residual Decoder for Neural Machine Translation (N18-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence networks with attention have been used for machine translation . however, the target-side context is limited and the model lacks the ability to capture non-syntactic dependencies among words. |
| Approach: | They propose a sequence-to-sequence network with attention that captures contextual information at each time-step prediction through an attention mechanism. |
| Outcome: | The proposed model outperforms a neural MT baseline and memory and self-attention network on three language pairs. |
Self-Attention Guided Copy Mechanism for Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Abstractive summarization models have been widely used to extract words from source into summary, but how to ensure that important words in source are copied remains a challenge. |
| Approach: | They propose a Transformer-based model to enhance copy mechanism by identifying the importance of each source word based on the degree centrality. |
| Outcome: | The proposed model outperforms baseline methods on CNN/Daily Mail and Gigaword datasets. |
Transfer Learning for a Letter-Ngrams to Word Decoder in the Context of Historical Handwriting Recognition with Scarce Resources (C18-1)
Copied to clipboard
| Challenge: | Lack of data can be an issue when beginning a new study on historical handwritten documents. |
| Approach: | They propose a character-based decoder for historical handwriting recognition on Italian Comedy Registers . they use untapped data from domains, periods, languages to obtain efficient system . |
| Outcome: | The character-based decoder can be used to learn historical handwriting on Italian Comedy registers . the results show that the system can be obtained by carefully selecting the datasets used . |
Tagging-Assisted Generation Model with Encoder and Decoder Supervision for Aspect Sentiment Triplet Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in the ASTE task have been driven by Natural Language Generation-based approaches, but most NLG methods overlook the supervision of the encoder-decoder hidden representations and fail to fully utilize the semantic information provided by the labels. |
| Approach: | They propose a tagging-assisted generation model with encoder and decoder supervision that enhances the supervision of the encoder-decoder through multiple-perspective tabbing assistance and label semantic representations. |
| Outcome: | The proposed model enhances the supervision of the encoder and decoder through multiple-perspective tagging assistance and label semantic representations. |
HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints (2021.emnlp-main)
Copied to clipboard
| Challenge: | HintedBT provides hints (as source tags on the encoder) about the quality of each source-target pair. |
| Approach: | They propose a method which provides hints to the encoder and decoder to improve the quality of BT data by providing hints about the quality. |
| Outcome: | The proposed method improves translation quality and performance in three low/medium-resource language pairs. |
Making Better Use of Bilingual Information for Cross-Lingual AMR Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on meaning representations for English and other languages finds that concepts in their predicted AMR graphs are less specific. |
| Approach: | They propose a cross-lingual AMR parser that can predict more precise concepts by translating translated texts and non-English texts. |
| Outcome: | The proposed model surpasses state-of-the-art parser by 10.6 points on Smatch F1 score. |
A Sequence-to-Sequence Approach to Dialogue State Tracking (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for dialogue state tracking are still challenging, but they are improving . a new approach to dialogue state monitoring is proposed, called Seq2Seq-DU . |
| Approach: | They propose a new dialogue state tracking module that formalizes DST as a sequence-to-sequence problem. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets in different settings. |
Pseudo-Bidirectional Decoding for Local Sequence Transduction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Local sequence transduction tasks involve massive overlapping between source and target sequences . experimental results show that Pseudo-Bidirectional Decoding improves performance of standard seq2seq models. |
| Approach: | They propose a simple but versatile approach for local sequence transduction tasks . they propose to copy source tokens to decoder as pseudo future context . |
| Outcome: | The proposed approach improves the performance of standard seq2seq models on LST tasks. |
Multi-Granularity Guided Fusion-in-Decoder (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-domain question answering requires deriving factual responses without explicit evidence . recent approaches combine retrieval of relevant information with response generation . |
| Approach: | They propose a model that concatenates multiple contexts in the decoding phase . they propose MGFiD, which harmonizes passage re-ranking with sentence classification . |
| Outcome: | The proposed model outperforms existing models on Natural Questions and TriviaQA datasets . it aggregates evident sentences into an anchor vector that instructs the decoder . |
Scene Graph and Dependency Grammar Enhanced Remote Sensing Change Caption Network (SGD-RSCCN) (2025.coling-main)
Copied to clipboard
| Challenge: | Remote sensing images are used for disaster assessment, urban planning and disaster response. |
| Approach: | They propose a Scene Graph and Dependency Grammar Enhanced Remote Sensing Change Caption Network to improve the accuracy and naturalness of extracting and describing change information from remote sensing images. |
| Outcome: | The proposed method improves the naturalness and accuracy of extracting and describing change information from remote sensing images. |
Unsupervised Relation Extraction: A Variational Autoencoder Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction use latent variables and supervised training which requires large datasets. |
| Approach: | They propose a VAE-based unsupervised relation extraction technique that uses latent variables as an intermediate variable instead of a latent variable. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the NYT dataset and outperformed existing methods. |
Encoders Help You Disambiguate Word Senses in Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Neural machine translation models can perform word sense disambiguation (WSD) however, it is unclear which component dominates the process of disambiguating words. |
| Approach: | They evaluate hidden states and investigate distributions of self-attention in NMT encoders and decoders to disambiguate word senses. |
| Outcome: | The proposed model outperforms encoder hidden states on large datasets . the model outpersforms decoders on large data sets . |
Deep Communicating Agents for Abstractive Summarization (N18-1)
Copied to clipboard
| Challenge: | Empirical results show that multiple communicating agents produce a better summary than extractive summarization. |
| Approach: | They propose an encoder-decoder architecture that uses deep communicating agents to represent a long document for abstractive summarization. |
| Outcome: | Empirical results show that multiple communicating agents produce a better summary than baselines. |
RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answering (2023.findings-acl)
Copied to clipboard
| Challenge: | Open-domain Question Answering (ODQA) systems rely on spurious features instead of genuine causal relationships to generate answers. |
| Approach: | They propose a model that leverages the encoders of FiD to distinguish between causal relationships and spurious features and guides the decoder to generate answers informed by this discernment. |
| Outcome: | The proposed model improves on two ODQA datasets and shows that it can identify causal relationships and identify spurious features. |
Learning Structural Information for Syntax-Controlled Paraphrase Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Syntax-controlled paraphrase generation aims to produce paraphrase conform to given syntactic patterns. |
| Approach: | They propose a model that captures parent-child and sibling relations and a syntax encoder to capture alignment relations. |
| Outcome: | The proposed model achieves state-of-the-art in terms of semantic and syntactic quality on two popular benchmark datasets. |
Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing (2021.findings-emnlp)
Copied to clipboard
Akshat Shrivastava, Pierce Chuang, Arun Babu, Shrey Desai, Abhinav Arora, Alexander Zotov, Ahmed Aly
| Challenge: | a novel approach to map utterances to semantic frames is based on non-autoregressive parsers that shift the decoding task from text generation to span prediction. |
| Approach: | They propose a non-autoregressive, task-oriented parser which shifts the decoding task from text generation to span prediction and produces endpoints as opposed to text. |
| Outcome: | The proposed model bridges the quality gap between non-autoregressive and autoregressive parsers, achieving 87 EM on TOPv2 and shows a 70% reduction in latency and 83% reduction in memory at beam size 5 compared to prior non-regressives. |
Span-based Hierarchical Semantic Parsing for Task-Oriented Dialog (D19-1)
Copied to clipboard
| Challenge: | Existing semantic parsers score intents and slots as labels of nesting nodes, but decode a valid tree globally. |
| Approach: | They propose a span-based semantic parser for parsing compositional utterances into Task Oriented Parse (TOP) the parsers score labels of the tree nodes covering each token span independently, but decode a valid tree globally. |
| Outcome: | The proposed parser outperforms previous methods on the TOP dataset in accuracy and training speed. |
Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing vision-based sign language production approaches suffer from out-of-vocabulary (OOV) and test-time generalization problems. |
| Approach: | They propose an avatar-based sign language production system that generates sign language videos from spoken language expressions. |
| Outcome: | The proposed system achieves higher BLEU-4 and higher ROUGE-L scores on a new Korean-Korean sign language dataset. |
Accelerating Neural Transformer via an Average Attention Network (P18-1)
Copied to clipboard
| Challenge: | Using parallelizable attention networks, the neural Transformer is slow to train due to auto-regressive architecture and self-attention in the decoder. |
| Approach: | They propose an average attention network to replace the original self-attention model in the decoder of the neural Transformer. |
| Outcome: | The proposed network can decode sentences over four times faster than the original version with almost no loss in training time and translation performance. |
Neural Machine Translation with Reordering Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing work exploits the reordering information in neural machine translation . experimental results show that the proposed methods can significantly improve the performance of the transformer translation system. |
| Approach: | They propose a reordering mechanism to learn the re ordering embedding of a word based on contextual information and stack them together with self-attention networks to learn sentence representation for machine translation. |
| Outcome: | The proposed method improves translation performance on English-to-German, NIST Chinese-to English, and WAT Japanese-toEnglish translation tasks. |
Focus-Constrained Attention Mechanism for CVAE-based Response Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models generate high-frequency but trivial responses such as "I don't know" or "I'm ok" due to the discrepancy in discourse-level information, standard models generate one-to-many relationships. |
| Approach: | They propose to transform coarse-grained discourse-level information into fine-grounded word-level knowledge by introducing a fine-grain focus signal and a focus-constrained attention mechanism to take full advantage of focus. |
| Outcome: | The proposed model can generate more diverse and informative responses compared with state-of-the-art models. |
Recursive Template-based Frame Generation for Task Oriented Dialog (2020.acl-main)
Copied to clipboard
| Challenge: | The input of an NLU component is a semantic frame that captures the intent and slot-labels provided by the user. |
| Approach: | They propose a recursive, hierarchical representation that captures the intent and slot-labels provided by the user and extend local tree-based loss functions with terms that provide global supervision. |
| Outcome: | The proposed representation improves on the widely used ATIS dataset and significantly improves the performance of the proposed framework. |
Generating Long and Informative Reviews with Aspect-Aware Coarse-to-Fine Decoding (P19-1)
Copied to clipboard
| Challenge: | Existing methods for review generation lack topical and syntactic characteristics of natural languages. |
| Approach: | They propose a review generation model that uses aspect semantics, syntactic sketch, and context information to generate a sentence and corresponding words. |
| Outcome: | The proposed model can generate long and informative review text for users given a product and her/his rating on it. |
TSDG: Content-aware Neural Response Generation with Two-stage Decoding Process (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical results show that generative models often use a single decoder to generate a complete response at a stroke. |
| Approach: | They propose a content-aware model with two-stage decoding process to separate content words from function words. |
| Outcome: | The proposed model outperforms competing models in automatic and human evaluation on two datasets. |
Towards Fine-grained Text Sentiment Transfer (P19-1)
Copied to clipboard
| Challenge: | Existing methods for fine-grained text sentiment transfer only reverse the sentiment polarity of text, but they lack a robust and parallel learning algorithm. |
| Approach: | They propose a novel fine-grained text sentiment transfer task that revises a sequence to satisfy a given sentiment intensity while preserving the original semantic content. |
| Outcome: | The proposed model outperforms existing methods by a large margin in automatic evaluation and human evaluation. |
CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) data are limited due to the limited resources. |
| Approach: | They propose a knowledge distillation framework for speech translation that integrates knowledge from machine translation and decouples knowledge from non-target class knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a benchmark dataset. |
Generating Knowledge Graph Paths from Textual Definitions using Sequence-to-Sequence Models (N19-1)
Copied to clipboard
| Challenge: | a novel method for mapping unrestricted text to knowledge graph entities is proposed . a proof-of-concept experiment has encouraging results comparable to those of state-of the-art systems. |
| Approach: | They propose a method for mapping unrestricted text to knowledge graph entities by framing the task as a sequence-to-sequence problem. |
| Outcome: | The proposed method produces highly interpretable predictions comparable to state-of-the-art systems. |
COMMIT: Code-Mixing English-Centric Large Language Model for Multilingual Instruction Tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | COMMIT improves the exact match score of low-resource language QA by 32x. |
| Approach: | They propose to specialize instruction tuning to deviate from English-centric instruction tuning . they propose to perform cross-lingual alignment to overcome data imbalance . |
| Outcome: | The proposed method improves the exact match score of low-resource language QA by 32x. |
Improving Abstractive Document Summarization with Salient Information Modeling (P19-1)
Copied to clipboard
| Challenge: | Abstractive document summarization is a task of natural language generation which generates fluent summaries with salient information automatically. |
| Approach: | They propose to incorporate a Gaussian focal bias on attention scores into an encoder to enhance the perception of local context and to distinguish salient information precisely. |
| Outcome: | The proposed framework outperforms state-of-the-art models on the CNN/Daily Mail benchmark and is based on a focus-attention mechanism and two new extensions. |
Improving Abstraction in Text Summarization (D18-1)
Copied to clipboard
| Challenge: | Abstractive text summarization models do not capture the abstractive nature of high quality summaries. |
| Approach: | They propose to decompose a decoder into a contextual network and a pretrained language model that incorporates prior knowledge about language generation. |
| Outcome: | The proposed model achieves comparable results to state-of-the-art models, based on ROUGE scores and human evaluations, while producing a significantly higher level of abstraction. |
MultiDM-GCN: Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research suggests that engaging conversations include visual cues (e.g., a video or images) or audio cue. |
| Approach: | They propose a multi-modal conversational framework that generates the responses following the different aspects of a product or service to cater to the user's needs. |
| Outcome: | The proposed framework outperforms baselines for the task-oriented dialogue setup. |
Breaking Down Multilingual Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Multilingual training is an essential ingredient in machine translation systems . but it has different effects in different multilingual settings, such as many-to-one, one-tomany and many- to-many learning . |
| Approach: | They compare multilingual training settings with encoders and decoders initialized by multilingual learning . they find important attention heads for each language pair and compare their correlations during inference . |
| Outcome: | The proposed models outperform the best models for high-resource languages and one-to-many models for low-resourced languages. |
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder (2021.emnlp-main)
Copied to clipboard
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, Arnold Overwijk
| Challenge: | Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space. |
| Approach: | They propose a self-learning method that pre-trains the autoencoder using a weak decoder to push the encoder to provide better sequence representations. |
| Outcome: | The proposed model significantly boosts the effectiveness and few-shot ability of dense retrieval models on web search, news recommendation, and open domain question answering. |
Multi-Input Attention for Unsupervised OCR Correction (P18-1)
Copied to clipboard
| Challenge: | Existing methods for OCR correction are mostly supervised methods that correct recognition errors in a single output. |
| Approach: | They propose a sequence-to-sequence model with attention and a decoder with attention averaging to search for consensus among multiple sequences. |
| Outcome: | The proposed methods cut the character and word error rates nearly in half on single inputs and can rival supervised methods. |
Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models are usually based on attention-based encoder-decoder frameworks. |
| Approach: | They introduce a seer decoder into the encoder-decoder framework during training . they force the conventional decoded decodes to simulate the behavior of the seer . |
| Outcome: | The proposed method outperforms baselines on Chinese, English and German translation tasks. |
AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech translation models learn acoustic representations from the encoder, which is not desirable for cross-modal and cross-lingual translation. |
| Approach: | They propose an adaptive speech-to-text translation model that dynamically adapts acoustic states in the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art speech translation models on two widely-used datasets. |
Semi-Supervised Dependency Parsing with Arc-Factored Variational Autoencoding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for dependency parsing use unlabelled data to compensate for the lack of training corpora. |
| Approach: | They propose semi-supervised dependency parsing methods that utilize unlabelled data to compensate for the scarcity of labelled training corpora. |
| Outcome: | The proposed model overcomes the tree constraint and the complexity of the training procedure while avoiding the challenges brought by the tree constraints. |
Ruleformer: Context-aware Rule Mining over Knowledge Graph (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on rule mining focuses on mining rules, but how to select appropriate rules for completion of different triplets has not been discussed. |
| Approach: | They propose to take context information into consideration when selecting suitable rules . they devise a transformer-based rule mining approach, Ruleformer . |
| Outcome: | The proposed model takes context information into consideration, which helps select suitable rules for inference tasks. |
Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training . |
| Approach: | They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments. |
| Outcome: | The proposed extensions improve compositional generalization on OOD compositions. |
Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios (2024.findings-acl)
Copied to clipboard
Lei Lin, Jiayi Fu, Pengli Liu, Qingyang Li, Yan Gong, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai
| Challenge: | chain-of-thought (CoT) prompting has been shown to be effective on complex reasoning tasks, but the naive greedy decoding used in CoT prompting causes the repetitiveness and local optimality. |
| Approach: | They propose a generalizable ensemble-optimization method that uses a set of reasoning paths to prompt a language model one more time to determine the optimal answer. |
| Outcome: | The proposed method can be generalized to almost all scenarios where the type of input questions and answer format of reasoning paths may be unknown. |
Abstractive Multi-Document Summarization via Joint Learning with Single-Document Summarization (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document summarization are extractive and abstractive. |
| Approach: | They propose to jointly learn an abstractive single-document decoder and a decoding controller to aggregate the decoded outputs for multiple input documents. |
| Outcome: | The proposed model outperforms several baselines on two multi-document summarization datasets and proves that it is useful for both tasks. |
Multilingual Translation via Grafting Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to graft pre-trained (masked) language models to multilingual data are limited, and they lack cross-attention component. |
| Approach: | They propose to graft separately pre-trained (masked) language models for machine translation using monolingual data and parallel data. |
| Outcome: | The proposed method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLUE in en2x directions compared with the multilingual Transformer of the same size. |
Bidirectional Transformer Reranker for Grammatical Error Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained seq2seq models suffer from a prediction bias due to their unidirectional decoding. |
| Approach: | They propose a bidirectional Transformer reranker that re-estimates the probability of each candidate sentence generated by pre-trained seq2seq models. |
| Outcome: | The proposed model improves on the original model and gives a 59.52 GLEU score on the JFLEG corpus. |
Synchronous Refinement for Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to decode target sentences face a one-pass issue . generated wrong words are added to the historical context to affect the generation of subsequent target words, which hinders the performance of machine translation. |
| Approach: | They propose a synchronous refinement method to revise potential errors in the generated words by considering part of the target future context. |
| Outcome: | The proposed method can refine generated target words and generate the next target word synchronously. |
Clues Before Answers: Generation-Enhanced Multiple-Choice QA (2022.naacl-main)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) uses text-to-text framework . but, there is an under-utilization of the decoder and knowledge that can be decoded . |
| Approach: | They propose a generative multiple-choice question answering model which generates a clue from the question and leverages it to enhance a reader for MCQA. |
| Outcome: | The proposed model outperforms text-to-text models on multiple MCQA datasets. |
Towards Reinterpreting Neural Topic Models via Composite Activations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Most Neural Topic Models (NTMs) use a variational auto-encoder framework producing K topics limited to the size of the encoder’s output. |
| Approach: | They propose a model-free two-stage process to reinterpret NTM and derive further insights on the state of the trained model. |
| Outcome: | The proposed model-free process decouples the strict interpretation of topics from the original NTM and evaluates them on a large external corpus. |
On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that PT and BT are nicely complementary to each other. |
| Approach: | They introduce two probing tasks for PT and BT respectively and investigate their complementarity. |
| Outcome: | The proposed methods establish state-of-the-art on the WMT16 English-Romanian and English-Russian benchmarks. |
Knowledge-enriched, Type-constrained and Grammar-guided Question Generation over Knowledge Bases (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for question generation over knowledge bases have low diversity and poor fluency due to the limited information contained in the subgraphs and semantic drift due to decoder’s oblivion of the semantics of the answer entity. |
| Approach: | They propose a knowledge-enriched, type-constrained and grammar-guided KBQG model that generates natural-language questions over a set of triples in the KB. |
| Outcome: | The proposed model outperforms existing methods on two widely-used benchmark datasets. |
Modelling Context Emotions using Multi-task Learning for Emotion Controlled Dialog Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent research has tackled this task using neural generative methods by augmenting emotion classes with the input sequences. |
| Approach: | They propose to use a self-attention based encoder and a decoder with dot product attention mechanism to generate a viable response with a specified emotion. |
| Outcome: | The proposed model outperforms baselines on automatic evaluation measures such as F1 and BLEU scores, thus resulting in more fluent and adequate responses. |
On Sparsifying Encoder Outputs in Sequence-to-Sequence Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Using sequence-to-sequence models, encoder outputs are usually transferred to the decoder for generation, but in this study, encoded outputs can be compressed to shorten the sequence for decoding. |
| Approach: | They propose to use a stochastic gate-based algorithm to mask encoder outputs to shorten the sequence delivered for decoding. |
| Outcome: | The proposed model can be used to shorten encoder outputs to short a sequence . the proposed model yields a speedup of up to 1.65 on document summarization and 1.20 on character-based machine translation tasks. |
Target Conditioning for One-to-Many Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models lack diversity in their generated translations, even when paired with search algorithm, like beam search. |
| Approach: | They propose to model one-to-many mapping by conditioning a decoder on a latent variable that represents the domain of target sentences. |
| Outcome: | The proposed method can scale to any number of domains without affecting performance or training time. |
Adaptive Weighting for Neural Machine Translation (C18-1)
Copied to clipboard
| Challenge: | Existing weighted sum models (WSMs) take inputs and generate one output, but they are independent of each other and are fixed for all inputs. |
| Approach: | They propose adaptive weighting for WSMs to control the contribution of each input and output state. |
| Outcome: | The proposed weighting improves translation accuracy by 1.49 and 0.92 BLEU points on Chinese-to-English translation and English-to German translation tasks. |
Exploring Unsupervised Pretraining Objectives for Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised cross-lingual pretraining has significantly reduced the need for large parallel data. |
| Approach: | They compare unsupervised cross-lingual pretraining with masking and reconstructing inputs in the decoder to produce real sentences. |
| Outcome: | The proposed methods produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. |
DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis (2022.coling-1)
Copied to clipboard
Xian Wu, Shuxin Yang, Zhaopeng Qiu, Shen Ge, Yangtian Yan, Xingwang Wu, Yefeng Zheng, S. Kevin Zhou, Li Xiao
| Challenge: | X-ray and CT are the gold standard for COVID-19 diagnosis and treatment . however, due to the excessive number of patients, writing reports becomes a heavy burden for radiologists. |
| Approach: | They propose to use X-ray and CT to generate medical reports automatically . they evaluate DeltaNet on a COVID-19 dataset, where it outperforms state-of-the-art approaches . |
| Outcome: | The proposed system outperforms state-of-the-art methods on a COVID-19 dataset. |
CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting (2024.lrec-main)
Copied to clipboard
Yuang Li, Yinglu Li, Min Zhang, Chang Su, Jiawei Yu, Mengyao Piao, Xiaosong Qiao, Miaomiao Ma, Yanqing Zhao, Hao Yang
| Challenge: | End-to-end automatic speech recognition systems struggle to recognize rare name entities such as personal names, organizations and terminologies that are not frequently encountered in the training data. |
| Approach: | They propose a convolutional neural network-based ASR system that performs open-vocabulary keyword-spotting before the decoder to match the features between the entities and the utterances. |
| Outcome: | The proposed system significantly improves mixed-error-rate (MER) and entity recall compared to the original Whisper model on three internal datasets and two publicly available datasets. |
Learning to Rewrite for Non-Autoregressive Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translations have poor inference speed but weak recognition of erroneous translation pieces. |
| Approach: | They propose an architecture to explicitly learn to rewrite the erroneous translation pieces. |
| Outcome: | The proposed architecture can achieve better performance while significantly reducing decoding time. |
Tailoring Neural Architectures for Translating from Morphologically Rich Languages (C18-1)
Copied to clipboard
| Challenge: | A morphologically complex word is a hierarchical constituent with meaning-preserving subunits, so word-based models which rely on surface forms might not be powerful enough to translate such structures. |
| Approach: | They propose a neural architecture which is designed to deal with morphological complexities on the source side and redesign the decoder accordingly to benefit from such information. |
| Outcome: | The proposed model outperforms existing subword- and character-based architectures and showed significant improvements on translating from German, Russian, and Turkish into English. |
Retrieval-augmented Image Captioning (2023.eacl-main)
Copied to clipboard
| Challenge: | Experimental results show that image captioning can be effectively formulated from this new perspective. |
| Approach: | They propose a pretrained visual and language decoders for image captioning that generate sentences from the input image and a set of captions retrieved from a datastore. |
| Outcome: | The proposed model generates sentences given the input image and retrieved captions, while the decoder attends to the multimodal encoder representations. |
Investigating the Emergent Audio Classification Ability of ASR Foundation Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Text and vision foundation models can perform many tasks in a zero-shot setting . however, there has been little work on the zero-shoot abilities of ASR foundation models . |
| Approach: | They investigate the ability of ASR foundation models to perform zero-shot audio classification using text prompts and a decoding probability generator. |
| Outcome: | The proposed model outperforms state-of-the-art models on audio classification datasets without training them on extra data or adding any parameters. |
Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems (N19-1)
Copied to clipboard
| Challenge: | Existing methods to solve math word problems require accurate natural language understanding to bridge texts and math expressions. |
| Approach: | They propose a neural approach to automatically solve math word problems by operating symbols according to their semantic meanings in texts. |
| Outcome: | The proposed model outperforms state-of-the-art models and the best non-retrieval-based models over 10% accuracy in a Math23K dataset. |
A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multi-modal neural machine translation models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities. |
| Approach: | They propose a graph-based multi-modal fusion encoder that exploits fine-grained semantic correspondences between different modalities. |
| Outcome: | The proposed encoder significantly extends the conventional text-based translation by taking images as additional inputs. |
Bridging the Gap between Prior and Posterior Knowledge Selection for Knowledge-Grounded Dialogue Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-grounded dialogue models lack prior and posterior knowledge selection . prior selection module may not learn to select knowledge properly because of lack of posterior information . |
| Approach: | They propose a knowledge distillation-based training strategy to remove the exposure bias of knowledge selection. |
| Outcome: | The proposed model improves on two knowledge-grounded dialogue datasets. |
FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoded beam search, and their generated output suffers from hallucinations. |
| Approach: | They propose to extend transformer encoders with the ability to fuse information from multiple passages to provide cross-sample attention over all tokens across samples. |
| Outcome: | The proposed method outperforms the current state-of-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference. |
Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Experimental results show that the Reinforce-NAT system surpasses the baseline NAT system by a significant margin on BLEU without decelerating the decoding speed. |
| Approach: | They propose a sequence-level training method and a Transformer decoder to fuse the target sequential information into the top layer of the decoded Transformer. |
| Outcome: | The proposed model surpasses the baseline NAT system on BLEU without decelerating the decoding speed and achieves comparable translation performance to the autoregressive Transformer model with considerable speedup. |
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text style transfer require style-labeled training data, but use only labeled data at inference time. |
| Approach: | They propose a method that uses readily-available unlabeled text to train style transfer . they use a style vector to condition a decoder to perform style transfer using unlabelled text . |
| Outcome: | The proposed method is competitive on sentiment transfer, even compared to models trained fully on labeled data. |
DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity. |
| Approach: | They propose a method to analyze encoder-decoder Transformers by using the decoder module Model Output encoder to cross-attend representations of intermediate encoder activations instead of using the default output. |
| Outcome: | The proposed method maps uninterpretable representations to human-interpreted sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models. |
Text-Only Training for Image Captioning using Noise-Injected CLIP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to image captioning requires large datasets of captioned images and is difficult to collect. |
| Approach: | They propose to use a decoder to translate CLIP textual embeddings back into text . they show that this intuition is “almost correct” because of a gap between the embeddable spaces . |
| Outcome: | The proposed approach shows that the intuition is “almost correct” because of a gap between the embedding spaces, and rectifies this via noise injection during training. |
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in machine translation (MT) have improved performance on low-resource language pairs. |
| Approach: | They propose to freeze most BART parameters and add new ones to fine-tune a model trained on MT. |
| Outcome: | The proposed model outperforms naive fine-tuning on Vietnamese to English on a training set for Vietnamese to Vietnamese . the proposed model is able to fine- tune on smaller datasets while still maintaining the same model performance. |
MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) based sparse architectures are prone to overfitting on low-resource language translation. |
| Approach: | They propose a modularized MNMT framework that flexibly assembles dense and MoE-based sparse modules to achieve the best of both worlds. |
| Outcome: | The proposed framework outperforms existing models on low-resource language translation and zero-shot translation on benchmark datasets. |
A Prospective-Performance Network to Alleviate Myopia in Beam Search for Response Generation (C18-1)
Copied to clipboard
| Challenge: | Existing methods to generate responses using beam search focus on current optimal results. |
| Approach: | They propose a beam search method that uses a Prospective-Performance Network to predict the future reward of a partially-generated response. |
| Outcome: | The proposed method can increase the quality and diversity of generated responses with high inference efficiency. |
Almost Free Semantic Draft for Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. |
| Approach: | They propose a method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. |
| Outcome: | Empirical results show that the proposed method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. |
Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing variational dialog models have pre-trained, restricting diversity of responses . a diffusion model increases complexity of prior distribution and its compatibility with PLMs . |
| Approach: | They propose a hierarchical conditional variational autoencoder with diffusion priors to address these challenges. |
| Outcome: | The proposed method generates more diverse responses without dialog pre-training. |
Sequence-to-Lattice Models for Fast Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive machine translation models have decoders that are difficult to port to NAT models. |
| Approach: | They propose a sequence-to-lattice model that replaces the decoder with a search lattice. |
| Outcome: | The proposed model is faster than past non-autoregressive generation approaches and more accurate than reducing the number of decoder layers. |
Text Generation Model Enhanced with Semantic Information in Aspect Category Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for ACSA fail to model relations of target words and opinion words in a sentence including multiple aspects. |
| Approach: | They propose to incorporate AMR into a text generation model to model relations of target words and opinion words in a sentence including multiple aspects. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three datasets. |
Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation (UNMT) can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. |
| Approach: | They propose a method to translate between 13 languages using a single encoder and a decoder . they propose two knowledge distillation methods to further enhance multilingual UNMT performance . |
| Outcome: | The proposed method improves translation performance for all languages using multilingual data. |
SrcMix: Mixing of Related Source Languages Benefits Extremely Low-resource Machine Translation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Multilingual models are widely used for machine translation, but their effectiveness for extremely low-resource languages (ELRLs) is dependent on how related languages are incorporated during fine-tuning. |
| Approach: | They propose a source-side mixing strategy that combines related ELRLs during fine-tuning while constraining the decoder to a single target language. |
| Outcome: | The proposed approach improves performance in high-resource to ELRL translations and in mid-resourced to MT translations. |
End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (D18-1)
Copied to clipboard
| Challenge: | Autoregressive decoding is the only part of sequence-to-sequence models that prevents massive parallelization at inference time. |
| Approach: | They propose a non-autoregressive architecture based on connectionist temporal classification . they conduct experiments on the WMT English-Romanian and English-German datasets . |
| Outcome: | The proposed model achieves a significant speedup over autoregressive models . the model can be trained end-to-end and maintains translation quality comparable to other models compared to autoregression models based on connectionist temporal classification . |
Multi-Task Learning with Language Modeling for Question Generation (D19-1)
Copied to clipboard
| Challenge: | Existing work on answer-aware questions generates a sentence and answer span as input . previous work on QG was mainly tackled by rule-based approach and neural-based one . |
| Approach: | They propose to incorporate an auxiliary task of language modeling to help question generation in a hierarchical multi-task learning structure. |
| Outcome: | The proposed model improves on SQuAD and MARCO datasets and human evaluation proves it. |
Contrastive Learning enhanced Author-Style Headline Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current work only uses the article itself in the headline generation, but have not taken the writing style of headlines into account. |
| Approach: | They propose a model which takes historical headlines into account to integrate the stylistic features of the author into the model and integrate them into the decoder. |
| Outcome: | The proposed model can integrate the stylistic features of the author into the model and generate a headline that is appropriate for the article and consistent with the author’s style. |
FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection (2024.acl-long)
Copied to clipboard
| Challenge: | Open Domain Question Answering (ODQA) is a longstanding task in Natural Language Processing that involves generating an answer solely based on a given question. |
| Approach: | They propose a novel approach that executes sentence selection on the encoded passages to enhance the inference speed while reducing the context length required for generating answers. |
| Outcome: | The proposed approach can increase inference speed by **2.3X-5.7X** while maintaining the model’s performance. |
Exploring Sequence-to-Sequence Learning in Aspect Term Extraction (P19-1)
Copied to clipboard
| Challenge: | Aspect term extraction (ATE) aims at identifying all aspect terms in a sentence . sequence labeling based methods cannot make full use of overall meaning of sentence if they have dependencies between labels. |
| Approach: | They propose to formalize ATE as a sequence-to-sequence (Seq2Seque) learning task . they propose gated unit networks and position-aware attention mechanism to make it suit to ATE . |
| Outcome: | The proposed learning task is effective when labels correspond to words one by one . the proposed learning system is gated unit networks and position-aware attention mechanism . |
Semi-supervised Autoencoding Projective Dependency Parsing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for semi-supervised dependency parsing use labeled data, but they require large amounts of labeles. |
| Approach: | They propose two end-to-end autoencoding models for semi-supervised graph-based projective dependency parsing. |
| Outcome: | The proposed models outperform a semi-supervised model on WSJ and UD dependency parsing data sets. |
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Using pre-training methods for NMT models is difficult because of the cross-attention module . cross-linguistic embeddings are not used to pretrain a decoder . |
| Approach: | They propose a semantic interface between pre-trained encoder and pre-train decoder to improve NMT performance. |
| Outcome: | The proposed method improves on translation and unsupervised translation tasks. |
ToXCL: A Unified Framework for Toxic Speech Detection and Explanation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models that focus on explicit toxic speech detection and explanation are prone to error propagation problems . et al., 2018) show that toxic speech models can be prone for generating errors . |
| Approach: | They propose a framework that can detect and explain toxic speech using a target group generator and an encoder-decoder model. |
| Outcome: | The proposed model outperforms baseline models and achieves state-of-the-art effectiveness . the proposed model generates a toxic explanation that matches the ground truth explanation . |
AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to extract product attribute values are limited by knowledge sharing across different attributes. |
| Approach: | They propose to use adaptive decoding to handle extraction of product attribute values by parameterizing the decoder with pretrained attribute embeddings, through a hypernetwork and a Mixture-of-Experts module. |
| Outcome: | The proposed model is able to handle multiple attributes without sharing the entire network parameters across all attributes. |
Entity Disambiguation via Fusion Entity Decoding (2024.naacl-long)
Copied to clipboard
Junxiong Wang, Ali Mousavi, Omar Attia, Ronak Pradeep, Saloni Potdar, Alexander Rush, Umar Farooq Minhas, Yunyao Li
| Challenge: | Existing generative approaches demonstrate improved accuracy compared to classification approaches under the standardized ZELDA benchmark. |
| Approach: | They propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. |
| Outcome: | The proposed model outperforms existing classification models on the ZELDA benchmark and on retrieval/reader frameworks. |
FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition (2021.findings-emnlp)
Copied to clipboard
Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Xiang-Yang Li, Tao Qin, Edward Lin, Tie-Yan Liu
| Challenge: | Error correction is widely used in automatic speech recognition (ASR) to post-process the generated sentence. |
| Approach: | They propose a fast correction model that takes multiple ASR candidates as input for better correction accuracy. |
| Outcome: | The proposed model can reduce the word error rate (WER) with multiple candidates by 3.2% and 2.6%. |
XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for active learning rely on model uncertainty or disagreement to pick unlabeled data, leading to over-confidence in superficial patterns and lack of exploration. |
| Approach: | They propose to use a bi-directional encoder and a uni-directional decoder to generate and score an explanation for low-resource text classification. |
| Outcome: | The proposed model improves on 9 strong baselines on six datasets and can generate explanations for its predictions. |
Improving the Robustness of Summarization Systems with Dual Augmentation (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that state-of-the-art summarization models have a significant decrease in performance on adversarial and noisy test sets. |
| Approach: | They propose a SummAttacker approach to generate adversarial samples based on pre-trained language models that can generate word-level synonym substitution and noise. |
| Outcome: | The proposed model performs better on noisy, attacked, and clean datasets than baseline models and is more robust on noisy and attacked datasets. |
Effectiveness of Data Augmentation and Pretraining for Improving Neural Headline Generation in Low-Resource Settings (2022.lrec-1)
Copied to clipboard
| Challenge: | Neural approaches for natural language generation (NLG) have mushroomed due to large textual resources. |
| Approach: | They propose to use a pretrained multilingual encoder-decoder model and a combination of two pretrained language models to train a model in a low-resource setting. |
| Outcome: | The proposed model outperforms the previous model on English and on a small subset of the same data. |
Text Summarization with Pretrained Encoders (D19-1)
Copied to clipboard
| Challenge: | Existing pretraining languages such as ELMo and GPT have advanced a wide range of tasks. |
| Approach: | They propose a novel document-level encoder based on BERT which can express the semantics of a document and obtain representations for its sentences. |
| Outcome: | The proposed model achieves state-of-the-art in extractive and abstractive settings. |
Context-Aware Cross-Attention for Non-Autoregressive Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies have shown that non-autoregressive translation models can predict all tokens independently and simultaneously. |
| Approach: | They propose to enhance signals of neighbour source tokens into conventional cross-attention to address a locality perception problem in NAT cross- attention. |
| Outcome: | The proposed approach improves translation quality over strong NAT baselines on representative datasets. |
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Encoder-decoder models for unsupervised sentence representation learning discard decoder after training . decoded sentences are often used to make better predictions of words in a given sentence . |
| Approach: | They propose two types of decoding functions whose inverse can be easily derived without expensive inverse calculation. |
| Outcome: | The proposed models can learn good representations from encoders and decoders without expensive calculations. |
Improving a Neural-based Tagger for Multiword Expressions Identification (L18-1)
Copied to clipboard
| Challenge: | MUMULS tagger for automatic detection of verbal multiword expressions is based on neural networks . character-level embeddings can improve the performance, reducing out-of-vocabulary rate . multiword Expressions are viewed by computational linguists as a "pain in the neck of NLP" |
| Approach: | They propose to improve MUMULS, a tagger for automatic detection of verbal multiword expressions. |
| Outcome: | The proposed tagger performed better on Czech language than the previous taggers. |
Keeping Notes: Conditional Natural Language Generation with a Scratchpad Encoder (P19-1)
Copied to clipboard
| Challenge: | Qualitative assessments in the form of human judgements (question generation), attention visualization (MT), and sample output (summarization) provide further evidence of the ability of Scratchpad to generate fluent and expressive output. |
| Approach: | They propose to use the encoder as a "scratchpad" memory to keep track of what has been generated and guide future generation. |
| Outcome: | The proposed mechanism improves the fluency of seq2seq models on three well-studied natural language generation tasks. |
Incorporating Inner-word and Out-word Features for Mongolian Morphological Segmentation (2020.coling-main)
Copied to clipboard
| Challenge: | Mongolian morphological segmentation is a crucial preprocessing step in many Mongolian related NLP applications. |
| Approach: | They propose a neural network incorporating inner-word and out-word features for Mongolian morphological segmentation. |
| Outcome: | The proposed network is compared with baselines and evaluates its performance. |
AdaNSP: Uncertainty-driven Adaptive Decoding in Neural Semantic Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing (SP) maps a natural language utterance into a formal language . standard Seq2Seq models ignore underlying grammars and may give ill-formed results. |
| Approach: | They propose an end-to-end model for semantic parsing that transduces a natural language sentence to the formal semantic representation. |
| Outcome: | The proposed model outperforms the state-of-the-art models and does not need expertise like predefined grammar or sketches in the meantime. |
TILGAN: Transformer-based Implicit Latent GAN for Diverse and Coherent Text Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models suffer from the exposure bias problem due to mismatches between training and generation stages. |
| Approach: | They propose a Transformerbased Implicit Latent GAN which combines a transformer autoencoder and GAN in the latent space with a novel design and distribution matching based on the Kullback-Leibler divergence. |
| Outcome: | The proposed model improves local and global coherence and quality-diversity trade-off on three benchmark datasets. |
On the Sub-layer Functionalities of Transformer Decoder (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation have focused on assessing the encoded representations or interpreting the multi-head self-attentions. |
| Approach: | They propose to use Transformer-based encoder-decoder architectures to analyze how information is propagated through each module of each decoder layer. |
| Outcome: | The proposed model can be dropped with minimal loss of performance on three translation datasets and can be used to train and inference faster. |
End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems (2020.emnlp-main)
Copied to clipboard
Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, Bing Xiang
| Challenge: | Existing approaches for synthetic QA data generation have limited or no success in improving the downstream Reading Comprehension task. |
| Approach: | They propose an end-to-end approach for synthetic QA data generation using a transformer-based encoder-decoder network that is trained end- to-end to generate both answers and questions. |
| Outcome: | The proposed model outperforms current state-of-the-art methods in the domain adaptation of QA models. |
Closed-Book Training to Improve Summarization Encoder Memory (D18-1)
Copied to clipboard
| Challenge: | Extractive summarization models pick words, phrases, and sentences from the source text to form a summary, while abstractive models generate (samples) words from a fixed-size vocabulary instead of copying from text directly. |
| Approach: | They propose to add an additional ‘closed-book’ decoder without attention and pointer mechanisms to the model to improve its memorization capabilities. |
| Outcome: | The proposed model outperforms the baseline model on the CNN/Daily Mail dataset in terms of ROUGE and METEOR metrics, and achieves higher scores in a test-only setup. |
Disentangling Pretrained Representation to Leverage Low-Resource Languages in Multilingual Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation requires an enormous dataset, leaving the low-resource language (LRL) underdeveloped. |
| Approach: | They evaluated five languages using a parallel corpus of 1,000 instances each and found a zero-shot improvement of 7.4 from the baseline score of 7.1 to a score of 15.5 at best. |
| Outcome: | The proposed model improves performance in the linguistically diverse country of Indonesia by 7.4 from baseline score of 7.1 to 15.5 at best. |
Answer-driven Deep Question Generation based on Reinforcement Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for deep question generation focus on enhancing document representations, but little attention is paid to the answer information. |
| Approach: | They propose a deep question generation model that makes better use of the target answer as a guidance to facilitate question generation. |
| Outcome: | The proposed model outperforms state-of-the-art models in automatic and human evaluations on the hotpotQA dataset. |
NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in neural topic models have focused on two main directions: the integration of the inference network with a pre-trained language model and the modeling of the relationship between words and topics in the generative model. |
| Approach: | They propose a framework to maximize mutual information between topics and PLMs . Experimental results indicate that NeuroMax reduces inference time . |
| Outcome: | The proposed framework reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings. |
Task-Oriented Conversation Generation Using Heterogeneous Memory Networks (D19-1)
Copied to clipboard
| Challenge: | Existing memory networks do not perform well when leveraging heterogeneous information from different sources. |
| Approach: | They propose to use user utterances, dialogue history and background knowledge tuples to integrate external knowledge into a neural dialogue model. |
| Outcome: | The proposed model outperforms the state-of-the-art data-driven task-oriented dialogue models on real-world datasets. |
RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to personalized dialogue generation rely on dialogue data paired with user traits, profiles or persona description sentences. |
| Approach: | They propose a hierarchical transformer retriever trained on dialogue domain data to perform personalized retrieval and a context-aware prefix encoder that fuses the retrieved information to the decoder more effectively. |
| Outcome: | The proposed model generates more fluent and personalized responses under a suite of human and automatic metrics and is superior to state-of-the-art baselines on English Reddit conversations. |
On Robustness of Finetuned Transformer-based NLP Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Transformer-based language models have been finetuned for a large number of tasks. |
| Approach: | They characterize changes between pretrained and finetuned models with CKA and STIR metrics. |
| Outcome: | The proposed models are more robust to perturbations than BERT and T5 on classification tasks and generation tasks. |
Learning Universal Sentence Representations with Mean-Max Attention Autoencoder (D18-1)
Copied to clipboard
| Challenge: | Existing methods to learn universal sentence representations focus on supervised learning. |
| Approach: | They propose a mean-max attention autoencoder that uses a multi-head mechanism to reconstruct the input sequence. |
| Outcome: | The proposed model outperforms state-of-the-art unsupervised single methods on a wide range of 10 transfer tasks. |
Dynamic Knowledge Prompt for Chest X-ray Report Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for radiology report generation fail to incorporate prior knowledge . data bias, sparse features of chest X-ray image make it difficult to generate reports . |
| Approach: | They propose a dynamically integrated framework for chest X-ray report generation that incorporates pulmonary lesion knowledge at the instance-level. |
| Outcome: | The proposed framework can dynamically incorporate pulmonary lesion knowledge at instance-level to facilitate report generation. |
Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous Speech Translation (SimulST) is a task focused on ensuring high-quality translation of speech in low-latency situations. |
| Approach: | They propose a token-level cross-modal alignment method to improve the translation of text to audio . they use audio transcription pairs to pre-train the encoder and a random wait-k-tokens strategy to optimize the task. |
| Outcome: | The proposed method achieves better trade-off between translation quality and latency. |
EDDA: An Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing text or data are limited by lack of logical connections between generated texts and training data. |
| Approach: | They propose an encoder-decoder data augmentation framework that combines large language models and chain-of-thought prompting to summarize texts into target-specific if-then rationales, establishing logical relationships. |
| Outcome: | The proposed framework significantly improves over state-of-the-art methods on benchmark datasets while enabling interpretable rationale-based learning. |
Getting More Data for Low-resource Morphological Inflection: Language Models and Data Augmentation (2020.lrec-1)
Copied to clipboard
| Challenge: | Morphological inflection is the process that generates the word form given its lexeme and morphological properties. |
| Approach: | They propose to use language models and data augmentation to improve morphological inflection without annotating more data. |
| Outcome: | The proposed model improves by 1.5% with the langauge model and by 9% with the data augmentation. |
Improving Seq2Seq Grammatical Error Correction via Decoding Interventions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to grammatical error correction (GEC) are sequence-to-sequence and sequence-edit. |
| Approach: | They propose a unified decoding intervention framework that employs an external critic to assess the appropriateness of the token to be generated incrementally. |
| Outcome: | The proposed framework outperforms baselines and state-of-the-art methods on English and Chinese datasets. |
Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence (2021.acl-long)
Copied to clipboard
| Challenge: | Existing generation models struggle to maintain a coherent event sequence throughout the generated text. |
| Approach: | They propose a long text generation model which can represent prefix sentences at sentence level and discourse level in the decoding process. |
| Outcome: | The proposed model can generate more coherent texts than state-of-the-art models. |
Exploring Recombination for Efficient Decoding of Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) decoder captures features of entire prediction history . some partial hypotheses with different prefixes will be regarded differently no matter how similar they are . |
| Approach: | They propose a method that uses a n-gram suffix to adapt it to beam search decoding. |
| Outcome: | The proposed method can obtain similar translation quality with a smaller beam size, making it more efficient. |
Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conversational question answering (CQA) requires models to extract answers from given contents to answer follow-up questions according to conversation history. |
| Approach: | They propose a novel architecture that integrates extractive MRC models into a generalized sequence-to-sequence framework. |
| Outcome: | The proposed architecture can use less storage space and consider historical memory deeply and selectively. |
The Transference Architecture for Automatic Post-Editing (2020.coling-main)
Copied to clipboard
| Challenge: | A research challenge is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions. |
| Approach: | They propose a multi-encoder based neural APE model that conditions post-editing decisions on both the source and machine translated text as inputs. |
| Outcome: | The proposed model outperforms the best performing systems by 1 BLEU point on the WMT 2016, 2017, and 2018 English–German APE shared tasks. |
Graph-to-Text Generation with Dynamic Structure Pruning (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies show that explicitly modeling the input graph structure can significantly improve the performance. |
| Approach: | They propose a structure-aware cross-attention mechanism to re-encode the graph representation conditioning on the newly generated context at each decoding step. |
| Outcome: | The proposed model improves performance on two graph-to-text datasets with only minor increase on computational cost. |
Guiding Neural Machine Translation with Semantic Kernels (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that our approach gains approximately an improvement of 1 BLEU score on most benchmarks over the Transformer baseline. |
| Approach: | They propose to extract several semantic kernels from a source sentence to capture global semantic information. |
| Outcome: | Empirical results show that the proposed approach improves 1 BLEU score on benchmarks . it is also 1.7 times faster than previous works on average at inference time . |
PromptRank: Unsupervised Keyphrase Extraction Using Prompt (2023.acl-long)
Copied to clipboard
| Challenge: | Existing keyphrase extraction methods struggle with document and candidate length discrepancies or fail to fully utilize the pre-trained language model without further fine-tuning. |
| Approach: | They propose an unsupervised keyphrase extraction approach that uses a pre-trained language model to rank candidates based on document embeddings. |
| Outcome: | The proposed approach outperforms the existing keyphrase extraction approach on six benchmarks. |
LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LT strategies cannot indicate the desired target language on zero-shot translation, i.e., the off-target issue. |
| Approach: | They propose a language converter strategy that embeds the target language into the top encoder layers to mitigate confusion in the encoder and ensures stable language indication for the decoder. |
| Outcome: | The proposed language converter strategy significantly mitigates off-target issue on multiUN, TED, and OPUS-100 datasets. |
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results show that our method outperforms full-model tuning in few-shot abstractive summarization tasks. |
| Approach: | They propose a soft prompts architecture with prompt pre-training and prompt fine-tuning paradigm to support few-shot abstractive summarization. |
| Outcome: | The proposed model outperforms Prompt Tuning and Profix-Tuning on CNN/DailyMail and XSum datasets and outperfies Profix Tuning by a large margin. |
DP-MLM: Differentially Private Text Rewriting Using Masked Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text privatization using Differential Privacy rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. |
| Approach: | They propose a method for differentially private text rewriting using masked language models to rewrite a text one token at a time. |
| Outcome: | The proposed method preserves utility at lower levels, compared to previous methods relying on autoregressive models with a decoder. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment (2023.acl-long)
Copied to clipboard
| Challenge: | Existing cognitive stimulation systems lack data on how to integrate emotional support and therapy principles into chit-chat dialogue systems. |
| Approach: | They propose a multi-source knowledge fusion method for CS dialogue to generate open-ended responses guided by the therapy principle and emotional support strategy. |
| Outcome: | The proposed method generates open-ended responses guided by the therapy principle and emotional support strategy of the target response. |
Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation (P19-1)
Copied to clipboard
| Challenge: | Disentangling the content and style in the latent space is prevalent in text style transfer . recurrent neural networks (RNN) based encoder and decoder cannot deal with the long-term dependency . |
| Approach: | They propose a style transformer which disentangles style information in latent space . they propose encoding and decoding methods that disentangle style information . |
| Outcome: | The proposed method can achieve better style transfer and better content preservation. |
Learning to Control the Fine-grained Sentiment for Story Ending Generation (P19-1)
Copied to clipboard
| Challenge: | Existing studies focus on controlling the sentiment of story endings. |
| Approach: | They propose a generic and novel framework which controls fine-grained sentiment intensity for automatic story ending generation without manually annotating sentiment labels. |
| Outcome: | The proposed framework can generate story endings which meet the given sentiment intensity better. |
Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to EL have been shown to be effective for both Entity Disambiguation and Entity Linking, but they suffer from high computational cost due to a complex (deep) decoder and the need for training on a large amount of data. |
| Approach: | They propose a method that parallelizes autoregressive linking across all potential mentions and relies on a shallow and efficient decoder. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on the English dataset AIDA-CoNLL and is >70 times faster and more accurate than the previous generative method. |
Semi-Supervised Semantic Dependency Parsing Using CRF Autoencoders (2020.acl-main)
Copied to clipboard
| Challenge: | Semantic dependency parsing allows words to have multiple dependency heads, resulting in graph-structured representations. |
| Approach: | They propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework. |
| Outcome: | The proposed model improves over the baseline model and is arc-factored. |
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Despite its success, multilingual neural machine translation suffers from the off-target issue, where the translation is in the wrong language. |
| Approach: | They propose a language-aware vocabulary sharing algorithm that can be used to increase the lexical distance between languages by isolating the vocab of different languages in the decoder. |
| Outcome: | The proposed algorithm reduces off-target rate for 90 translation tasks from 29% to 8%, while improving overall BLEU score by an average of 1.9 points without extra training cost or sacrificing the supervised directions’ performance. |
Neural Topic Modeling based on Cycle Adversarial Training and Contrastive Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Neural topic models have been widely used to extract common topics across documents. |
| Approach: | They propose a framework to apply contrastive learning directly to the decoder . they propose 'self-supervised' contrastive loss to make the generator capture similar topic information . |
| Outcome: | The proposed framework outperforms baselines on four benchmark datasets. |
An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation (D19-1)
Copied to clipboard
| Challenge: | Existing methods to generate paraphrases are not trivial and often fail in practice. |
| Approach: | They propose to use imitation learning to boost the performance of generating paraphrases by using a pointer-generator model. |
| Outcome: | The proposed model outperforms the state-of-the-art methods on the benchmark datasets. |
Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to unsupervised style transfer lack fine-grained control of the influence from the target style. |
| Approach: | They propose a model that exploits the relevance of each output word to the target style . they pretrain a style classifier and train an attentional Seq2seq model to reconstruct input sentences . |
| Outcome: | The proposed model achieves state-of-the-art performance in terms of transfer accuracy and content preservation. |
Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generating additional parallel sentences are aimed at expanding the support of the empirical data distribution by generating new sentence pairs that contain infrequent words. |
| Approach: | They propose to use data augmentation techniques to generate additional parallel sentences by reversing the order of the target sentence to produce unfluent target sentences. |
| Outcome: | The proposed approach improves on six low-resource translation tasks and the baseline and over DA methods. |
Parallel Refinements for Lexically Constrained Text Generation with BART (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work injects lexical constraints into the output, which generates generic or ungrammatical sentences and has high computational complexity. |
| Approach: | They propose a model that incorporates pre-specified keywords into the output to control the generated text. |
| Outcome: | The proposed model decomposes the generated text into two sub-tasks and improves the sentence quality. |
Hard-Coded Gaussian Attention for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has questioned the importance of multi-headed attention in achieving high translation quality. |
| Approach: | They develop a “hard-coded” attention variant without any learned parameters. |
| Outcome: | The proposed model reduces BLEU scores by adding a single learned cross attention head to an otherwise hard-coded Transformer. |
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
From Sights to Insights: Towards Summarization of Multimodal Clinical Documents (2024.acl-long)
Copied to clipboard
| Challenge: | a recent WHO report highlights a drastic doctor-to-patient ratio . telehealth is one of the most impactful sectors where AI advances can bring a significant revolution . |
| Approach: | They propose an image-guided encoder-decoder model that uses contextual attention to create detailed visual-guides for multimodal documents. |
| Outcome: | The proposed model outperforms state-of-the-art models on multimodal question and dialogue summarization tasks. |
Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Text error correction methods usually use the source (incorrect) sentence as encoder input and generate the target (correct) sentences through the decoder. |
| Approach: | They propose a method to correct errors in text sequences by randomly masking out the correct tokens in the source sentence. |
| Outcome: | The proposed method improves accuracy on Mandarin and English datasets with autoregressive and non-autoregressive generation models. |
Collaborative Learning of Bidirectional Decoders for Unsupervised Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for unsupervised text style transfer struggle to achieve high style conversion rate and low content loss. |
| Approach: | They propose a collaborative learning framework for unsupervised text style transfer using a pair of bidirectional decoders. |
| Outcome: | The proposed framework achieves strong empirical results on style compatibility and content preservation. |
PASTE: A Tagging-Free Decoding Framework Using Pointer Networks for Aspect Sentiment Triplet Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for tagging opinion triplets fail to capture the strong interdependence between the three opinion factors, whereas grid tabbing fails to capture span-level semantics while predicting sentiment between an aspect-opinion pair. |
| Approach: | They propose a tagging-free approach to extracting opinion triplets using a pointer network decoding framework that captures the interdependence between the three elements of an opinion triple. |
| Outcome: | The proposed architecture captures the interdependence between the aspect and opinion triplets while predicting their connecting sentiment. |
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2023.findings-acl)
Copied to clipboard
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
| Challenge: | Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model . however, the architecture used for FiD was not designed for retrieval augmented models . |
| Approach: | They propose to make FiD a modified retrieval-augmented language model with a large decoder and memory bandwidth constraints to alleviate memory bandwidth limitations. |
| Outcome: | The proposed architecture outperforms existing models on knowledge-intensive tasks even on large models on many knowledge-based tasks. |
UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on decoding word-level fMRI volumes from a restricted vocabulary. |
| Approach: | They propose an open-vocabulary task to bridge fMRI time series and human language . they use a pre-trained language model to construct a robust encoder for cognitive signals . |
| Outcome: | The proposed task bridges fMRI time series and human language with a baseline model. |
Revisiting Non-Autoregressive Translation at Scale (2023.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on two advanced NAT models show scaling can improve translation performance. |
| Approach: | They empirically examine the impact of scaling on NAT behaviors on a large-scale WMT dataset. |
| Outcome: | The proposed model can achieve comparable performance with the scaling model while maintaining the superiority of decoding speed with standard NAT models. |
On Pursuit of Designing Multi-modal Transformer for Video Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video grounding are not end-to-end, i.e., they rely on time-consuming post-processing steps to refine predictions. |
| Approach: | They propose an end-to-end multi-modal Transformer model that uses two encoders and a cross-modal decoder for grounding prediction. |
| Outcome: | The proposed model is 4.9% faster than existing models and is based on a set of encodings and decoders. |
Improving Role-Oriented Dialogue Summarization with Interaction-Aware Contrastive Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for encoding dialogues do not capture interaction information between roles, thus ignore interaction-related key information. |
| Approach: | They propose a contrastive learning based interaction-aware model for the role-oriented dialogue summarization namely CIAM and use it to train the decoder to learn role-level interaction. |
| Outcome: | The proposed model captures interaction information between different roles and produces informative summaries on two public datasets. |
DisCo_Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec (2026.acl-long)
Copied to clipboard
Tao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui, Yong Ma, Yingying Gao, Chao Deng, Shilei Zhang, Junlan Feng
| Challenge: | DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. |
| Approach: | They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens . |
| Outcome: | DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. |
PIXAR: Auto-Regressive Language Modeling in Pixel Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows the possibility of building open-vocabulary large language models that operate on pixel representations. |
| Approach: | They propose a pixel-based autoregressive LLM that performs generative tasks . they propose an adversarial pretraining stage that improves the readability and accuracy of PIXAR by 8.1 on LAMBADA and 8.5 on bAbI. |
| Outcome: | The proposed model performs free-form generative tasks while keeping the number of parameters on par with previous encoder-decoder models. |
Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages (2025.acl-long)
Copied to clipboard
| Challenge: | XLM-R and mBART have advanced multilingualism in NLP, but low-resource languages such as Tibetan, Uyghur, Kazakh, and Mongolian are underserved. |
| Approach: | They propose a framework for adapting multilingual encoders to text generation in extremely low-resource languages by reusing the weights between the encoder and the decoder. |
| Outcome: | The proposed framework performs better on various downstream tasks even when compared with much larger models. |
MccSTN: Multi-Scale Contrast and Fine-Grained Feature Fusion Networks for Subject-driven Style Transfer (2024.lrec-main)
Copied to clipboard
| Challenge: | Stylistic style transfer is an important part of the image processing field . due to the low semantic similarity between the original image and the style image, many fine-grained style features are discarded. |
| Approach: | They propose a new style representation and transfer framework that can be adapted to existing image style transfers. |
| Outcome: | The proposed framework can be adapted to existing image style transfers. |
GenKIE: Robust Generative Multimodal Document Key Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Key information extraction (KIE) is a key application for information retrieval and text mining. |
| Approach: | They propose a novel generative end-to-end model, named GenKIE, to address the KIE task. |
| Outcome: | The proposed model generalizes over different types of documents and achieves state-of-the-art results. |
Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in acquiring diverse knowledge, making them highly effective across a wide range of tasks. |
| Approach: | They propose a flipped knowledge distillation paradigm where LLM learns from SLM . they propose to reinterpret LLMs as encoder-decoder models using LoRA . |
| Outcome: | The proposed model has been deployed in an online application environment and validated on financial and healthcare benchmarks and real-world applications. |
Read As Human: Compressing Context via Parallelizable Close Reading and Skimming (2026.acl-long)
Copied to clipboard
Jiwei Tang, Shilei Liu, Zhicheng Zhang, Qingsong Lv, Runsong Zhao, Tingwei Lu, Langming Liu, Haibin Chen, Yujin Yuan, Hai-Tao Zheng, Wenbo Su, Bo Zheng
| Challenge: | Existing task-aware methods require loading the entire input sequence at once for compression, which suffer from computational inefficiency. |
| Approach: | They propose a framework that adopts an adaptive hybrid reading strategy to reduce computational inefficiency and redundant information in long-context scenarios. |
| Outcome: | Experiments show that RAM outperforms baselines on multiple question answering and summarization benchmarks while delivering up to a 12x speedup on long inputs. |
Masked Diffusion Captioning for Visual Feature Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Unlike autoregressive captioning, the strength of the visual learning signal in MDC does not depend on each token’s position in the sequence, reducing the need for auxiliary objectives. |
| Approach: | a decoder conditioned on visual features is trained to reconstruct the original text. |
| Outcome: | masked diffusion captioning (MDC) is a form of image-conditioned captioning that can be applied to visual tasks. |
Interpreting Style Representations via Style-Eliciting Prompts (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has attempted to explain learning of style representations by generating natural language descriptions with large language models (LLMs) conditioned on input text. |
| Approach: | They propose a framework for interpreting style representations through style-eliciting prompts by prompting an LLM to generate text conditioned on these features. |
| Outcome: | The proposed framework outperforms baselines that directly prompt LLMs with target text, and achieves superior performance in both style description and style imitation. |