Papers with encoders
Domain Adaptation of Image Encoder for Multimodal Manga Translation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing machine translation systems lack sufficient manga comprehension capabilities when utilizing image information. |
| Approach: | They propose a domain-adapted image encoder training method for manga . the method trains encoders to acquire visual features that consider the structural and sequential characteristics of the manga based on a Japanese-English translation task. |
| Outcome: | The proposed method improves translation evaluation metrics in Japanese-English translation task compared to the conventional method . |
A Frustratingly Easy Approach for Entity and Relation Extraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on end-to-end relation extraction models combine two tasks: named entity recognition and relation extraction. |
| Approach: | They propose a pipelined approach for entity and relation extraction that uses two independent encoders to construct the relation model. |
| Outcome: | The proposed approach achieves an 8.16 speedup with a slight reduction in accuracy on standard benchmarks. |
Plane Geometry Problem Solving with Multi-modal Reasoning: A Survey (2026.findings-eacl)
Copied to clipboard
| Challenge: | Plane geometry problem solving has gained significant attention as a benchmark to assess the multi-modal reasoning capabilities of large vision-language models. |
| Approach: | They present a systematic review of existing work in PGPS and summarize their results. |
| Outcome: | The proposed frameworks are compared with existing frameworks and analyze them according to their architectural designs. |
Machine Reading, Fast and Slow: When Do Models “Understand” Language? (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models of reading comprehension score highly on NLU benchmarks, but they are often 'read fast', i.e. rely on shallow patterns. |
| Approach: | They propose a definition for the reasoning steps expected from a system that would be 'reading slowly' they compare that behavior with five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. |
| Outcome: | The proposed model is compared with five models of the BERT family of various sizes, and compared using saliency scores and counterfactual explanations. |
Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)
Copied to clipboard
Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang
| Challenge: | Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks. |
| Approach: | They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator. |
| Outcome: | The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings. |
Mention Flags (MF): Constraining Transformer-based Text Generators (2021.acl-long)
Copied to clipboard
| Challenge: | Constrained decoding algorithms produce hypotheses satisfying all constraints, but they are computationally expensive and can lower the generated text quality. |
| Approach: | They propose a Mention Flag mechanism which traces whether lexical constraints are satisfied in outputs of an S2S decoder. |
| Outcome: | The proposed models maintain higher constraint satisfaction and text quality than baseline models and other constrained decoding algorithms. |
INDUS: Effective and Efficient Language Models for Scientific Applications (2024.emnlp-industry)
Copied to clipboard
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grezes, Megan Ansdell, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Lima, Panagiotis Vagenas, S. Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Tsengdar Lee
| Challenge: | Large language models trained on general domain corpora showed remarkable results on natural language processing tasks. |
| Approach: | They develop a suite of large language models trained on general domain corpora that address NLP tasks and smaller versions of them created using knowledge distillation. |
| Outcome: | The proposed models outperform general-purpose and domain-specific encoders on new and existing tasks and in industrial settings. |
Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale two-stream pre-trained models like CLIP have achieved tremendous success in image-text retrieval. |
| Approach: | They propose a cross-modal framework for image-text retrieval using two-stream pre-trained models . they embed images and texts into instance representations with two separate encoders . experimental results on MSCOCO and Flickr30k reveal the effectiveness of their framework . |
| Outcome: | The proposed framework improves image-text retrieval performance on two popular cross-modal retrieval benchmarks. |
Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification (2025.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that sentiment analysis datasets lack context in which an opinion was expressed and are limited by a few emotion categories. |
| Approach: | They propose to ground an LLM-based model into a corpus of narratives to generate stories-character-centered utterances with unique contexts over 28 emotion classes. |
| Outcome: | The proposed model generates non-repetitive story-character-centered utterances with unique contexts over 28 emotion classes. |
Graph-based Filtering of Out-of-Vocabulary Words for Encoder-Decoder Models (P18-3)
Copied to clipboard
| Challenge: | Encoder-decoder models employ words that are frequently used in the training corpus but may still include noisy words. |
| Approach: | They propose a method for selecting more suitable words for learning encoders by utilizing co-occurrence information. |
| Outcome: | The proposed method outperforms the baseline method in Japanese-to-English translation and grammatical error correction tasks with an F-measure of 1.48 points higher. |
Inducing Language-Agnostic Multilingual Representations (2021.starsem-1)
Copied to clipboard
| Challenge: | Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world, but they currently require large pretraining corpora or access to typologically similar languages. |
| Approach: | They propose to remove language identity signals from multilingual embeddings by re-aligning vector spaces of target languages to a pivot source language and removing language-specific means and variances. |
| Outcome: | The proposed approaches reduce cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages. |
Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are highly sensitive to end-of-input artifacts in fine-tuning and inference data, e.g., whether input sequences end with punctuation or newline characters. |
| Approach: | They propose to convert generative LVLMs into vision-language encoders via contrastive learning objectives and use supervised contrastive objectives to train them. |
| Outcome: | The proposed approach improves visual and text representations and improves retrieval and (semantic) similarity tasks. |
Joint Aspect Extraction and Sentiment Analysis with Directional Graph Convolutional Networks (2020.coling-main)
Copied to clipboard
| Challenge: | End-to-end aspect-based sentiment analysis uses two sub-tasks to extract aspect terms . experimental results demonstrate the effectiveness of our approach on all datasets . |
| Approach: | They propose to combine aspect extraction and sentiment analysis with encoding syntactic information to improve model's representation of input sentences. |
| Outcome: | The proposed approach achieves state-of-the-art on three benchmark datasets. |
Cross-Thought for Sentence Encoder Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models to pretrain sentence encoders with large unlabeled corpus are lacking in linguistic information retrieval. |
| Approach: | They propose a novel approach to pre-training sequence encoder using transformers . they propose to train a Transformer-based sequence encoded over a large set of short sequences based on a set of masked words . |
| Outcome: | The proposed approach outperforms state-of-the-art encoders on hotpotQA by improving intermediate information retrieval performance. |
Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis (2022.aacl-main)
Copied to clipboard
| Challenge: | Multimodal aspect-based sentiment analysis (MABSA) aims to extract aspect terms from text and image pairs, and then analyze their corresponding sentiment. |
| Approach: | They propose a dual-encoder transformer with cross-modal alignment to extract aspect terms from text and image pairs and then analyze their corresponding sentiments. |
| Outcome: | The proposed approach outperforms existing methods on two benchmarks. |
AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have shown great power in natural language understanding . fine-grained tokenizations have advantages and disadvantages for learning of pre-tried models . |
| Approach: | They propose a pretrained language model based on both fine-grained and coarse-grain tokenizations . they propose to use both tokenization techniques to learn pre-trained models . |
| Outcome: | The proposed model outperforms BERT on benchmark datasets for Chinese and English . it can perform better with the same computational cost as BERT, the authors show . |
Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that attention cannot be considered as a faithful explanation across encoders and tasks. |
| Approach: | They propose a new family of Task-Scaling mechanisms that scale attention weights across tasks and two attention mechanisms. |
| Outcome: | The proposed models improve explanation faithfulness across two attention mechanisms, five encoders and five text classification datasets without sacrificing predictive performance. |
UER: An Open-Source Toolkit for Pre-training Models (D19-3)
Copied to clipboard
Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du
| Challenge: | Existing work on pre-training models have shown that it is important to use a framework to deploy various pre- training models efficiently. |
| Approach: | They propose an assemble-on-demand pre-training toolkit that assembles pre-trained models on demand and encapsulates them with rich modules. |
| Outcome: | The proposed framework can reproduce state-of-the-art models or develop models that remain unexplored. |
Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding (2023.acl-industry)
Copied to clipboard
Jonathan Hueser, Judith Gaspers, Thomas Gueudre, Chandana Prakash, Jin Cao, Daniil Sorokin, Quynh Do, Nicolas Anastassacos, Tobias Falke, Turan Gojayev
| Challenge: | Larger encoders can improve accuracy for spoken language understanding (SLU) but are difficult to use given the inference latency constraints of online systems. |
| Approach: | They propose to use a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU. |
| Outcome: | The proposed encoders achieve state-of-the-art performance on numerous NLP tasks. |
Bootstrapping a Crosslingual Semantic Parser (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in semantic parsing are limited to English but professional translation can be prohibitively expensive. |
| Approach: | They adapt a semantic parser trained on a single language to new languages and multiple domains with minimal annotation. |
| Outcome: | The proposed approach achieves parsing accuracy within 2% of translation using only 50% of training data. |
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity (2022.tacl-1)
Copied to clipboard
| Challenge: | averaging hard attention is used to recognize formal languages that UHAT and GUHAT cannot recognize. |
| Approach: | They analyze three formal Transformer encoders that differ in the form of their self-attention mechanism . they find that UHAT and GUHAT Transformers can only recognize formal languages in AC0 . |
| Outcome: | The proposed models can recognize languages that UHAT and GUHAT cannot . the proposed models are based on the DYCK and PARITY languages . |
Representing Numbers in NLP: a Survey and a Vision (2021.naacl-main)
Copied to clipboard
| Challenge: | Numeracy is an essential skill for language understanding since numbers are often interspersed in text. |
| Approach: | They propose a comprehensive taxonomy of tasks and methods to represent numbers in text . they synthesize best practices for representing numbers in texts and articulate a vision for holistic numeracy . |
| Outcome: | The proposed model synthesizes best practices for representing numbers in text . it argues that the model is more effective than other approaches . |
End-to-End Argument Mining as Biaffine Dependency Parsing (2021.eacl-main)
Copied to clipboard
| Challenge: | Argumentation mining (AM) is a new field of research that uses dependency parsing to analyse arguments. |
| Approach: | They propose a neural end-to-end approach to argument mining based on dependency parsing . their model is biaffine dependency parsed and outperforms the current state-of-the-art . |
| Outcome: | The proposed model outperforms the state-of-the-art in component identification and relation identification. |
Decoder-Only LLMs can be Masked Auto-Encoders (2025.acl-short)
Copied to clipboard
Dan Qiao, Yuan Gao, Zheming Yang, Di Yang, Ziheng Wu, Pengcheng Lu, Minghui Qiu, Juntao Li, Min Zhang
| Challenge: | Modern NLP workflows require different models for generation and embedding tasks. |
| Approach: | They propose a method that transforms an LLM into a Uni-Directional Masked Auto-Encoder. |
| Outcome: | The proposed method achieves state-of-the-art under unsupervised conditions with merely 100 training steps. |
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024.findings-eacl)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | Embeddings in transformers encode vast amounts of linguistic nuances and patterns. |
| Approach: | They investigate the anisotropy dynamics and intrinsic dimension of embeddings in transformers . they found that transformer decoders exhibit a bell-shaped anisotropie profile . |
| Outcome: | The investigated embeddings exhibit a bell-shaped curve with the highest anisotropy concentrations in the middle layers . the intrinsic dimension increases in the initial phases of training, indicating an expansion into higher-dimensional space. |
Permutation Invariant Strategy Using Transformer Encoders for Table Understanding (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for encoding text in tables require additional training and require additional pretraining. |
| Approach: | They propose a novel encoding strategy that preserves the critical property of permutation invariance across rows or columns. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on three table interpretation tasks: column type annotation, relation extraction, and entity linking. |
Medical Spoken Named Entity Recognition (2025.naacl-industry)
Copied to clipboard
Khai Le-Duc, David Thulke, Hung-Phong Tran, Long Vo-Dang, Khai-Nguyen Nguyen, Truong-Son Hy, Ralf Schlüter
| Challenge: | Named Entity Recognition (NER) aims to extract named entities from speech and categorise them into types like person, location, organization, etc. |
| Approach: | They present a spoken NER dataset in the medical domain using pre-trained models that are encoder-only and sequence-to-sequence. |
| Outcome: | The dataset is the largest spoken NER dataset in the world regarding the number of entity types, featuring 18 distinct types. |
End-to-End Neural Bridging Resolution (2022.coling-1)
Copied to clipboard
| Challenge: | state-of-the-art resolvers for bridging resolution are weaker than entity coreference resolution. |
| Approach: | They evaluate bridging resolvers in an end-to-end setting and strengthen them with better encoders . they also try to gain a better understanding of them through perturbation experiments . |
| Outcome: | bridging resolvers are evaluated in an end-to-end setting and strengthened with better encoders . bribridging resolution is the task of identifying briating anaphors and linking them to their antecedents - a paper by the journal bribing resolution argues . |
Line Graph Enhanced AMR-to-Text Generation with Mix-Order Graph Attention Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Existing graph-to-sequence approaches use graph neural networks as encoders, but they lack the structure information needed to translate AMR into the graph-based data. |
| Approach: | They propose a graph-to-sequence task which aims to recover natural language from Abstract Meaning Representations (AMR) they adopt graph attention networks with higher-order neighborhood information to explore the edge relations in AMR graphs. |
| Outcome: | The proposed framework achieves state-of-the-art performance on English AMR benchmark datasets and is able to translate the AMR semantics into the natural language. |
A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection tasks (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Virtual assistants often can handle only a limited scope of intents. |
| Approach: | They propose to combine out-of-scope (OOS) rejection with intent classification . they propose to regularize cross-entropy loss with an in-scope embedding reconstruction loss . |
| Outcome: | The proposed method achieves a 1-4% improvement in the area under the precision-recall curve for rejecting out-of-sample instances without compromising intent classification performance. |
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023.findings-acl)
Copied to clipboard
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu
| Challenge: | a new method for embedding text is developed for tasks that require specialized encoders . INSTRUCTOR is a single embedder that can generate text embeddables tailored to different tasks and domains based on instruction finetuning . |
| Approach: | They introduce a new method for computing text embeddings given task instructions . they first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture . |
| Outcome: | The proposed method improves on 70 embedding evaluation tasks with fewer parameters than the previous best model. |
MCP: Self-supervised Pre-training for Personalized Chatbots with Multi-level Contrastive Sampling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on generating implicit user profiles from the user’s dialogue history, thus it suffers from data sparsity and performance degradation. |
| Approach: | They propose a self-supervised learning framework MCP for capturing better representations from users’ dialogue history for personalized chatbots. |
| Outcome: | The proposed model improves on two real-world datasets. |
EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal models that depend on encoders like CLIP or ImageBind need ample amounts of training data to bridge modalities. |
| Approach: | They propose an efficient model that leverages bidirectional conditional diffusion model to foster more efficient modality interactions. |
| Outcome: | The proposed model is able to train a projection layer linking an LLM and an adapter to align the LLM’s text space with the bidirectional diffusion model. |
Multi-Operational Mathematical Derivations in Latent Space (2024.naacl-long)
Copied to clipboard
| Challenge: | Using a symbolic engine, we investigate the possibility of approximating multiple mathematical operations in latent space for expression derivation. |
| Approach: | They propose to model mathematical operations as explicit geometric transformations by leveraging a symbolic engine and a large-scale dataset. |
| Outcome: | The proposed paradigms can be used to approximate multiple mathematical operations in latent space, while discriminating the conclusions for a single operation is achievable in the original expression encoder. |
FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Tables store rich numerical data, but numerical reasoning over tables is still a challenge. |
| Approach: | They propose a spreadsheet formula is a valuable supervision for numerical reasoning in tables. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three representative datasets of formula prediction, question answering, and cell type classification. |
ConvFiT: Conversational Fine-Tuning of Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, Tsung-Hsien Wen
| Challenge: | Existing Transformer-based language models (LMs) are not effective as sentence encoders when used off-the-shelf. |
| Approach: | They propose a method which turns a pretrained LM into a universal conversational encoder and task-specialised sentence encoder. |
| Outcome: | The proposed framework achieves state-of-the-art ID performance across the board with particular gains in the most challenging, few-shot setups. |
Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for Word Sense Disambiguation are not uniformly distributed on rare or unseen senses. |
| Approach: | They propose a bi-encoder model that embeds the target word with its context and the dictionary definition, or gloss, of each sense. |
| Outcome: | The proposed model outperforms previous state-of-the-art models on English all-words WSD, with a 31.1% error reduction on less frequent senses over prior work. |
Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful on a wide range of NLP tasks . however, contextual representations from pre-trated models contain entangled semantic and syntactic information. |
| Approach: | They propose a semantic sentence embedding model that disentangles semantics and syntax from pre-trained models. |
| Outcome: | The proposed model outperforms state-of-the-art models on unsupervised semantic similarity tasks. |
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained Masked Language Models are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. |
| Approach: | They propose a contrastive learning technique which turns pretrained MLMs into effective universal lexical and sentence encoders without additional data. |
| Outcome: | The proposed technique can turn MLMs into effective universal lexical and sentence encoders even without additional data. |
Trainable Hard Negative Examples in Contrastive Learning for Unsupervised Abstractive Summarization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for contrastive learning rely on manual negative examples and are poor in quality and adaptability during training. |
| Approach: | They propose a framework that learns trainable negative examples for contrastive learning in unsupervised abstractive summarization by combining a negative example network and a representation network. |
| Outcome: | The proposed approach eliminates the need for manual negative example design and improves on two benchmark datasets. |
Personalized Review Generation By Expanding Phrases and Attending on Aspect-Aware Representations (P18-2)
Copied to clipboard
| Challenge: | Existing systems that use user and item identity as inputs for review generation are lacking in the field of natural language processing. |
| Approach: | They propose an encoder-decoder framework that generates personalized reviews by expanding short phrases provided as input to the system. |
| Outcome: | The proposed model learns representations capable of generating coherent and diverse reviews. |
The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for summarization systems measure semantic overlap between a system summary and a human reference on word-string level. |
| Approach: | They propose to use distributed representations to evaluate system summary and human reference on word-string level. |
| Outcome: | The proposed representations outperform ROUGE on recent corpora but are less good on test data used in previous studies. |
Implicit n-grams Induced by Recurrence (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that self-attention based models have limitations on modeling sequential transformations. |
| Approach: | They propose to extract some explainable features from trained RNNs that are reminiscent of classical n-grams features. |
| Outcome: | The proposed models can model interesting linguistic phenomena such as negation and intensification. |
Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER (2025.emnlp-industry)
Copied to clipboard
| Challenge: | nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance. |
| Approach: | They propose a framework that enables a single shared encoder backbone with modular adapters. |
| Outcome: | The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint. |
“All that Glitters”: Techniques for Evaluations with Unreliable Model and Human Annotations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using standard metrics in the presence of poor labels masks label and model quality . evaluation techniques accounting for unreliable labels reveal important flaws, including spurious correlations and nonrandom racial biases . |
| Approach: | They analyze human labels, GPT model ratings, and transformer encoder model ratings . they show that standard metrics in the presence of poor labels mask label and model quality . |
| Outcome: | The proposed methods mask label and model quality even in the presence of poor models. |
An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition (2023.acl-short)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is the task to detect and classify entity spans in text. |
| Approach: | They propose to use Convolutional Neural Network to model spatial relations in NER . they use three commonly used nested NER datasets to compare their results . |
| Outcome: | The proposed model outperforms several proposed methods with the same pre-trained encoders in three nested NER datasets. |
Genre Separation Network with Adversarial Training for Cross-genre Relation Extraction (D18-1)
Copied to clipboard
| Challenge: | Existing methods to extract genre-specific and genre-agnostic features require great human effort. |
| Approach: | They propose to use two encoders to explicitly extract genre-specific and genre-agnostic features. |
| Outcome: | The proposed approach outperforms the state-of-the-art by 1.7% on three distinct genres. |
Generating Weather Comments from Meteorological Simulations (2021.eacl-main)
Copied to clipboard
Soichiro Murakami, Sora Tanaka, Masatsugu Hangyo, Hidetaka Kamigaito, Kotaro Funakoshi, Hiroya Takamura, Manabu Okumura
| Challenge: | generating weather-forecast comments from meteorological simulations is labor intensive and requires a solid knowledge of meteorology. |
| Approach: | They propose a data-to-text model that incorporates three types of encoders for numerical forecast maps, observation data, and meta-data. |
| Outcome: | The proposed model performs best against baselines in terms of informativeness . it is available online and the results are available to the general public . |
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2025.acl-long)
Copied to clipboard
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, Iacopo Poli
| Challenge: | Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks compared to larger decoder models. |
| Approach: | They introduce a new transformer model, ModernBERT, which brings modern model optimizations to encoder-only transformer models. |
| Outcome: | The proposed model improves on the BERT transformer model and is faster and more memory efficient than the older models. |
Pre-training Entity Relation Encoder with Intra-span and Inter-span Information (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models do not handle text spans and relation among text span pairs. |
| Approach: | They propose to integrate span-related information into pre-trained encoder for entity relation extraction task. |
| Outcome: | The proposed pre-training method outperforms distantly supervised pre-trained models on two entity relation extraction benchmark datasets. |
Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders (2020.emnlp-main)
Copied to clipboard
| Challenge: | Named entity recognition and relation extraction are two important fundamental problems. |
| Approach: | They propose to design two separate encoders to capture two different types of information in the representation learning process. |
| Outcome: | The proposed methods show significant improvements on standard datasets. |
Case-Based Abductive Natural Language Inference (2022.coling-1)
Copied to clipboard
| Challenge: | Recent approaches for multi-hop inference construct explanations considering each test case in isolation, but semantic drift causes wrong conclusions. |
| Approach: | They propose an abductive framework for multi-hop NLI exploring the retrieve-reuse-refine paradigm in Case-Based Reasoning. |
| Outcome: | The proposed model can be integrated with sparse and dense pre-trained encoders to improve multi-hop inference, or adopted as an evidence retriever for Transformers. |
Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical knowledge is encoded in large pre-trained language models (LMs) this is done through supervised classification tasks to predict the grammamatical properties of a span using only the token representations coming from the LM encoder. |
| Approach: | They propose to use a supervised 'edge probing' task to detect grammatical knowledge in large pre-trained language models (LMs) this is done by encoding grammamatical properties using only token representations coming from the LM encoder. |
| Outcome: | The proposed model performs well when fine-tuned or in adversarial situations where the model is forced to learn wrong correlations. |
ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models lack feature representations that capture the deep semantics of language and sensitivity to minor input variations, resulting in significant changes in the generated text. |
| Approach: | They propose an end-to-end model architecture called ASEM that performs emotion analysis on top of sentiment analysis for open-domain chatbots. |
| Outcome: | The proposed model outperforms existing models for generating empathetic embeddings, providing e-mpathetic and diverse responses. |
Self-supervised Rewiring of Pre-trained Speech Encoders:
Towards Faster Fine-tuning with Less Labels in Speech Processing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech encoders have facilitated great success across various speech processing tasks, but fine-tuning them for downstream tasks requires large training data to converge or to achieve state-of-the-art. |
| Approach: | They propose to rewire pre-trained speech encoders to improve their representation space without task-specific labels by neutrally synthesising audio inputs and frame masking. |
| Outcome: | The proposed model shows consistent improvement in isotropy in the representation space on 6 speech processing tasks. |
RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech Transformers in speech translation systems have facilitated state-of-the-art (SotA) results, but their computational cost is high. |
| Approach: | They propose a Reducer Adaptor block that could be seamlessly integrated within any Transformer-based speech encoding architecture. |
| Outcome: | The proposed Reducer Adaptor block outperforms the existing SotA architecture by an average of 0.68 BLEU score on 8 language pairs from Must-C. |
Encoders Help You Disambiguate Word Senses in Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Neural machine translation models can perform word sense disambiguation (WSD) however, it is unclear which component dominates the process of disambiguating words. |
| Approach: | They evaluate hidden states and investigate distributions of self-attention in NMT encoders and decoders to disambiguate word senses. |
| Outcome: | The proposed model outperforms encoder hidden states on large datasets . the model outpersforms decoders on large data sets . |
Deep Communicating Agents for Abstractive Summarization (N18-1)
Copied to clipboard
| Challenge: | Empirical results show that multiple communicating agents produce a better summary than extractive summarization. |
| Approach: | They propose an encoder-decoder architecture that uses deep communicating agents to represent a long document for abstractive summarization. |
| Outcome: | Empirical results show that multiple communicating agents produce a better summary than baselines. |
Encoding Conversation Context for Neural Keyphrase Extraction from Microblog Posts (N18-1)
Copied to clipboard
| Challenge: | Existing keyphrase extraction methods suffer from data sparsity problem when conducted on short and informal texts. |
| Approach: | They propose a neural keyphrase extraction framework for microblog posts that takes conversation context into account and uses four types of neural encoders to represent conversation context. |
| Outcome: | The proposed framework outperforms state-of-the-art keyphrase extraction methods on Twitter and Weibo datasets. |
Vec2Sent: Probing Sentence Embeddings with Natural Language Generation (2020.coling-main)
Copied to clipboard
| Challenge: | a new unsupervised probing task is able to retrieve black-box sentence embeddings . a variety of problems surround probing tasks, including manual construing . |
| Approach: | They propose a method to generate black-box sentence embeddings by conditionally generating from them . they also illustrate how the language generated from different encoders differs . |
| Outcome: | The proposed probing task improves the performance of black-box sentence embeddings . the proposed task is based on a conditional natural language generation approach . |
Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new study aims to extend multilingual representation learning beyond the hundred most frequent languages . current work on multilingual sentence representations has focused on training one model which handles all languages of interest . |
| Approach: | They propose a teacher-student approach to extend existing monolingual sentence embedding space to new languages. |
| Outcome: | The proposed model outperforms the original LASER encoder in 44 African languages . the model can be used to train multiple languages and learn new languages if they have the same training data . |
MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing UMR methods can be broadly divided into two categories: early-fusion approaches, such as Marvel, which projects visual features into the language model space for integrating with text modality, and late-fusion methods, such UniVL-DR, which encode visual and textual inputs using separate encoders and obtain fused embeddings through addition. |
| Approach: | They propose to map different modalities into a shared embedding space for multi-modal retrieval. |
| Outcome: | Experiments on the WebQA+ and EVQA+ datasets show that MiMIC outperforms both early- and late-fusion approaches. |
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering (2023.acl-short)
Copied to clipboard
| Challenge: | Existing dense retrieval models are parameter-inefficient and underperform sparse counterparts. |
| Approach: | They propose a task-aware specialization for dEnse Retrieval architecture that enables parameter sharing by interleaving shared and specialized blocks in a single encoder. |
| Outcome: | The proposed architecture surpasses BM25 on questions and passages using 60% of the parameters as bi-encoder dense retrievers. |
Audiocite.net : A Large Spoken Read Dataset in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning methods for speech processing have proved difficult to apply to French due to the scarcity of large speech datasets. |
| Approach: | They present a corpus of 6,682 hours of audiobooks from 130 readers . they describe the creation process and final statistics of the corpus . |
| Outcome: | The proposed model based on the audiocite.net corpus, which contains 6,682 hours of audiobooks, was able to perform in 14k version. |
LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern automatic speech recognition systems rely on encoder-decoder architectures and their encoders are a critical bottleneck for efficient deployment due to high computational intensity. |
| Approach: | They propose a low-rank compression scheme for ASR encoders that leverages the strong low-ranked properties observed in intermediate activations and approximates linear transformations with a chain of low-Rank matrix multiplications. |
| Outcome: | The proposed method reduces inference costs while maintaining transcription accuracy while preserving low-rank properties observed in intermediate activations. |
Exploring Logically Dependent Multi-task Learning with Causal Inference (2020.emnlp-main)
Copied to clipboard
| Challenge: | Hierarchical multi-task learning models can utilize task dependencies by stacking encoders and outperform democratic ones. |
| Approach: | They propose a model that utilizes the labels of all lower-level tasks and a Gumbel sampling model to deal with cascading errors. |
| Outcome: | The proposed model outperforms democratic models on six out of seven subtasks and achieves state-of-the-art on the two English and one Chinese datasets. |
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (D19-1)
Copied to clipboard
| Challenge: | Stack Exchange has fewer than 160 user-labeled duplicates, and 25% have fewer. |
| Approach: | They propose a framework that combines sentence encoders with unlabeled data to solve the problem of duplicate question detection in Community Question Answering forums. |
| Outcome: | The proposed framework outperforms BM25, a single-view system and a supervised domain-adversarial DQD method on the CQADupStack corpus and on Stack Exchange forums. |
Beyond Coherence: Improving Temporal Consistency and Interpretability in Dynamic Topic Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing topic models capture bag-of-words statistics but lack semantic priors . interpretability remains shallow, relying on noisy top-word lists that obscure thematic clarity. |
| Approach: | They propose a variational framework to capture more faithful temporal trajectories . they propose to use entropy-regularized optimal transport to align entire topic constellations . |
| Outcome: | The proposed framework captures more faithful temporal trajectories and improves interpretability. |
BanglaAutoKG: Automatic Bangla Knowledge Graph Construction with Semantic Neural Graph Filtering (2024.lrec-main)
Copied to clipboard
| Challenge: | Bangla is underrepresented in KGs due to lack of comprehensive datasets, encoders, NER models, part-of-speech taggers, and lemmatizers. |
| Approach: | Bangla is underrepresented in KGs due to lack of comprehensive datasets, encoders, NER models, part-of-speech taggers, and lemmatizers. authors propose a framework that can automatically construct Bengali KG from any Bangla text. |
| Outcome: | The proposed framework can automatically construct Bengali KGs from any Bangla text. |
Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for Sequence Labeling require high-quality annotations, but imperfect annotations are relatively easy to obtain from crowdsourcing (noisy labels) Existing approaches to learn a model without knowing the underlying ground truth label sequences in the target domain are expensive and time-consuming. |
| Approach: | They propose a framework Consensus Network that can be trained on annotations from multiple sources. |
| Outcome: | The proposed framework improves on learning with crowd annotations and unsupervised cross-domain model adaptation in two practical settings. |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
Syntax-Infused Variational Autoencoder for Text Generation (P19-1)
Copied to clipboard
| Challenge: | Experimental results demonstrate the generative superiority of SIVAE on both reconstruction and targeted syntactic evaluations. |
| Approach: | They propose a syntax-infused variational autoencoder that integrates sentences with their syntactic trees to improve the grammar of generated sentences. |
| Outcome: | The proposed model improves the grammar of generated sentences by integrating sentences with syntactic trees. |
ZeroAE: Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text classification use only encoders or decoders that do not allow for the use of labels in unseen domains. |
| Approach: | They propose an autoencoder that encodes text into two disentangled spaces and decodes it to generate text with labels in the unseen domains. |
| Outcome: | The proposed model outperforms the existing methods in label-partially-unseen and label-fully-un-seeen scenarios and even outperfects the SOTA methods. |
MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for incorporating external attribute knowledge into deep neural networks are concatenating multiple attributes to word/text representation or treating them as biases to adjust attention distribution. |
| Approach: | They propose a multi-attribute BERT to incorporate external attribute knowledge into deep neural networks. |
| Outcome: | The proposed method outperforms existing models and models on three benchmark datasets. |
Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adaptive Retention is a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. |
| Approach: | They propose a probabilistic token selection mechanism that learns which representations to keep under a strict global budget M. |
| Outcome: | The proposed method reduces memory usage by 35–45% while improving throughput by 1.8. |
CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval (2026.findings-acl)
Copied to clipboard
Zhipeng Qian, Zihan Liang, Yufei Ma, Ben Chen, Huangyu Dai, Yiwei Ma, Jiayi Ji, Chenyi Lei, Han Li, Xiaoshuai Sun
| Challenge: | Existing approaches to search for images using single-modality are limited by representation space fragmentation. |
| Approach: | They propose a unified representation framework that achieves efficient query-target alignment . they introduce a multi-level Chain-of-Thought prompting strategy that guides MLMs to generate discriminative, semantically compatible captions for target images . |
| Outcome: | The proposed framework achieves efficient query-target alignment through synergistic components. |
Breaking Down Multilingual Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Multilingual training is an essential ingredient in machine translation systems . but it has different effects in different multilingual settings, such as many-to-one, one-tomany and many- to-many learning . |
| Approach: | They compare multilingual training settings with encoders and decoders initialized by multilingual learning . they find important attention heads for each language pair and compare their correlations during inference . |
| Outcome: | The proposed models outperform the best models for high-resource languages and one-to-many models for low-resourced languages. |
A Hybrid Detection and Generation Framework with Separate Encoders for Event Extraction (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work on event extraction tasks has been based on classification-based methods . a new generation-based method is being developed to extract event triggers and event arguments from plain text. |
| Approach: | They propose to use independent encoders to model event detection and event argument extraction, respectively, and use token-level features to precisely control the fusion between two encoder. |
| Outcome: | The proposed method avoids feature interference and achieves joint training . it is compared with other methods and achieved competitive results on standard benchmarks . |
FusionDTI: Fine-grained Binding Discovery with Token-level Fusion for Drug-Target Interaction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | despite advances in DTI models, models often struggle to capture fine-grained interactions between drugs and proteins. |
| Approach: | They propose a novel drug-target interaction model that uses a token-level module to learn fine-grained information for drug-target interactions. |
| Outcome: | The proposed model learns fine-grained information for drug-target interaction . it mitigates sequence fragment invalidation and incorporates the structure-aware vocabulary of target proteins . |
Improving English-Arabic Transliteration with Phonemic Memories (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing neural approaches to transliterate names from English to Arabic are limited and focus on leveraging the phonemic association between English and Arabic. |
| Approach: | They propose a model for English-Arabic transliteration using a memory module modeling the phonemic association between English and Arabic to guide the transliterations process. |
| Outcome: | The proposed model improves on EANames corpus, which better represents names in the general public than linked Wikipedia entries that are always names of famous people. |
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing studies on crosslingual transfer have focused on word-level information sharing, but words are not independent in sentences; their combinations form larger linguistic units, known as context. |
| Approach: | They propose to use orderagnostic models to transfer word order to distant languages . they train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages. |
| Outcome: | The proposed model performs better on languages with different word orders than on other languages. |
Rethinking Smoothness for Fast and Adaptable Entity Alignment Decoding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for integrating knowledge graphs rely on entity and relation embeddings . Fig. 1 shows how to decode knowledge graph in under 6 seconds . |
| Approach: | They propose a framework that only utilizes entity embeddings to decode knowledge graphs. |
| Outcome: | The proposed framework reconstructs KG representation by maximizing smoothness of entity embeddings. |
Contextual Text Style Transfer (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for text style transfer are limited by the lack of parallel data. |
| Approach: | They propose a task to translate a sentence into a desired style with its surrounding context taken into account. |
| Outcome: | The proposed model outperforms state-of-the-art methods across style accuracy, content preservation and contextual consistency metrics. |
deepQuest: A Framework for Neural-based Quality Estimation (C18-1)
Copied to clipboard
| Challenge: | Predicting Machine Translation (MT) quality has been limited to word and sentence-level prediction. |
| Approach: | They propose a framework that can generalize neural QE approaches to the level of documents. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on document-level quality estimates and is 40 times faster to train. |
Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations (2024.acl-long)
Copied to clipboard
| Challenge: | Vision-and-language models with separate encoders for each modality are limited in availability. |
| Approach: | They propose a multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation. |
| Outcome: | The proposed model outperforms models on English and low-resource languages. |
FOOL ME IF YOU CAN! An Adversarial Dataset to Investigate the Robustness of LMs in Word Sense Disambiguation (2024.emnlp-main)
Copied to clipboard
Mohamad Ballout, Anne Dedert, Nohayr Abdelmoneim, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger
| Challenge: | Word sense disambiguation (WSD) is a key task in natural language processing . however, these models struggle with recognizing semantic boundaries in adversarial contexts . |
| Approach: | They propose to use a coarse-grained WSD dataset to assess model robustness . they found that some models struggled to correctly disambiguate homonyms in adversarial contexts . |
| Outcome: | The proposed dataset includes four test sets to assess the robustness of language models in WSD tasks. |
MURAL: Multimodal, Multitask Representations Across Languages (2021.findings-emnlp)
Copied to clipboard
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, Jason Baldridge
| Challenge: | Image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. |
| Approach: | They propose a dual encoder that integrates image-text matching and translation pairs to solve two tasks by learning from billions of pairs. |
| Outcome: | The proposed encoder outperforms ALIGN's cross-modal retrieval performance on well-resourced languages and significantly improves on under-resource languages. |
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained Transformers are remarkably robust to pruning. |
| Approach: | They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance. |
| Outcome: | The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance. |
MaskLoRA: Low-Rank Subspace–Induced Token Masking for Efficient and Faithful Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | MASKLORA is a plug-and-play masking mechanism that can be used to mask lowrank subspaces. |
| Approach: | They propose a plug-and-play masking mechanism that transforms PEFT's lowrank subspace into a faithful token selector. |
| Outcome: | The proposed masking mechanism matches full-model accuracy while yielding 1.3-2.6 speedups. |
A Simple and Effective Unified Encoder for Document-Level Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for document-level machine translation use two separate encoders to model the source sentences and document- level contexts. |
| Approach: | They propose a unified encoder that can outperform existing models of dual-encoder models . they propose to use document-level contexts to model the interaction between the contexts and the source sentences . |
| Outcome: | The proposed model outperforms baseline models of dual-encoder models in terms of BLEU and METEOR scores. |
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances have been achieved in Visual Speech Recognition (VSR) despite the lack of data, there is no clear comparison between different types of decoders for certain languages and tasks. |
| Approach: | They focused on how the conventional DNN-HMM decoder behaves depending on the amount of data used for their estimation. |
| Outcome: | The proposed model improves the CTC/Attention model in data-scarcity scenarios while requiring less training time and fewer parameters. |
Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | In encoder-decoder neural models, multiple encoders are used to represent contextual information in addition to the individual sentence. |
| Approach: | They propose to use multiple context encoders to encode the individual sentences in document-level neural machine translation (NMT) They propose a noisy dropout setup and a single-encoder approach to encode context sentences. |
| Outcome: | The proposed approach encodes the context and the current sentence without contexts. |
SparseFlow: Accelerating Transformers by Sparsifying Information Flows (2024.acl-long)
Copied to clipboard
| Challenge: | SparseFlow is an efficient method to sparsify the dense information flows within transformers. |
| Approach: | They propose a method to sparsify the dense pathways of token representations across all transformer blocks by parameterizing them to be sparse. |
| Outcome: | The proposed method reduces computational costs by half on average without compromising task accuracy. |
Multi-Source Syntactic Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to integrate source syntax into neural machine translations use linearized parses. |
| Approach: | They propose a linearized parsed neural machine translation technique that integrates source syntax into neural machine learning. |
| Outcome: | The proposed model improves over seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. |
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining and multitask learning are widely used to improve the speech translation performance. |
| Approach: | They propose to train a speech translation model along with an auxiliary text translation task. |
| Outcome: | The proposed method improves translation quality by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs. |
Decomposable Neural Paraphrase Generation (P19-1)
Copied to clipboard
| Challenge: | Existing models learn to generate paraphrases by mapping a sequence to another, with each word processed and generated in a uniform way. |
| Approach: | They propose a Transformer-based model that can learn and generate paraphrases at different levels of granularity in a disentangled way. |
| Outcome: | The proposed model achieves competitive in-domain performance compared to state-of-the-art models and significantly better performance when adapting to a new domain. |
Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models (LMs) are trained in autoregressive manner, conditioned on the prefix. sequence labeling (SL) tasks assign labels to each individual input token, naturally benefiting from bidirectional context. |
| Approach: | They explore sequence repetition (SR) as a less invasive alternative to decoder-only models . they show that increasing the number of repetitions does not degrade SL performance . |
| Outcome: | The proposed technique improves the quality of token-level embeddings and surpasses encoders and unmasked decoders. |
Alleviating Performance Degradation Caused by Out-of-Distribution Issues in Embedding-Based Retrieval (2025.findings-emnlp)
Copied to clipboard
Haotong Bao, Jianjin Zhang, Qi Chen, Weihao Han, Zhengxin Zeng, Ruiheng Chang, Mingzheng Li, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Recent studies reveal query out-of-distribution issues degrading ANN performance . a distribution regularizer is introduced into the encoder training objective to encourage alignment between query and base embeddings. |
| Approach: | They introduce a distribution regularizer into the encoder training objective to encourage alignment between query and base embeddings. |
| Outcome: | The proposed method consistently improves retrieval performance across multiple datasets. |
Massive End-to-end Speech Recognition Models with Time Reduction (2024.naacl-long)
Copied to clipboard
Weiran Wang, Rohit Prabhavalkar, Haozhe Shan, Zhong Meng, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Chengjian Zheng, Yanzhang He, Tara Sainath, Pedro Moreno Mengibar
| Challenge: | Using the neural architecture of Google’s universal speech model, we reduce the frame rate and speed up training and inference. |
| Approach: | They propose to use the neural architecture of Google’s universal speech model with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. |
| Outcome: | The proposed methods work with both connectionist temporal classification (CTC) and RNN-Transducer (RNN-T) and over two domains. |
Modelling Commonsense Properties Using Pre-Trained Bi-Encoders (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models can capture commonsense properties that are rarely expressed in text. |
| Approach: | They propose to fine-tune language models to explicitly model commonsense properties . they train separate concept and property encoders on extracted hyponym-hypernym pairs and generic sentences . |
| Outcome: | The proposed model can capture commonsense properties with higher accuracy than human models . a new study shows that the model can model commonsensence properties with much higher accuracy . |
ECC: Synergizing Emotion, Cause and Commonsense for Empathetic Dialogue Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Empathy improves human-machine dialogue systems by enhancing the user's experience. |
| Approach: | They propose a framework that leverages specialized encoders to capture the key features of emotion, cause, and commonsense and collaboratively models these through a Conditional Variational Auto-Encoder. |
| Outcome: | Empirical results show that the framework outperforms baseline models and offers a robust solution for empathetic dialogue generation. |
A Transition-based Method for Complex Question Understanding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on complex question understanding does not model intermediate states and does not provide step-wise information. |
| Approach: | They propose a transition-based method where a decider predicts a sequence of actions to build the graph node-by-node. |
| Outcome: | The proposed method parses complex questions to QDMR using atomic operators . it has transparent and human-readable intermediate results, showing improved interpretability . |
Modeling Multi-Granularity Hierarchical Features for Relation Extraction (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work on relation extraction focuses on constructing explicit structured features using knowledge graph and dependency tree. |
| Approach: | They propose a method to extract multi-granularity features based solely on the original input sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models that even use external knowledge on three public benchmarks: SemEval 2010 Task 8, Tacred, and Tacred Revisited. |
BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla (2025.findings-naacl)
Copied to clipboard
Fabiha Haider, Fariha Tanjim Shifat, Md Farhan Ishmam, Md Sakib Ul Rahman Sourove, Deeparghya Dutta Barua, Md Fahim, Md Farhad Alam Bhuiyan
| Challenge: | Existing work on monolingual or binary hate classification in Bangla has not addressed the challenge of multi-label hate speech classification in underrepresented languages. |
| Approach: | They propose a multi-label transliterated Bangla hate speech dataset that translates or transliterates under-resourced text to higher-resource text before classifying the hate group(s). |
| Outcome: | The proposed approach outperforms other methods in the zero-shot setting while achieving state-of-the-art performance. |
Incorporating Inner-word and Out-word Features for Mongolian Morphological Segmentation (2020.coling-main)
Copied to clipboard
| Challenge: | Mongolian morphological segmentation is a crucial preprocessing step in many Mongolian related NLP applications. |
| Approach: | They propose a neural network incorporating inner-word and out-word features for Mongolian morphological segmentation. |
| Outcome: | The proposed network is compared with baselines and evaluates its performance. |
DET: A Dual-Encoding Transformer for Relational Graph Embedding (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to graph representation only consider the local neighbors, sacrificing the Transformer’s ability to attend to elements at any distance. |
| Approach: | They propose a dual-encoding Transformer architecture that uses a structural encoder and a semantic encoder to seek for semantically relevant nodes. |
| Outcome: | The proposed architecture achieves superior performance compared to state-of-the-art attention-based methods on complex relational graphs like KGs and citation networks. |
ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained text encoders can perform many NLP tasks with less resource. |
| Approach: | They propose a BERT-based Chinese text encoder enhanced by n-gram representations . they show reasonable performance when ZEN is trained on a small corpus . |
| Outcome: | The proposed encoder incorporates the comprehensive information of both the character sequence and words or phrases it contains. |
MCSE: Multimodal Contrastive Learning of Sentence Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to learning semantically meaningful sentence embeddings are limited by the complexity of pre-trained models. |
| Approach: | They propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. |
| Outcome: | The proposed approach improves the state-of-the-art average Spearman’s correlation by 1.7% on a variety of semantic textual similarity tasks. |
ALIGN-SIM: A Task-Free Test Bed for Evaluating and Interpreting Sentence Embeddings through Semantic Similarity Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Sentence embeddings play a pivotal role in a wide range of NLP tasks . evaluating and interpreting these dense vectors remains an open challenge to date . |
| Approach: | They propose a task-free test bed for evaluating and interpreting sentence embeddings . they examined five classical and eight LLM-induced sentence embedders based on semantic similarity alignment criteria . |
| Outcome: | The proposed test bed consists of five semantic similarity alignment criteria . it shows that none of the embeddings aligned with the criteria compared to other benchmarks . |
Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that humans are not supervised by the natural language inference . |
| Approach: | They propose to solve the natural language inference problem via task-agnostic multimodal pretraining. |
| Outcome: | The proposed network outperforms fully-supervised BiLSTM and BiLS+ELMO on plain text inference datasets. |
HVGuard: Utilizing Multimodal Large Language Models for Hateful Video Detection (2025.emnlp-main)
Copied to clipboard
Yiheng Jing, Mingming Zhang, Yong Zhuang, Jiacheng Guo, Juan Wang, Xiaoyang Xu, Wenzhe Yi, Keyan Guo, Hongxin Hu
| Challenge: | Existing methods for hateful video detection rely on unimodal analysis or feature fusion . Existing tools struggle to capture cross-modal interactions and reason through implicit hate in sarcasm and metaphor . |
| Approach: | They propose a reasoning-based hateful video detection framework with multimodal large language models . they integrate Chain-of-Thought reasoning to enhance multimodal interaction modeling . |
| Outcome: | The proposed framework outperforms existing tools on two public datasets covering English and Chinese. |
Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing bi-encoder architectures do not allow any sharing between text and knowledge graphs . john sutter: experimental results show that enabling full interaction yields strong improvements. |
| Approach: | They propose cross-stitch bi-encoders that allow full interaction between text and KG . they say the amount of sharing is dynamically controlled via cross-attention-based gates . |
| Outcome: | Experimental results show that bi-encoder architectures yield strong improvements . cross-stitch mechanism allows sharing and updating representations between two encoders . |
xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to find relevant passages using sparse keywords are not effective for open domain question answering. |
| Approach: | They propose a new contrastive learning method for learning a dual-encoder model for question-passage matching using a large pool of negative samples. |
| Outcome: | The proposed method maintains large pool of negative samples and optimizes question-to-passage and passage-to question matching tasks. |
Supertagging Combinatory Categorial Grammar with Attentive Graph Convolutional Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on supertagging but have not tapped into contextual information. |
| Approach: | They propose to build a graph from chunks extracted from a lexicon and apply attention over it to enhance supertagging by leveraging contextual information. |
| Outcome: | The proposed approach outperforms previous studies in terms of supertagging and parsing. |
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text (2026.acl-long)
Copied to clipboard
| Challenge: | CR methods originally designed for English struggle with Morphologically Rich Languages (MRLs) a single token in Hebrew may consist of multiple anaphors, and word/morpheme boundary discrepancies make mention detection and coreference resolution difficult in MRLs. |
| Approach: | They propose a CR dataset that identifies mentions at word, sub-word and multi-word levels and an evaluation protocol that directly addresses word/morpheme boundary discrepancies. |
| Outcome: | The proposed evaluation protocol directly addresses word/morpheme boundary discrepancies in Modern Hebrew, an MRL rich with complex words and pronominal clitics. |
On Tree-Based Neural Sentence Modeling (D18-1)
Copied to clipboard
| Challenge: | Existing tree-based sentence modeling approaches adopt syntactic parsing trees as the explicit structure prior. |
| Approach: | They replace parsing trees with trivial trees to study their effectiveness . they found that tree-based sentence modeling gives better results when crucial words are closer to the final representation . |
| Outcome: | The proposed tree-based sentences have shown better results on many downstream tasks. |
Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | 'context mixing' is a feature of Transformers that is used to build up representations of acoustic and linguistic structure in speech models. |
| Approach: | They propose to use a French spelling quirk to probe context mixing in speech models to find out how to translate spoken words into written equivalents. |
| Outcome: | The proposed model incorporates cues to identify correct transcription, whereas encoder-decoder models relegate task to decoder modules. |
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (D19-1)
Copied to clipboard
| Challenge: | Existing models with better representations of visual content and language have been developed for visual-content understanding. |
| Approach: | They propose a framework to learn vision-and-language connections from Transformers models . they pre-train a large-scale Transformer model with large amounts of image-and sentence pairs . |
| Outcome: | The proposed model improves state-of-the-art on two visual-reasoning tasks by 22% . the proposed model is based on a large-scale Transformer model with three encoders . |
The Transference Architecture for Automatic Post-Editing (2020.coling-main)
Copied to clipboard
| Challenge: | A research challenge is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions. |
| Approach: | They propose a multi-encoder based neural APE model that conditions post-editing decisions on both the source and machine translated text as inputs. |
| Outcome: | The proposed model outperforms the best performing systems by 1 BLEU point on the WMT 2016, 2017, and 2018 English–German APE shared tasks. |
Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling (D19-1)
Copied to clipboard
| Challenge: | Existing techniques for relevance and semantic matching cannot be easily adapted to the other. |
| Approach: | They propose a model that incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness. |
| Outcome: | The proposed model incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness. |
TWIST: Text-encoder Weight-editing for Inserting Secret Trojans in Text-to-Image Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Trojan attacks require extensive training data and poor generalization, limiting effectiveness and scalability. |
| Approach: | They propose a method for embedding Trojans into plugins using a single edit layer . they find that the method reduces modified parameters by 8-fold and cuts injection time to 25 seconds . |
| Outcome: | The proposed method achieves an average attack success rate of 91%, a 78% improvement over the state-of-the-art (SOTA) method. |
Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks (D19-1)
Copied to clipboard
| Challenge: | State-of-the-art SRL models do not model non-local interaction between arguments . e.g., LSTMs do not allow for efficient inference . |
| Approach: | They propose a new approach to model interactions between arguments using capsule networks . they analyze errors in the refinement procedure by capturing intuition in a flexible way . |
| Outcome: | The proposed model outperforms the baseline model on all 7 languages and achieves state-of-the-art results on 5 languages including English. |
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for review rating prediction ignore hierarchies among data . paper review rating predictions are important for improving paper review process . |
| Approach: | They propose a Hierarchical bi-directional self-attention Network framework for paper review rating prediction and recommendation . they leverage hierarchical structure of paper reviews with three levels of encoders . |
| Outcome: | The proposed approach can be used to make an effective decision-making tool for the academic paper review process. |
IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models on context-dependent text-to-SQL task focus on utilizing historic user inputs. |
| Approach: | They propose a database schema interaction graph encoder to utilize historic user inputs. |
| Outcome: | The proposed model outperforms previous state-of-the-art models on two datasets . it also outperformed existing models on the benchmark SParC and CoSQL datasets. |
Less Mature is More Adaptable for Sentence-level Language Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies fine-tune encoders or contrastive learning approaches to learn sentence representations. |
| Approach: | They propose to use sentence-level models to study how sentence representations influence downstream task performance. |
| Outcome: | The proposed models outperform token-level models in terms of time and data efficiency. |
A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment (2023.acl-long)
Copied to clipboard
| Challenge: | Existing cognitive stimulation systems lack data on how to integrate emotional support and therapy principles into chit-chat dialogue systems. |
| Approach: | They propose a multi-source knowledge fusion method for CS dialogue to generate open-ended responses guided by the therapy principle and emotional support strategy. |
| Outcome: | The proposed method generates open-ended responses guided by the therapy principle and emotional support strategy of the target response. |
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models that use heuristics to shorten sequence lengths are computationally prohibitive. |
| Approach: | They propose a new method to shorten sequence lengths by transforming tokens through encoders and a core-set based token selection method that avoids expensive pre-training and fine tuning. |
| Outcome: | The proposed model outperforms existing models on GLUE benchmarks and Long Range Arena datasets and demonstrates that it is cost-effective and space-efficient. |
Exploring Dual Encoder Architectures for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dual encoders have been used for question-answering and information retrieval tasks with good results. |
| Approach: | They propose to use two different versions of dual encoders for QA retrieval tasks . they propose to share parameters in projection layers between two encoder towers . |
| Outcome: | The proposed architectures outperform SDE and ADE on QA retrieval tasks. |
Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy (2026.acl-long)
Copied to clipboard
| Challenge: | a method to compare and visualise sentence encoders at scale is proposed . we map encoder LLMs using QRE-based feature vectors, which are then projected to 2D . |
| Approach: | They propose a method to compare and visualise sentence encoders at scale by creating a map of encoder . they construct a QRE-based map of sentences covering 1101 publicly available sentence encoded sentences . |
| Outcome: | The proposed method compares sentence encoders at scale by creating a map of encoder models . it shows that the map accurately reflects relationships between encoder and unit base encoder . |
Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has exposed text encoders for replicating discriminatory social biases which may cause unintended and undesired model behaviors with respect to social groups. |
| Approach: | They propose a method to reduce social stereotypes by redistributing attention scores of a text encoder so it forgets any preference to historically advantaged groups and attends to all social classes with the same intensity. |
| Outcome: | The proposed method reduces stereotypes and inflicts no semantic damage on pre-trained encoders. |
CTC-based Non-autoregressive Speech Translation (2023.acl-long)
Copied to clipboard
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Logic-Guided Message Generation from Raw Real-Time Sensor Data (2022.lrec-1)
Copied to clipboard
| Challenge: | Developing a natural language generation model to enable human pilots to communicate with drones is challenging because of its redundant nature and diversity. |
| Approach: | They propose a corpus for a specific domain that instantiates these properties by combining sensor data with text. |
| Outcome: | The proposed model can alert the human pilot of the system state and environment in preparation of handover of control. |
Improving Radiology Summarization with Radiograph and Anatomy Prompts (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on automatic impression generation, but this task is time-consuming and in high demand. |
| Approach: | They propose to use an anatomy-enhanced multimodal model to generate automatic impressions by combining radiology images with textual features. |
| Outcome: | The proposed model achieves state-of-the-art on two benchmark datasets and compares with existing models. |
On Pursuit of Designing Multi-modal Transformer for Video Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video grounding are not end-to-end, i.e., they rely on time-consuming post-processing steps to refine predictions. |
| Approach: | They propose an end-to-end multi-modal Transformer model that uses two encoders and a cross-modal decoder for grounding prediction. |
| Outcome: | The proposed model is 4.9% faster than existing models and is based on a set of encodings and decoders. |
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning (2023.acl-long)
Copied to clipboard
Xiao Xu, Bei Li, Chenfei Wu, Shao-Yen Tseng, Anahita Bhiwandiwalla, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan
| Challenge: | Two-Tower Vision-Language models suffer from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of unil-modal knowledge. |
| Approach: | They propose a model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels to facilitate more comprehensive cross-modal alignment and fusion. |
| Outcome: | The proposed model outperforms baselines with and without Vision-Language Pre-training (VLP) with 4M VLP data. |
Visually Grounded Reasoning across Languages and Cultures (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new protocol allows for a multilingual hierarchy of concepts and images based on native speakers . the results suggest that the current models are not robust enough to handle multilingual data . |
| Approach: | They propose a protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. |
| Outcome: | The proposed protocol lets the selection of concepts and images be entirely driven by native speakers, rather than scraping them automatically. |
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025.findings-acl)
Copied to clipboard
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam, Jesus-German Ortiz-Barajas, Rendi Chevi, Alham Fikri Aji
| Challenge: | Large Language Models excel in zero-shot and few-shot tasks, but their architecture makes them difficult to use. |
| Approach: | They adapt Large Language Models (LLMs) for zero-shot generalization using Statement Tuning . they find encoders can achieve zero- shot cross-lingual generalization . |
| Outcome: | The proposed model generalizes well across languages while being more efficient. |
ImpliHateVid: A Benchmark Dataset and Two-stage Contrastive Learning Framework for Implicit Hate Speech Detection in Videos (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on text-based hate speech detection focus on video-based approaches . however, hateful content remains a persistent challenge due to the vast amount of data generated every day. |
| Approach: | They propose a novel two-stage contrastive learning framework for hate speech detection in videos . they train modality-specific encoders for audio, text, and image using contrastive loss . |
| Outcome: | The proposed framework is based on two datasets, ImpliHateVid and HateMM datasets. |
Probing Representations for Document-level Event Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level information extraction tasks require a more comprehensive understanding that often extends to the entire input document. |
| Approach: | They propose to use probing to analyze document-level information extraction representations by embedding probes into a standard dataset. |
| Outcome: | The proposed models improve argument detections but struggle with document length and cross-sentence discourse. |
BanglaTLit: A Benchmark Dataset for Back-Transliteration of Romanized Bangla (2024.findings-emnlp)
Copied to clipboard
| Challenge: | low-resource languages like Bangla are limited by the lack of datasets. |
| Approach: | They propose a large-scale transliteration dataset and a pre-training corpus on romanized Bangla. |
| Outcome: | The proposed datasets show that the proposed methods can enrich romanized Bangla. |
EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained vision-language models have achieved impressive results in a range of vision-linguistic tasks. |
| Approach: | They propose a distilling then pruning framework to compress large vision-language models into smaller, faster ones. |
| Outcome: | The proposed framework reduces the size of a pre-trained large vision-language model and improves its performance on vision-linguistic tasks. |
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained large language models can be fine-tuned with instruction tuning to align the model responses with human intentions. |
| Approach: | They investigate the effectiveness of parameter efficient fine-tuning (PEFT) of the Q-Former with visual reasoning benchmarks ScienceQA and IconQA. |
| Outcome: | The proposed model achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. |
DNCASR: End-to-End Training for Speaker-Attributed ASR (2025.acl-long)
Copied to clipboard
| Challenge: | DNCASR is a novel end-to-end trainable system for speaker clustering and automatic speech recognition . it is designed to transcribe long multi-party meetings with speaker-attributed transcriptions . |
| Approach: | They propose a joint neural speaker clustering and automatic speech recognition system . DNCASR uses two separate encoders to independently encode speaker characteristics and local waveform information . |
| Outcome: | The proposed system outperforms a parallel system that does not have links between speaker and ASR decoders. |
Privacy Implications of Retrieval-Based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of retrieval-based language models shows improved interpretability, factuality, and adaptability compared to parametric counterparts . kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models . |
| Approach: | They present the first study of privacy risks in retrieval-based language models . they aim to strike a balance between utility and privacy in domains where privacy is of concern . |
| Outcome: | The proposed methods improve interpretability, factuality, and adaptability compared to parametric models . the study finds that kNN-LMs are more susceptible to leaking private data than parametric ones . |
Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on coarse-grained hallucination detection and fail to capture hallucinics . vision encoders exhibit unique hallucinian characteristics, but suboptimal of simple feature fusion. |
| Approach: | They propose a visual encoder that employs different training paradigms to instill inductive biases in visual encoded models. |
| Outcome: | The proposed system reduces hallucinations and improves model performance. |
Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work addresses problem of compression of tokens into shorter sequence of real-valued vectors . attainable lossless compression ratio is typically not higher than x10 . |
| Approach: | They propose to compress a sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings or key-value cache. |
| Outcome: | The proposed algorithms reduce the amount of compute in existing language models rather than minimizing number of bits needed to store text. |
MedMT5: An Open-Source Multilingual Text-to-Text LLM for the Medical Domain (2024.lrec-main)
Copied to clipboard
Iker García-Ferrero, Rodrigo Agerri, Aitziber Atutxa Salazar, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana Ramirez-Romero, German Rigau, Jose Maria Villa-Gonzalez, Serena Villata, Andrea Zaninello
| Challenge: | Existing studies on large language models for medical applications have focused on a single language . medical mT5 outperforms both encoders and similar sized text-to-text models in English, French, and Italian benchmarks . |
| Approach: | They propose to train Medical mT5, the first open-source text-to-text multilingual model for the medical domain. |
| Outcome: | The proposed model outperforms encoders and similar sized models on the Spanish, French, and Italian benchmarks while being competitive with current state-of-the-art models in English. |
Lemma Dilemma: On Lemma Generation Without Domain- or Language-Specific Training Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can generate lemmas in context without prior fine-tuning. |
| Approach: | They compare in-context lemma generation with traditional fully supervised approaches . they use encoder-only supervised methods and cross-lingual methods . |
| Outcome: | The proposed model outperforms the traditional fully supervised approach in the context of lemmatization tasks. |
Modeling Complex Semantics Relation with Contrastively Fine-Tuned Relational Encoders (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning relational embeddings fail to capture nuanced representations and rich semantics. |
| Approach: | They propose different relational encoders designed to capture diverse relational aspects and semantic properties of entity pairs. |
| Outcome: | The proposed encoders capture diverse relational aspects and semantic properties of entity pairs. |
Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a limited number of text encoders are able to recognize fine-grained entities or events within encoded semantics. |
| Approach: | They propose a new evaluation dataset to examine embeddings' ability to recognize fine-grained entities or events within encoded semantics. |
| Outcome: | The proposed dataset shows embeddings struggle with fine-grained matching . the proposed encoder outperforms the state-of-the-art 7B model in a small sample . |
EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current voice conversion methods struggle in zero-shot cross-lingual settings . authors develop a method that can be used in zero shot cross-linguistic settings despite advances in technology . |
| Approach: | They propose a voice-conversion model that combines discrete speech representations with a non-autoregressive speech decoder. |
| Outcome: | The proposed approach excels in zero-shot cross-lingual settings even for unseen languages and accents. |
xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art trainable machine translation evaluation metrics rely on large encoders . this makes them computationally expensive and inaccessible to researchers with limited resources. |
| Approach: | They propose a method to extract knowledge stored in large encoders and a pipeline for efficient black-box distillation. |
| Outcome: | The proposed model surpasses COMET-22 and BLEURT-20 on the WMT22 dataset by 6.4%. |
Revisiting Context Choices for Context-aware Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work has cast doubt on whether context-aware machine translation models learn useful signals from context or are improvements in automatic evaluation metrics just a side-effect. |
| Approach: | They propose to use separate encoders for source sentence and context as multiple sources for one target sentence to train context-aware machine translation models. |
| Outcome: | The proposed model improves translation quality even with empty lines as context, but the correct context improves it and random out-of-domain context degrades it. |
SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have increased their vulnerability to adversarial manipulation. |
| Approach: | They propose to target audio-only adversarial attacks on multimodal audio–video–language models . they show that attacks can be successful at low perceptual distortions . |
| Outcome: | The proposed models achieve up to 96% success rate under realistic conditions . the proposed models are more robust to noise than to noise and distortion than to speech recognition systems . |
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for video-text retrieval capture fine-grained semantic concepts . however, they lack the ability to capture finer-grain concepts such as objects and actions. |
| Approach: | They propose a dual-encoder architecture for fast video-text retrieval that learns lexicon representations to capture fine-grained semantics. |
| Outcome: | The proposed framework outperforms existing methods with 4.8% and 8.2% improvement on MSR-VTT and DiDeMo respectively. |
GRAM: Generative Recommendation via Semantic-aware Multi-granular Late Fusion (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies rely on item metadata to construct abbreviated item IDs, leading to a loss of valuable details. |
| Approach: | They propose a Generative Recommender via semantic-aware multi-granular late fusion to integrate rich semantics efficiently with minimal information loss. |
| Outcome: | The proposed model outperforms eight state-of-the-art recommendation models on four benchmark datasets and achieves significant improvements of 11.5-16.0% in Recall@5 and 5.3-13.6% in NDCG@5. |
ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing speech-to-speech large language models rely on ASR transcription or use encoders to extract latent representations, weakening affective information and contextual coherence in multi-turn dialogues. |
| Approach: | They propose a framework for speech-based empathetic response generation that captures turn-level affective states and dialogue-level emotional dynamics. |
| Outcome: | The proposed framework outperforms baselines in automatic and human evaluations and remains robust across different Large Language Model (LLM) backbones. |
Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable capabilities across a wide range of general tasks, from question answering to code generation. |
| Approach: | They use a paragraph-level pipeline to annotate PubMed Central paragraphs . they use XLM-RoBERTa to fine-tune the pipeline and propagate annotations to the full corpus . |
| Outcome: | The proposed approach improves performance on 11 tasks while using 2.5x fewer tokens and only public data. |
Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the extent to which SLMs encode nuanced syntactic and conceptual features . acoustic and phonetic features are shallow, but the extent of nuance is unclear . |
| Approach: | a new study evaluates contextual syntactic and semantic features in transformer-based speech language models . authors compare SLMs to linguistic competence assessments for large language models. |
| Outcome: | a new study compares SLMs with linguistic competence assessments to assess speech recognition and understanding . the results show that SLM models encode grammatical features more robustly than conceptual ones . |
Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods struggle with content-style entanglement, leading to poor generalization across domains. |
| Approach: | They propose an explanation-by-design framework that explicitly disentangles style from content through architectural separation-by design. |
| Outcome: | The proposed framework disentangles style from content through architectural separation-by-design. |