LATENCY

Predicting Entity Salience in Extremely Short Documents (2024.emnlp-industry)

Challenge:	Loki is an open-source fact-checking tool designed to address the growing problem of misinformation.
Approach:	They propose a tool that breaks down the fact-checking task into five steps . they propose LOKI, which offers a semiautomated, human-in-the-loop approach .
Outcome:	a new open-source tool is designed to address the growing problem of misinformation . the tool breaks down the fact-checking task into five steps to assist human judgment .

Copied to clipboard

Benjamin Bullough, Harrison Lundberg, Chen Hu, Weihang Xiao

Challenge:	False positive: ES is a natural language understanding task that determines which entities are most salient to a passage . Falsity: Popsicle, Frank Epperson and San Francisco are salient entities .
Approach:	They propose a lightweight and data-efficient approach for entity salience detection on short documents . they propose he use of a human-labeled dataset to evaluate entity salient on short questions .
Outcome:	The proposed approach achieves competitive performance over state-of-the-art models at significant cost and latency advantages.

SLTEV: Comprehensive Evaluation of Spoken Language Translation (2021.eacl-demos)

Copied to clipboard

Ebrahim Ansari, Ondřej Bojar, Barry Haddow, Mohammad Mahmoudi

Machine Translation

Challenge:	Spoken Language Translation (SLT) evaluation of machine translation (MT) quality has been investigated for decades.
Approach:	They propose an open-source tool for assessing machine translation (MT) quality based on time-stamped transcripts and reference translations.
Outcome:	The proposed evaluation tool is based on time-stamped transcripts and reference translations into a target language.

Does Simultaneous Speech Translation need Simultaneous Models? (2022.findings-emnlp)

Copied to clipboard

Sara Papi, Marco Gaido, Matteo Negri, Marco Turchi

SLATE: A Sequence Labeling Approach for Task Extraction from Free-form Inked Content (2022.emnlp-industry)

Challenge:	Simultaneous speech translation (SimulST) systems strive for high output quality but also low latency.
Approach:	They propose to train SimulST offline without additional training or adaptation . they also show offline training achieves similar or better quality compared to offline training .
Outcome:	The proposed model can serve both offline and simultaneous applications without additional training or adaptation.

Copied to clipboard

Apurva Gandhi, Ryan Serrao, Biyi Fang, Gilbert Antonius, Jenna Hong, Tra My Nguyen, Sheng Yi, Ehi Nosakhare, Irene Shaffer, Soundararajan Srinivasan

Information Extraction & Text Mining

Challenge:	SLATE is a sequence labeling approach for extracting tasks from free-form content . past approaches for task extraction from typed content focus on building separate sentence-level task classification models.
Approach:	They propose a sequence labeling approach for extracting tasks from free-form content . they use a single, low-latency sequence labelling approach to perform sentence segmentation and classification .
Outcome:	The proposed model outperforms a baseline model and achieves 84.4% task F1 score and 88.4% boundary similarity score.

CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction (2025.acl-industry)

Copied to clipboard

Harsh Maheshwari, Srikanth Tenneti, Alwarappan Nakkiran

Select-then-Route : Taxonomy guided Routing for LLMs (2025.emnlp-industry)

Challenge:	Retrieval Augmented Generation (RAG) is a powerful application of Large Language Models (LLMs).
Approach:	They propose to use BERTScore to fine-tune Large Language Models on domain-specific data to improve citation accuracy.
Outcome:	The proposed approach improves citation accuracy by 15.46% with minimal latency and cost.

Copied to clipboard

Soham Shah, Kumar Shridhar

Challenge:	Large language models have boosted performance across a broad spectrum of tasks . sending each query to the most suitable model is prohibitively expensive .
Approach:	They propose a framework that selects a small pool of LLMs and routes queries through an adaptive cascade.
Outcome:	The proposed framework improves accuracy and latency by 4X while reducing inference cost.

Opportunistic Decoding with Timely Correction for Simultaneous Translation (2020.acl-main)

Copied to clipboard

Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Liang Huang

Learning When to Translate for Streaming Speech (2022.acl-long)

Challenge:	Existing approaches to balancing translation quality and latency are either too aggressive or too conservative.
Approach:	They propose an opportunistic decoding technique that always (over-)generates a certain mount of extra words at each step to keep the audience on track with the latest information.
Outcome:	The proposed technique reduces latency and increases BLEU with no over-generating . it also corrects mistakes in the overgenerated words when observing more context .

Copied to clipboard

Qian Dong, Yaoming Zhu, Mingxuan Wang, Lei Li

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment (2026.acl-industry)

Challenge:	Existing methods waiting-and-translating for a fixed duration break speech acoustic units . Existing models waiting-for a set duration and generating partial sentences are not effective .
Approach:	They propose a monotonic segmentation module inside an encoder-decoder model to detect proper speech unit boundaries for a streaming speech input.
Outcome:	The proposed method outperforms existing methods on a speech translation dataset and achieves the best trade-off between translation quality and latency.

Copied to clipboard

Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway, Patrick Huber, Chinnadhurai Sankar, Ayushi Dalmia, Zechun Liu, Lemeng Wu, Tarek Elgamal, Adithya Sagar, Vikas Chandra, Raghuraman Krishnamoorthi

Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection (2022.naacl-main)

Challenge:	MobileLLM-Flash is a family of foundation models for efficient on-device use with strong capabilities.
Approach:	They propose a method for designing on-device large language models under mobile latency constraints using hardware-in-the-loop architecture search.
Outcome:	The proposed model is amenable to industry-scale deployment and is compatible with mobile runtimes like Executorch.

Copied to clipboard

Angelica Chen, Vicky Zayats, Daniel Walker, Dirk Padfield

Metamo: Empowering Large Language Models with Psychological Distortion Detection for Cognition-aware Coaching (2025.emnlp-demos)

Challenge:	a recent study shows that current NLP models operate non-incrementally, causing unacceptable delays for the user.
Approach:	They propose a streaming BERT-based sequence tagging model that detects disfluencies in real-time . they train the model to decide whether to immediately output a prediction or wait for further context .
Outcome:	The proposed model produces accurate predictions sooner than baselines, with lower flicker . disfluencies hurt readability of ASR transcripts, erode model performance on downstream tasks .

Copied to clipboard

Hajime Hotta, Huu-Loi Le, Manh-Cuong Phan, Minh-Tien Nguyen

SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction (2026.acl-industry)

Challenge:	Metamo is a browser-based dialogue system that transforms an off-the-shelf large language model into an empathetic coach for everyday workplace concerns.
Approach:	They propose a browser-based dialogue system that first identifies the cognitive distortion behind an emotion, then recognizes the user’s emotion, and finally produces a question-centered reply that invites reflection.
Outcome:	Empirical tests on public corpora showed that the proposed system improved emotionrecognition quality and response diversity without sacrificing latency.

Copied to clipboard

Sin Yu Bonnie Ho, Arlie Coles, Erik Larsson, Eric Marshall, Nathan Bodenstab, Paul Vozila

Information Extraction & Text Mining

Challenge:	Structured information extraction (IE) pairs values from unstructured text with schema-defined keys.
Approach:	They propose a retrieval-augmented generation framework that prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and few-shot examples.
Outcome:	The proposed framework can achieve up to 8.8% increase in micro-F1, 47% reduction in latency, and 48% reduction in token costs on real-world healthcare and e-commerce datasets.

Learning from LLM Agents: In-Context Generative Models for Text Casing in E-Commerce Ads (2025.emnlp-industry)

Copied to clipboard

Yingxue Zhou, Tan Zhu, Tao Zeng, Zigeng Wang, Wei Shen

Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark (2026.findings-eacl)

Challenge:	Existing NER-based transformer models are expensive and lack contextual dependencies, making them less reliable when handling unseen or ad-specific terms, e.g., brand names.
Approach:	They propose a two-stage approach to casing correction in e-commerce ad content that leverages Chain-of-Actions to enforce content policies while accurately handling ads-specific terms.
Outcome:	The proposed model outperforms existing NER-based models and achieves near-LLM performance at a fraction of the cost.

Copied to clipboard

Jieun Park, KyungTae Lim, Joon-ho Lim

Numerical Reasoning

Challenge:	Recent advances in LLMs have significantly improved mathematical problem-solving, with models like GPT-4 achieving human-level performance.
Approach:	They propose a bilingual English-Korean dataset enriched with teacher solutions, student solutions, and annotations marking students’ initial errors.
Outcome:	The proposed model achieves high agreement with human judgments and lower latency and resource usage than commercial APIs, demonstrating strong computational efficiency.

The Impact of Inference Acceleration on Bias of LLMs (2025.naacl-long)

Copied to clipboard

Elisabeth Kirsten, Ivan Habernal, Vedant Nanda, Muhammad Bilal Zafar

Challenge:	Recent work suggests strategies to increase inference efficiency with LLMs . however, these strategies may inadvertently lead to some side-effects.
Approach:	They propose to optimize inference acceleration strategies such as quantization, pruning, and caching to reduce inference cost and latency while maintaining predictive performance.
Outcome:	The proposed strategies reduce cost and latency while maintaining predictive performance while preserving the model size.

CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)

Copied to clipboard

Divya Jyoti Bajpai, Manjesh Hanawal

Structured Pruning Learns Compact and Accurate Models (2022.acl-long)

Challenge:	Pre-trained Language Models suffer in inference latency due to their large size.
Approach:	They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers.
Outcome:	The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy.

Copied to clipboard

Mengzhou Xia, Zexuan Zhong, Danqi Chen

LEASH: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model (2026.acl-long)

Challenge:	Pre-trained language models have high costs in terms of storage, memory, and computation time.
Approach:	They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency.
Outcome:	The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data.

Copied to clipboard

Yanhao Li, Lu Ma, Jiaran Zhang, Lexiang Tang, Wentao Zhang, Guibo Luo

LLMInit: A Free Lunch from Large Language Models for Selective Initialization of Recommendation (2025.emnlp-industry)

Challenge:	Existing approaches to long reasoning traces are hard to tune and fail to adapt to evolving LLMs.
Approach:	They propose a reinforcement learning framework that optimizes the length of reasoning traces by a Lagrangian primal–dual method.
Outcome:	The proposed framework reduces the average reasoning length by 60% across diverse tasks while maintaining competitive performance.

Copied to clipboard

Weizhi Zhang, Liangwei Yang, Wooseong Yang, Henry Peng Zou, Yuqing Liu, Ke Xu, Sourav Medya, Philip S. Yu

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)

Challenge:	Existing algorithms for collaborative filtering are limited by their computational demands and latency.
Approach:	They propose a framework to integrate pre-trained LLM embeddings into CF models through selective initialization strategies.
Outcome:	The proposed framework improves recommendation performance while maintaining low computational costs.

Copied to clipboard

Anuj Diwan, Eunsol Choi, David Harwath

InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model (2025.findings-acl)

Challenge:	Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks.
Approach:	They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks.
Outcome:	The proposed model is based on a self-attention-based model with a high input length threshold.

Copied to clipboard

Siqi Ouyang, Xi Xu, Lei Li

FormGym: Doing Paperwork with Agents (2026.eacl-long)

Challenge:	Existing models for simultaneous speech translation assume pre-segmented speech, limiting their real-world applicability.
Approach:	They propose a multi-turn dialogue task that can translate unbounded streaming speech . they construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a cache management strategy to facilitate efficient inference.
Outcome:	The proposed approach reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines.

Copied to clipboard

Matthew Toles, Isaac Song, Rattandeep Singh, Zhou Yu

Challenge:	Existing studies focus on parsing, extraction and web form interaction, rather than end-to-end completion of document-style forms.
Approach:	They propose a benchmark formulation of the end-to-end form filling task that evaluates form completion and accuracy.
Outcome:	The proposed task is based on three existing datasets and adds one new dataset to achieve more challenging, diverse, and realistic test cases.

Learning Adaptive Segmentation Policy for Simultaneous Translation (2020.emnlp-main)

Copied to clipboard

Ruiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, Haifeng Wang

Text Generation

Challenge:	Experimental results show that adaptive segmentation policies for simultaneous translation are more accurate than current methods . if translation starts before adequate source content is delivered, the quality of translation degrades . waiting for too much source text increases latency, which would hurt accuracy .
Approach:	They propose a new adaptive segmentation policy for simultaneous translation based on human interpreters . it learns to segment the source text by considering possible translations produced by the translation model .
Outcome:	Experimental results show that the proposed method achieves better accuracy-latency trade-off over state-of-the-art methods.

Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers (2025.emnlp-industry)

Copied to clipboard

Zhiyuan Peng, Ting-Ruen Wei, Tingyu Song, Yilun Zhao

Gaussian Multi-head Attention for Simultaneous Machine Translation (2022.findings-acl)

Challenge:	Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency and the number of forward passes.
Approach:	They propose to use a large language model to evaluate the efficiency of LLM-based rerankers . they propose to measure ranking quality and query processing efficiency using an interpretable FLOPs estimator .
Outcome:	The proposed metrics evaluate LLM-based rerankers with different architectures without running any experiments.

Copied to clipboard

Shaolei Zhang, Yang Feng

Machine Translation

Challenge:	Existing methods for siMT do not explicitly model the alignment to perform the control.
Approach:	They propose to model alignment and translation in a unified manner by Gaussian Multi-head Attention (GMA) they propose to integrate alignment-related priors into the translation model to determine final attention.
Outcome:	The proposed method outperforms strong baselines on trade-off between translation and latency.

EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation (2025.findings-acl)

Copied to clipboard

Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, Jong C. Park

Question Answering

Challenge:	Current retrieval-augmented generation systems struggle when retrieval models fail to rank the most relevant documents . existing extractive methods reduce latency but rely on independent, non-adaptive sentence selection .
Approach:	They introduce an extractive context compression framework that enhances retrieval-augmented generation in question answering.
Outcome:	EXIT surpasses existing compression methods and uncompressed baselines in QA accuracy . the framework reduces inference time and token count while preserving contextual dependencies .

MoDification: Mixture of Depths Made Easy (2025.naacl-long)

Copied to clipboard

Chen Zhang, Meizhi Zhong, Qimeng Wang, Xuantao Lu, Zheyu Ye, Chengqiang Lu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang, Dawei Song

Non-autoregressive Streaming Transformer for Simultaneous Translation (2023.emnlp-main)

Challenge:	Long-context efficiency is a trending topic in large language model (LLM) serving.
Approach:	They propose a method to combine long-context efficiency and mixture of depths to bring down both latency and memory.
Outcome:	The proposed method achieves 1.2 speedup in latency and 1.8 reduction in memory compared to original LLMs especially in long-context applications.

Copied to clipboard

Zhengrui Ma, Shaolei Zhang, Shoutao Guo, Chenze Shao, Min Zhang, Yang Feng

EfficientLLM: Unified Pruning-Aware Pretraining for Auto-Designed Compact Language Models (2026.acl-long)

Challenge:	Simultaneous machine translation models are trained to strike a balance between latency and translation quality.
Approach:	They propose a non-autoregressive streaming Transformer which generates blank tokens and decodes repetitive tokens to adjust its READ/WRITE strategy flexibly.
Outcome:	The proposed model outperforms previous strong autoregressive models on various benchmarks on siMT.

Copied to clipboard

Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Haokun Lin, Xianlin Zeng, Guoqi Li, Jiajun Zhang

Look Before You Leap: A Lookahead Reasoning Quality Gate for Speculative Decoding (2026.eacl-long)

Challenge:	Large language models (LLMs) driven by scaling laws can be developed in large model sizes.
Approach:	They propose a pruning-aware pretraining approach that decouples LLM pruning from direct pretraining.
Outcome:	The proposed model outperforms pretraining models with 100M 1B parameters in commen sense benchmarks.

Copied to clipboard

Hiroaki Kingetsu, Kaoru Yokoo, Kenji Fukumizu, Manohar Kaul

Deep Model Compression Also Helps Models Capture Ambiguity (2023.acl-long)

Challenge:	Unlike token-level likelihood search, which is myopic and often rewards verbosity, our approach works at an intermediate granularity.
Approach:	They propose a lookahead quality gate for speculative decoding that accepts the longest reliable prefix of each k-token lookaheaded draft.
Outcome:	The proposed method improves accuracy over baselines while achieving 2.6-7.9 faster generation on math and science benchmarks.

Copied to clipboard

Hancheol Park, Jong Park

Challenge:	Experimental results show that our method reduces the model size significantly and improves latency.
Approach:	They propose a method to capture the degree of relationship between a sample and its candidate classes by deep model compression.
Outcome:	The proposed method reduces model size significantly and improves latency.

AdaptMerge: Inference Time Adaptive Visual and Language-Guided Token Merging for Efficient Large Multimodal Models (2025.findings-emnlp)

Copied to clipboard

Zahidul Islam, Mrigank Rochan

Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)

Challenge:	Existing token reduction methods ignore image complexity and vision-language interactions, ignoring image complexity.
Approach:	They propose a training-free, inference-time token merging strategy that adaptively reduces visual tokens by leveraging feature diversity and language-guided relevance.
Outcome:	The proposed approach outperforms state-of-the-art token reduction methods on Google’s Gemma 3 models while achieving reduced computational costs and improved performance.

Copied to clipboard

Ileana Rugina, Rumen Dangovski, Li Jing, Preslav Nakov, Marin Soljacic

Challenge:	Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns .
Approach:	They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask.
Outcome:	The proposed approach saves 90% of computations and maintains quality of results.

On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation (2025.findings-naacl)

Copied to clipboard

Xiaonan Jing, Srinivas Billa, Danny Godbout

BASS: Batched Attention-optimized Speculative Sampling (2024.findings-acl)

Challenge:	Hallucination is a popular topic in natural language generation (NLG).
Approach:	They propose to use large language models to evaluate faithfulness of guided NLGs by a rubric template and large language inference models to score the generation on quantifiable scales.
Outcome:	The proposed system can provide accurate judgement and explain whether a source and generation are factually consistent.

Copied to clipboard

Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras

Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models (2026.findings-acl)

Challenge:	Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models.
Approach:	They propose a batched speculative decoding system that generates sequences at an average speed of 5.8ms per token and a batch size of 8 at a 2.15 speed-up over optimized regular decoding.
Outcome:	The proposed system achieves state-of-the-art latency and speed-up over optimized regular decoding.

Copied to clipboard

Sachin Kumar

Structured Pruning of Large Language Models (2020.emnlp-main)

Challenge:	Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms across four benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode.
Approach:	They compare hypernetwork-based LoRA adaptation against carefully designed few-shot prompting in a controlled experiment . they find that few- shot prompting contributes +21.5% to performance and documentation contributes 0% .
Outcome:	The hypernetwork-based LoRA adaptation provides no measurable improvement over few-shot prompting alone.

Copied to clipboard

Ziheng Wang, Jeremy Wohlwend, Tao Lei

Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation (2022.acl-long)

Challenge:	Recent advances in language modeling have led to remarkable improvements on a variety of tasks.
Approach:	They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training.
Outcome:	The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference.

Copied to clipboard

Ruiqing Zhang, Zhongjun He, Hua Wu, Haifeng Wang

Speech & Audio in NLP

Challenge:	Existing methods to perform simultaneous speech-to-text translation ignore contextual information and suffer from low translation quality.
Approach:	They propose an adaptive segmentation policy for simultaneous speech-to-text translation . it learns to segment the source streaming speech into meaningful units .
Outcome:	The proposed method achieves a good accuracy-latency trade-off over state-of-the-art methods on English-German and Chinese-English.

MixLLM: Dynamic Routing in Mixed Large Language Models (2025.naacl-long)

Copied to clipboard

Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen

RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents (2026.acl-long)

Challenge:	Large Language Models (LLMs) exhibit potential artificial generic intelligence, however, their usage is costly with high response latency.
Approach:	They develop a dynamic contextual-bandit-based routing system for query-LLM assignment that leverages query tags to enhance query embeddings.
Outcome:	The proposed model maximizes response quality and minimizes cost and latency.

Copied to clipboard

Jize Wang, Han Wu, Zhiyuan You, Yiming Song, Yijun Wang, Zifei Shan, Yining Li, Songyang Zhang, Xinyi Le, Cailian Chen, Xinping Guan, Dacheng Tao

LLM-as-Scheduler: Agentic Workflow Dynamic Scheduling (2026.acl-long)

Challenge:	Existing methods for mixing-of-agents (MoA) lack model selection criteria and struggle with large model pools.
Approach:	They propose a mixture-of-agents framework with dynamic routing that uses a lightweight scorer to perform initial screening and refines the model scores through self- and cross-assessment.
Outcome:	The proposed framework outperforms existing methods for large model pools and tasks . it reduces cost by 89.8% and latency by 63.6% in the large-scale model pool.

Copied to clipboard

Dawei Xiang, Kexin Chu, Wenyan Xu, Wenhui Zhang, Wei Zhang

Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy (2021.emnlp-main)

Challenge:	Experiments show that LAS cuts token usage by 43% and reduces end-to-end latency by more than 36%, while causing at most a 1.4 percentage-point drop in accuracy compared with a strong fixed workflow.
Approach:	They propose a system that dynamically chooses the right workflow for each query.
Outcome:	Experiments show that LAS cuts token usage by 43% and reduces end-to-end latency by more than 36% while causing at most a 1.4 percentage-point drop in accuracy compared with a strong fixed workflow.

Copied to clipboard

Shaolei Zhang, Yang Feng

LLM Performance Predictors are good initializers for Architecture Search (2024.findings-acl)

Challenge:	Existing methods for simultaneous machine translation require multiple models for different latency levels, resulting in large computational costs.
Approach:	They propose a universal SiMT model with Mixture-of-Experts Wait-k Policy to achieve the best translation quality under arbitrary latency with only one model.
Outcome:	The proposed model outperforms all the strong baselines under different latency levels including the state-of-the-art adaptive policy.

Copied to clipboard

Ganesh Jawahar, Muhammad Abdul-Mageed, Laks Lakshmanan, Dujian Ding

When Internalization Fails: Finding Better Targets for Reasoning Compression (2026.findings-acl)

Challenge:	Large language models (LLMs) have diverse applications, encompassing both open-ended tasks (e.g., brainstorming and chat) and closed-ended ones (eg. question answering).
Approach:	They construct PP prompts for Large Language Models (LLMs) that estimate the performance of specific deep neural network architectures on downstream tasks.
Outcome:	The proposed model achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors in machine translation tasks.

Copied to clipboard

Mourad Heddaya, Manley Roberts, Rohan Wadhawan, Chenhao Tan

SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation (2025.acl-long)

Challenge:	Reasoning language models generate long reasoning traces that increase latency and cost.
Approach:	They compare three approaches to shorten reasoning traces by inference-time truncation . they use Implicit Chain-of-Thought-style curricula that progressively shorten the teacher trace .
Outcome:	The proposed methods work well on GSM8K and multiplication tasks.

Copied to clipboard

Keqi Deng, Wenxi Chen, Xie Chen, Phil Woodland

Learning Language-Specific Layers for Multilingual Machine Translation (2023.acl-long)

Challenge:	Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency.
Approach:	They propose to train LLMs offline and employ a test-time policy to guide simultaneous inference by extracting boundary-aware speech prompts that allow it to be better matched with text input data.
Outcome:	The proposed model trains speech LLMs offline and employs a test-time policy to guide simultaneous inference.

Copied to clipboard

Telmo Pires, Robin Schmidt, Yi-Hsiu Liao, Stephan Peitz

Toward Interactive Dictation (2023.acl-long)

Challenge:	Multilingual Machine Translation (MNMT) is a promising new approach to improve translation quality between non-English languages.
Approach:	They propose a language-specific transformer layer to increase model capacity while keeping computation and parameters constant.
Outcome:	The proposed approach improves translation quality by 1.3 chrF (1.5 spBLEU) over not using LSLs on a separate decoder architecture.

Copied to clipboard

Belinda Z. Li, Jason Eisner, Adam Pauls, Sam Thomson

KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation (2025.findings-acl)

Challenge:	Existing systems that allow both dictation and editing-by-voice restrict their command language to flat templates invoked by trigger words.
Approach:	They propose to allow users to interrupt dictation with spoken editing commands in open-ended natural language.
Outcome:	The proposed system can predict edited text with large pre-trained models and predict small programs.

Copied to clipboard

Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, Murali Annavaram

FLARE: Fine-Grained Length-Aware Routing for Resource-Efficient Heterogeneous LLM Serving (2026.acl-long)

Challenge:	Existing methods to inference large language models are limited by CPU capabilities and memory constraints.
Approach:	They propose an efficient I/O-aware LLM inference method that overlaps GPU computation with KV cache transfer to minimize idle GPU time.
Outcome:	The proposed method reduces the cost of auto-regressive decoding by 35.8% . it also achieves 46.2% higher throughput during decoding compared to state-of-the-art methods.

Copied to clipboard

Yujia Fu, Heming Zhong, Dan Huang, Yutong Lu

LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM (2025.findings-acl)

Challenge:	Existing routers lack fine-grained resource awareness across deployment settings, which degrades efficiency metrics in real-world serving.
Approach:	They propose a length-centric, resource-aware multi-LLM routing framework that uses length-based models to estimate per-query latency and cost.
Outcome:	Experiments show that FLARE reduces latency and cost by up to 68% and 75% while maintaining competitive accuracy.

Copied to clipboard

Sambal Shikhar, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jean Lahoud, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal

Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Models (2025.findings-emnlp)

Challenge:	Existing speech-enabled LLMs degrade conversational quality by modifying the LLM, compromising its linguistic capabilities.
Approach:	They propose a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS system that generates high-quality speech with low latency.
Outcome:	The proposed system achieves a significantly lower word error rate compared to speech-enabled LLMs while operating at comparable latency.

Copied to clipboard

Makesh Narsimhan Sreedhar, Traian Rebedea, Christopher Parisien

Simul-MuST-C: Simultaneous Multilingual Speech Translation Corpus Using Large Language Model (2024.emnlp-main)

Challenge:	Recent research shows that reasoning-based language models offer significant benefits for LLM safety and guardrail applications.
Approach:	They conduct an analysis of reasoning-based guardrail models for content moderation . they find reasoning models exhibit strong sample efficiency and inference efficiency .
Outcome:	The reasoning-based guardrail models show strong performance across domains . the models achieve competitive performance with significantly fewer training examples .

Copied to clipboard

Mana Makinae, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing (2026.findings-acl)

Challenge:	Simultaneous speech translation (SiST) begins translating before the entire source input is received.
Approach:	They propose a dataset that rearranges sentences into segmented monotonic data for simultaneous speech translation using the Large Language Model.
Outcome:	The proposed dataset improves quality and latency in siST translations by rearranging sentences into segmented monotonic data.

Copied to clipboard

Wei Cheng, Yongchang Cao, Chen Shen, Binhua Li, Jue Chen, Yongbin Li, Wei Hu

Programming Languages in NLP

Challenge:	Large Language Models (LLMs) are increasingly used for code editing, yet the full-code generation paradigm suffers from severe efficiency bottlenecks.
Approach:	They propose to use a structure-aware diff format to train LLMs to choose the most token-efficient format between a given diff format and full code.
Outcome:	The proposed approach matches the most token-efficient format with full-code generation while reducing latency and cost by over 30% on long-code editing tasks.

DrFrattn: Directly Learn Adaptive Policy from Attention for Simultaneous Machine Translation (2025.emnlp-main)

Copied to clipboard

Libo Zhao, Jing Li, Ziqian Zeng

Machine Translation

Challenge:	Existing approaches to learn read/write policies from attention mechanism may compromise effectiveness of attention mechanism .
Approach:	They propose a method that directly learns adaptive policies from the attention mechanism . experimental results demonstrate that the method achieves an improved balance between translation accuracy and latency.
Outcome:	The proposed method achieves improved balance between translation accuracy and latency.

PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation (2026.acl-long)

Copied to clipboard

Yuma Ichikawa, Naoya Takagi, Takumi Nakagawa, Yuzi Kanazawa, Akira Sakai

REaR : Retrieve, Expand and Refine for Effective Multitable Retrieval (2026.acl-long)

Challenge:	Experimental results show that PHOTON is superior to competitive Transformer-based language models . despite their capabilities, the inference cost increases with context length under recent serving workloads.
Approach:	They propose a hierarchical autoregressive model that replaces horizontal scanning with vertical context scanning.
Outcome:	Experimental results show that PHOTON is superior to Transformer-based language models . it reduces decode-time KV-cache traffic yielding up to 103 higher throughput per unit memory .

Copied to clipboard

Rishita Agarwal, Himanshu Singhal, Peter Baile Chen, Manan Roy Choudhury, Dan Roth, Vivek Gupta