Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)
Copied to clipboard
| Challenge: | 108 papers were selected for presentation at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). |
| Approach: | 108 papers were selected for presentation at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). |
| Outcome: | The industry track attracted 421 paper submissions at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). |
Copied to clipboard
| Challenge: | Existing approaches prioritize performance but overlook the balance between effectiveness and computational cost. |
| Approach: | They propose a plug-and-play framework that integrates with existing search strategies to improve LLM decision-making while maintaining efficiency. |
| Outcome: | The proposed framework reduces costs to 1/10 of the original search framework while maintaining effectiveness. |
Copied to clipboard
| Challenge: | Existing methods for detecting ads video violations lack precise temporal grounding, noisy annotations, and limited generalization. |
| Approach: | They propose a framework that integrates curriculum reinforcement learning with large language models to enhance reasoning and cognitive capabilities for violation detection. |
| Outcome: | The proposed framework achieves superior performance in violation category accuracy and temporal interval localization. |
Copied to clipboard
| Challenge: | Existing studies on distilled lightweight LLMs have focused on transferring knowledge from a larger model (the teacher) to a smaller model (sector). |
| Approach: | They propose a family of distilled, lightweight LLMs derived from Qwen2.5 models. |
| Outcome: | Experimental results show that the distilled models have significantly stronger instruction-following capabilities than the original models. |
Copied to clipboard
| Challenge: | Recommender systems are a key component of our day-to-day lives, but evaluation remains a challenge due to the gap between offline metrics and online behaviors. |
| Approach: | They propose a framework that enables users to build believable human proxies from historical data. |
| Outcome: | The proposed framework exhibits closer alignment with real humans than previous work, both at micro and macro levels. |
Copied to clipboard
| Challenge: | a new language model that supports 512K context lengths addresses practical limitations in long-context training . a competitive 35% score on 512k-token BABILong tasks without RAG or task-specific tuning is achieved . |
| Approach: | They present a language model that supports 512K-token context length . they evaluated its long-context learning performance on three benchmarks . |
| Outcome: | The model outperforms open-source models on three long-context benchmarks . it achieves a competitive 35% score on 512K-token BABILong tasks without RAG or fine-tuning . |
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) struggle with identifying and categorizing student errors in multimodal mathematical contexts. |
| Approach: | They propose a new framework that decomposes error detection into three phases with specialized agents. |
| Outcome: | The proposed framework shows higher accuracy in error step identification and 3% improvement in error categorization on real-world educational data. |
Copied to clipboard
| Challenge: | Existing models require dataset-specific training, causing costly procedures and performance bottlenecks. |
| Approach: | They propose a log anomaly detection model with semantic relational reasoning that extracts cross-system semantic patterns and encodes them as high-dimensional learnable vectors. |
| Outcome: | The proposed model extracts cross-system semantic patterns and encodes them as high-dimensional learnable vectors. |
Copied to clipboard
| Challenge: | e-commerce payment fraud detection is a new area for reinforcement learning (RL) and Large Language Models (LLMs). |
| Approach: | They propose to integrate reinforcement learning (RL) with Large Language Models (LLMs) by framing transaction risk as a multi-step Markov Decision Process (MDP), RL optimizes risk detection across multiple payment stages. |
| Outcome: | The proposed approach improves fraud detection accuracy and demonstrates zero-shot capability. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising results in various domains, but their practical application in industry-relevant operations research presents significant challenges and opportunities. |
| Approach: | They propose a cognitive-inspired framework that enhances optimization through counterfactual reasoning . they use a workflow that transforms requirements into mathematical models and executable solver code . |
| Outcome: | Experiments show that ORMind outperforms existing methods in the NL4Opt dataset and ComplexOR dataset. |
Copied to clipboard
| Challenge: | System-level testing is a critical phase in the development of large, safety-dependent systems, such as those in the automotive industry. |
| Approach: | They propose an AI-powered assistant to aid users in creating test specifications for system-level requirements. |
| Outcome: | The proposed system reduces the effort required to derive test specifications by 30% in ROUGE-L. |
Copied to clipboard
| Challenge: | Existing LLMs are unable to match outputs due to their open-ended nature . |
| Approach: | They propose a meta-evaluation strategy PromptCUE to evaluate cutting-edge LAJ-MT models such as GEMBA-MQM and a rubric-style prompt tailored to the characteristics of LLMs. |
| Outcome: | The proposed model is able to predict scores or identify errors for individual sentences and is reliable in the real world. |
Copied to clipboard
| Challenge: | Social media platforms have enabled large-scale influence campaigns, impacting democratic processes. |
| Approach: | They propose a system to enhance diversity and realism of the generated content while ensuring its adherence to the original scenario. |
| Outcome: | The proposed system improves diversity and realism while ensuring its adherence to the original scenario. |
Copied to clipboard
| Challenge: | Behavioral therapy notes are important for legal compliance and patient care, but quality standards for them remain underdeveloped. |
| Approach: | They propose a rubric for evaluating therapy notes across key dimensions: completeness, conciseness, faithfulness. |
| Outcome: | The proposed evaluation framework improves on therapist-written notes and LLM-generated notes. |
Copied to clipboard
| Challenge: | Existing studies on low-rank adapter training use the default chain of operations while calculating the output. |
| Approach: | They propose a framework that allows for efficient LoRA implementations by introducing low-rank adapters to linear layers and selecting the best forward and backward graphs based on FLOPs and time estimations. |
| Outcome: | The proposed framework significantly improves the speed of neural network training and fine-tuning with low-rank adapters. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require high quality instruction data for effective alignment, especially in code generation tasks where expert curated datasets are expensive to produce. |
| Approach: | They propose a scalable algorithm for synthesizing large-scale, high quality coding instructions using evolutionary principles. |
| Outcome: | The proposed approach generates 7.5 million coding instructions with a small seed population and is highly parallelizable and effective even with weaker generator models. |
Copied to clipboard
| Challenge: | Existing methods to train a model on a mixture of domain datasets require separate correction language models. |
| Approach: | They propose a multi-task correction MoE that trains experts to become an "expert" of speech-to-text, language-totext and vision-to text datasets by learning to route each dataset’s tokens to its mapped expert. |
| Outcome: | The proposed model outperforms GPT-3.5 and Claude-3.5-Sonnet on the Open ASR Leaderboard and reaches an average relative 5.0% WER reduction and substantial improvements in BLEU scores. |
Copied to clipboard
| Challenge: | API specifications are often presented as unstructured HTML pages, requiring external users to manually convert it into a structured format. |
| Approach: | They propose a framework that transforms long API documentation pages into consistent, machine-readable API specifications. |
| Outcome: | The proposed framework generalizes well across hundreds of APIs and produces valid OpenAPI specifications that encapsulate most of the information from the original documentation. |
Copied to clipboard
| Challenge: | Existing methods to optimize geospatial repartition rely on manual adjustments by experts or algorithmic optimization using limited offline operational metrics. |
| Approach: | They propose a framework that calibrates LLM uncertainty to enable robust geospatial repartition by integrating historical data with LLM-generated candidates. |
| Outcome: | The proposed framework calibrates LLM uncertainty to enable robust geospatial repartition by integrating historical data with LLM-generated candidates. |
Copied to clipboard
| Challenge: | General-purpose LLMs and their multimodal counterparts provide a crucial advantage in process automation. |
| Approach: | They propose a model that can be finetuned and deployed on a single 24GB GPU . it provides reliable confidence scores and quick inferences for processing files in large-scale or time-sensitive environments. |
| Outcome: | The proposed model achieves state-of-the-art results on seven diverse benchmarks and provides reliable confidence scores and quick inferences. |
Copied to clipboard
| Challenge: | Wikidata is a large open-source structured knowledge base that is used by search engines, robots, and scripts. |
| Approach: | They propose a vandalism detection system for Wikidata that converts edits into a single space using a method called Graph2Text. |
| Outcome: | The proposed system outperforms the current production system and is released under an open license. |
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework . |
| Approach: | They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. |
| Outcome: | The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences. |
Copied to clipboard
| Challenge: | Retrieval Augmented Generation (RAG) is a powerful application of Large Language Models (LLMs). |
| Approach: | They propose to use BERTScore to fine-tune Large Language Models on domain-specific data to improve citation accuracy. |
| Outcome: | The proposed approach improves citation accuracy by 15.46% with minimal latency and cost. |
Copied to clipboard
| Challenge: | Experimental results show that opensource curriculum training is more effective when distinct datasets are available for different training stages. |
| Approach: | They propose an opensource suite for training long reasoning models using publicdata and models. |
| Outcome: | The proposed model outperforms DeepSeek-R1-DistillQwen-32B models in math reasoning. |
Copied to clipboard
| Challenge: | Out-of-scope (OOS) intent detection is critical in task-oriented dialogue systems . without effective OOS detection, such inputs could lead to incorrect responses, reduced user trust, and eventual system failures. |
| Approach: | They propose a modular framework that combines uncertainty modeling with fine-tuned large language models (LLMs) their method yields state-of-the-art results on key OOS detection benchmarks . |
| Outcome: | The proposed framework yields state-of-the-art results on key OOS detection benchmarks including real-world OOS data. |
Copied to clipboard
| Challenge: | Podcasts, videos, and other long-form talk content requires significant time investment to assess their relevance. |
| Approach: | They propose an LLM-based approach for generating podcast episode previews and deploy it at scale, serving hundreds of thousands of podcast previews in a real-world application. |
| Outcome: | The proposed approach outperforms a baseline built on top of various ML expert models and offers a 4.6% increase in user engagement with preview content and a 5x boost in processing efficiency. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have transformed how complex tasks can be automated . traditional cloud computing operations involve complex manual configurations . |
| Approach: | They propose a pipeline for generating K8s manifests directly from user-described intents expressed in natural language using LLMs. |
| Outcome: | The proposed pipeline can generate K8s manifests directly from user-described intents expressed in natural language using LLMs. |
Copied to clipboard
| Challenge: | Tabular data analysis is crucial in many scenarios, yet its complexity and density can make it challenging to determine the most appropriate analysis operations for a new table. |
| Approach: | They propose a tabular data analysis framework that recommends query-code-result triplets for new tables . they propose Rec-Align, a method to further improve recommendation quality . |
| Outcome: | The proposed framework achieves 77.0% top-5 recommendation recall on a dataset designed for tabular data analysis recommendation. |
Copied to clipboard
| Challenge: | Anomaly Detection (AD) focuses on detecting samples that differ from the standard pattern, making it vital for quality control and process optimization. |
| Approach: | They propose a framework that provides industrial operators with explanations for logical anomalies by compiling automatically generated questions into a checklist and collecting responses. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) Logical AD performance on public benchmarks, MVTec LOCO AD, with an AUROC of 87.6% and an F1-max of 88.0% along with the explanations of anomalies. |
Copied to clipboard
| Challenge: | Existing knowledge editing approaches struggle with sequential editing scenarios and harm the general capabilities of the model. |
| Approach: | They propose a framework that combines robust supervised fine-tuning and model merging for knowledge editing to combine supervised and supervised learning. |
| Outcome: | The proposed approach outperforms existing methods in sequential editing while preserving the original performance of the model. |
Copied to clipboard
| Challenge: | Generative retrieval (GR) is an emerging search paradigm for food delivery search. |
| Approach: | They propose a method that harnesses the advanced query understanding capabilities of large language models to enhance the retrieval of results for complex and long-tail queries in food delivery search scenarios. |
| Outcome: | The proposed method increases the number of online orders by 0.68% for complex search intents. |
Copied to clipboard
| Challenge: | Large language models for code (LLMs) are gaining more and more attention due to their wide applicability. |
| Approach: | They propose a method which extracts overlapping contexts from training data using variable-length stride. |
| Outcome: | The proposed method outperforms the conventional approach of controlling the number of epochs in terms of the pass@k rate. |
Copied to clipboard
| Challenge: | DataMorgana generates synthetic Q&A benchmarks tailored to RAG applications . lexical, syntactic, and semantic diversity of generated benchmarks exceeds existing tools . |
| Approach: | They propose a tool for generating synthetic Q&A benchmarks tailored to RAG applications in enterprise settings. |
| Outcome: | The proposed tool surpasses existing tools in terms of lexical, syntactic, and semantic diversity while maintaining high quality. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown capabilities in various natural language processing tasks, yet struggle with logical reasoning. |
| Approach: | They propose to combine Large Language Models with symbolic reasoners to improve syntactic correctness and semantic accuracy in logical parsing tasks. |
| Outcome: | The proposed approach improves syntactic correctness and semantic accuracy in logical parsing tasks. |
Copied to clipboard
| Challenge: | Large language models (LLMs) are used to summarize large volumes of textual information into a smaller, more manageable size. |
| Approach: | They propose a large language model-based summarization system for regulated banking environments that generates accurate, privacy-compliant summaries of customer-advisor conversations. |
| Outcome: | The proposed system achieves 94% factual consistency rate and significant reduction in hallucination rate. |
Copied to clipboard
| Challenge: | Existing de-identification methods suffer from recall errors, limited generalization, and inefficiencies, limiting their real-world applicability. |
| Approach: | They propose a multi-modal framework for de-identifying electronic health records using a retrieval-based entity relexicalization approach. |
| Outcome: | The proposed framework achieves competitive performance while optimizing token usage to reduce LLM costs. |
Copied to clipboard
| Challenge: | Xu et al., 2024) argue that LLMs can be learned via conceptual consistency. |
| Approach: | They propose a method that takes concept hierarchies from a knowledge graph and generates benchmarks that test conceptual consistency in LLMs. |
| Outcome: | The proposed method reveals rates of conceptual inconsistencies in several state-of-the-art LLMs. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely used for relevance assessment in information retrieval, but maintaining and updating such models is resource-intensive, limiting their feasibility in dynamic and multilingual search environments. |
| Approach: | They propose to combine a generative SLM with an embedding-based SLM to achieve higher relevance judgment accuracy while reducing computational costs. |
| Outcome: | The proposed approach outperforms state-of-the-art LLMs in relevance assessment tasks while reducing computational costs. |
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models’ performance as correctors on Chinese Grammatical Error Correction (CGEC) remains unsatisfactory due to the challenging nature of the task. |
| Approach: | They propose a training framework EXAM that uses LLMs as explainers to enhance CGEC small models and a novel evaluation method SEE that utilizes LLM as evaluators to bring more reasonable evaluations. |
| Outcome: | The proposed methods improve the performance of LLMs on Chinese Grammatical Error Correction (CGEC) task. |
Copied to clipboard
| Challenge: | Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks . alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructures. |
| Approach: | They propose a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. |
| Outcome: | The proposed solution achieves comparable performance to baseline Transformer-based LLMs while optimizing memory consumption and time to first token. |
Copied to clipboard
| Challenge: | Large language models (LLMs) are designed to handle complex task requests, but lack of specific datasets for training and evaluation of such systems . |
| Approach: | They propose a framework to generate a dataset for in-vehicle speech recognition systems . they train an in-car context sensor that correctly identifies the functional intent of the driver . |
| Outcome: | The proposed framework outperforms baseline models across experimental settings. |
Copied to clipboard
| Challenge: | Text-to-speech (TTS) systems are limited by limited data and linguistic complexities. |
| Approach: | They propose a data-optimized framework with an advanced acoustic model to build high-quality TTS systems for low-resource scenarios. |
| Outcome: | The proposed framework enables zero-shot voice cloning and improved performance across diverse client applications, including finance, healthcare, education, and law. |
Copied to clipboard
| Challenge: | drafting method statements is labor-intensive and time-consuming . traditional methods involve using static templates filled in manually by engineers . |
| Approach: | They propose a framework that automates method statement generation by using multi-agent collaboration. |
| Outcome: | The proposed framework achieves 4.38 ContentScore, excelling in specialization, completeness, organization, and clarity. |
Copied to clipboard
| Challenge: | Current Large Language Model (LLM) evaluation frameworks use the same static prompt template across all models under evaluation. |
| Approach: | They investigate the effect of PO on LLM evaluations by using public academic and internal benchmarks to optimize the prompt for each model. |
| Outcome: | The proposed frameworks use the same static prompt template across all models under evaluation, but have some drawbacks. |
Copied to clipboard
| Challenge: | Large Language Models exhibit subjective preferences, opinions, and beliefs, which may shape their behavior, influence advice and recommendations, and potentially reinforce certain viewpoints. |
| Approach: | They developed a benchmark to assess LLMs’ subjective inclinations across societal, cultural, ethical, and personal domains. |
| Outcome: | The proposed benchmark assesses LLMs’ subjective inclinations across societal, cultural, ethical, and personal domains. |
Copied to clipboard
| Challenge: | Electronic Discovery (eDiscovery) requires identifying relevant documents from vast collections for legal production requests. |
| Approach: | They propose a system that integrates knowledge graphs for enhanced document ranking and classification, augmented by LLM-driven reasoning. |
| Outcome: | The proposed system outperforms baselines in F1-score, precision, and recall across balanced and imbalanced datasets. |
Copied to clipboard
| Challenge: | Recent efforts focus on automatic summarization of individual cases, which condense the content of a single case, making it easier for legal professionals to grasp key points. |
| Approach: | They propose a pipeline to generate multi-case structured reports using entire body of case law on user-specified topics within the European Court of Human Rights. |
| Outcome: | The proposed pipeline generates structured reports that enhance efficient, scalable legal analysis. |
Copied to clipboard
| Challenge: | Rapid growth of digital applications has intensified the demand for real-time natural language processing (NLP) capabilities. |
| Approach: | They propose a framework that combines Medusa and knowledge distillation to achieve compounded benefits in both model size and inference speed. |
| Outcome: | The proposed framework reduces inference latency by 10-20x while maintaining the student model’s performance quality. |
Copied to clipboard
| Challenge: | Antimicrobial resistance (AMR) is one of the top ten global public health threats . pharmaceutical industries face high costs (over $1 billion), long timelines, and a high failure rate . |
| Approach: | They propose an LLM-based pipeline that acts as an alert system, detecting prior evidence of antibiotic activity to prevent costly rediscoveries. |
| Outcome: | The proposed system integrates literature on organisms and chemicals into a Knowledge Graph (KG), ensuring taxonomic resolution, synonym handling, and multi-level evidence classification. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced multi-turn conversation systems, emphasizing the need for proactive guidance to enhance users’ interactions. |
| Approach: | They propose a goal-adaptive supervised fine-tuning framework that generates proactive guidance for users to click for the next turn of the conversation. |
| Outcome: | The proposed framework achieves 86.10% accuracy in offline evaluation (+23.95% over baseline) and 25.28% CTR in online deployment (149.06% relative improvement). |
Copied to clipboard
| Challenge: | Text-to-Speech (TTS) training requires extensive and diverse text and speech data. |
| Approach: | They propose a synthetic speech data generation pipeline that generates multilingual, domain-specific datasets for TTS training. |
| Outcome: | The proposed pipeline generates data that is 10–48% more diverse than baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. |
Copied to clipboard
| Challenge: | Existing methods for identifying subgroups raise privacy concerns and gather sensitive information at runtime might be impractical. |
| Approach: | They propose a method to identify and train underperforming subgroups and train a model to predict if an utterance belongs to these subgroup. |
| Outcome: | The proposed method reduces biases and improves performance on intent classification and automatic speech recognition tasks. |
Copied to clipboard
| Challenge: | Existing methods for Community Question Answering (CQA) focus on static knowledge, limiting their applicability to real-world scenarios. |
| Approach: | They propose a retrieval-augmented generation framework for real-time industrial CQA that integrates static knowledge with dynamic historical QA pairs via a centroid-based memory mechanism. |
| Outcome: | The proposed framework outperforms baselines on three industrial CQA datasets and achieves 25.9% improvement in vector similarity, reducing latency by 8.7%–23.3%, and lowering chunk growth from 20.23% to 2.06% over iterations. |
Copied to clipboard
| Challenge: | Empirical tests demonstrate that PlanGPT framework has achieved advanced performance, providing comprehensive support that significantly enhances professional planning efficiency. |
| Approach: | They propose a specialized AI agent framework tailored for urban and spatial planning that integrates a customized local database retrieval system and domain-specific knowledge activation capabilities. |
| Outcome: | Empirical tests show that PlanGPT framework significantly improves planning efficiency . it integrates a customized database retrieval system, domain-specific knowledge activation capabilities, and advanced tool orchestration mechanisms. |
Copied to clipboard
| Challenge: | a recent study shows that LLMs are useful for automating taxonomies from a seed taxonomy to a set of known concepts. |
| Approach: | They propose to use Large Language Models for automated taxonomy generation and completion. |
| Outcome: | The proposed approach is based on an open-source LLM (Llama-3). |
Copied to clipboard
| Challenge: | Using large language models and human-in-the-loop quality control, we enrich multilingual e-book and audio book data and make our product catalog easier to navigate for children. |
| Approach: | They propose a user-centered, empirically guided approach to multilingual metadata enrichment for children’s books using large language models and human-in-the-loop quality control. |
| Outcome: | The proposed approach delivers high-quality labels and improves user experience in real-world production environments. |
Copied to clipboard
| Challenge: | a lack of publicly available datasets for training and benchmarking limits current AI techniques' effectiveness in industry-specific applications. |
| Approach: | They propose an email automation pipeline that automates email response generation at scale in real-world enterprise settings. |
| Outcome: | The proposed pipeline automates email response generation at scale in real-world environments. |
Copied to clipboard
| Challenge: | Modern text processing pipelines require robust methods to remove extraneous content while preserving a document’s core message. |
| Approach: | They propose a method that leverages multilingual sentence embeddings and approximate nearest-neighbor search to identify and excise unwanted text segments. |
| Outcome: | Experiments on HTML datasets show that SORE outperforms structural methods and yields high precision in diverse scenarios. |
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) for low-resource variants of English remains challenging, as most models are trained on datasets predominantly focused on American or British English. |
| Approach: | They propose a new output format for Named Entity Recognition (NER) that achieves a three-fold reduction in inference time compared to JSON format. |
| Outcome: | The proposed output format achieves a three-fold reduction in inference time on average compared to JSON format, which is widely used for structured outputs. |
Copied to clipboard
| Challenge: | Empirical studies on their effectiveness and acceptance in real-world classrooms are limited, leaving their practical impact uncertain. |
| Approach: | They develop an LLM-based virtual teaching assistant and deploy it in an introductory AI programming course with 477 graduate students. |
| Outcome: | The proposed system is tested in an introductory AI programming course with 477 graduate students. |
Copied to clipboard
| Challenge: | Currently, practitioners working on dense retrieval face a bewildering number of choices. |
| Approach: | They propose a framework for thinking about retrieval in terms of nearest-neighbor search over vector representations where these representations can be dense (typically called embeddings, generated from transformers) or flat (with brute-force search) |
| Outcome: | The proposed model explicates tradeoffs between HNSW and flat indexes from the perspectives of indexing time, query evaluation performance, and retrieval quality. |
Copied to clipboard
| Challenge: | Effective content moderation is essential for video platforms to safeguard user experience and uphold community standards. |
| Approach: | They propose a method to transform a generative MLLM into a multimodal classifier using minimal discriminative training data. |
| Outcome: | The proposed method improves F1 score by 66.50% over traditional classifiers while requiring only 2% of the fine-tuning data. |
Copied to clipboard
| Challenge: | Ambiguous user queries pose a challenge in task-oriented dialogue systems . Large Language Models (LLMs) rely on the top-k retrieved documents for clarification . traditional approaches lack principled mechanisms to determine when to use broad domain knowledge vs specific retrieved document context for clarification. |
| Approach: | They propose a hybrid approach that dynamically chooses between document-based or aspect-based clarification based on query ambiguity. |
| Outcome: | The proposed approach shows significant improvements over baselines on product troubleshooting and product search datasets. |
Copied to clipboard
| Challenge: | Existing approaches to train large-scale models with extensive datasets are limited by their inadequate planning capabilities compared to humans. |
| Approach: | They propose a paradigm that enhances the performance of Large Language Models (LLMs) they use look-ahead planning to refine action selection and LEAN to streamline navigation through agile prompt construction. |
| Outcome: | The proposed framework outperforms agents trained via imitation learning, reinforcement learning, and reasoning-based approaches without any fine-tuning. |
Copied to clipboard
| Challenge: | Existing methods for retrieval of recommendation systems rely on collaborative filtering signals and lacks similarity for long-tail items. |
| Approach: | They propose a Motivation-aware Retrieval for Long-Tail Recommendation that integrates purchase motivations with traditional item features to capture similarity among long-tail items. |
| Outcome: | The proposed model captures similarity between long-tail items while maintaining collaborative filtering advantages for popular items. |
Copied to clipboard
| Challenge: | EHRs contain vast amounts of valuable clinical data, stored as unstructured text. |
| Approach: | They propose a method that uses existing NER+L methods to classify medical entities at scale using a named entity recognition and linking task. |
| Outcome: | The proposed model outperforms Bi-LSTM in minority class tasks with up to 28% of the time and 32% faster training time. |
Copied to clipboard
| Challenge: | Suboptimal prompts can introduce biases, inconsistencies, and unreliable evaluations. |
| Approach: | They propose an active-sampling-based framework for automatic prompt optimization . they use a small, diverse subset of samples to guide prompt refinement . |
| Outcome: | The proposed framework outperforms baselines on four popular LLMs and three real-world datasets. |
Copied to clipboard
| Challenge: | With the emergence of ChatGPT, transformer-only models have significantly advanced text classification and related tasks. |
| Approach: | They propose to use prompt engineering and supervised fine-tuning methods for transformer-based text classification in industrial applications. |
| Outcome: | The proposed models perform well in a variety of industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. |
Copied to clipboard
| Challenge: | Existing methods for text chunking struggle with document structure and noise . Existing approaches struggle with maintaining semantic coherence while handling complex documents. |
| Approach: | They propose a bottom-up approach to chunking that combines document structure awareness with noise elimination. |
| Outcome: | The proposed method outperforms existing methods in noise reduction, completeness, context coherence, task relevance, and retrieval performance. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be used to broaden user experiences beyond established preferences and reinforce feedback loops. |
| Approach: | They propose a hierarchical approach that combines hierarchic planning with LLM inference-time scaling to improve recommendation relevancy without compromising novelty. |
| Outcome: | The proposed approach shows significant gains in both user satisfaction and exploration diversity. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enable natural language to SQL conversion, but generating accurate, efficient queries is challenging due to ambiguous intent, domain knowledge requirements and database constraints. |
| Approach: | They propose a system for reliable SQL generation that integrates Table Onboarder, SQL Generator and Feedback Augmentation. |
| Outcome: | The proposed system surpasses the best single-LLM baseline by 21.5% and the strongest pipeline competitor by 5.3% on public benchmarks and internal datasets. |
Copied to clipboard
| Challenge: | Existing methods for lexical retrieval struggle due to semantic mismatches and overlapping terminologies, and ambiguous abbreviations common in specialized fields like finance and cloud computing. |
| Approach: | They propose a scalable hard-negative mining framework that dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. |
| Outcome: | The proposed framework improves on public domain datasets and shows that it is generalizable and ready for real-world applications. |
Copied to clipboard
| Challenge: | Traditionally, company comparisons rely on relative returns and discrete classifications, or a combination of both. |
| Approach: | They propose to use clusters of embeddings to enhance the interpretability of Large Language Models by decomposing Large Language models activations into interpretable features. |
| Outcome: | The proposed clusters of embeddings capture the internal representation of a company description, rather than just semantic similarity alone. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have greatly improved the performance on most natural language tasks, and often show surprisingly good zero-shot generalization to new domains. |
| Approach: | They propose to continuously pretrain the Llama 3.1 base models on 1 trillion tokens of e-commerce data to introduce domain specific knowledge into the model while at the same time keeping the general capabilities intact. |
| Outcome: | The proposed model can be adapted to the new domain without sacrificing performance on general domain tasks. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used as computer-use agents . authors present a novel attack framework that bypasses refusal-trained safeguards . |
| Approach: | They propose a new attack framework that bypasses refusal-trained safeguards in LLMs . SUDO iteratively refines its attacks based on a built-in refusal feedback . authors highlight need for robust, context-aware safeguards if LLM is to be used . |
| Outcome: | The proposed framework bypasses refusal-trained safeguards in commercial agents . it achieves a stark attack success rate of 24.41% (with no refinement) and up to 41.33% (by iterative refinement). |
Copied to clipboard
| Challenge: | Existing systems focus primarily on assessment rather than treatment planning. |
| Approach: | They propose a framework that structures LLM reasoning to align with real-life workflows. |
| Outcome: | The proposed framework outperforms baseline approaches in assessment accuracy and treatment plan quality. |
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models for specific tasks require extensive seed datasets or struggle to balance task relevance and data diversity. |
| Approach: | They propose a data synthesis framework that uses a multi-hop process to expand very few seed data points while ensuring data diversity and task relevance. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in task-specific fine-tuning by over 30%. |
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have achieved impressive performance on many language tasks. |
| Approach: | They synthesize a high-quality dataset of error correction pairs to evaluate and improve LLMs for mobile applications by reweighting the sample. |
| Outcome: | The proposed model improves on offline evaluation and live A/B testing, given the LLM performance on offline data and scores from a small privacy-preserving on-device language model. |
Copied to clipboard
| Challenge: | Multilingual automatic speech recognition (ASR) in the medical domain is a critical foundational task, serving a wide range of downstream applications such as speech translation, spoken language understanding, and voice-activated assistants. |
| Approach: | They present the first multilingual medical ASR dataset and the first collection of small-to-large end-to end medical APR models spanning five languages: Vietnamese, English, German, French, and Mandarin Chinese. |
| Outcome: | The proposed model covers Vietnamese, English, German, French, and Mandarin Chinese, and is the first multilingual ASR dataset across five languages. |
Copied to clipboard
| Challenge: | Existing visual attribute value extraction methods rely on product images and textual information, which can be ambiguous, inaccurate, or unavailable. |
| Approach: | They propose a framework that leverages a curated pool of image captioning models to generate accurate captions from product images. |
| Outcome: | The proposed framework significantly improves state-of-the-art large multimodal models in zero-shot and fine-tuning settings. |
Copied to clipboard
| Challenge: | Existing tools to evaluate large language models in the financial domain are limited by the inherently closed nature of the financial industry. |
| Approach: | They present the first open leaderboard for evaluating Korean large language models focused on finance. |
| Outcome: | The proposed model is FINKRX, a fully open and transparent LLM built using these best practices. |
Copied to clipboard
| Challenge: | Sentiment Reasoning is an auxiliary task in sentiment analysis where the model predicts both the sentiment label and generates the rationale behind it based on the input transcript. |
| Approach: | They propose a task - Sentiment Reasoning - for both speech and textmodalities and propose 'multimodal multitask framework' . they propose to use a model that generates the rationale behind each predicted label and provides a rationale for model prediction with quality semantically comparable to humans. |
| Outcome: | The proposed task improves model transparency by providing rationale for model prediction with quality semantically comparable to humans while improving model’s classification performance. |
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are expensive and time-consuming to evaluate . however, they are limited in their use in industrial settings due to their limited availability and limited resources. |
| Approach: | They evaluate 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. |
| Outcome: | The proposed models can be used to assess chart comprehension and reasoning tasks, but they are expensive and time-consuming. |
Copied to clipboard
| Challenge: | Experimental evaluation on 2,589 occupational health cases demonstrates that OccuTriage outperforms single-agent approaches with a 20.16% average discordance rate compared to baseline rates of 43.05% . |
| Approach: | They propose to use specialized LLM agents, retrieval augmentation, and a bidirectional decision architecture to simulate healthcare professionals’ reasoning. |
| Outcome: | The proposed framework outperforms single-agent approaches with a 20.16% average discordance rate while matching or exceeding human expert performance. |
Copied to clipboard
| Challenge: | Existing large reasoning models are limited by their closed nature and high API costs and safety issues. |
| Approach: | They propose to build a long CoT dataset with existing short CoT LLMs that are not trained for inference-time scaling. |
| Outcome: | The proposed model achieves quality comparable to—or slightly below—R1 and is able to think longer and provide control over the thought budget to better manage the overthinking problem. |
Copied to clipboard
| Challenge: | Empirical studies show that SingaKids provides effective dialogic teaching, benefiting learners at different performance levels. |
| Approach: | They propose a dialogic tutor designed to facilitate language learning through picture description tasks. |
| Outcome: | Empirical studies show that SingaKids provides effective dialogic teaching, benefiting learners at different performance levels. |
Copied to clipboard
| Challenge: | Existing frameworks for streaming and non-streaming ASR models have been used to reduce development, training and deployment costs. |
| Approach: | They propose to use dynamic right-context through chunked attention masking to train zipformer-based ASR models. |
| Outcome: | The proposed framework reduces word error by relative 7.9% with a small degradation in user-perceived latency. |
Copied to clipboard
| Challenge: | Existing query classification methods rely on posterior click behavior to construct training samples, resulting in insufficient prior information for modeling. |
| Approach: | They propose a semi-supervised scaleable unified framework that integrates enhanced modules to unify query classification tasks. |
| Outcome: | The proposed framework outperforms the state-of-the-art models in offline and online A/B experiments. |
Copied to clipboard
| Challenge: | CodeIF assesses the ability of large language models to adhere to task-oriented instructions in code generation tasks. |
| Approach: | They introduce a benchmark designed to assess LLMs' ability to adhere to task-oriented instructions within diverse code generation scenarios. |
| Outcome: | The proposed benchmark assesses LLMs' ability to adhere to task-oriented instructions in code generation tasks across a wide range of complexity levels and programming domains. |
Copied to clipboard
| Challenge: | Existing benchmarks focus on isolated components rather than addressing the broader needs of BI users. |
| Approach: | They propose a holistic, end-to-end benchmarking framework that categorizes queries into descriptive, diagnostic, predictive, and prescriptive types, aligning with practical BI needs. |
| Outcome: | The proposed framework assesses BI systems on quality, relevance, depth of insights based on queries categorized into descriptive, diagnostic, predictive, and prescriptive types . |
Copied to clipboard
| Challenge: | Existing methods do not incorporate feedback from the query relevance model, limiting their ability to generate queries that enhance product retrieval. |
| Approach: | They propose an adversarial reinforcement learning framework that exposes weaknesses in query classification models by creating synthetic queries that augment the classifier's training set. |
| Outcome: | The proposed framework improves query generation performance on public datasets and on proprietary datasets. |
Copied to clipboard
| Challenge: | Automating benefit verification phone calls saves time and improves patient care. |
| Approach: | They propose a second-stage postprocessing pipeline that reduces manual effort while maintaining a high bar for accuracy. |
| Outcome: | The proposed system significantly reduces manual effort while maintaining a high bar for accuracy while reducing noise and jargon. |
Copied to clipboard
| Challenge: | In-car AI assistants struggle with multi-turn conversations and fail to handle cognitively complex follow-up questions. |
| Approach: | They propose a framework that leverages Bloom's Taxonomy to generate follow-up questions with increasing cognitive complexity and a Gricean-inspired evaluation framework to assess their Logical Consistency, Informativeness, Relevance, and Clarity. |
| Outcome: | The proposed framework validates both LLM-based and human evaluations and identifies the specific cognitive complexity level at which in-car AI assistants begin to falter information. |
Copied to clipboard
| Challenge: | Existing work on large language models lacks a realistic environment and parallelized framework to support complex interactions between agents and environments. |
| Approach: | They propose a framework that integrates realistic societal environments and parallelized interactions to support simulations of large-scale agents. |
| Outcome: | The proposed framework can support simulations of 30,000 agents faster than the wall-clock time with 24 NVIDIA A800 GPUs and the performance increases linearly with the increase of LLM computational resources. |
Copied to clipboard
| Challenge: | Recent advances in large language models have drawn attention for their potential to automate and optimize processes across diverse sectors. |
| Approach: | They propose a specialized LLM for plant construction engineering that delivers optimized responses to plant engineers by leveraging enriched domain knowledge. |
| Outcome: | The proposed model delivers optimized responses to plant engineers by leveraging enriched domain knowledge. |
Copied to clipboard
| Challenge: | Existing methods fail to reconcile click-through rate (CTR) optimization with topic expansion. |
| Approach: | They propose a query generation framework that aligns click-through rate and topic expansion goals through an online DPO paradigm. |
| Outcome: | The proposed approach achieves significant CTR gains (+2.3%) and higher human-rated query quality compared to state-of-the-art methods. |
Copied to clipboard
| Challenge: | Compared to math word problems, geometry problems emphasize multi-modal formats and the translation between informal and formal languages. |
| Approach: | They propose a symbolic deduction engine-based geometry problem generation framework that leverages a symbolic deduction engine to generate geometry problems. |
| Outcome: | The proposed method avoids inherent biases in translating natural language into formal language and guarantees to control the generated problems in terms of knowledge points and difficulties by an elaborate checking function. |
Copied to clipboard
| Challenge: | Structured table extraction from unstructured text is critical for automating data processing tasks across industries where accuracy and reliability are paramount. |
| Approach: | They propose a natural language-based method for extracting structured tables from text . they use Python classes or SQL statements to explicitly construct table structures . |
| Outcome: | The proposed method improves F1 scores and mitigates hallucinations . it integrates with standard SQL databases and Python workflows, ensuring seamless deployment . |
Copied to clipboard
| Challenge: | Clinical note generation (CNG) tools are being developed to address extended working hours and healthcare provider fatigue. |
| Approach: | They evaluate the reliability of 12 open-weight and proprietary LLMs from Anthropic, Meta, Mistral, and OpenAI in CNG in terms of their ability to generate notes that are string equivalent (consistency rate), have the same meaning (semantic consistency) and are correct (symbol similarity) |
| Outcome: | The results show that the LLMs generated notes that are string equivalent (consistency rate), have the same meaning (semantic consistency) and are correct (symbol similarity) overall, Meta’s Llama 70B was the most reliable, followed by Mistral’s Small model. |
Copied to clipboard
| Challenge: | Existing Document AI frameworks lack the capability to structurally organize and manage document information. |
| Approach: | They propose a framework that corrects OCR errors at the character, word, and structural levels and a synthetic data generation strategy that realistically simulates such errors to train an effective correction model. |
| Outcome: | The proposed framework improves document retrieval and question answering tasks by correcting errors introduced by OCR errors at the character, word, and structural levels. |
Copied to clipboard
| Challenge: | key-value caches in large language models consume memory, posing a major challenge for scalable deployment. |
| Approach: | They propose a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. |
| Outcome: | The proposed technique reduces the KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. |
Copied to clipboard
| Challenge: | Emotional support conversation (ESC) aims to alleviate the emotional distress of individuals through effective conversations. |
| Approach: | They propose a framework that bootstraps the planning during ESC and determines the optimal strategy based on long-term returns. |
| Outcome: | The proposed framework outperforms baseline models on ESC datasets and can be used to guide the LLM to response. |
Copied to clipboard
| Challenge: | generative AI is revolutionizing how users interact with smartphones, transforming how they interact with them. |
| Approach: | They propose a framework for task instruction recommendation that enables intuitive one-touch AI tasking on smartphones. |
| Outcome: | The proposed framework shows significant improvements in recommendation accuracy and coherence and intent alignment with predefined instruction candidates. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated potential in code generation and natural language understanding, but they struggle with code constraints. |
| Approach: | They propose to use Large Language Models to handle constraints represented in code . they use JSON, YAML, XML, Python, and natural language to test their effectiveness . |
| Outcome: | The proposed benchmark shows that LLMs can handle code constraints better than natural language . the results suggest that conscious choice of representations can lead to optimal use of LLM in enterprise use cases involving code constraints. |
Copied to clipboard
| Challenge: | Dual quality is a problem where products with identical ingredients or characteristics are sold under the same brand and similar packaging in different markets, but are significantly altered in composition or quality parameters. |
| Approach: | They propose to use natural language processing to detect inconsistent product quality by analyzing a Polish-language dataset and using different approaches. |
| Outcome: | The proposed approach can detect and address inconsistent product quality in Polish and other languages. |
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) systems aim to evaluate the quality of candidate writing using computational methods. |
| Approach: | They propose a model that assigns a confidence score to each automated score to ensure it meets high reliability standards. |
| Outcome: | The proposed model achieves an F1 score of 0.97 and releases 47% of predicted scores with 100% CEFR agreement and 99% with at least 95% CEFR agreeance compared to the standalone model where all predicted scores are released. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have led to significant improvements in various service domains, including search, recommendation, and chatbot applications. |
| Approach: | They propose a framework for developing scalable, controllable, and reliable AI-driven agents that can be applied to real-world applications. |
| Outcome: | The proposed framework bridges the gap between academic research and real-world application, and enables scalable, controllable, and reliable AI-driven agents. |
Copied to clipboard
| Challenge: | Existing document question answering methods reduce inference costs and input tokens. |
| Approach: | They propose a retrieval-augmented generation method that automatically extracts useful entities and generates summaries from documents. |
| Outcome: | The proposed method surpasses baseline retrieval-augmented generation (RAG) and long-context question answering (LC) methods achieve higher accuracy by processing entire documents, but at the cost of increased computational Corresponding authors. |
Copied to clipboard
| Challenge: | Recent advances in Retrieval-Augmented Generation (RAG) frameworks and Vision-Language Models (VLMs) have improved retrieval performance on multimodal documents by processing pages as images. |
| Approach: | They propose a cost-effective multimodal document processing system that dynamically selects the processing modalities for each page as an image or text based on page characteristics and query intent. |
| Outcome: | The proposed system reduces average query processing latency by 2.29 and cost by up to 10 . it reduces cost and latency while maintaining high performance on large scale deployments . |