Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Optimizing Entity Resolution in Voice Interfaces: An ASR-Aware Entity Reference Expansion Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) errors in voice-based dialog systems pose significant impediments to downstream tasks. |
| Approach: | They propose an automatic speech recognition (ASR) error-aware loss function to inject failed mentions and resolved entity names into the knowledge graph to enhance its awareness of unresolved mentions. |
| Outcome: | The proposed system enhances the knowledge graph's awareness of unresolved mentions by injecting pairs of failed mentions and resolved entities into the knowledge map. |
Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing solutions for hallucination detection do not consider latency, train or evaluate on production data. |
| Approach: | They propose to use customer service conversation data to evaluate existing methods . they propose to train small encoder models on a new dataset to outperform existing methods. |
| Outcome: | The proposed model outperforms existing methods and highlights the value of combining small amounts of in-domain data with public datasets. |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate concise summarizations require extensive modifications to the architecture. |
| Approach: | They propose a lightweight model that can be finetuned to extract salient keyphrases from the source document to enhance ROUGE F1 and recall. |
| Outcome: | The proposed model can be finetuned to extract salient keyphrases without any LLM customization. |
Predicting Entity Salience in Extremely Short Documents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | False positive: ES is a natural language understanding task that determines which entities are most salient to a passage . Falsity: Popsicle, Frank Epperson and San Francisco are salient entities . |
| Approach: | They propose a lightweight and data-efficient approach for entity salience detection on short documents . they propose he use of a human-labeled dataset to evaluate entity salient on short questions . |
| Outcome: | The proposed approach achieves competitive performance over state-of-the-art models at significant cost and latency advantages. |
Don’t Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) are becoming increasingly popular in business scenarios, but maintaining topic continuity is a challenge. |
| Approach: | They propose a topic continuity model that assesses whether a response aligns with the initial conversation topic using a Naive Bayes approach. |
| Outcome: | The proposed model outperforms existing models in handling lengthy and complex conversations. |
Retrieval Augmented Spelling Correction for E-Commerce Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | e-commerce spelling correction services face a challenge with new brand names . we propose a new approach that uses a fine-tuned retrieval algorithm to correct for brand names. |
| Approach: | They propose a method that uses product names to be incorporated into a large language model to do contextual spelling correction. |
| Outcome: | The proposed approach improves performance with only minor latency increases . the proposed approach is more efficient than a stand-alone LLM . |
Scaling Parameter-Constrained Language Models with Quality Data (2024.emnlp-industry)
Copied to clipboard
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra
| Challenge: | Scaling laws in language modeling quantify training loss as a function of dataset size and model parameters, but neglect the critical role of data quality in model generalization. |
| Approach: | They propose to use effective training tokens as a combination of text diversity and syntheticity as measured by a teacher model to calculate scaling laws. |
| Outcome: | The proposed term effective training tokens is a combination of two readily-computed indicators of text diversity and syntheticity as measured by a teacher model. |
INDUS: Effective and Efficient Language Models for Scientific Applications (2024.emnlp-industry)
Copied to clipboard
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grezes, Megan Ansdell, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Lima, Panagiotis Vagenas, S. Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Tsengdar Lee
| Challenge: | Large language models trained on general domain corpora showed remarkable results on natural language processing tasks. |
| Approach: | They develop a suite of large language models trained on general domain corpora that address NLP tasks and smaller versions of them created using knowledge distillation. |
| Outcome: | The proposed models outperform general-purpose and domain-specific encoders on new and existing tasks and in industrial settings. |
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantization-aware Training (QAT) is a popular technique for reducing memory usage and improving computational efficiency in large language models. |
| Approach: | They propose a weight-decomposed low-rank quantization-aware training approach that integrates QAT with a group-specific quantization magnitude adjustment. |
| Outcome: | The proposed method outperforms the state-of-the-art method on LLaMA and LLama2 models. |
Hybrid-RACA: Hybrid Retrieval-Augmented Composition Assistance for Real-time Text Prediction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) enhanced with retrieval augmentation have shown great performance in many applications, but their computational overhead and additional retrieval step limit their effectiveness in real-time tasks. |
| Approach: | They propose a system that combines a cloud-based LLM with a smaller client-side model through retrieval augmented memory to provide real-time text prediction. |
| Outcome: | The proposed system can generate better responses from the cloud-based model while maintaining low latency. |
LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit (2024.emnlp-industry)
Copied to clipboard
Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Dacheng Tao, Xianglong Liu
| Challenge: | Existing quantization techniques have been categorized as 'simple' and 'highly efficient' however, their configurations vary from each other and cannot be fairly compared . |
| Approach: | They propose a plug-and-play compression toolkit to explore the impact of quantization. |
| Outcome: | The proposed toolkit explores the impact of quantization on large language models. |
PDFTriage: Question Answering over Long, Structured Documents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to document QA use a pre-retrieval step to retrieve the relevant context from documents, but this is incongruous with the user's mental model of the document. |
| Approach: | They propose an approach called PDFTriage that enables models to retrieve the context based on either structure or content. |
| Outcome: | The proposed approach can retrieve context based on structure or content across several classes of questions where existing retrieval-augmented LLMs fail. |
Fairness-Aware Online Positive-Unlabeled Learning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Positive-unlabeled (PU) learning is a new approach to improve text classification by analyzing the impact of the online setting on fairness. |
| Approach: | They propose to extend Positive-Unlabeled (PU) learning to online learning by analyzing the impact of the online setting on fairness. |
| Outcome: | The proposed approach improves fairness in PU learning in both offline and online settings by using only labeled positive and unlabeled samples. |
SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to enhance mathematical reasoning and problem-solving abilities of Large Language Models (LLMs) despite their remarkable performance across domains, a notable challenge persists in the realm of mathematical reasoning. |
| Approach: | They propose a sequential learning approach that integrates the Chain-of-Thought and the Program-ofThough. |
| Outcome: | The proposed approach achieves state-of-the-art (SOTA) performance by integrating CoT and PoT learning. |
Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Increasing use of large language models (LLMs) require performant guardrails to ensure the safety of inputs and outputs . when these guardrail are trained on imbalanced data, they can learn the societal biases resulting from the model's performance. |
| Approach: | They propose a method for mitigating counterfactual fairness in closed-source text safety classifiers by using a debiasing regularizer and a threshold-agnostic metric. |
| Outcome: | The proposed method outperforms classifiers and acts as a debiasing regularizer . it uses threshold-agnostic metrics and Fair Data Reweighting (FDW) to assess the counterfactual fairness of a model . |
Centrality-aware Product Retrieval and Ranking (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Ambiguity and complexity of user queries often lead to mismatch between user’s intent and retrieved product titles or documents. |
| Approach: | They propose a user-intent centrality optimization approach which optimizes for the user intent in semantic product search. |
| Outcome: | The proposed approach improves product ranking efficiency for ambiguous queries and lexical terms with alphanumeric characters. |
Fusion-Eval: Integrating Assistant Evaluators with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent studies have employed large language models (LLMs) as reference-free metrics for NLG evaluation, enhancing adaptability to new tasks tasks. |
| Approach: | They propose a method that leverages large language models to integrate insights from various assistant evaluators. |
| Outcome: | The proposed approach achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.7444 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. |
Investigating the Personality Consistency in Quantized Role-Playing Dialogue Agents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Using the Big Five personality traits model, we evaluate how stable assigned personalities are for Quantized Role-Playing Dialog Agents (QRPDA) during multi-turn interactions. |
| Approach: | They propose a non-parametric method to evaluate the stability of assigned personalities in quantized large language models (LLMs) for role-playing scenarios. |
| Outcome: | The proposed method shows that it maintains consistent personality traits in QRPDA, and it is more reliable in real-world applications. |
Robust ASR Error Correction with Conservative Data Filtering (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Error correction (EC) based on large language models is an emerging technology to enhance the performance of automatic speech recognition systems. |
| Approach: | They propose to pair large set of ASR hypotheses with gold references to improve linguistic acceptability over sources and be inferable from available context. |
| Outcome: | The proposed approach significantly reduces overcorrection and improves quality in out-of-domain (OOD) settings. |
Code Representation Pre-training with Complements from Program Executions (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing languages have syntactic representations of code to improve code intelligence, but they are difficult to learn from code. |
| Approach: | They propose to embed dynamic information of programs revealed by their test cases into feature representations of code as complements. |
| Outcome: | The proposed method yields 6%/19% mAP improvements over its masked language modeling counterparts. |
ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency (2024.emnlp-industry)
Copied to clipboard
Yuhang Yao, Han Jin, Alay Shah, Shanshan Han, Zijian Hu, Dimitris Stripelis, Yide Ran, Zhaozhuo Xu, Salman Avestimehr, Chaoyang He
| Challenge: | Large language models (LLMs) are widely used in commercial applications . low latency is crucial due to system latency, query concurrency, and computational resources constraints. |
| Approach: | They propose a system that can be resource-efficiently served by addressing bottlenecks beyond LLM inference . they propose 4.3 speed up over vLLM and 1.5 higher throughput . |
| Outcome: | The proposed system outperforms state-of-the-arts with 1.5 higher throughput . it achieves 4.3 speed up with 64 concurrent requests on Mixtral 8x7B . |
Context Matters: Pushing the Boundaries of Open-Ended Answer Generation with Graph-Structured Knowledge Context (2024.emnlp-industry)
Copied to clipboard
| Challenge: | GraphContextGen outperforms dominant text-based retrieval systems in domain specific community question answering platforms like AskUbuntu, Unix, and ServerFault. |
| Approach: | They propose a framework that combines graph-driven context retrieval with knowledge graphs based enhancement to improve the proficiency of LLMs. |
| Outcome: | The proposed framework outperforms dominant text-based retrieval systems in open-ended questions. |
SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply Chain Disruptions (2024.emnlp-industry)
Copied to clipboard
Zhi-Qi Cheng, Yifei Dong, Aike Shi, Wei Liu, Yuzhi Hu, Jason O’Connor, Alexander Hauptmann, Kate Whitefoot
| Challenge: | EV battery supply chain is vulnerable to disruptions caused by natural disasters and geopolitical tensions. |
| Approach: | They propose a system integrating Large Language Models with domain expertise for EV supply chain risk assessment. |
| Outcome: | Evaluated on 12,070 paragraphs from 365 sources (2022-2023), SHIELD outperforms baseline GCNs and LLM+prompt methods in disruption prediction. |
Divide-Conquer-Reasoning for Consistency Evaluation and Automatic Improvement of Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality and consistency of text generated by Large Language Models are not effective. |
| Approach: | They propose a divide-conquer-reasoning approach to evaluate LLM-generated texts using a split-and-conquers evaluator and an automatic metric converter to facilitate this approach. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by a large margin on multiple benchmarks and reduces 90% of output inconsistencies in one iteration. |
Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. |
| Approach: | They propose to fine tune financial text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents. |
| Outcome: | The proposed embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embeddING from OpenAI. |
TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024.emnlp-industry)
Copied to clipboard
Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang
| Challenge: | Large language models have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools. |
| Approach: | They propose a framework to enhance the task planning and tool usage abilities of LLMs in industrial systems. |
| Outcome: | The proposed framework enhances the task planning and tool usage abilities of LLM-based agents in industrial systems. |
Detecting Ambiguous Utterances in an Intelligent Assistant (2024.emnlp-industry)
Copied to clipboard
| Challenge: | ambiguous utterances can be interpreted as either chat or task intents in intelligent assistants . ambiguity of intent is particularly noticeable in intelligent devices where task-oriented and non-task-oriented utterrances are mixed and most utterations are short due to characteristics of devices. |
| Approach: | They propose to feed sentence embeddings developed from microblogs and search logs with a self-attention mechanism to detect ambiguous utterances robustly. |
| Outcome: | The proposed model outperforms baselines and a strong LLM-based model. |
GeoIndia: A Seq2Seq Geocoding Approach for Indian Addresses (2024.emnlp-industry)
Copied to clipboard
| Challenge: | a new geocoding system for Indian addresses addresses is needed for logistics, urban planning and location-based services. |
| Approach: | They propose a geocoding system for Indian addresses using hierarchical H3-cell prediction using a Seq2Seq framework. |
| Outcome: | The proposed system outperforms existing geocoding platforms in accuracy and reliability across multiple Indian states. |
Moleco: Molecular Contrastive Learning with Chemical Language Models for Molecular Property Prediction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Pre-trained chemical language models (CLMs) are highly effective in predicting molecular properties, but implicitly contain limited structural information. |
| Approach: | They propose a contrastive learning framework to enhance the understanding of molecular structures within chemical language models by measuring the similarity of fingerprint vectors among different molecules. |
| Outcome: | The proposed framework outperforms state-of-the-art models in molecular property prediction. |
SEED: Semantic Knowledge Transfer for Language Model Adaptation to Materials Science (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to adapt pre-trained language models to materials science rely on frequency information from limited downstream datasets. |
| Approach: | They propose a vocabulary expansion method to adapt pre-trained language models to materials science by incorporating latent materials knowledge of lightweight embeddings into PLMs. |
| Outcome: | The proposed method mitigates the limitations of existing adaptation methods and can be used in materials science. |
News Risk Alerting System (NRAS): A Data-Driven LLM Approach to Proactive Credit Risk Monitoring (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Credit risk monitoring is an essential process for financial institutions to evaluate the creditworthiness of borrowing entities. |
| Approach: | They propose a system which proactively alerts Credit Officers to credit-relevant news events . the system has been deployed for nearly three years and has an estimated precision of 77% . |
| Outcome: | The new system has alerted Credit Officers to over 2700 credit-relevant events with an estimated precision of 77%. |
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Unlike other modalities, speech has unique temporal dependencies, making efficient inference methods unexplored. |
| Approach: | They propose a weighted token merging framework specifically designed for speech-related tasks to improve the trade-off between efficiency and performance. |
| Outcome: | The proposed method achieves state-of-the-art efficiency-performance trade-off on speech-related tasks. |
TensorOpera Router: A Multi-Model Router for Efficient LLM Inference (2024.emnlp-industry)
Copied to clipboard
Dimitris Stripelis, Zhaozhuo Xu, Zijian Hu, Alay Shah, Han Jin, Yuhang Yao, Jipeng Zhang, Tong Zhang, Salman Avestimehr, Chaoyang He
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance across a diverse set of domain-specific tasks. |
| Approach: | They propose a non-monolithic LLM querying system that seamlessly integrates various LLM experts into a single query interface and dynamically routes incoming queries to the most high-performant expert based on query’s requirements. |
| Outcome: | The proposed model improves query efficiency by 40% and costs by 30% while maintaining or enhancing model performance by 10%. |
Prompt-Tuned Muti-Task Taxonomic Transformer (PTMTTaxoFormer) (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for Hierarchical Text Classification (HTC) are expensive and require explicit injection of the hierarchy, verbalizers, and/or prompt engineering. |
| Approach: | They propose a hierarchical text classification system that uses a single classifier to predict one or more topics using differentiable prompts and labels that are learnt through backpropagation. |
| Outcome: | The proposed model outperforms existing models on several benchmarks that span a range of topics consistently. |
Arcee’s MergeKit: A Toolkit for Merging Large Language Models (2024.emnlp-industry)
Copied to clipboard
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
| Challenge: | Open-source language models can merge their parameters to improve performance and versatility without additional training. |
| Approach: | They propose to integrate model checkpoints into powerful multitask models without additional training. |
| Outcome: | the library has facilitated the merging of thousands of models, contributing to some of the world’s most powerful open-source model checkpoints. |
Personal Large Language Model Agents: A Case Study on Tailored Travel Planning (2024.emnlp-industry)
Copied to clipboard
Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj, Homa Fashandi, Kevin Ferreira, Chul Lee
| Challenge: | Large Language Models (LLMs) are becoming more autonomous and capable of handling real-world tasks through their access to tools, various planning strategies, and memory, referred to as LLM agents. |
| Approach: | They introduce a personalized version of TravelPlanner and establish baselines for personal LLM agents by comparing generic and personal plans. |
| Outcome: | The proposed model encapsulates user-related information, preferences, and personal concepts and provides baselines for personal LLM agents. |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |
ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in the quality and accessibility of large language models have precipitated a surge in user-facing tools for content generation. |
| Approach: | They propose a pipeline framework for verifiable human-in-the-loop table-to-text generation that is based on a domain specific language and a set of modules that use it as a representation for generating verifierable commentary. |
| Outcome: | The proposed framework learns from human feedback in real-time, needing only a few samples to improve performance. |
BPID: A Benchmark for Personal Identity Deduplication (2024.emnlp-industry)
Copied to clipboard
Runhui Wang, Yefan Tao, Adit Krishnan, Luyang Kong, Xuanqing Liu, Yuqian Deng, Yunzhao Yang, Henrik Johnson, Andrew Borthwick, Shobhit Gupta, Aditi Gundlapalli, Davor Golac
| Challenge: | Data deduplication is a critical task in data management and mining, focused on consolidating duplicate records that refer to the same entity. |
| Approach: | They propose to use a dataset with 1,000,000 unlabeled synthetic PII profiles and a subset of 10,000 pairs curated and labeled as matches or non-matches. |
| Outcome: | The proposed datasets contain synthetic profiles built from publicly available sources that do not represent real individuals. |
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in text-video retrieval neglect the crucial user perspective, leading to discrepancies between user queries and content retrieved. |
| Approach: | They propose a novel, training-free pipeline that leverages Large Language Models for iterative feedback learning. |
| Outcome: | Experimental results show that MERLIN significantly outperforms existing systems in video retrieval. |
Identifying High Consideration E-Commerce Search Queries (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Identifying high consideration queries is essential for e-commerce sites to better serve user needs . ecommerce sites can create or serve customized content for specific queries . |
| Approach: | They propose an engagement-based Query Ranking approach to identify potential engagement levels with query-related shopping knowledge content during product search. |
| Outcome: | The proposed method outperforms human-selected queries in terms of customer impact . human evaluation shows a precision of 96% for HC queries identified by the model . |
Sample Design Engineering: An Empirical Study on Designing Better Fine-Tuning Samples for Information Extraction with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt Engineering (PE) is renowned for improving IE performance through prompt modifications, but the realm of sample design for downstream fine-tuning remains unexplored. |
| Approach: | They propose a methodical approach to enhancing LLMs’ post-tuning performance by refining input, output, and reasoning designs. |
| Outcome: | The proposed approach outperforms heuristic design strategies on three complex IE tasks with four additional LLMs. |
Refining App Reviews: Dataset, Methodology, and Evaluation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | App reviews are valuable for developers, but often contain typos, spelling errors, and complex sentences, hindering efficient interpretation and slowing down app improvement processes. |
| Approach: | They propose to refine 10,000 original and refined reviews from 10 mobile apps using an annotated dataset of 10,000 pairs of reviews. |
| Outcome: | The proposed dataset includes 10,000 original and refined reviews from 10 mobile applications. |
TelBench: A Benchmark for Evaluating Telco-Specific Large Language Models (2024.emnlp-industry)
Copied to clipboard
Sunwoo Lee, Dhammiko Arya, Seung-Mo Cho, Gyoung-eun Han, Seokyoung Hong, Wonbeom Jang, Seojin Lee, Sohee Park, Sereimony Sek, Injee Song, Sungbin Yoon, Eric Davis
| Challenge: | a growing demand for Large Language Models (LLMs) is requiring specialized models to augment customer service agents' skills. |
| Approach: | They propose a methodology for developing a specialized Telecommunications LLM . they use a dataset to evaluate customer service expertise in the telecommunications domain . |
| Outcome: | The proposed model improves the efficiency of customer service agents and reduces response times. |
RRADistill: Distilling LLMs’ Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine (2024.emnlp-industry)
Copied to clipboard
Nayoung Choi, Youngjune Lee, Gyu-Hwung Cho, Haeyu Jeong, Jungmin Kong, Saehun Kim, Keunchan Park, Sarah Cho, Inchang Jeong, Gyohee Nam, Sunghoon Han, Wonil Yang, Jaeho Choi
| Challenge: | Large Language Models excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. |
| Approach: | They propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. |
| Outcome: | The proposed method improves re-ranking for long-tail queries on a Korean-based search platform. |
KorSmishing Explainer: A Korean-centric LLM-based Framework for Smishing Detection and Explanation Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models that detect smishing have a high accuracy but lack interpretability, which undermines user trust and practical applicability. |
| Approach: | They propose an explainable smishing detection framework that adapts to a Korean-centric large language model for sys-phishing detection. |
| Outcome: | The proposed framework achieves 15% improvement in accuracy over existing models and produces high-quality explanatory text. |
Time Matters: An End-to-End Solution for Temporal Claim Verification (2024.emnlp-industry)
Copied to clipboard
| Challenge: | despite growing interest, the verification of temporal claims has not received much attention in the digital space. |
| Approach: | They propose an end-to-end solution that takes cues of temporal information and extracts relevant evidence sentences from temporal claims. |
| Outcome: | The proposed method significantly improves accuracy of temporal claim verification. |
MILD Bot: Multidisciplinary Childhood Cancer Survivor Question-Answering Bot (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing systems in the United States provide limited psychosocial support for childhood cancer survivors due to stigma and disclosing medical history. |
| Approach: | They propose a multidisciplinary chILDhood cancer survivor question-answering bot that uses an intent classifier and a semantic textual similarity model to provide timely, accurate, and high-quality information. |
| Outcome: | The proposed model shows significant promise in offering timely, accurate, and high-quality information, effectively addressing a critical need for support among childhood cancer survivors. |
Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval (2024.emnlp-industry)
Copied to clipboard
Zhirui Kuai, Zuxu Chen, Huimu Wang, Mingming Li, Dadong Miao, Wang Binbin, Xusong Chen, Li Kuang, Yuxing Han, Jiaxing Wang, Guoyu Tang, Lin Liu, Songlin Wang, Jingwei Zhuo
| Challenge: | Generative retrieval (GR) is a transformative paradigm in search and recommender systems . however, data sparsity and long-tailed distribution hinder the full utilization of GR . |
| Approach: | They propose a method to reduce the "Hourglass" phenomenon in RQ-SID where codebook tokens become overly concentrated. |
| Outcome: | The proposed methods improve retrieval efficiency and generalization capabilities. |
Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to named entity recognition are domain specific and require a domain specific architecture. |
| Approach: | They propose a retrieval augmented large language model for Named Entity Recognition . the model uses word-embedding over sentence-level embedding to fine tune . |
| Outcome: | The proposed model outperforms existing models on the CrossNER dataset. |
IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing (2024.emnlp-industry)
Copied to clipboard
Kang Chen, Qing Zhang, Chengbao Lian, Yixin Ji, Xuwei Liu, Shuguang Han, Guoqiang Wu, Fei Huang, Jufeng Chen
| Challenge: | Unlike professional Business-to-Consumer (B2C) e-commerce platforms, consumer-to consumer (C2C), is mainly targeting individual sellers. |
| Approach: | They develop an intelligent product listing tool that generates product descriptions using various product attributes such as category, brand, color, condition, etc. |
| Outcome: | The proposed tool outperforms the base model in domain-specific tasks while producing less hallucination. |
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
Improving Hierarchical Text Clustering with LLM-guided Multi-view Cluster Representation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | a multi-stage approach to hierarchical clustering of interaction drivers in contact centers is proposed . silhouette score and human preference score are improved by 36.7% for top-level clusters compared to standard agglomerative clustering . |
| Approach: | They propose a multi-stage approach that introduces different perspectives or views to improve the quality of hierarchical clustering of interaction drivers in a contact center. |
| Outcome: | The proposed approach improves the quality of generated clusters on public datasets with minimal query time compared to the current state-of-the-art approaches. |
PARA: Parameter-Efficient Fine-tuning with Prompt-Aware Representation Adjustment (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning excel in the context of single-backbone multi-tenant applications. |
| Approach: | They propose to integrate a lightweight vector generator within each Transformer layer to improve prompt-aware representation adjustment. |
| Outcome: | The proposed method surpasses current benchmarks in terms of performance despite having a similar number of adjustable parameters. |
RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenance (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have improved the open-domain QA’s performance, but how to efficiently handle enterprise-exclusive corpora and build domain-specific QA systems are still not studied for industrial applications. |
| Approach: | They propose a general and comprehensive framework based on Retrieval Augmented Generation (RAG) and facilitate the whole business process of establishing QA systems for IT operations and maintenance. |
| Outcome: | The proposed framework achieves superior results on two kinds of QA tasks. |
ULMR: Unlearning Large Language Models via Negative Response and Model Parameter Average (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have attracted significant interest from the research community due to their broad applicability in many language-oriented tasks. |
| Approach: | They propose a framework which uses pre-training datasets to rewrite instructions and generate negative responses to preserve the performance of the original LLM. |
| Outcome: | The proposed framework can erase the pre-training data while maintaining the performance of the original model. |
Pretraining and Finetuning Language Models on Geospatial Networks for Accurate Address Matching (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to address matching and building authoritative address catalogues are limited in data quality and require labeling effort to develop accurate models. |
| Approach: | They propose to view addresses as an address graph and curate inputs by placing geospatially linked addresses in the same context. |
| Outcome: | The proposed framework improves address matching and fine-tuning language models. |
SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a profound impact on a wide range of applications. |
| Approach: | They propose a framework to mitigate the tool-abuse behavior of Large Language Models and propose SMARTCAL to mitigate this issue. |
| Outcome: | The proposed framework improves the performance of LLMs on three datasets with two mainstream tool-use frameworks and shows an 8.6% increase in QA performance and 21.6 percent lower expected calibration error (ECE) than existing methods. |
Probing the Depths of Language Models’ Contact-Center Knowledge for Quality Assurance (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in large Language Models (LMs) have significantly enhanced their capabilities across various domains, including natural language understanding and domain knowledge. |
| Approach: | They propose methods to transfer domain-specific knowledge to smaller models by leveraging evaluation plans generated by more knowledgeable models with optional human-in-the-loop refinement to enhance the capabilities of smaller models. |
| Outcome: | The proposed models improve 18.95% on an in-house QA dataset on a contact-center quality assurance task. |
Intelligent Predictive Maintenance RAG framework for Power Plants: Enhancing QA with StyleDFS and Domain Specific Instruction Tuning (2024.emnlp-industry)
Copied to clipboard
Seongtae Hong, Joong Shin, Jaehyung Seo, Taemin Lee, Jeongbae Park, Cho Young, Byeongho Choi, Heuiseok Lim
| Challenge: | Existing off-premise Question-Answering systems based on Large Language Models face data leakage and domain-specific tuning challenges. |
| Approach: | They propose an on-premise intelligent PMS framework based on a chunking method . they propose instruction tuning using relevant domain-specific data improves LLM performance . |
| Outcome: | The proposed framework improves performance even under limited data conditions. |
Structured Object Language Modeling (SO-LM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Structured objects generation is a challenging problem for existing Large Language Models. |
| Approach: | They propose a self-supervised method to train an LLM to perform the task natively without prompt-engineering. |
| Outcome: | The proposed method matches or outperforms prompt-engineered state-of-the-art models while being more cost-efficient. |
Assisting Breastfeeding and Maternity Experts in Responding to User Queries with an AI-in-the-loop Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | a breastfeeding and maternity expert is a scarce resource and engaging in a conversation with mothers is time-consuming. |
| Approach: | They propose to use RAG to allow experts to draft their responses from a generated text or to answer from scratch. |
| Outcome: | The proposed solution is based on a RAG approach to response generation . experts can draft their response from the generated text or answer from scratch . |
A Hassle-free Algorithm for Strong Differential Privacy in Federated Learning Systems (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Differential privacy (DP) and federated learning (FL) are used for language models training in production mobile keyboard applications. |
| Approach: | They propose a variant of DP-FTRL that uses a correlated noise mechanism to train on-device language models. |
| Outcome: | The proposed method improves privacy-utility trade-off and memory efficiency over existing FL methods while simplifying usage requirements and reducing memory. |
ProConSuL: Project Context for Code Summarization with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental results show that ProConSuL significantly improves code summaries and reduces the number of hallucinations. |
| Approach: | They propose a framework to provide a large language model with precise information about the code structure from program analysis methods. |
| Outcome: | The proposed framework significantly improves code summaries and reduces hallucinations compared to the base model. |
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. |
| Approach: | They propose a method that routes queries to RAG or LC based on model self-reflection. |
| Outcome: | The proposed method significantly reduces the computation cost while maintaining a comparable performance to RAG. |
MARS: Multilingual Aspect-centric Review Summarisation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for summarizing customer feedback are not able to extract actionable reviews into a specific target language. |
| Approach: | They propose a framework involving extract-then-summarise to summariser customer feedback into a specific language. |
| Outcome: | The proposed framework improves abstractive baselines and efficiency to real-time systems. |
A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection tasks (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Virtual assistants often can handle only a limited scope of intents. |
| Approach: | They propose to combine out-of-scope (OOS) rejection with intent classification . they propose to regularize cross-entropy loss with an in-scope embedding reconstruction loss . |
| Outcome: | The proposed method achieves a 1-4% improvement in the area under the precision-recall curve for rejecting out-of-sample instances without compromising intent classification performance. |
Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for meeting summarization rely on transcripts and generate generic summaries, failing to contextualize long discussions and to tailor information to individual preferences and productivity requirements. |
| Approach: | They propose a multi-source approach that considers supplementary materials and generates a summary from this enriched transcript. |
| Outcome: | The proposed model increases summary relevance by 9% and produces more content-rich outputs. |
Detecting LLM-Assisted Cheating on Open-Ended Writing Tasks on Language Proficiency Tests (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used for open-ended writing tasks . however, there are limitations in detecting LLM-generated samples . |
| Approach: | They propose a framework for training LLM-generated text detectors that can detect LLM generated samples after being copy-typed. |
| Outcome: | The proposed model outperforms the transformer-based classifier on a high-stakes online English proficiency test. |
Can Machine Unlearning Reduce Social Bias in Language Models? (2024.emnlp-industry)
Copied to clipboard
Omkar Dige, Diljot Arneja, Tsz Fung Yau, Qixuan Zhang, Mohammad Bolandraftar, Xiaodan Zhu, Faiza Khattak
| Challenge: | Existing methods for mitigating bias in language models are expensive and time-consuming . comparative studies have not evaluated their respective advantages and disadvantages . |
| Approach: | They propose to use Partitioned Contrastive Gradient Unlearning and Negation via Task Vector to reduce social biases in open-source language models. |
| Outcome: | The proposed methods outperform PCGU and DPO in debiasing models . the proposed methods can be easily tuned to balance the trade-off between bias reduction and generation quality . |
Don’t be my Doctor! Recognizing Healthcare Advice in Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming increasingly popular in everyday use, especially in highly regulated domains such as healthcare, where misleading advice may influence users to commit malpractice. |
| Approach: | They present a large-scale health-advice benchmark dataset that evaluates large language models' ability to recognize health-related advice in industrial settings. |
| Outcome: | The proposed model can be misinterpreted as direct advice in highly regulated domains such as healthcare, but the results are not enough to protect them from misinterpreting them as medical advice. |
Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models that are pre-trained on a general domain can deteriorate performance due to domain shift when applied to new domains. |
| Approach: | They propose to train a multilingual non-profit IR system for the Islamic domain using Rust Language capabilities. |
| Outcome: | The proposed model outperforms models pre-trained on general domains and on resource-constrained devices. |
Adapting LLMs for Structured Natural Language API Integration (2024.emnlp-industry)
Copied to clipboard
Robin Chan, Katsiaryna Mirylenka, Thomas Gschwind, Christoph Miksovic, Paolo Scotton, Enrico Toniato, Abdel Labbi
| Challenge: | API integration is crucial for enterprise systems, but there are challenges in combining APIs based on user intent. |
| Approach: | They propose a framework that leverages large language models to integrate APIs based on natural language input. |
| Outcome: | The proposed framework improves performance over existing methods and RAGs based on open APIs . it can learn structural API constraints implicitly during training and retain structured knowledge . |
OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to QA require multiple modalities and a broad pool of information sources to generate coherent answers. |
| Approach: | They propose a new resource to evaluate the effectiveness of question answering systems that perform retrieval augmented generation in scenarios that demand reasoning on multi-modal, multi-document contexts. |
| Outcome: | The proposed method evaluates question answering systems that perform retrieval augmented generation (RAG) in open-domain questions . it requires systems to navigate diverse modalities and a broad pool of information sources, making it uniquely challenging. |
Survival of the Safest: Towards Secure Prompt Optimization through Interleaved Multi-Objective Evolution (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental evaluations across diverse benchmark datasets affirm ‘s efficacy in delivering high performance and notably enhancing safety and security compared to single-objective methods. |
| Approach: | They propose a multi-objective prompt optimization framework that integrates semantic, feedback, and crossover mutations to effectively traverse the prompt landscape. |
| Outcome: | The proposed framework enhances both performance and security in large language models while keeping computational demands low. |
Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantitative investing relies on extracting quantitative features or signals from various data sources including market prices, economic indicators, financial text, etc. |
| Approach: | They propose to integrate LLMs’ token-level embeddings into a forecasting module and compare their results to those of encoder-only and decoder-based models. |
| Outcome: | The proposed model outperforms conventional sentiment scores on multiple investment universes and is based on encoder-only and decoder-based models. |
AmazonQAC: A Large-Scale, Naturalistic Query Autocomplete Dataset (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing systems that provide a graphical representation of QAC are limited in their ability to provide real-time data. |
| Approach: | They introduce a new QAC dataset sourced from Amazon Search logs . they assess Prefix Trees, semantic retrieval, and Large Language Models with and without finetuning . |
| Outcome: | The proposed system can predict search terms based on user-typed prefixes . the proposed system achieves only half of what is theoretically possible on the test data . |
Language, OCR, Form Independent (LOFI) pipeline for Industrial Document Information Extraction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models for low-resource language (LRL) documents are limited in semantic entity extraction, and there are limitations in SER from word level results. |
| Approach: | They propose a pipeline for Document Information Extraction (DIE) in low-resource language (LRL) business documents that solves language, Optical Character Recognition (OCR), and form dependencies through flexible model architecture, a token-level box split algorithm, and the SPADE decoder. |
| Outcome: | Experiments on Korean and Japanese documents show that the pipeline performs well in the Semantic Entity Recognition task without pre-training. |
The State of the Art of Large Language Models on Chartered Financial Analyst Exams (2024.emnlp-industry)
Copied to clipboard
Mahmoud Mahfouz, Ethan Callanan, Mathieu Sibue, Antony Papadimitriou, Zhiqiang Ma, Xiaomo Liu, Xiaodan Zhu
| Challenge: | Chartered Financial Analyst (CFA) program is widely recognized globally . study compares state-of-the-art large language models with open-source models . proprietary models pass levels I and II, but fail at level III due to essay questions . |
| Approach: | They benchmark five leading proprietary models and eight open-source models on mock CFA exams to provide an overview of their financial analysis capabilities. |
| Outcome: | The models on the mock CFA exams pass the highest scores, but fail at the lowest levels due to essay questions. |
Value Alignment from Unstructured Text (2024.emnlp-industry)
Copied to clipboard
Inkit Padhi, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Manish Nagireddy, Pierre Dognin, Kush Varshney
| Challenge: | Currently, alignment of large language models to value systems relies on the availability of supervised and preference data. |
| Approach: | They propose a systematic approach for aligning large language models to values in unstructured text data using synthetic data generation techniques. |
| Outcome: | The proposed approach shows improved performance on the Mistral-7B-Instruct model compared to other approaches, as quantified through the use of automatic metrics and win rates. |
LARA: Linguistic-Adaptive Retrieval-Augmentation for Multi-Turn Intent Classification (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Multi-turn intent classification is challenging due to the complexity and evolving nature of conversational contexts . lack of data on multi-turn datasets makes it difficult to collect multi-turned datasets a challenge . |
| Approach: | They propose a framework for multi-turn intent classification that integrates a retrieval-augmented mechanism with a fine-tuned smaller model. |
| Outcome: | The proposed framework improves accuracy on multi-turn intent classification tasks across six languages. |
Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models (2024.emnlp-industry)
Copied to clipboard
James Fletcher, Nicholas Dehnen, Seyed Nima Tayarani Bathaie, Aijun An, Heidar Davoudi, Ron DiCarlantonio, Gary Farmaner
| Challenge: | Existing image description systems are trained mainly on natural images, whereas icon images are drawings. |
| Approach: | They propose to use a dataset to generate both visual and functional icon descriptions based on the icon image and its context information in the car manual. |
| Outcome: | The proposed model performs well on the dashboard icon description task while the third model perform poorly. |
Athena: Safe Autonomous Agents with Verbal Contrastive Learning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing safety benchmarks on the ability of large language models to perform tasks are lacking. |
| Approach: | They propose a framework that leverages verbal contrastive learning to guide agents towards safety . they use past safe and unsafe trajectories as in-context examples to guide them towards safety. |
| Outcome: | The proposed framework leverages verbal contrastive learning to guide agents towards safety while performing tasks. |
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024.emnlp-industry)
Copied to clipboard
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G P Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi
| Challenge: | Existing research explores the use of Large Language Models (LLMs) as the backbone of agentic systems. |
| Approach: | They propose a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling that has better generalizability on multiple tasks across seven evaluation benchmarks. |
| Outcome: | The proposed model outperforms more than 15 other models on out-of-domain datasets and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL). |
Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing LLMs require a new call to the inference endpoint/API for each new query . repeated calls to the endpoints/AP Is expensive and impractical for many real-world use cases. |
| Approach: | They compare the performance of various LLMs for query-based meeting summarization . they find that combining queries for the same context in a single prompt can be used to minimize repeated calls. |
| Outcome: | The proposed approach reduces the number of calls to the inference endpoints/APIs in meeting summarization tasks. |
DiAL : Diversity Aware Listwise Ranking for Query Auto-Complete (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Query Auto-Complete (QAC) is an essential search feature that helps users articulate their query by suggesting relevant completions as they type. |
| Approach: | They propose a new framework that explicitly optimizes for diversity alongside customer feedback signals to balance relevance and diversity. |
| Outcome: | The proposed framework yields an improvement of 8.5% in MRR and 22.8% in NDCG compared to the pairwise ranking approach on an eCommerce dataset. |
Systematic Evaluation of Long-Context LLMs on Financial Concepts (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Long-context large language models (LC LLMs) are promising for tasks with long context windows . however, their ability to reliably utilize their growing context windows remains under investigation . |
| Approach: | They evaluate the performance of long-context large language models using a real-world financial news dataset. |
| Outcome: | The proposed models exhibit brittleness at longer context lengths even for simple tasks, the authors show . they advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall) |
ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models (2024.emnlp-industry)
Copied to clipboard
Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li
| Challenge: | Knowledge Graphs (KGs) are a powerful tool for capturing structured representations of the world. |
| Approach: | They propose a scalable method for generating up-to-date and configurable conversational KGQA datasets that adheres to human interaction configurations and operates at a significantly larger scale. |
| Outcome: | Qualitative psychometric analyses show that ConvKGYarn produces high-quality data comparable to popular conversational KGQA datasets across various metrics. |
Knowledge-augmented Financial Market Analysis and Report Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to generate financial market analysis text require extensive financial knowledge and skill of financial analysts. |
| Approach: | They propose a task to generate financial market analysis reports using financial market data using a financial knowledge graph. |
| Outcome: | The proposed framework outperforms large-scale language models and retrieval-augmented baselines in the financial market analysis generation task. |
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance. (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Structured generation is used to extract key output information from large language models (LLMs). |
| Approach: | They examine whether constraints on generation space impact LLMs’ abilities, including reasoning and domain knowledge comprehension. |
| Outcome: | The proposed model is based on a few-shot in-context learning and instruction-following capabilities. |
ASTRA: Automatic Schema Matching using Machine Translation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Currently, eCommerce platforms use schema matching to structure product information from disparate sources. |
| Approach: | They propose to model the schema matching problem as a neural machine translation task . they propose to use open-source seq2seq models fine-tuned on product attribute mappings to build a framework . |
| Outcome: | The proposed model achieves a significant performance boost (15% precision and 7% recall uplift) it can support new attributes with precision 95% using only five labeled samples per attribute. |
Neural Search Space in Gboard Decoder (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Gboard decoder uses context, a lexicon and language models to provide a user-friendly keyboard. |
| Approach: | They propose a Neural Search Space which replaces an N-gram LM with a neural network LM and dynamically constructs the search space during decoding. |
| Outcome: | The proposed system improves the quality of the decoded keyboards on various locales with acceptable latency increases. |
Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt leakage poses a compelling security and privacy threat in LLM applications. |
| Approach: | They propose a model which leverages the LLM sycophancy effect and a threat model which fine tunes an open-source model to defend against prompt leakage attempts. |
| Outcome: | The proposed model elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. |
Sequential LLM Framework for Fashion Recommendation (2024.emnlp-industry)
Copied to clipboard
Han Liu, Xianfeng Tang, Tianlang Chen, Jiapeng Liu, Indu Indu, Henry Zou, Peng Dai, Roberto Galan, Michael Porter, Dongmei Jia, Ning Zhang, Lian Xiong
| Challenge: | Existing fashion recommendation systems struggle with the unique challenges of the fashion domain. |
| Approach: | They propose a sequential fashion recommendation framework that leverages a pre-trained large language model enhanced with recommendation-specific prompts. |
| Outcome: | The proposed framework significantly improves fashion recommendation performance on Amazon fashion. |
Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Using natural language as an interaction medium for video editing can mitigate the complexity of video editing. |
| Approach: | They propose a method to fine-tune LLMs for invoking tools in real-time applications by interpreting user stylistic requests in natural language. |
| Outcome: | The proposed model matches the performance of the teacher model significantly, reducing costs and latency. |
Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for fact checking RAG outputs rely on large language models. |
| Approach: | They propose a method that computes a factuality score that can be thresholded to yield a binary decision to check RAG outputs. |
| Outcome: | The proposed method is low latency and low cost at run-time and no need for LLM fine-tuning. |
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model (2024.emnlp-industry)
Copied to clipboard
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar
| Challenge: | Prior work on LLMs focused on models that combine text and one other modality, such as image encoders or proprietary models that are not open sourced. |
| Approach: | They propose a unified model that reasons over diverse input modality signals and generates textual responses. |
| Outcome: | The proposed model performs better on multimodal tasks than industry-leading models . |
SLM as Guardian: Pioneering AI Safety with Small Language Model (2024.emnlp-industry)
Copied to clipboard
Ohjoon Kwon, Donghyeon Jeon, Nayoung Choi, Gyu-Hwung Cho, Hwiyeol Jo, Changbong Kim, Hyunwoo Lee, Inho Kang, Sun Kim, Taiwoo Park
| Challenge: | Prior safety research on large language models focused on aligning them to safety requirements, but internalizing such safeguard features into larger models brought challenges of higher training cost and unintended degradation of helpfulness. |
| Approach: | They propose a multi-task learning mechanism that integrates harmful query detection and safeguard response into a single model. |
| Outcome: | The proposed approach outperforms the publicly available LLMs in harmful query detection and safeguard response generation. |
Hyper-QKSG: Framework for Automating Query Generation and Knowledge-Snippet Extraction from Tables and Lists (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Featured snippets are a compressed excerpt that contains the answer to a user's query . knowledge-snippet is a useful tool for generating information retrieval services such as google. |
| Approach: | They propose to automatically extract query-knowledge snippet pairs from structured HTML documents using a new Language Model. |
| Outcome: | The proposed framework improves the quality of generated knowledge-snippets in real-world environments. |
Patentformer: A Novel Method to Automate the Generation of Patent Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Patentformer is a novel method for generating patent specification by fine-tuning the generative models with diverse sources of information, e.g., patent claims, drawing text, and brief descriptions of the drawings. |
| Approach: | They propose a method for generating patent specification by fine-tuning generative models with diverse sources of information, e.g., patent claims, drawing text, and brief descriptions of the drawings. |
| Outcome: | The proposed method generates patent specification in legal writing style and human-like quality may be better than the actual specification. |
MARCO: Multi-Agent Real-time Chat Orchestration (2024.emnlp-industry)
Copied to clipboard
Anubhav Shrimal, Stanley Kanagaraj, Kriti Biswas, Swarnalatha Raghuraman, Anish Nediyanchath, Yi Zhang, Promod Yenigalla
| Challenge: | MARCO is a multi-agent real-time chat orchestration framework for automating workflows that require interactions with tools, reasoning, and human collaboration. |
| Approach: | They propose a multi-agent real-time chat orchestration framework for automating workflows using LLMs. |
| Outcome: | The proposed framework performs with 94.48% accuracy and 92.74% accuracy on restaurant and retail conversations datasets and 44.91% improved latency and 33.71% cost reduction in a production setting. |
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024.emnlp-industry)
Copied to clipboard
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, Min Zhang
| Challenge: | Existing models for text retrieval are based on a multi-stage process that involves retrieving documents from a large corpus. |
| Approach: | They propose to build a multilingual text representation model and a cross-encoder reranker from scratch for text retrieval. |
| Outcome: | The proposed models outperform the state-of-the-art models on long-context retrieval benchmarks. |
ItiNera: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planning (2024.emnlp-industry)
Copied to clipboard
Yihong Tang, Zhaokai Wang, Ao Qu, Yihao Yan, Zhaofeng Wu, Dingyi Zhuang, Jushi Kai, Kebing Hou, Xiaotong Guo, Jinhua Zhao, Zhan Zhao, Wei Ma
| Challenge: | Existing urban itinerary planning studies focus on traditional tourism, but they lack the precision and accuracy needed to create a personalized itinerary. |
| Approach: | They propose an open-domain urban itinerary planning system that integrates spatial optimization with large language models to provide customized urban itineraries based on user needs. |
| Outcome: | The proposed system can generate personalized urban itineraries based on user needs and scale with existing methods. |
RESTful-Llama: Connecting User Queries to RESTful APIs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated exceptional performance in zero-shot learning and reasoning tasks. |
| Approach: | They propose a framework that transforms natural language instructions into effective RESTful API calls and a method to generate fine-tuning datasets from public API documentation. |
| Outcome: | The proposed framework improves performance in a 31.9% improvement in robustness and 2.33x increase in efficiency compared to existing methods. |
A Cost-Efficient Modular Sieve for Extracting Product Information from Company Websites (2024.emnlp-industry)
Copied to clipboard
Anna Hätty, Dragan Milchevski, Kersten Döring, Marko Putnikovic, Mohsen Mesgar, Filip Novović, Maximilian Braun, Karina Borimann, Igor Stranjanac
| Challenge: | Existing methods for extracting product information are resource-intensive and computationally prohibitive due to website structure differences and numerous non-product pages. |
| Approach: | They propose a modular method that leverages low-cost classification models to filter out company web pages. |
| Outcome: | The proposed method improves on a new dataset of 7000 product and non-product web pages and reduces computational time and costs. |
CharacterGLM: Customizing Social Characters with Large Language Models (2024.emnlp-industry)
Copied to clipboard
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Character-based dialogue systems (CharacterDial) allow users to customize social characters for social interactions. |
| Approach: | They will collect a large-scale Chinese corpus of characters with diverse categories and behaviors and develop CharacterGLM models to address these challenges. |
| Outcome: | Experiments show that CharacterGLM outperforms most popular open- and closed-source LLMs and performs comparable to GPT-4. |
RAC: Retrieval-augmented Conversation Dataset for Open-domain Question Answering in Conversational Settings (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies constrain questions and answers within predefined contexts, excluding the retrieval process. |
| Approach: | They present a retrieval-augmented conversation dataset that addresses key challenges . they propose a system that combines query rewriting and retrieval with reranking . |
| Outcome: | The proposed system improves query rewriting, retrieval, reranking, and response generation performance. |
Improving Retrieval in Sponsored Search by Leveraging Query Context Signals (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models for retrieving relevant bid keywords fail to capture nuanced user intent . a new approach to enhance query understanding uses contextual signals . |
| Approach: | They propose a method to augment queries with rich contextual signals from web search results and large language models stored in an online cache. |
| Outcome: | The proposed approach outperforms context-free models in retrieving relevant bid keywords for user queries. |
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data (2024.emnlp-industry)
Copied to clipboard
Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, null Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong
| Challenge: | Large language models exhibit significant performance discrepancies between high- and low-resource languages. |
| Approach: | They present an open-source multilingual LLM with 8 billion parameters and a multilingual instruction dataset. |
| Outcome: | The proposed model achieves consistent multilingual representations across languages. |
QUIS: Question-guided Insights Generation for Automated Exploratory Data Analysis (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Exploratory Data Analysis (EDA) is a complex task that requires thorough exploration and analysis of the data. |
| Approach: | They propose a fully automated EDA system that generates questions in iterations and analyzes data to produce multiple relevant insights. |
| Outcome: | QUIS is a fully automated EDA system that generates questions in iterations without human intervention or manually curated examples. |
PEARL: Preference Extraction with Exemplar Augmentation and Retrieval with LLM Agents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing systems specialize in extracting customer preferences from standalone queries . absence of a conversational interface often leaves customers feeling the need for humanlike assistance . |
| Approach: | They propose a shopping assistant chatbot that extracts customer preferences as key-value filters from a multi-turn conversation on an e-commerce website. |
| Outcome: | The proposed solution improves performance on exact match by 10% compared to baselines and improves inference latency by 1%. |
RAG-HAT: A Hallucination-Aware Tuning Pipeline for LLM in Retrieval-Augmented Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) has emerged as a significant advancement in the field of large language models (LLMs). |
| Approach: | They propose a method that uses hallucination detection labels to correct hallucines by integrating up-to-date information into their initial training. |
| Outcome: | The proposed method is based on the Retrieval Augmented Generation (RAG) method, which has shown to be effective in mitigating hallucinations and improving answer quality. |
Intent Detection in the Age of LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Traditional approaches to intent detection struggle with out-of-scope (OOS) detection. |
| Approach: | They propose to use adaptive in-context learning and chain-of-thought prompting to detect intent in SOTA LLMs. |
| Outcome: | The proposed system achieves 2% of native accuracy with 50% less latency. |
Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Aegis is an advanced LLM-based multi-agent for intelligent functional safety engineering that can perform all phases of a vehicle's lifecycle, including design, development, production, operation, and decommissioning. |
| Approach: | They introduce Aegis: An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering. |
| Outcome: | The proposed solution can perform Hazard Analysis and Risk Assessment (HARA), document Functional Safety Requirements (FSR), and plan test cases for Automatic Emergency Braking (AEB) systems. |
Efficient Answer Retrieval System (EARS): Combining Local DB Search and Web Search for Generative QA (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Developing a virtual assistant is crucial for supporting clients as it provides 24/7 assistance . factual questionanswering system is capable of handling all user queries . |
| Approach: | They propose a production-ready factual question answering system that combines local knowledge base search with generative, context-based QA. |
| Outcome: | The proposed system boosts local knowledge base retrieval by 23% . the system is language-agnostic and can be applied to any data domain . |
GraphQL Query Generation: A Large Training and Benchmarking Dataset (2024.emnlp-industry)
Copied to clipboard
Manish Kesarwani, Sambit Ghosh, Nitin Gupta, Shramona Chakraborty, Renuka Sindhgatta, Sameep Mehta, Carlos Eberhardt, Dan Debrunner
| Challenge: | GraphQL is a powerful query language for APIs, but crafting complex GraphqL queries can be challenging. |
| Approach: | a team of researchers has created a large-scale, cross-domain text-to-GraphQL query operation dataset . the dataset includes 10,940 training triples spanning 185 cross-source data stores and 957 test triples over 14 data stores. |
| Outcome: | The proposed dataset includes 10,940 training triples and 957 test triples over 14 data stores. |
Mixture of Diverse Size Experts (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown superior performance in a variety of tasks due to the sub-linearly increasing computational costs. |
| Approach: | They propose a new MoE architecture with designed layers where experts have different sizes to mitigate this defect. |
| Outcome: | The proposed architecture surpasses existing MoEs by adaptively assigning the parameter budget to experts while maintaining the same total parameter size and number of experts. |
Course-Correction: Safety Alignment Using Synthetic Preferences (2024.emnlp-industry)
Copied to clipboard
Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Liu Yan, Tianwei Zhang, Wei Xu, Han Qiu
| Challenge: | Recent studies show that large language models generate harmful content, but the potential for generating harmful content is an escalating concern. |
| Approach: | They propose to fine-tune LLMs with preference learning to emphasize the preference for timely course-correction by using an automated pipeline. |
| Outcome: | The proposed model improves course-correction skills without affecting general performance and resists jailbreak attacks. |
GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable performance in OpenQA, but for practical deployment, knowledge distillation is crucial to maintain high performance while operating under computational constraints. |
| Approach: | They propose an algorithm to perform unsupervised knowledge distillation without the guidance of labels to achieve 99.5% of performance. |
| Outcome: | The proposed algorithm achieves 99.5% of performance in a commercial question-answering system. |
PRISM: A New Lens for Improved Color Understanding (2024.emnlp-industry)
Copied to clipboard
| Challenge: | PRISM is a visual representation learner that can grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |
| Approach: | They propose a method that extends CLIP's ability to grasp the nuances of precise colors by utilizing a curated dataset of 100 image-text pairs that can be effortlessly repurposed for fine-tuning. |
| Outcome: | The proposed method improves CLIP's ability to grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |