The economic trade-offs of large language models: A case study (2023.acl-industry)
Copied to clipboard
Kristen Howell, Gwen Christian, Pavel Fomitchov, Gitit Kehat, Julianne Marzulla, Leanne Rolston, Jadin Tredup, Ilana Zimmerman, Ethan Selfridge, Joseph Bradley
| Challenge: | Large Language Models (LLMs) are a natural fit for contact-based customer service, but their efficacy must be balanced with the cost of training and serving them. |
| Approach: | They propose a cost framework for evaluating an NLP model’s utility for the enterprise as a function of the usefulness of the responses that they generate. |
| Outcome: | The proposed model can be used to help human agents handle complex customer service calls and can be modified to improve their performance. |
Similar Papers
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size (2025.coling-industry)
Copied to clipboard
Ashok Urlana, Charaka Vinayak Kumar, Bala Mallikarjunarao Garlapati, Ajeet Kumar Singh, Rahul Mishra
| Challenge: | Large language models are playing a pivotal role in deploying strategic use cases across organizations . lack of studies examining potential challenges and risks associated with LLMs . |
| Approach: | They propose a case study and a practical guide for industries to utilize LLMs more efficiently. |
| Outcome: | The proposed study examines the literature on large language models with industry practitioners and examines industrial publications to address these questions. |
Large Human Language Models: A Need and the Challenges (2024.naacl-long)
Copied to clipboard
| Challenge: | a growing recognition of the importance of modeling human and social factors into human-centered NLP models . authors advocate for three positions toward creating large human language models based on psychological and behavioral sciences . |
| Approach: | et al. advocate for three positions toward creating large human language models . they argue that LM training should include the human context and recognize that people are more than their group . |
| Outcome: | a new study shows that learning language from linguistic signals alone is not adequate, according to a recent paper . authors advocate for three positions toward creating large human language models . a human-centered model should include the human context, and account for the dynamic nature of the human environment, they say . |
Can a Large Language Model Keep My Secrets? A Study on LLM-Controlled Agents (2025.acl-srw)
Copied to clipboard
| Challenge: | Using large language models, agents can assist with natural language tasks when given access to confidential data. |
| Approach: | They created a synthetic dataset consisting of confidentiality-aware planning and deduction tasks in organizational access control. |
| Outcome: | The proposed model can perform tasks similar to humans when given access to confidential data. |
Assistive Large Language Model Agents for Socially-Aware Negotiation Dialogues (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that virtual agents can help humans achieve task and social goals. |
| Approach: | They propose a tuning-free and label-free method to identify high-quality ICL exemplars for the remediator agent and propose measurable criteria to measure the quality of the negotiation outcomes. |
| Outcome: | The proposed model is able to improve negotiation outcomes across three negotiation topics. |
7 Points to Tsinghua but 10 Points to ? Assessing Large Language Models in Agentic Multilingual National Bias (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, but studies on risks associated with cross biases are limited to immediate context preferences. |
| Approach: | They investigate multilingual bias in state-of-the-art Large Language Models by analyzing their responses to decision-making tasks across multiple languages. |
| Outcome: | The proposed model can provide personalized advice across university applications, travel, and relocation scenarios. |
Harnessing Large Language Models as Post-hoc Correctors (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their effectiveness in a wide range of tasks, including machine translation and commonsense reasoning. |
| Approach: | They propose a training-free framework that can work as a post-hoc corrector to propose corrections for ML models. |
| Outcome: | The proposed framework improves the performance of a number of models by up to 39% on text analysis and the challenging molecular predictions. |
Evaluating Large Language Models with Enterprise Benchmarks (2025.naacl-industry)
Copied to clipboard
Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, Yada Zhu
| Challenge: | Existing benchmarks lack domain-specific datasets for evaluating large language models . existing benchmarks often lack domain specific datasets, which can be difficult to convert to standardized metrics or regulatory issues. |
| Approach: | They propose to use 25 publicly available domain-specific English benchmarks from diverse domains . they propose to combine a wide range of natural language processing tasks for holistic evaluation . |
| Outcome: | The proposed framework includes 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks. |
Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing research focuses on enhancing large language models through scaling laws or fine-tuning strategies, but ignores the potential of using agent paradigms to compensate for the inherent weaknesses of small models. |
| Approach: | They propose to use structured agent frameworks to improve effectiveness over direct prompting . they also propose to employ routing-based multi-agent systems with collaborative capabilities . |
| Outcome: | The proposed model significantly outperforms direct prompting with single-agent systems . the proposed model is more reliable and cost-effective than other models . |
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |