Papers with Large-scale
FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)
Copied to clipboard
| Challenge: | Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error . |
| Approach: | They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions. |
| Outcome: | The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant. |
CLEVA: Chinese Language Models EVAluation Platform (2023.emnlp-demo)
Copied to clipboard
Yanyang Li, Jianqiao Zhao, Duo Zheng, Zi-Yuan Hu, Zhi Chen, Xiaohui Su, Yongfeng Huang, Shijia Huang, Dahua Lin, Michael Lyu, Liwei Wang
| Challenge: | Large language models (LLMs) have revolutionized natural language processing. |
| Approach: | They propose a Chinese-based platform that assesses Chinese LLMs using a standardized workflow and a unique sampling strategy. |
| Outcome: | CLEVA evaluates Chinese LLMs on a standardized workflow and a competitive leaderboard with minimal coding. |
Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Large-scale conversational assistants process every utterance using multiple models for domain, intent and named entity recognition. |
| Approach: | They combine utterance encodings from a RoBERTa model with the Nbest hypothesis produced by the production system to detect domain classification errors. |
| Outcome: | The proposed approach outperforms bi-LSTM models and a standalone model by 2.2% to 32.2% by ensembling multiple models. |