Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers
When does text prediction benefit from additional context? An exploration of contextual signals for chat and email messages (2021.naacl-industry)
Copied to clipboard
| Challenge: | Prior-message context provides the greatest lift in Teams (chat) scenario. |
| Approach: | They compare prior-message context with email and chat messages from Microsoft Teams and Outlook. |
| Outcome: | The proposed model outperforms existing models on two of the largest commercial communication platforms: Microsoft Teams and Outlook. |
Identifying and Resolving Annotation Changes for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Annotation conflict resolution is crucial for machine learning, says a new study . past work on annotation conflict resolution assumed data is collected at once . a a supervised neural model can resolve conflicts in data annotation but requires access to high-quality data . |
| Approach: | They propose an approach to resolve annotation conflicts in a real-world context using a German dialog system. |
| Outcome: | The proposed approach improves on a real-world dataset with 3.5M utterances in German. |
Optimizing NLU Reranking Using Entity Resolution Signals in Multi-domain Dialog Systems (2021.naacl-industry)
Copied to clipboard
Tong Wang, Jiangning Chen, Mohsen Malmir, Shuyan Dong, Xin He, Han Wang, Chengwei Su, Yue Liu, Yang Liu
| Challenge: | In dialog systems, the Natural Language Understanding component makes the interpretation decision before the mentioned entities are resolved. |
| Approach: | They propose to leverage Entity Resolution (ER) features in NLU reranking to learn model weights . they propose a score distribution matching method to ensure the models are calibrated . |
| Outcome: | The proposed approach outperforms the baseline model on multiple domain evaluations. |
Entity Resolution in Open-domain Conversations (2021.naacl-industry)
Copied to clipboard
Mingyue Shang, Tong Wang, Mihail Eric, Jiangning Chen, Jiyang Wang, Matthew Welch, Tiantong Deng, Akshay Grewal, Han Wang, Yue Liu, Yang Liu, Dilek Hakkani-Tur
| Challenge: | Recent work on incorporating external knowledge into the response generation models has attracted great interest. |
| Approach: | They propose a neural entity linking approach to incorporate external knowledge into the response generation models to improve the relevancy of retrieved knowledge. |
| Outcome: | The proposed approach outperforms the baseline model by 62.8% relative to the baseline. |
Pretrain-Finetune Based Training of Task-Oriented Dialogue Systems in a Real-World Setting (2021.naacl-industry)
Copied to clipboard
| Challenge: | a challenge in building task-oriented dialogue systems is the limited amount of supervised training data available. |
| Approach: | They propose a method for training retrieval-based dialogue systems using annotated data and a larger, unlabeled dataset. |
| Outcome: | The proposed method improves model performance offline and online compared with no pretraining . the model is deployed in an agent-support application and evaluated on live customer service contacts . |
Contextual Domain Classification with Temporal Representations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing studies that incorporate context in SLU have focused on domains where context is limited to a few minutes. |
| Approach: | They propose temporal representations that combine wall-clock second difference and turn order offset information to utilize both recent and distant context in a novel large-scale setup. |
| Outcome: | The proposed model reduces 13.04% of classification errors compared to baseline . previous studies have focused on domains where context is limited to a few minutes . |
Bootstrapping a Music Voice Assistant with Weak Supervision (2021.naacl-industry)
Copied to clipboard
| Challenge: | Music listening is among the top-5 reasons of daily usage of voice assistants in the US. |
| Approach: | They propose a weakly-supervised method to label large amounts of voice query logs . they show that slot tagging models outperform models trained on hand-annotated or synthetic data . |
| Outcome: | The proposed method outperforms models trained on hand-annotated or synthetic data at a lower cost. |
Continuous Model Improvement for Language Understanding with Machine Translation (2021.naacl-industry)
Copied to clipboard
Abdalghani Abujabal, Claudio Delli Bovi, Sungho Ryu, Turan Gojayev, Fabian Triefenbach, Yannick Versley
| Challenge: | a simple translation-test approach would fail the latency requirements of a live environment. |
| Approach: | They show that annotating unlabeled utterances offline can improve performance . they demonstrate that an extrinsic evaluation can improve the performance if manual data is available . |
| Outcome: | The proposed method improves performance in an extrinsic evaluation setting with real-world commercial dialog system in german. |
A Hybrid Approach to Scalable and Robust Spoken Language Understanding in Enterprise Virtual Agents (2021.naacl-industry)
Copied to clipboard
Ryan Price, Mahnoosh Mehrabani, Narendra Gupta, Yeon-Jun Kim, Shahab Jalalvand, Minhua Chen, Yanjie Zhao, Srinivas Bangalore
| Challenge: | Spoken language understanding (SLU) extracts the intended mean- ing from a user's utterance. |
| Approach: | They propose a framework for intent and entity extraction utilizing a hybrid of statistical and rule-based approaches. |
| Outcome: | The proposed framework can be deployed quickly for a large class of EVA applications with little need for human intervention. |
Proteno: Text Normalization with Limited Data for Fast Deployment in Text to Speech Systems (2021.naacl-industry)
Copied to clipboard
| Challenge: | Developing Text Normalization systems for Text-to-Speech (TTS) on new languages is hard. |
| Approach: | They propose a novel architecture to facilitate Text Normalization systems for TTS on new languages . they use a granular tokenization mechanism that enables the system to learn majority of classes . |
| Outcome: | The proposed architecture performs comparable with the state-of-the-art systems on English . the proposed system learns most classes from training data and precodes them for other classes . |
Addressing the Vulnerability of NMT in Input Perturbations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in NMT have improved translation quality but are vulnerable to input perturbations. |
| Approach: | They propose a method to reduce the effect of noisy inputs by using a Context-Enhanced Reconstruction approach. |
| Outcome: | The proposed approach improves robustness on Chinese-English and French-English translation tasks. |
Cross-lingual Supervision Improves Unsupervised Neural Machine Translation (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing models that use only monolingual data have not been fully duplicated in the vast majority of language pairs, especially for zero-source languages. |
| Approach: | They propose to leverage the corpus from En-Fr and En-De to collectively train the translation from one language into many languages under one model. |
| Outcome: | The proposed model significantly improves translation quality with a big margin in the benchmark unsupervised translation tasks and achieves comparable performance to supervised NMT. |
Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent studies using pretrain-finetuning approach have achieved state-of-the-art (SOTA) performance in many natural language processing tasks. |
| Approach: | They propose a new tokenization method that combines morphology-considered subword tokenization and vocabulary methods to address this limitation. |
| Outcome: | The proposed method can be used without modifying the model structure. |
Autocorrect in the Process of Translation — Multi-task Learning Improves Dialogue Machine Translation (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing neural machine translation models are not able to translate dialogues in real life scenarios. |
| Approach: | They propose a joint learning method to identify omission and typos and utilize context to translate dialogue utterances. |
| Outcome: | The proposed method improves translation quality by 3.2 BLEU over baselines and recovers omitted pronouns by 47.16%. |
LightSeq: A High Performance Inference Library for Transformers (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing inference frameworks for natural language processing are not the best choice for online service of sequence processing problems. |
| Approach: | They propose a highly efficient inference library for Transformer models that includes GPU optimization techniques to streamline computation and reduce memory footprint. |
| Outcome: | The proposed library achieves 14x speedup compared with TensorFlow and 1.4x speed up compared to a concurrent CUDA implementation. |
Practical Transformer-based Multilingual Text Classification (2021.naacl-industry)
Copied to clipboard
| Challenge: | XNLI does not reflect the data availability and task variety of industry applications. |
| Approach: | They compare transformer-based text classification methods to multilingual models in five different languages . they use a task- and domain-adaptive pretraining and data augmentation technique . |
| Outcome: | The proposed methods outperform monolingual models on two tasks in five languages . the results show that practical modifications can improve model performance without labeling . |
An Emotional Comfort Framework for Improving User Satisfaction in E-Commerce Customer Service Chatbots (2021.naacl-industry)
Copied to clipboard
| Challenge: | E-commerce has grown rapidly over the last several years, and chatbots for intelligent customer service are simultaneously drawing attention. |
| Approach: | They propose a framework to obtain proper answer to customers’ emotional questions using emotion classification model and text matching. |
| Outcome: | The proposed framework is very promising on real online systems. |
Language Scaling for Universal Suggested Replies Model (2021.naacl-industry)
Copied to clipboard
Qianlan Ying, Payal Bajaj, Budhaditya Deb, Yu Yang, Wei Wang, Bojia Lin, Milad Shokouhi, Xia Song, Yang Yang, Daxin Jiang
| Challenge: | We consider scaling automated suggested replies (SR) to multiple languages for a commercial email application. |
| Approach: | They propose a multi-lingual multi-task continual learning framework with auxiliary tasks and language adapters to train universal language representation across regions. |
| Outcome: | The proposed model reduces catastrophic forgetting and improves cross-lingual transfer across languages while reducing training costs. |
Graph-based Multilingual Product Retrieval in E-Commerce Search (2021.naacl-industry)
Copied to clipboard
| Challenge: | Modern e-commerce search systems require product retrieval under multilingual scenarios. |
| Approach: | They propose a universal multilingual retrieval system that captures interactions between search queries and items in e-commerce search. |
| Outcome: | The proposed system outperforms state-of-the-art retrieval models on five countries and has been deployed in production for multiple countries. |
Query2Prod2Vec: Grounded Word Embeddings for eCommerce (2021.naacl-industry)
Copied to clipboard
| Challenge: | Query2Prod2Vec is a model that grounds lexical representations for product search in product embeddings. |
| Approach: | They propose a model that grounds lexical representations for product search in product embeddings. |
| Outcome: | The proposed model is more accurate than existing methods from the literature . it is also more efficient than existing embedding methods in the context of high-traffic websites. |
An Architecture for Accelerated Large-Scale Inference of Transformer-Based Language Models (2021.naacl-industry)
Copied to clipboard
| Challenge: | a recent paper shows that attention-based language models can be used to train, evaluate, and perform inference on predictive models. |
| Approach: | They develop a machine learning architecture that can scale to a large volume of requests . they use a BERT model that is fine-tuned for emotion analysis . |
| Outcome: | The proposed architecture can scale to a large volume of requests with a minimum of 96 hours of running time. |
When and Why a Model Fails? A Human-in-the-loop Error Detection Framework for Sentiment Analysis (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods for sentiment analysis are difficult to assess for erroneous predictions that might exist prior to deployment. |
| Approach: | They propose a framework for error detection based on explainable features that can detect erroneous model predictions on unseen data with high precision. |
| Outcome: | The proposed framework detects erroneous model predictions on unseen data with high precision, given limited human-in-the-loop intervention, and can be deployed on unselected data with a high accuracy. |
Technical Question Answering across Tasks and Domains (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods for technical QA have a limited data size and question and answer overlaps . |
| Approach: | They propose a framework of deep transfer learning to address technical QA across tasks and domains using document retrieval and reading comprehension tasks. |
| Outcome: | The proposed framework performs better than state-of-the-art methods on the TechQA task. |
Cost-effective Deployment of BERT Models in Serverless Environment (2021.naacl-industry)
Copied to clipboard
| Challenge: | a large upfront infrastructure investment makes machine learning models difficult to deploy . however, serverless architectures have strict limits on the size of the deployment package . |
| Approach: | They propose to fine-tune BERT-style models on proprietary datasets for tasks . they use knowledge distillation to obtain models that are tuned for a specific domain . |
| Outcome: | The proposed model deployments report acceptable latency levels and cost-effectiveness without infrastructure overhead. |
Noise Robust Named Entity Understanding for Voice Assistants (2021.naacl-industry)
Copied to clipboard
Deepak Muralidharan, Joel Ruben Antony Moniz, Sida Gao, Xiao Yang, Justine Kao, Stephen Pulman, Atish Kothari, Ray Shen, Yinying Pan, Vivek Kaul, Mubarak Seyed Ibrahim, Gang Xiang, Nan Dun, Yidan Zhou, Andy O, Yuan Zhang, Pooja Chitkara, Xuan Wang, Alkesh Patel, Kushal Tayal, Roger Zheng, Peter Grasch, Jason D Williams, Lin Li
| Challenge: | Named Entity Recognition and Entity Linking are challenging for voice assistants . utterances are relatively short, so there is not much context to help disambiguate . |
| Approach: | They propose a Named Entity Understanding system that combines NER and EL in a joint reranking module. |
| Outcome: | The proposed framework improves NER accuracy by up to 3.13% and EL accuracy by 3.6% in F1 score . it also leads to better accuracies in other natural language understanding tasks . |
Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems (2021.naacl-industry)
Copied to clipboard
| Challenge: | Named entity linking (NEL) is a preprocessing step in commercial systems . a small organization or individual could use an off-the-shelf system to accomplish the same objectives . |
| Approach: | They examine how to repurpose off-the-shelf NEL systems to correct sport-related errors. |
| Outcome: | The proposed model can improve sports question-answering accuracy by 25% . the proposed model is based on the best available model . |
Intent Features for Rich Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | generic dialog systems, or chatbots, are increasingly popular, but most industrial dialog systems are built for specific clients and use cases. |
| Approach: | They propose a new neural network architecture that allows for domain and topic agnostic properties of intents that can be learnt from syntactic cues only. |
| Outcome: | The proposed model improves on baselines for identifying intent features in a deployed, multi-intent natural language understanding module. |
Development of an Enterprise-Grade Contract Understanding System (2021.naacl-industry)
Copied to clipboard
Arvind Agarwal, Laura Chiticariu, Poornima Chozhiyath Raman, Marina Danilevsky, Diman Ghazi, Ankush Gupta, Shanmukha Guttula, Yannis Katsis, Rajasekar Krishnamurthy, Yunyao Li, Shubham Mudgal, Vitobha Munigala, Nicholas Phan, Dhaval Sonawane, Sneha Srinivasan, Sudarshan R. Thitte, Mitesh Vasa, Ramiya Venkatachalam, Vinitha Yaski, Huaiyu Zhu
| Challenge: | Currently, legal contract review remains an expensive and arduous process. |
| Approach: | They describe a commercial system designed and deployed for contract understanding that enables legal professionals to review contracts. |
| Outcome: | The proposed system is used by a wide range of enterprise users and solves three major challenges. |
Discovering Better Model Architectures for Medical Query Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has attracted intense attention in computer vision and NLP. |
| Approach: | They propose to use neural architecture search to optimize model architectures for medical questions . they propose to modify the ENAS method to accelerate and stabilize the search results . |
| Outcome: | The proposed approach outperforms baseline models on two medical questions . it is compared with other NAS methods and shows that it provides the best results . |
OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing models for OOD detection work with text, but they do not work directly with the text. |
| Approach: | They propose to use a sequential generative adversarial network (SeqGAN) based model to generate OOD data for a given domain automatically. |
| Outcome: | The proposed model outperforms state-of-the-art in OOD detection metrics for ROSTD and OSQ datasets. |
Coherent and Concise Radiology Report Generation via Context Specific Image Representations and Orthogonal Sentence States (2021.naacl-industry)
Copied to clipboard
| Challenge: | Neural models for text generation are often designed in an end-to-end fashion, limiting their practical usability in downstream applications. |
| Approach: | They propose a method to compute image representations specific to each sentential context and exploiting diverse sentence states to ensure topical continuity and content diversity of generated radiology reports. |
| Outcome: | The proposed method outperforms baselines on objective metrics and human evaluations by 18% and 29% respectively in the evaluation for informativeness and content ordering respectively. |
An Empirical Study of Generating Texts for Search Engine Advertising (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing studies on neural language generation have not evaluated the effect of generated ads with actual serving included because it requires a large amount of training data and a particular environment. |
| Approach: | They propose to integrate a reinforcement learning framework into an end-to-end sequence-tosequence (Seq2S) model and demonstrate how to improve the ads’ impact, deploy models to a product, and evaluate the generated ads. |
| Outcome: | The proposed method improves the ads’ impact, deploys the models to a product, and evaluates the generated ads. |
Ad Headline Generation using Self-Critical Masked Language Model (2021.naacl-industry)
Copied to clipboard
| Challenge: | We propose a programmatic solution to generate product advertising headlines using retail content. |
| Approach: | They propose a programmatic solution to generate product advertising headlines using retail content . they use Reinforcement Learning (RL) Policy gradient methods on Transformer . |
| Outcome: | The proposed method outperforms existing methods in overlap metrics and quality audits. |
LATEX-Numeric: Language Agnostic Text Attribute Extraction for Numeric Attributes (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods for training numeric attributes are based on manual labeling and distant supervision leads to incomplete training annotations. |
| Approach: | They propose a multi-task learning architecture to deal with missing attribute values in training data, removing dependency on manual annotations. |
| Outcome: | The proposed framework improves on 20 numeric attributes extracted from 5 product categories and 3 english marketplaces with language-agnostic performance. |
Training Language Models under Resource Constraints for Adversarial Advertisement Detection (2021.naacl-industry)
Copied to clipboard
| Challenge: | e-commerce and social media sites require content moderation to ensure ethical standards . a tiered moderation workflow with automated components complements human experts . |
| Approach: | They propose techniques for training text classification models under resource constraints . they use weak supervision, curriculum learning and multi-lingual training to fine-tune BERT . |
| Outcome: | The proposed techniques detect adversarial ads with a substantial gain over baseline . the authors show that the proposed methods can be applied to multiple languages . |
Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have improved the performance of virtual assistants . however, meager training data is often a key bottleneck in creating voice-enabled applications . |
| Approach: | They propose to use unsupervised and semi-supervised techniques to improve NLU accuracy . they incorporate anonymized, unlabeled and automatically transcribed user utterances into training . |
| Outcome: | The proposed methods improve NLU accuracy in low-resource settings by integrating unsupervised and SSL techniques. |
Label-Guided Learning for Item Categorization in e-Commerce (2021.naacl-industry)
Copied to clipboard
| Challenge: | a recent study shows that item categorization uses the semantic information of the labels to guide the classification task. |
| Approach: | They investigate whether using the semantic information of the labels can improve item categorization systems in e-commerce. |
| Outcome: | The proposed methods improve item categorization performance on a real data set from a major e-commerce company in Japan. |
Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Intent detection models require large amounts of labeled data to achieve high accuracy, and in practical scenarios it is more common to find small, unbalanced, and noisy datasets. |
| Approach: | They benchmark intent detection methods on a variety of datasets and found that Watson Assistant's model outperforms other commercial solutions. |
| Outcome: | The proposed model outperforms pretrained language models on a variety of datasets while requiring only a fraction of computational resources and training data. |
Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Obtaining human annotation is expensive and time-consuming process. |
| Approach: | They propose a semi-supervised learning pipeline which leverages millions of unlabeled examples to improve natural language understanding tasks. |
| Outcome: | The proposed pipeline can be used to improve natural language understanding tasks. |