Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)

A logical-based corpus for cross-lingual evaluation (D19-61)

Challenge:	Recent proposed approaches to perform unsupervised language adaptation lack annotated resources in less-resourced languages.
Approach:	They propose to use Adversarial Training, Sentence Encoder Alignment and Shared-Private Architecture to perform unsupervised language adaptation without using aligned sentences.
Outcome:	The proposed approaches are more suitable when the source and target language datasets contain other variations in content besides the language shift.

Copied to clipboard

Felipe Salvatore, Marcelo Finger, Roberto Hirata Jr

Syntactic Text Processing

Challenge:	a recent study shows that deep learning models can be used to solve textual inference tasks using simple linguistic patterns.
Approach:	They propose a set of syntactic tasks focused on contradiction detection that exploit linguistic patterns.
Outcome:	The proposed tasks can be implemented in English and Portuguese.

Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in Distributional Semantic Models (D19-61)

Copied to clipboard

Jeroen Van Hautte, Guy Emerson, Marek Rei

Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)

Challenge:	Word embeddings are an essential component of many natural language processing applications.
Approach:	They propose 3 new tasks to obtain higher-quality vectors for word embeddings . they use word forms in training data that are related to word forms themselves .
Outcome:	The proposed methods improve the performance of both baseline and advanced models on 4 out of 6 tasks.

Copied to clipboard

Seth Ebner, Felicity Wang, Benjamin Van Durme

BERT is Not an Interlingua and the Bias of Tokenization (D19-61)

Challenge:	Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework .
Approach:	They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations.
Outcome:	The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size.

Copied to clipboard

Jasdeep Singh, Bryan McCann, Richard Socher, Caiming Xiong

Syntactic Text Processing

Challenge:	Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space.
Approach:	They propose to use a multilingual BERT model to partition representations for each language rather than using a common, shared, interlingual space.
Outcome:	The results show that the model partitions representations for each language rather than using a common, shared, interlingual space.

Cross-lingual Joint Entity and Word Embedding to Improve Entity Linking and Parallel Sentence Mining (D19-61)

Copied to clipboard

Xiaoman Pan, Thamme Gowda, Heng Ji, Jonathan May, Scott Miller

Deep Bidirectional Transformers for Relation Extraction without Supervision (D19-61)

Challenge:	Entities can be used as effective signals to generate less ambiguous semantic representations and align multiple languages.
Approach:	They propose a method to generate cross-lingual data that is a mix of entities and contextual words based on Wikipedia.
Outcome:	The proposed method can generate cross-lingual data that is a mix of entities and contextual words based on Wikipedia . it provides reliable alignment on word/entity level and sentence level, and thus can be used for unsupervised cross-linguistic entity linking.

Copied to clipboard

Yannis Papanikolaou, Ian Roberts, Andrea Pierleoni

Domain Adaptation with BERT-based Domain Classification and Data Selection (D19-61)

Challenge:	Existing frameworks for relation extraction use distant supervision instead of annotated data.
Approach:	They propose a framework to deal with relation extraction tasks without supervision . they use syntactic parsing and pre-trained word embeddings to extract relations .
Outcome:	The proposed framework outperforms baselines on four biomedical datasets and achieves slightly worse results than the state-of-the-art in three out of four data sets.

Copied to clipboard

Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh Nallapati, Bing Xiang

Empirical Evaluation of Active Learning Techniques for Neural MT (D19-61)

Challenge:	Modern deep neural models with millions of parameters can easily adapt to a new learning task and dataset when enough supervision is given.
Approach:	They propose a domain adaptation framework based on curriculum learning and domain-discriminative data selection.
Outcome:	The proposed framework outperforms discrepancy-based methods on transfer tasks while consuming only fraction of training budget.

Copied to clipboard

Xiangkai Zeng, Sarthak Garg, Rajen Chatterjee, Udhyakumar Nallasamy, Matthias Paulik

Text Generation

Challenge:	Several active learning (AL) algorithms for machine translation (MT) have been well-studied for phrase-based MT.
Approach:	They propose to use a phrase-based algorithm to compare different AL methods in a simulated AL framework to demonstrate how unsupervised pre-training and paraphrastic embedding can be used to improve existing AL methods.
Outcome:	The proposed method outperforms existing methods in the context of phrase-based MT and is based on a simulated phrase-driven dataset.

Fast Domain Adaptation of Semantic Parsers via Paraphrase Attention (D19-61)

Copied to clipboard

Avik Ray, Yilin Shen, Hongxia Jin

Few-Shot and Zero-Shot Learning for Historical Text Normalization (D19-61)

Challenge:	Semantic parsers are used to convert user’s natural language commands to executable logical form in intelligent personal agents. Labeled datasets required to train such parser are expensive to collect, and are never comprehensive.
Approach:	They propose to use a sequence-to-sequence/tree attention based attention-based sequence-based parsers which support fast near real time retraining.
Outcome:	The proposed parsers can maintain high accuracy and fast retraining time while leveraging paraphrases already present in the training dataset.

Copied to clipboard

Marcel Bollmann, Natalia Korchagina, Anders Søgaard

From Monolingual to Multilingual FAQ Assistant using Multilingual Co-training (D19-61)

Challenge:	Historical text normalization often relies on small training datasets.
Approach:	They evaluate 63 multi-task learning configurations for sequence-to-sequence-based historical text normalization across ten datasets from eight languages.
Outcome:	The proposed learning architecture outperforms the simple, but strong identity baseline.

Copied to clipboard

Mayur Patidar, Surabhi Kumari, Manasi Patwardhan, Shirish Karande, Puneet Agarwal, Lovekesh Vig, Gautam Shroff

Multilinguality

Challenge:	Recent research on cross-lingual transfer shows state-of-the-art results on benchmark datasets using pre-trained language representation models like BERT.
Approach:	They propose a method to augment an annotated dataset with machine translations in target languages and fine-tune the PLRM jointly.
Outcome:	The proposed approach provides consistent gains on multiple benchmark datasets while requiring a single model for multiple languages.

Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)

Copied to clipboard

Luke Melas-Kyriazi, George Han, Celine Liang

Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (D19-61)

Challenge:	Recent research points to knowledge distillation as a potential solution for NLU tasks.
Approach:	They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples.
Outcome:	The proposed approach outperforms BERT training approaches while using 300 times fewer parameters.

Copied to clipboard

He He, Sheng Zha, Haohan Wang

Metric Learning for Dynamic Text Classification (D19-61)

Challenge:	Statistical natural language inference models are susceptible to learning dataset bias.
Approach:	They propose a debiasing algorithm that debiases models that use only known dataset biases . they use two benchmark datasets to train three high-performing NLI models .
Outcome:	The proposed learning objective improves model performance on challenge datasets while maintaining reasonable performance on original datasets.

Copied to clipboard

Jeremy Wohlwend, Ethan R. Elenberg, Sam Altschul, Shawn Henry, Tao Lei

Information Retrieval

Challenge:	Traditional text classifiers are limited to predicting over a fixed set of labels, but real-world applications require dynamic classification.
Approach:	They propose to replace the traditional fixed-size output layer with a learned metric space . they propose to add or remove support points in the metric and fine-tune the resulting metric .
Outcome:	The proposed method is robust to changes in the label space and improves performance in low data regime.

Evaluating Lottery Tickets Under Distributional Shifts (D19-61)

Copied to clipboard

Shrey Desai, Hongyuan Zhan, Ahmed Aly

Cross-lingual Parsing with Polyglot Training and Multi-treebank Learning: A Faroese Case Study (D19-61)

Challenge:	Recent research suggests deep neural networks are dramatically over-parametrized.
Approach:	They propose that large, over-parameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better) test accuracy.
Outcome:	The proposed models can achieve commensurate performance using the same initialization as the original model.

Copied to clipboard

James Barry, Joachim Wagner, Jennifer Foster

Syntactic Text Processing

Challenge:	Cross-lingual dependency parsing involves transferring syntactic knowledge from one language to another.
Approach:	They compare two approaches to cross-lingual dependency parsing using monolingual source models and a polyglot model which is trained on the combination of all source languages.
Outcome:	The proposed methods improve low-resource dependency parsers by transferring syntactic knowledge from one language to another.

Inject Rubrics into Short Answer Grading System (D19-61)

Copied to clipboard

Tianqi Wang, Naoya Inoue, Hiroki Ouchi, Tomoya Mizumoto, Kentaro Inui

Challenge:	Short Answer Grading (SAG) is a task of scoring students’ answers in examinations. Existing SAG systems only predict scores based on the answers, but they ignore important evaluation criteria such as rubrics.
Approach:	They propose to inject rubrics into SAG models by introducing word-level attention mechanism into the model to locate information in each answer that are highly related to the score.
Outcome:	The proposed model outperforms the state-of-the-art model on the widely used ASAP-SAS dataset under low-resource settings.

Instance-based Inductive Deep Transfer Learning by Cross-Dataset Querying with Locality Sensitive Hashing (D19-61)

Copied to clipboard

Somnath Basu Roy Chowdhury, Annervaz M, Ambedkar Dukkipati

Multimodal, Multilingual Grapheme-to-Phoneme Conversion for Low-Resource Languages (D19-61)

Challenge:	Existing methods to train supervised learning models rely on labeled data, which is expensive or impossible to acquire.
Approach:	They propose an inductive transfer learning method that can augment learning models by infusing similar instances from different learning tasks in Natural Language Processing domain.
Outcome:	The proposed method improves the performance of three major news classification datasets by reducing dependency on labeled data by a significant margin.

Copied to clipboard

James Route, Steven Hillis, Isak Czeresnia Etinger, Han Zhang, Alan W Black

Multimodality

Challenge:	Grapheme-to-phoneme conversion (g2p) is a task of predicting the pronunciation of words from their orthographic representation.
Approach:	They propose to leverage audio data as an auxiliary modality in a multi-task training process to learn a more optimal grapheme representation.
Outcome:	The proposed model reduces phoneme error rate to 2.46% on in-domain test set compared to unimodal spelling- pronunciation model.

Natural Language Generation for Effective Knowledge Distillation (D19-61)

Copied to clipboard

Raphael Tang, Yao Lu, Jimmy Lin

Neural Unsupervised Parsing Beyond English (D19-61)

Challenge:	Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks.
Approach:	They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques .
Outcome:	The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability.

Copied to clipboard

Katharina Kann, Anhad Mohananey, Samuel R. Bowman, Kyunghyun Cho

Reevaluating Argument Component Extraction in Low Resource Settings (D19-61)

Challenge:	Unsupervised parsing is a task that can be learned without substantial prior knowledge.
Approach:	They train an unsupervised model for Arabic, Chinese, English, and German to learn syntactic structure from unlabeled text.
Outcome:	The PRPN architecture outperforms trivial baselines and acquires at least some parsing ability for all languages.

Copied to clipboard

Anirudh Joshi, Timothy Baldwin, Richard Sinnott, Cecile Paris

Information Extraction & Text Mining

Challenge:	Argument component extraction is a challenging and complex high-level semantic extraction task.
Approach:	They propose to use character-level, GloVe, ELMo, and BERT encodings to compare arguments extracted using standard BiLSTM-CRF encoders.
Outcome:	The proposed approaches perform better than baselines in higher-level semantic extraction tasks and suggest future improvements.

Reinforcement-based denoising of distantly supervised NER with partial annotation (D19-61)

Copied to clipboard

Farhad Nooralahzadeh, Jan Tore Lønning, Lilja Øvrelid

Information Extraction & Text Mining

Challenge:	Existing named entity recognition systems rely on large amounts of human-labeled data for supervision, but the result is noisy.
Approach:	They propose to use partial annotation to address false negative cases and implement a reinforcement learning strategy to identify false positive instances.
Outcome:	The proposed model reduces the amount of manually annotated data required to perform NER in a new domain.

Samvaadhana: A Telugu Dialogue System in Hospital Domain (D19-61)

Copied to clipboard

Suma Reddy Duggenpudi, Kusampudi Siva Subrahamanyam Varma, Radhika Mamidi

Natural Language Interfaces

Challenge:	a dialogue system for Hospital domain in Telugu is a resource-poor Dravidian language . the system handles various hospital and doctor related queries .
Approach:	They propose to model a dialogue system for Hospital domain in Telugu which is a resource-poor Dravidian language.
Outcome:	The proposed system achieves a high overall rating and a significantly accurate context-capturing method.

Towards Zero-resource Cross-lingual Entity Linking (D19-61)

Copied to clipboard

Shuyan Zhou, Shruti Rijhwani, Graham Neubig

Low-Resource NLP

Challenge:	XEL is challenging for most languages because of limited availability of requisite resources . simulated environments that use significant resources are not available in truly low-resource languages .
Approach:	They propose improvements to entity candidate generation and disambiguation to make better use of the limited resources available in low-resource languages.
Outcome:	The proposed model gains 6-20% end-to-end linking accuracy on four low-resource languages.

Transductive Auxiliary Task Self-Training for Neural Multi-Task Models (D19-61)

Copied to clipboard

Johannes Bjerva, Katharina Kann, Isabelle Augenstein

Weakly Supervised Attentional Model for Low Resource Ad-hoc Cross-lingual Information Retrieval (D19-61)

Challenge:	Multi-task learning and self-training are two common ways to improve a machine learning model’s performance in settings with limited training data.
Approach:	They propose a transductive auxiliary task self-training procedure that trains a model on auxiliary tasks and test instances with auxiliary labels generated by a single-task version of the model.
Outcome:	The proposed method improves accuracy by 9.56% over the pure multi-task model for dependency relation tagging and 13.03% for semantic taging.

Copied to clipboard

Lingjun Zhao, Rabih Zbib, Zhuolin Jiang, Damianos Karakos, Zhongqiang Huang

Multilinguality

Challenge:	Low resource languages often lack relevance annotations for cross-lingual information retrieval . when available, the training data has limited coverage for possible queries .
Approach:	They propose a weakly supervised neural model for Cross-lingual information retrieval from low-resource languages using weak supervision instead of relevance annotations.
Outcome:	The proposed model achieves 19 MAP points improvement compared to CNNs and 12 points improvement from machine translation-based CLIR models.

X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension (D19-61)

Copied to clipboard

Mostafa Abdou, Cezar Sas, Rahul Aralikatte, Isabelle Augenstein, Anders Søgaard

Multilinguality

Challenge:	Existing knowledge bases are heavily biased towards English, but Wikipedias cover very different topics in different languages.
Approach:	They propose a multilingual dataset that frams relation extraction as a machine reading problem.
Outcome:	The proposed model can be used to transfer models cross-lingually and improves knowledge base completion across languages.

Zero-Shot Cross-lingual Name Retrieval for Low-Resource Languages (D19-61)

Copied to clipboard

Kevin Blissett, Heng Ji

Low-Resource NLP

Challenge:	a novel name retrieval method is proposed for languages with no annotations or training data.
Approach:	They propose a method which relies on zero annotation or resources from the target language . they pre-train an orthographic encoder using Wikipedia inter-lingual links from dozens of languages .
Outcome:	The proposed method shows 11.6% improvement over state-of-the-art methods.

Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations (D19-61)

Copied to clipboard

Ke Tran, Arianna Bisazza