Papers by Jian Pei
Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train cross-lingual pre-trained language models have shown great success in cross-linguistic sequence labeling tasks. |
| Approach: | They propose a cross-lingual language informative span masking task to eliminate the objective gap between pre-training and fine-tuning stages. |
| Outcome: | The proposed method surpasses the state-of-the-art methods on multiple benchmarks even with limited pre-training data. |
Generating Informative Responses with Controlled Sentence Function (P18-1)
Copied to clipboard
| Challenge: | Sentence function is a significant factor to achieve the purpose of the speaker, but has not been touched in large-scale conversation generation. |
| Approach: | They propose a model to generate informative responses with controlled sentence function using a latent variable and a type controller to deal with compatibility. |
| Outcome: | The proposed model outperforms state-of-the-art models and generates responses with controlled sentence function and informative content. |
Lexicon-Enhanced Self-Supervised Training for Multilingual Dense Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent multilingual pre-trained models perform poorly on multilingual retrieval tasks due to lack of multilingual training data. |
| Approach: | They propose to mine and generate self-supervised training data based on large-scale unlabeled corpus and introduce query generator to generate more queries in target languages for unlabed passages. |
| Outcome: | The proposed method performs better than baselines on a Mr. TYDI dataset and an industrial dataset from a commercial search engine. |
Reasoning over Entity-Action-Location Graph for Procedural Text Understanding (2021.acl-long)
Copied to clipboard
| Challenge: | Procedural text understanding aims at tracking the states and locations of entities mentioned in a paragraph. |
| Approach: | They propose a framework to model entities-entity, action, and location relations using a graph neural network. |
| Outcome: | The proposed approach outperforms strong baselines on two datasets, ProPara and Recipes. |
A Graph Representation of Semi-structured Data for Web Question Answering (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies treat semi-structured data as flat documents with pieces of text . semi-structural data is more effective to represent rich relational information . question answering is an important feature in most search engines . |
| Approach: | They propose a graph representation of Web tables and lists based on categorization of components and their relations . they also develop reasoning techniques on the graph model for the question answering task . |
| Outcome: | The proposed graph improves F1 score by 3.90 points over the state-of-the-art baselines on real datasets. |
Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to translate spoken language understanding into low-resource languages are limited to implicit alignment and disregard the inherent semantic structure in SLU. |
| Approach: | They propose to model utterance-slot-word structure by a multi-level contrastive learning framework . they also propose a label-aware joint model leveraging label semantics to enhance alignment . |
| Outcome: | The proposed model improves performance on two zero-shot cross-lingual datasets. |
ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods (2024.emnlp-main)
Copied to clipboard
| Challenge: | ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities. |
| Approach: | They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities. |
| Outcome: | The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
Modeling Event-Pair Relations in External Knowledge Graphs for Script Reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods focus on graph triples with event overlap, but ignore more supportive triples . Script reasoning relies on understanding the relationship between two events . |
| Approach: | They propose a model to learn the inferential relations between events from the whole eventuality KG . they propose 'script adapter' to extend the model to infer the associated relations between an event chain and a subsequent event candidate. |
| Outcome: | The proposed model is compared with baselines using external KG or not on a script reasoning task. |
Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation (2020.coling-main)
Copied to clipboard
| Challenge: | Cross-lingual Machine Reading Comprehension (CLMRC) is a challenging problem due to the lack of large-scale annotated datasets in low-source languages, such as Arabic, Hindi, and Vietnamese. |
| Approach: | They propose a novel approach to augment cross-lingual machine reading comprehension by combining knowledge from multiple language branch models into a single model for all target languages. |
| Outcome: | Extensive experiments on two CLMRC benchmarks show the proposed method is effective and robust to data noises. |
Structural Contrastive Pretraining for Cross-Lingual Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train multilingual language models using pretraining tasks like mask language modeling have yielded promising results on a wide range of downstream tasks. |
| Approach: | They propose a new task to align the structural words in a parallel sentence, enhancing models’ ability to comprehend cross-lingual representations. |
| Outcome: | The proposed task improves model's ability to comprehend cross-lingual representations by increasing the frequency of negative pairings. |
Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to scale out spoken language understanding to low-resource languages are noisy. |
| Approach: | They propose a method for mitigating noise in augmented data by training models with augmented datasets. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two benchmark datasets. |
Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA (2026.acl-long)
Copied to clipboard
Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, Zhenbo Luo, Jian Luan, Wei Chen, Xiang Bai
| Challenge: | Existing OCR-free approaches to document visual question answering are brittle and passive. |
| Approach: | They propose an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. |
| Outcome: | The proposed framework outperforms open-source and proprietary models in five benchmarks and improves out-of-domain performance by 47.9% over baseline. |
Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features (N19-2)
Copied to clipboard
Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, Jimmy Lin
| Challenge: | e-commerce companies often have the option of escalating complaints by filing grievances with a government authority . this is detrimental to an ecommerce company, but this problem is challenging to solve by integrating recurrent neural networks with manually-engineered features. |
| Approach: | They propose a model that integrates recurrent neural networks with manually-engineered features to identify cases where the customer expresses such an intent. |
| Outcome: | The proposed model outperforms baseline models and provides better recall and triage for specialized agents. |