Papers by Jinfeng Rao

Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives (P19-1)

Copied to clipboard

Yi Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, Aston Zhang

Challenge:	Using a pointer-generator framework for reading/sampling over large documents, we propose a framework for learning over long narratives where documents easily span over thousands of tokens.
Approach:	They propose a curriculum learning (CL) based pointer-generator framework for reading/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual difficulty.
Outcome:	The proposed framework improves on the NarrativeQA reading comprehension benchmark and reaches state-of-the-art performance.

Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling (D19-1)

Copied to clipboard

Jinfeng Rao, Linqing Liu, Yi Tay, Wei Yang, Peng Shi, Jimmy Lin

Semantic Text Processing

Challenge:	Existing techniques for relevance and semantic matching cannot be easily adapted to the other.
Approach:	They propose a model that incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness.
Outcome:	The proposed model incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness.

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)

Copied to clipboard

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler

Language Models

Challenge:	Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour.
Approach:	They conduct extensive experiments to understand scaling behaviour of ten different model architectures.
Outcome:	The results show that the best performing model can fluctuate at different scales.

The OSU/Facebook Realizer for SRST 2019: Seq2Seq Inflection and Serialized Tree2Tree Linearization (D19-63)

Copied to clipboard

Kartikeya Upasani, David King, Jinfeng Rao, Anusha Balakrishnan, Michael White

Morphology

Challenge:	Existing linearization methods for shallow surface realization tasks are not available for all languages.
Approach:	They propose a system that implements morphological inflection with a baseline linearizer for a shallow surface realization task.
Outcome:	The proposed system is competitive across languages, but poor on longer sentences.

DSI++: Updating Transformer Memory with New Documents (2023.emnlp-main)

Copied to clipboard

Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler

Semantic Text Processing

Challenge:	Differentiable Search Indices (DSIs) encode a corpus of documents and use the same model to map queries directly to relevant document identifiers.
Approach:	They propose a continual learning challenge for Differentiable Search Indices (DSIs) they propose to continuously index new documents while answering queries related to previously and newly indexed documents.
Outcome:	The proposed model stably memorizes more documents and improves the average Hits@10 by +21.1% over baselines.

Transcending Scaling Laws with 0.1% Extra Compute (2023.emnlp-main)

Copied to clipboard

Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani

Language Models

Challenge:	Existing scaling of language models is expensive and requires significant computational costs.
Approach:	They propose a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute.
Outcome:	The proposed method significantly improves existing language models and their scaling curves with a relatively tiny amount of extra compute.

Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling (D19-1)

Copied to clipboard

Linqing Liu, Wei Yang, Jinfeng Rao, Raphael Tang, Jimmy Lin

Semantic Text Processing

Challenge:	Semantic similarity modeling is central to many NLP problems such as question answering.
Approach:	They propose a pairwise word interaction model with syntactic structure priors to explore their effectiveness.
Outcome:	Extensive evaluations on eight benchmark datasets show that incorporating structural information improves over strong baselines.

Simple Attention-Based Representation Learning for Ranking Short Social Media Posts (N19-1)

Copied to clipboard

Peng Shi, Jinfeng Rao, Jimmy Lin

Representation Learning

Challenge:	Existing approaches to ranking short social media posts are complex and require different components to capture a multitude of relevance signals.
Approach:	They propose a word-level Siamese architecture with attention-based mechanisms for capturing semantic "soft" matches between query and post tokens.
Outcome:	The proposed model is faster and simpler than existing models and more efficient than existing approaches.

Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (P19-1)

Copied to clipboard

Yi Tay, Aston Zhang, Anh Tuan Luu, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, Siu Cheung Hui

Green & Sustainable NLP

Challenge:	Existing models for natural language processing are heavily parameterized and memory inefficient.
Approach:	They propose a series of lightweight and memory efficient neural architectures for NLP tasks . they propose quaternion algebra and hypercomplex spaces for computation .
Outcome:	The proposed models enable expressive inter-component interactions and significantly reduce parameter size without loss of performance.

Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue (P19-1)

Copied to clipboard

Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, Rajen Subba

Semantic Text Processing

Challenge:	Generating fluent natural language responses from structured semantic representations is a critical step in task-oriented conversational systems.
Approach:	They propose using tree-structured semantic representations for better discourse-level structuring and sentence-level planning and introduce a challenging dataset using this representation for the weather domain.
Outcome:	The proposed model improves discourse-level structuring and sentence-level planning on a weather domain and can be decoded to improve semantic correctness.

Improving Compositional Generalization with Self-Training for Data-to-Text Generation (2022.acl-long)

Copied to clipboard

Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, Emma Strubell

Text Generation

Challenge:	Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs).
Approach:	They propose a template-based input representation that greatly improves the model’s generalization capability.
Outcome:	The proposed model improves tree accuracy by 46%+ and reduces slot error rates by 73%+ over the strong baselines on SGD and Weather benchmarks.