Papers by Yixin Nie
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural network models have significantly pushed forward performance on natural language processing benchmarks with the development of largescale language model pre-training. |
| Approach: | They find that models on natural language inference and reading comprehension are unstable . they propose to use a model-selection routine to analyze the model's instability . |
| Outcome: | The proposed models can perform poorly on two language-related tasks, the authors show . they also show that the model selection routine is unstable, and that it is not reliable . |
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Multilingual transformers have been shown to have remarkable transfer skills in zero-shot settings. |
| Approach: | They investigate cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference using a large scale Chinese dataset. |
| Outcome: | The proposed model trains on Chinese and English natural language inference datasets. |
Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)
Copied to clipboard
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams
| Challenge: | Dynabench is an open-source platform for dynamic dataset creation and model benchmarking. |
| Approach: | They propose an open-source platform for dynamic dataset creation and model benchmarking. |
| Outcome: | The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning (2022.findings-acl)
Copied to clipboard
| Challenge: | Using distributed NLI, we show that models can capture human judgement distribution more effectively than the softmax baseline. |
| Approach: | They propose a new NLU task to predict the distribution of human judgements . they propose Monte Carlo, Deep Ensemble, Re-Calibration and Distribution Distillation methods to capture human judgement distributions. |
| Outcome: | The proposed methods perform better than the softmax baseline, but the results are still far below the estimated human upper-bound. |
ConjNLI: Natural Language Inference Over Conjunctive Sentences (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing stress tests do not consider non-boolean usages of conjunctions and use templates . large-scale pre-trained models do not understand conjunctive semantics well, we find . |
| Approach: | They propose a stress-test for natural language inference over conjunctive sentences where the premise differs from the hypothesis by conjunctions removed, added, or replaced. |
| Outcome: | The proposed stress-test for natural language inference over conjunctive sentences is challenging . it finds that pre-trained models do not understand conjunction semantics well . |
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances on neural approaches to natural language processing have triggered a resurgent interest on building intelligent open-domain chatbots. |
| Approach: | They propose a dialoguE COntradiction DEtection task and a conversational dataset . they show that their best contradiction detection model correlates well with human judgments . |
| Outcome: | The proposed model is more robust and generalizes well on analysis and out-of-distribution dialogues than standard (unstructured) Transformer models that explicitly hinge on utterance structures are more robust, the study shows . |
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (D19-1)
Copied to clipboard
| Challenge: | Recent advances in representation learning have separated progress in both IR and MC . few studies have examined the relationship between retrieval and comprehension at different levels of granularity for development of MRS systems. |
| Approach: | They propose a simple yet effective pipeline system with consideration on hierarchical semantic retrieval at both paragraph and sentence level and their potential effects on the downstream task. |
| Outcome: | The proposed system achieves state-of-the-art on the leaderboard test sets of both FEVER and HOTPOTQA. |
UNICORN: A Unified Causal Video-Oriented Language-Modeling Framework for Temporal Video-Language Tasks (2024.emnlp-main)
Copied to clipboard
Yuanhao Xiong, Yixin Nie, Haotian Liu, Boxin Wang, Jun Chen, Rong Jin, Cho-Jui Hsieh, Lorenzo Torresani, Jie Lei
| Challenge: | Recent advances in large multimodal models have encouraged the development of large multi-modal models . however, it is unclear how to extend these models to the more complex video domain . |
| Approach: | They propose a visual instruction tuning framework to address temporal video-language tasks . they collect a dataset and fine-tune the framework on instruction-following data . |
| Outcome: | The proposed model can perform better on established temporal video-language tasks without training objectives and intensive pre-training. |
Text-guided 3D Human Generation from 2D Collections (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 3D human modeling is used for engaging interaction in gaming, film, and animation. however, the customization of characters is crucial for creativity and scalability. |
| Approach: | They propose a 3D human generation using fashion descriptions to enhance 3D geometry transformation and fine-grained consistency. |
| Outcome: | The proposed model can generate a 3D human, guided by a fashion description, with high efficiency. |