Papers by Pengcheng Yin
Omni-RewardBench: Toward a Comprehensive Evaluation of Generative Reward Models Across Modalities (2026.acl-long)
Copied to clipboard
Chi-Min Chan, Yujin Zhou, Pengcheng Wen, Boqin Yin, Jiaming Ji, Juntao Dai, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Existing evaluation benchmarks for ORMs are largely text-centric or limited to bimodal tasks . a new study examines the effectiveness of Omni-RewardBench for ORms across modalities . |
| Approach: | They propose a hybrid automatic-annotation and human-verification pipeline to construct high-quality evaluation data. |
| Outcome: | The proposed model is the first benchmark for comprehensive evaluation of ORMs across modalities. |
SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-SQL aims to automate the process of generating SQL queries on a database from natural language text. |
| Approach: | They propose a method to improve few-shot prompting capabilities of Text-to-SQL for Large Language Models (LLMs) they propose 'SQlPrompt' which aims to diversify the SQL proposals during consistency selection with different prompt designs and foundation models. |
| Outcome: | The proposed method outperforms previous approaches for in-context learning with zero labeled data by a large margin, closing the gap with finetuning state-of-the-art with thousands of labeles. |
A Tree-based Decoder for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing work on adding syntactic information to NMT systems is limited to linguistically-inspired tree structures. |
| Approach: | They propose an NMT model that can naturally generate the topology of an arbitrary tree structure on the target side. |
| Outcome: | The proposed model outperforms standard seq2seq models by 2.1 BLEU points and other methods for incorporating target-side syntax by 0.7 BLUE points. |
Retrieval-Based Neural Code Generation (D18-1)
Copied to clipboard
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, Graham Neubig
| Challenge: | Existing methods to generate program source code from natural language are not able to generate complex code due to a lack of ability to memorize large and complex structures. |
| Approach: | They propose a method that uses subtree retrieval to explicitly reference existing code examples within a neural code generation model. |
| Outcome: | The proposed method improves performance on two code generation tasks by up to +2.6 BLEU. |
StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing (P18-1)
Copied to clipboard
| Challenge: | Semantic parsing is the task of transducing natural language (NL) utterances into formal meaning representations (MRs), commonly represented as tree structures. |
| Approach: | They propose a variational auto-encoding model for semi-supervised semantic parsing which learns from limited amounts of parallel data and readily-available unlabeled NL utterances. |
| Outcome: | Experiments on ATIS domain and Python show that with extra unlabeled data, StructVAE outperforms strong supervised models. |
TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation (D18-2)
Copied to clipboard
| Challenge: | Existing neural semantic parsers only focus on a small subset of tasks, such as SQL queries, robotic commands, and even general-purpose programming languages like Java. |
| Approach: | They propose a transition-based neural semantic parser that maps natural language utterances into formal meaning representations (MRs) they use an abstract syntax description language to constrain the output space and model the information flow. |
| Outcome: | Experiments on four different semantic parsing and code generation tasks show that the proposed system is generalizable, extensible, and effective. |
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention (2021.naacl-main)
Copied to clipboard
Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, Jacob Andreas
| Challenge: | Existing approaches to compositional generalization in semantic parsers focus on word-level alignments, but they focus on spans. |
| Approach: | They propose a span-level supervised attention loss that improves compositional generalization in semantic parsers by focusing on spans. |
| Outcome: | The proposed method improves on three benchmarks of compositional generalization. |
On The Ingredients of an Effective Zero-shot Semantic Parser (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have performed zero-shot learning by synthesizing training examples of canonical utterances and programs from a grammar, and further paraphrasing these utterrances to improve linguistic diversity. |
| Approach: | They propose to bridge gaps between canonical and real-world user-issued examples by using stronger paraphrasers and improved grammars. |
| Outcome: | The proposed model achieves strong performance on two semantic parsing benchmarks with zero labeled data. |
Reranking for Neural Semantic Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing is the task of transducing natural language utterances into machine executable meaning representations (e.g., Python code). |
| Approach: | They propose to rerank an n-best list of predicted MRs and use features to fix observed problems with baseline models to improve parser performance. |
| Outcome: | The proposed method outperforms the best published neural parser on four datasets and improves the baseline parsing performance by 5.7% and 2.9%. |
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language understanding tasks. |
| Approach: | They propose a pretrained language model that jointly learns representations for NL sentences and (semi-)structured tables. |
| Outcome: | The proposed model performs best on the weakly-supervised semantic parsing benchmark WikiTableQuestions while performing competitively on the text-to-SQL dataset Spider. |
Improving Open Information Extraction via Iterative Rank-Aware Learning (P19-1)
Copied to clipboard
| Challenge: | Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences. |
| Approach: | They propose an additional binary classification loss to calibrate the extraction likelihood . they propose an iterative learning process where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. |
| Outcome: | Experiments on open information extraction (IE) show that the extraction likelihood is not well calibrated when comparing quality of extracted assertions. |
Natural Language to Code Generation in Interactive Data Science Notebooks (2023.acl-long)
Copied to clipboard
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, Charles Sutton
| Challenge: | Data scientists use computational notebooks to perform data wrangling and analytic tasks. |
| Approach: | They build a benchmark program that synthesizes programs given NL intents from users by using a Python code language model. |
| Outcome: | The proposed model outperforms public code LMs in a dataset of 1078 code generation problems using the pandas data analysis framework in data science notebooks. |
Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data (2022.acl-long)
Copied to clipboard
| Challenge: | Existing work has treated procedures as shallow structures without modeling the parent-child relation. |
| Approach: | They propose to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow . they link steps in an article to other articles with similar goals, recursively building the KB . |
| Outcome: | The proposed method significantly outperforms baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval. |
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)
Copied to clipboard
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu
| Challenge: | Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately . |
| Approach: | They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes . |
| Outcome: | The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show . |
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on open-domain code generation focuses on limited domains or domain-specific languages with limited set of operators. |
| Approach: | They incorporate external knowledge into NL-to-code generation by combining StackOverflow and programming language API documentation with data augmentation and retrieval-based data re-sampling. |
| Outcome: | The proposed approach improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. |
ContextCheck: Sentence-Level Faithfulness Verification with Context-Aware Disambiguation (2026.findings-acl)
Copied to clipboard
Yueqin Yin, Yaxi Li, Xin Liu, Xun Wang, Kaiqiang Song, Simin Ma, Shujian Liu, Sathish Reddy Indurthi, Haoyun Deng, Pengcheng He, Mingyuan Zhou, Song Wang
| Challenge: | Large language models often hallucinate, producing content that is factually incorrect or not grounded in the sources. |
| Approach: | They propose a framework for sentence-level faithfulness verification with context-aware disambiguation. |
| Outcome: | The proposed framework improves Macro F1 by over 10 points compared to baselines on three context-dependent datasets. |