Papers by Zhengbao Jiang
CoRI: Collective Relation Integration with Data Augmentation for Open Information Extraction (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to integrate extracted knowledge from the Web to knowledge graphs (KGs) however, the predictions are made independently, which can be mutually inconsistent. |
| Approach: | They propose a relation integration model that aligns free-text relations to relations in a target KG . they propose combining two stages to make independent predictions and a collective model that accesses all candidate predictions. |
| Outcome: | The proposed model outperforms baseline models on two datasets and improves AUC from .677 to .748 and from 1.716 to 1.780. |
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) capture factual knowledge by filling in the blanks of cloze-style prompts. |
| Approach: | They propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
| Outcome: | The proposed method improves the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
Beyond Memorization: The Challenge of Random Memory Access in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Language Models (LMs) have shown their effectiveness in knowledge-intensive tasks. |
| Approach: | They investigate whether a generative language model is able to access its memory sequentially or randomly. |
| Outcome: | The proposed LMs are able to access memory sequentially or randomly. |
Generalizing Natural Language Analysis through Span-relation Representations (2020.acl-main)
Copied to clipboard
| Challenge: | a large number of natural language processing tasks are generated with specially designed architectures. |
| Approach: | They propose to represent a wide variety of tasks in a single unified format . they perform extensive experiments to demonstrate benefits of multi-task learning . |
| Outcome: | The proposed model performs comparable to state-of-the-art models on 10 tasks . it also shows that it can analyze differences and similarities in how the model handles different tasks compared to other models . |
GSum: A General Framework for Guided Neural Abstractive Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Abstractive summarization models are flexible, but they can be difficult to control. |
| Approach: | They propose a general and extensible guided summarization framework that takes different kinds of guidance as input and perform experiments across different varieties. |
| Outcome: | The proposed framework can generate more faithful summaries and different types of guidance generate qualitatively different summary. |
Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Generative question answering (QA) models generate answers to complex questions, but their mechanism for doing so is still poorly understood. |
| Approach: | They decompose multi-hop questions into multiple corresponding single-hop question chains and find marked inconsistency in QA models’ answers on these pairs of ostensibly identical question chains. |
| Outcome: | The proposed models lack zero-shot multi-hop reasoning ability when trained on single-hop questions and on logical forms. |
Improving Open Information Extraction via Iterative Rank-Aware Learning (P19-1)
Copied to clipboard
| Challenge: | Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences. |
| Approach: | They propose an additional binary classification loss to calibrate the extraction likelihood . they propose an iterative learning process where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. |
| Outcome: | Experiments on open information extraction (IE) show that the extraction likelihood is not well calibrated when comparing quality of extracted assertions. |
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | eschewing separate architecture and training for knowledge-intensive tasks is cumbersome . end-to-end training only based on supervision from the end task is awkward . |
| Approach: | They propose a single Transformer that performs retrieval as attention and end-to-end training solely based on supervision from the end QA task. |
| Outcome: | The proposed model outperforms state-of-the-art retrievers and readers on in-domain datasets. |
Instruction-tuned Language Models are Better Knowledge Learners (2024.acl-long)
Copied to clipboard
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, Srini Iyer
| Challenge: | Large language models store factual knowledge in parameters, but it can become outdated as the work evolves . pre-instruction-tuning improves ability of LLMs to absorb knowledge from new documents . |
| Approach: | They propose a method that instruction-tunes on questions prior to training on documents . they propose to use QA pairs to update factual knowledge of large language models . |
| Outcome: | The proposed method outperforms instruction-tuning on documents by 17.8%. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering (2022.naacl-main)
Copied to clipboard
| Challenge: | a table-based question answering system requires complex reasoning and alignment between questions and tables. |
| Approach: | They propose a table-based QA model that consumes both natural and synthetic data . they combine retrieval with masking to pair natural sentences with QA . |
| Outcome: | The proposed model outperforms existing models in few-shot and full settings and on WikiTableQuestions. |
GPTScore: Evaluate as You Desire (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for text generation are not adequate to assess the quality of the generated outputs. |
| Approach: | They propose a framework that utilizes emergent abilities of generative pre-trained models to evaluate generated texts. |
| Outcome: | The proposed evaluation framework can achieve what one desires to evaluate for texts simply by natural language instructions. |
SPE: Symmetrical Prompt Enhancement for Fact Probing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work probes PLMs for the extent of factual knowledge through prompts . however, these methods do not consider symmetry of the task: object and subject prediction. |
| Approach: | They propose a continuous prompt-based method that leverages symmetry of the task by constructing symmetrical prompts for subject and object prediction. |
| Outcome: | The proposed method improves on a popular factual probing dataset on lAMA. |
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on open-domain code generation focuses on limited domains or domain-specific languages with limited set of operators. |
| Approach: | They incorporate external knowledge into NL-to-code generation by combining StackOverflow and programming language API documentation with data augmentation and retrieval-based data re-sampling. |
| Outcome: | The proposed approach improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. |
Active Retrieval Augmented Generation (2023.emnlp-main)
Copied to clipboard
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig
| Challenge: | Generative language models (LMs) have a tendency to hallucinate and create inaccurate output. |
| Approach: | They propose a method which iteratively uses a prediction of the upcoming sentence to anticipate future content. |
| Outcome: | The proposed method achieves superior or competitive performance on all tasks . iteratively uses a prediction of the upcoming sentence to anticipate future content . |