Papers by Jeff Pan
AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing state-of-the-art Large Language Models (LLMs) still cannot perform well in this situation even with the help of in-context learning and finetuning. |
| Approach: | They propose a benchmark to evaluate LLMs’ ability to plan and execute multiple APIs from various sources in order to complete the user’s task. |
| Outcome: | The proposed benchmarks show that the existing state-of-the-art LLMs still cannot perform well in this situation even with in-context learning and finetuning. |
Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text-to-SQL semantic parsing are limited to retrieving schemata based on a single query. |
| Approach: | They propose a text-to-sql semantic parser that uses abstract syntax trees to select few-shot examples for retrieval-augmented generation. |
| Outcome: | The proposed model can generate approximated versions of SQL queries in parallel, and shows improvements over state-of-the-art benchmarks. |
Trigger-Argument based Explanation for Event Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing works on ED use words or phrases to explain models’ inner mechanisms, but for ED, the event structure is more enlightening clues to explain model behaviors. |
| Approach: | They propose a Trigger-Argument based Explanation method which can utilize event structure knowledge to uncover a faithful interpretation for existing ED models at neuron level. |
| Outcome: | The proposed method can reveal the process by which the model predicts on the large-scale MAVEN and the widely-used ACE 2005 datasets. |
Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs (2024.findings-emnlp)
Copied to clipboard
Junjie Wang, Mingyang Chen, Binbin Hu, Dan Yang, Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff Pan, Wen Zhang, Huajun Chen
| Challenge: | Recent studies have attempted to enhance the performance of large language models (LLMs) in complex question-answering (QA) tasks by combining step-wise planning with external retrieval. |
| Approach: | They propose a framework for enhancing LLMs’ planning capabilities by using planning data derived from knowledge graphs (KGs). |
| Outcome: | The proposed framework improves LLMs’ planning capabilities by using knowledge graphs (KGs) the proposed framework is compared with existing frameworks on multiple datasets and shows that it is effective for large language models. |
UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have investigated the potential of language models as knowledge bases and the existence of severe biases when extracting factual knowledge. |
| Approach: | They propose an adapter-based framework for generalised factual knowledge extraction using simple methods without introducing extra parameters. |
| Outcome: | The proposed framework improves the model’s out-of-domain generalisation and consistency under various prompts. |
Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies suggest that Knowledge Graphs (KGs) contain valuable external knowledge for LLMs. |
| Approach: | They propose to model a conditional subgraph retrieval task handled by small language models and use a subgraph identifier as a special token to retrieve subgraphs. |
| Outcome: | The proposed model achieves competitive retrieval performance compared to state-of-the-art models relying on 7B parameters. |
Code-Switching with Word Senses for Pretraining in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT) many state-of-the-art (SOTA) NMT systems struggle to handle polysemous words . |
| Approach: | They propose an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. |
| Outcome: | The proposed approach improves translation quality and scales to various data and resource-strapped scenarios. |
BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for commonsense reasoning use knowledge graphs to train models . however, it is not always possible to have relevant training data available . |
| Approach: | They propose to transform a question-answer task into a binary classification task by ranking all candidate answers according to their reasonableness. |
| Outcome: | The proposed approach is less data hungry than existing methods using KGs. |
Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing text-to-SQL datasets that capture complex reasoning are limited by their execution accuracy. |
| Approach: | They present a bilingual text-to-SQL dataset specific to complex reasoning . their evaluation shows that Archer challenges the capabilities of current models . |
| Outcome: | The proposed dataset challenges state-of-the-art models with 6.73% execution accuracy . the dataset contains 1,042 English and 1,042, Chinese questions and 521 unique SQL queries . |
Improving Sequential Model Editing with Fact Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to fix erroneous knowledge in Pre-trained Language models experience a performance decline when the number of edits increases. |
| Approach: | They propose a framework that leverages factual information to enhance editing generalization and guide the identification of edits by retrieving related facts from the fact-patch memory. |
| Outcome: | The proposed framework can improve model generalization and accuracy even with thousands of edits. |
Multi-view Contrastive Learning for Entity Typing over Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to knowledge graph entity typing ignore the way types can be clustered together. |
| Approach: | They propose a method that effectively encodes coarse-grained knowledge from clusters into entity and type embeddings. |
| Outcome: | The proposed method encodes coarse-grained knowledge from clusters into entity and type embeddings. |
Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to hierarchical multi-label text classification (HMTC) ignore the correlation between similar samples and introduce noise . |
| Approach: | They propose a semi-supervised method that uses a label hierarchy to bring text and label embeddings closer to each other by supervised contrastive learning. |
| Outcome: | The proposed method bridges the gap between supervised contrastive learning and HMTC by bringing text and label embeddings closer. |
Transformer-based Entity Typing in Knowledge Graphs (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge graphs encoding entity types are far from complete, since in real-world applications they are continuously emerging. |
| Approach: | They propose a transformer-based approach to infer plausible entity types by encoding neighbours' information by a local transformer and a global transformer. |
| Outcome: | The proposed approach outperforms the state-of-the-art on two real-world datasets. |
CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge rewriting methods may include irrelevant information, omit crucial details, or fail to align with the question’s semantics. |
| Approach: | They propose a new rewriting method CoTKR for generating reasoning traces and corresponding knowledge in an interleaved manner, thereby mitigating the limitations of single-step knowledge rewrite. |
| Outcome: | The proposed method mitigates the limitations of single-step knowledge rewriting and bridges the preference gap between the knowledge reactor and the question answering (QA) model. |
Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to abstract inference ignore the *polysemy* and *hierarchical nature of concepts* . prevailing approaches disregard how arguments might entail differently across various concept levels, thereby missing potential enlargement connections. |
| Approach: | They propose a framework that organizes arguments hierarchically and delves into entailment relations at diverse concept levels. |
| Outcome: | The proposed framework improves the model's generalization and reasoning prowess in natural language inference. |
InstructEd: Soft-Instruction Tuning for Model Editing with Hops (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for model editing are limited due to excessive memorization and knowledge conflict issues. |
| Approach: | They propose to insert soft instructions into the attention module to facilitate interactions between instructions and questions and to understand and utilize new facts. |
| Outcome: | The proposed method achieves 10% improvement in one-hop (multi-hop) model editing on three datasets with LLaMAs and GPT2 . |
A Usage-centric Take on Intent Understanding in E-Commerce (2024.emnlp-main)
Copied to clipboard
| Challenge: | Identifying and understanding user intents is a crucial task for E-Commerce. |
| Approach: | They propose to use intent understanding as a natural language reasoning task independent of product ontologies to identify and understand user intents. |
| Outcome: | The proposed framework can't be used to strongly align user intents with products with desirable properties and recommend useful products across diverse categories. |
An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks. |
| Approach: | They propose to employ four PEFT methods to fine-tune the LLM component of open-source MLLMs. |
| Outcome: | The proposed method is the best performing on seven datasets, while fine-tuning the connector layers leads to improved performance in most MLLMs. |