Papers by Jiaming Shen
Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion (2022.findings-acl)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) aims to extract semantic relations among entity pairs in a document. |
| Approach: | They propose an evidence-enhanced framework that empowers document-level relation extraction (DocRE) Eider efficiently extracts evidence and effectively fuses extracted evidence in inference. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three benchmark datasets. |
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). |
| Approach: | They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations. |
| Outcome: | The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference. |
Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved competitive performance with limited labeled data for many NLP tasks. |
| Approach: | They propose a prompt-based data selection method for pre-trained language models fine-tuning under cold-start scenarios. |
| Outcome: | The proposed method outperforms the strongest cold-start data selection baselines on six text classification datasets with 128 labels. |
Predicting Text Preference Via Structured Comparative Reasoning (2024.acl-long)
Copied to clipboard
Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurzion, Alexander Rush, Jialu Liu, Michael Bendersky
| Challenge: | Existing approaches to comparative reasoning rely on pretraining or fine-tuning models at the cost of massive human annotation and computation. |
| Approach: | They propose a model that prompts LLMs to generate structured intermediate comparisons by proposing aspects for comparison, followed by generating textual comparisons under each aspect. |
| Outcome: | The proposed model significantly reduces hallucination and improves consistency across various NLP tasks. |
SynSetExpan: An Iterative Framework for Joint Entity Set Expansion and Synonym Discovery (2020.emnlp-main)
Copied to clipboard
| Challenge: | Entity set expansion and synonym discovery are two critical NLP tasks that are often performed separately, without exploring their interdependencies. |
| Approach: | They propose a framework that enables two tasks to mutually enhance each other by including popular entities’ infrequent synonyms into the set, which boosts set expansion recall. |
| Outcome: | The proposed framework can be used to enhance two NLP tasks by including popular entities’ infrequent synonyms into the set, which boosts set expansion recall. |
Multilingual Fine-Grained News Headline Hallucination Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models to generate news headlines often suffer from the "hallucination" problem, where the produced headline does not fully align with the source article's content. |
| Approach: | They propose to use a multilingual, fine-grained dataset to detect news headlines in 5 languages using supervised fine-tuning techniques and coarse-to-fine prompting to boost the few-shot detection performance. |
| Outcome: | The proposed methods boost the few-shot hallucination detection performance in terms of the example-F1 metric. |
Eliciting Knowledge from Experts: Automatic Transcript Parsing for Cognitive Task Analysis (P19-1)
Copied to clipboard
| Challenge: | Cognitive task analysis (CTA) is a type of analysis used to elicit and represent the knowledge and thought processes of domain experts. |
| Approach: | They propose a weakly-supervised framework for automated CTA transcript parsing . they partition the parser process into a sequence labeling task and a text span-pair relation extraction task with distant supervision from human-curated protocol files. |
| Outcome: | The proposed framework reduces human labor and scales the task to a small scale. |
Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Linguistic steganography studies how to hide secret messages in natural language cover texts. |
| Approach: | They propose a method which encodes secret messages using self-adjusting arithmetic coding based on a neural language model. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on four datasets by 15.3% and 38.9% in terms of bits/word and KL metrics. |
LiPO: Listwise Preference Optimization through Learning-to-Rank (2025.naacl-long)
Copied to clipboard
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, Xuanhui Wang
| Challenge: | Recent work on language models with curated feedback provides promising alternatives to RLHF . multiple responses can be ranked by reward models or AI feedback, but there is no study on directly fitting upon a list of responses. |
| Approach: | They propose a method that aligns language models with curated human feedback . they propose SLiC and DPO as promising alternatives to traditional RLHF . |
| Outcome: | The proposed method outperforms DPO and SLiC on several preference alignment tasks with curated and real rankwise preference data. |
Empower Entity Set Expansion via Language Model Probing (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for expanding seed entities with new entities belong to the same semantic class are difficult to implement and can lead to accumulative errors. |
| Approach: | They propose an iterative set expansion framework that leverages automatically generated class names to address the semantic drift issue. |
| Outcome: | The proposed framework generates high-quality class names and outperforms state-of-the-art methods significantly. |
TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names (2021.naacl-main)
Copied to clipboard
| Challenge: | Hierarchical multi-label text classification (HMTC) aims to assign each text document to a set of relevant classes from a taxonomy. |
| Approach: | They propose to conduct HMTC based on only class surface names as supervision signals to mimic human experts. |
| Outcome: | The proposed framework outperforms the best existing method by 25% on two challenging datasets. |
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024.findings-acl)
Copied to clipboard
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang
| Challenge: | Knowledge distillation (KD) is a technique for transferring expertise from large teacher models to compact student models with reduced memory footprints and inference costs. |
| Approach: | They propose to transfer knowledge from large teacher models to compact student models by exploiting teacher-student capacity discrepancies to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
| Outcome: | The proposed framework exploits teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. |
| Approach: | They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures. |
| Outcome: | The proposed method improves the identification of high-level structures using phrase-guided masking. |
Training ELECTRA Augmented with Multi-word Selection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-training methods for NLP tasks require massive computation resources. |
| Approach: | They propose a method that trains a discriminator to detect replaced tokens and select original tokens from candidate sets. |
| Outcome: | The proposed method improves ELECTRA based on multi-task learning on GLUE and SQUAD datasets. |
Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs’ Instruction Following Capability (2026.findings-acl)
Copied to clipboard
Jiaming Wang, Yunke Zhao, Peng Ding, Jun Kuang, Yibin Shen, Zhe Tang, Yilin Jin, ZongYu Wang, Xiaoyu Li, Xuezhi Cao
| Challenge: | Existing models lack the ability to adhere to instructions, resulting in suboptimal performance. |
| Approach: | They propose an automated iterative instruction-following benchmark with integrated feedback mechanism. |
| Outcome: | The proposed benchmark identifies erroneous components in model responses and provides feedback accurately. |
ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that large pretrained language models can generate training data with no task-specific or cross-task data. |
| Approach: | They propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. |
| Outcome: | The proposed framework achieves 4.3% gain over baselines and saves 70% of time compared with baselines using large language models. |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024.findings-naacl)
Copied to clipboard
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky
| Challenge: | Existing methods to rank documents using large language models do not understand these challenging ranking formulations. |
| Approach: | They propose to use Pairwise Ranking Prompting to improve ranking performance . they propose to outperform fine-tuned baseline rankers on benchmark datasets . |
| Outcome: | The proposed technique outperforms supervised baselines on benchmark datasets and outperformed other LLM-based solutions by over 10% on average. |
Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for topic taxonomies focus on frequent terms and local topic-subtopic relations, which leads to limited topic term coverage. |
| Approach: | They propose a framework for topic taxonomy expansion that directly generates topic-related terms belonging to new topics. |
| Outcome: | The proposed framework outperforms baseline methods on two real-world text corpora. |
Corpus-based Open-Domain Event Type Induction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing event extraction methods require predefined event types and their annotations to learn event extractors. |
| Approach: | They propose to represent each event type as a cluster of predicate sense, object head> pairs. |
| Outcome: | The proposed method can discover salient and high-quality event types on three datasets from different domains. |
End-to-End Reinforcement Learning for Automatic Taxonomy Induction (P18-1)
Copied to clipboard
| Challenge: | Existing methods for automating taxonomy induction often divide the problem into two subtasks . a novel end-to-end reinforcement learning approach is proposed to improve the accuracy of such methods. |
| Approach: | They propose an end-to-end reinforcement learning approach to automatic taxonomy induction from a set of terms. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on two public datasets of different domains. |
Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning (2024.acl-long)
Copied to clipboard
Yue Yu, Jiaming Shen, Tianqi Liu, Zhen Qin, Jing Nathan Yan, Jialu Liu, Chao Zhang, Michael Bendersky
| Challenge: | Recent advances in natural language processing (NLP) have witnessed the remarkable capabilities of Large Language Models (LLMs). |
| Approach: | They propose an Explanation-Aware Soft Ensemble framework to empower in-context learning with Large language models. |
| Outcome: | The proposed framework can be used to enhance in-context learning on seven natural language understanding tasks and four varying-size LLMs. |