Papers by Pei Dong

10 papers
AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models (2024.findings-emnlp)

Copied to clipboard

Challenge: Large Language Models (LLMs) exhibit significant but subtle weaknesses, such as mistakes in instruction-following or coding tasks.
Approach: They propose a framework to automatically expose weaknesses in Large Language Models (LLMs) they use three LLM-powered agents to perform comprehensive weakness identification .
Outcome: The proposed framework shows that it is more effective than untargeted data augmentation methods like Self-Instruct to identify weaknesses in LLMs.
AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)

Copied to clipboard

Challenge: Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment.
Approach: They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.
Outcome: The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references.
Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents (2026.acl-long)

Copied to clipboard

Challenge: Tool-calling agents are increasingly deployed in real-world customer-facing workflows . but most studies on tool-callers focus on idealized settings with general, fixed, and well-specified tasks.
Approach: They propose a tool-calling agent-based data pipeline that converts trajectories into user-facing tasks with controlled intent adaptations.
Outcome: The proposed pipeline can be used to study tool use under three scenarios.
DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024.acl-long)

Copied to clipboard

Challenge: Numerous code large language models (LLMs) have been proposed to enhance code generation performance.
Approach: They propose a diverse instruction model DolphCoder with self-evaluating for code generation that learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Outcome: The proposed model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work.
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)

Copied to clipboard

Challenge: Large language models are often not well aligned with human intents, which requires additional training.
Approach: They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents.
Outcome: The proposed model outperforms existing models and is model-agnostic.
Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT (2023.emnlp-main)

Copied to clipboard

Challenge: Existing methods to fine-tune discriminative models address these challenges by focusing on in-domain intents.
Approach: They evaluate ChatGPT on OOD intent discovery and generalized intent discovery tasks . they outline the strengths and weaknesses of ChatGPt and outline their results .
Outcome: The proposed task aims to extend a closed intent classifier to open-world intent sets.
Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions (2025.findings-emnlp)

Copied to clipboard

Challenge: Reinforcement learning (RL) for large language models typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling.
Approach: They propose a mixed-data approach to training large language models with varying reward clarity . they combine Multiple-choice questions (MCQs) with OE questions for which they use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators.
Outcome: The mixed-data approach improves medical question-answering performance across model scales.
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)

Copied to clipboard

Challenge: Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references.
Approach: They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings.
Outcome: The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading.
CharacterGLM: Customizing Social Characters with Large Language Models (2024.emnlp-industry)

Copied to clipboard

Challenge: Character-based dialogue systems (CharacterDial) allow users to customize social characters for social interactions.
Approach: They will collect a large-scale Chinese corpus of characters with diverse categories and behaviors and develop CharacterGLM models to address these challenges.
Outcome: Experiments show that CharacterGLM outperforms most popular open- and closed-source LLMs and performs comparable to GPT-4.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations