Papers by Wen Huang
Copied to clipboard
| Challenge: | Existing methods for multi-turn function calling are limited by redundancy and lack explicit integration of progress awareness into training. |
| Approach: | They propose a framework that explicitly integrates progress awareness into LLM training for multi-turn function calling. |
| Outcome: | Empirical results show that Progra outperforms existing methods on two public benchmarks. |
Copied to clipboard
| Challenge: | Evaluating role-playing capabilities in large language models is challenging due to complex dynamics involved in role-playering. |
| Approach: | They propose a simulation sandbox that generates situational fine-grained character behavior trajectories to enhance LLM performance. |
| Outcome: | The proposed model generates situational fine-grained character behavior trajectories to enhance performance. |
Copied to clipboard
| Challenge: | Temporal knowledge graphs (TKGs) require predicting future facts by modeling structural dependencies within each snapshot and temporal evolution across snapshots. |
| Approach: | They propose an encoder-agnostic framework that provides persistent entity states . EST maintains a global state buffer and aligns structural evidence with sequential signals . |
| Outcome: | Experiments show that EST improves diverse backbones and achieves state-of-the-art performance. |
Copied to clipboard
| Challenge: | Existing defenses, including post-training alignment and prompt engineering, struggle with adaptability to out-of-distribution (OOD) attacks. |
| Approach: | They propose an adversarial game-based defense method that dynamically adjusts LLMs’ internal representations to achieve a balanced trade-off between helpfulness and harmlessness. |
| Outcome: | The proposed method improves LLMs’ safety over all baselines. |
Copied to clipboard
| Challenge: | Using language models (LMs) to solve complex problems, humans might struggle to understand and repair flawed ones. |
| Approach: | They propose to automatically decompose complex problems into simpler pieces that correspond to specific subtasks and measure their assistive value. |
| Outcome: | The proposed method enables non-experts to solve 33.3% more problems and speeds them up by 3.3x . |
Copied to clipboard
| Challenge: | a recent study shows that language models are prone to self-contradiction during dialogues. |
| Approach: | They propose a red teaming framework that detects and attempts to explain dialogues, then modifies existing contradictory content using the explanation. |
| Outcome: | The proposed task improves the ability to detect contradictory dialogues and provides valid explanations. |
Copied to clipboard
| Challenge: | Existing models for text retrieval are based on a multi-stage process that involves retrieving documents from a large corpus. |
| Approach: | They propose to build a multilingual text representation model and a cross-encoder reranker from scratch for text retrieval. |
| Outcome: | The proposed models outperform the state-of-the-art models on long-context retrieval benchmarks. |
Copied to clipboard
| Challenge: | Existing code translation benchmarks focus on individual functions, overlooking repository-level challenges like intermodule coherence and dependency management. |
| Approach: | They propose a framework for benchmarking Java-to-C# translation at the repository level . it uses a translation framework guided by skeletons and fine-grained quality evaluation . |
| Outcome: | The proposed framework improves Java-to-C# translation quality at the repository level. |
Copied to clipboard
| Challenge: | Existing studies highlight that dependency-related issues cause over 40% of observed runtime errors on the generated repository. |
| Approach: | They propose a large-scale benchmark and evaluation framework specifically designed to assess LLMs’ capability on dependency inference. |
| Outcome: | The proposed model achieves only a 48% execution pass rate on Python, indicating room for improvement. |
Copied to clipboard
| Challenge: | Existing unsupervised reinforcement learning methods lack the capacity to adapt to the model’s evolving reasoning capabilities during training. |
| Approach: | They propose an unsupervised reinforcement learning algorithm that adapts rewards to balance consensus and exploration based on the Free Energy Principle. |
| Outcome: | Empirical evaluations on nine datasets show that FREIA outperforms baseline methods on reasoning tasks. |
Copied to clipboard
| Challenge: | Existing approaches to program repair are based on correctness alone. |
| Approach: | They propose a framework that mitigates over-editing and improves repair accuracy by generating buggy programs and re-edits. |
| Outcome: | The proposed framework improves repair precision by 31.4% under fix1@1, a metric that considers repair correctness and extent, and significantly increases decoding throughput when combined with speculative editing. |
Copied to clipboard
| Challenge: | Existing methods for web scraping suffer from limited adaptability and scalability when faced with a new website. |
| Approach: | They propose a framework that generates web scrapers with large language models and a new executability metric to measure the performance of web scraper generation tasks. |
| Outcome: | The proposed framework can handle diverse web environments more efficiently. |
Copied to clipboard
| Challenge: | MPLSandbox is an out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler and analysis tools for Large Language Models (LLMs). |
| Approach: | They propose a multi-programming language sandbox that provides unified feedback from compilers and analysis tools for Large Language Models. |
| Outcome: | The proposed multi-language sandbox can provide comprehensive feedback from compilers and analysis tools for large language models (LLMs). |
Copied to clipboard
| Challenge: | Existing large language models (LLMs) do not align with psychiatric diagnostic protocols. |
| Approach: | They propose a framework that transforms the Mini International Neuropsychiatric Interview into automatic computational workflows through coordinated multi-agent collaboration. |
| Outcome: | The proposed framework transforms the gold-standard Mini International Neuropsychiatric Interview (MINI) into automatic computational workflows through coordinated multi-agent collaboration. |
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
Copied to clipboard
| Challenge: | Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. |
| Approach: | They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. |
| Outcome: | The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references. |
Copied to clipboard
| Challenge: | Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data. |
| Approach: | They propose a method that uses a dynamic graph to organize auxiliary languages in prompts to improve LRL translations. |
| Outcome: | The proposed method improves translation accuracy in low-resource languages (LRLs) using auxiliary language pairs and synthetic pseudo-parallel data. |
Copied to clipboard
| Challenge: | Existing research on inference scaling focuses on unstructured output generation tasks, such as mathematical problems. |
| Approach: | They propose an inference-scaling framework that combines fine-grained beam search with ToolPRM, a process reward model scoring each intra-call decision. |
| Outcome: | The proposed framework outperforms outcome and coarse-grained reward models in predictive accuracy and yields consistent test-time gains on multiple function-calling benchmarks. |
Copied to clipboard
| Challenge: | Existing methods for detecting large language models (LLMs) generate fluent text, but they only use a few tokens due to the short length or insufficient information in some texts. |
| Approach: | They propose a method that leverages external text corpora to evaluate the difference in logit distribution of input text under retrieved human-written and LLM-rewritten contexts. |
| Outcome: | The proposed method achieves state-of-the-art performance in AUROC on five public datasets with three widely-used source LLMs. |
Copied to clipboard
| Challenge: | Existing augmentation techniques manipulate words in the original text that break the semantic coherence of the text, or exploit generative models that ignore preserving entities in the text. |
| Approach: | They propose a novel Entity-to-Text based data augmentation technique called EnTDA to add, delete, replace or swap entities in the original text. |
| Outcome: | The proposed technique generates semantically coherent and entity preserving texts on thirteen NER tasks and two settings. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been widely adopted in real-world dialogue applications, but their robustness is criticized all along. |
| Approach: | They propose to use play-by-play text commentary to build a multi-turn athletic real-world scenario dialogue benchmark to evaluate three critical aspects of multi-turned conversations: ultra multi- turn, interactive multi-twist, and cross-turn tasks. |
| Outcome: | The proposed benchmarks outperform open-source LLMs on three critical aspects of multi-turn conversations: ultra multi-turned, interactive multi- turn, and cross-turn tasks. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have driven the rise of agentic workflows . yet, how can we attribute performance gains to individual upgrades and their interactions? |
| Approach: | They propose a game-theoretic framework that models component upgrades as players and evaluates component coalitions to compute Shapley values. |
| Outcome: | The proposed framework provides interaction-aware attribution and recommendation for model allocation under a fixed workflow structure. |
Copied to clipboard
| Challenge: | Existing large language models (LLMs) exhibit hallucinations when analyzing logs due to the implicit knowledge and rules in logs that LLMs cannot capture. |
| Approach: | They propose a lightweight log analysis framework that generates and utilizes rules through LLMs. |
| Outcome: | The proposed framework outperforms LLM-based methods in log parsing and anomaly detection tasks and achieves better performance compared to case-based approaches. |
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on language abilities and knowledge, often overlooking the assessment of ICL ability. |
| Approach: | They propose to evaluate the ICL ability of Large Language Models (LLMs) using the ICLEval benchmark. |
| Outcome: | The proposed benchmark demonstrates that ICL ability is universally present in different LLMs and model size is not the sole determinant of ICL efficacy. |
Copied to clipboard
| Challenge: | Non-collaborative dialogue agents are expected to engage in strategic conversations with diverse users, and this poses two main challenges for existing dialogue agents: 1) the inability to integrate user-specific characteristics into the strategic planning; 2) the difficulty of training strategic planners that can be generalized to diverse users. |
| Approach: | They propose to integrate a user-aware strategic planning module and a population-based training paradigm into a non-collaborative dialogue agent for securing a mutual agreement that leans favorably towards the system's objectives. |
| Outcome: | The proposed model can be used to achieve a mutual agreement that leans favorably towards the system's objectives. |
Copied to clipboard
| Challenge: | Drug-drug interactions arise when multiple drugs are administered concurrently. |
| Approach: | They propose a pairwise knowledge-augmented generative method for DDIE text generation that integrates biological functions from a knowledge set into a language model. |
| Outcome: | The proposed method outperforms existing methods in DDIE text generation on two professional datasets. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. |
| Approach: | They propose a framework to train critic models using refinement signals to generate feedback loops where critiques guide the model in refining its responses. |
| Outcome: | The proposed framework outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. |
Copied to clipboard
| Challenge: | Existing Mixture-of-Experts training frameworks use a micro-batch to calculate LBL . micro-batches are restricted to a single sequence, preventing expert specialization . |
| Approach: | They propose to use a global-batch to loosen the load balance constraint for MoEs models . they propose to synchronize fi across micro-batches and then use it to calculate the LBL . |
| Outcome: | The proposed global-batch LBL improves the domain specialization of experts . the micro-battery LBL is almost at the sequence level, and the router is pushed to distribute the token evenly . |
Copied to clipboard
| Challenge: | a new framework for evaluation of exhaustive search capabilities is needed . high-entropy enumeration tasks make such ground truth impossible for humans to create . VERITAS is a framework built on the principle of computationally irreducible constraints . |
| Approach: | They propose a framework that uses non-optimizable constraints to create verifiable searches . VERITAS can generate infinite number of test cases with perfect ground truth and precise difficulty control . |
| Outcome: | a new evaluation framework for large language models is based on non-optimizable constraints . the framework can generate infinite number of test cases with perfect ground truth and precise difficulty control . |
Copied to clipboard
| Challenge: | Existing relevance models rely on query-keyword pairs but keywords are usually short texts with scarce semantic information, which may not accurately reflect the underlying advertising purposes. |
| Approach: | They propose a bidding-graph augmented triple-based relevance model with three towers to deeply fuse the bidding graphs and semantic textual data. |
| Outcome: | The proposed model outperforms existing models on a large industry dataset and consistently outperformed existing models. |
Copied to clipboard
| Challenge: | Event information is a type of common sense knowledge that helps people understand how stories evolve and provides predictive hints for future events. |
| Approach: | They propose a temporal event understanding pipeline that integrates state-of-the-art components. |
| Outcome: | The proposed pipeline can be easily adapted to other domains, including biomedical domains. |
Copied to clipboard
| Challenge: | Existing methods for data-to-text generation are insufficient to produce long and diverse texts. |
| Approach: | They propose a planning-based hierarchical variational model that plans a sequence of groups and then realizes each sentence conditioned on the planning result and the previously generated context. |
| Outcome: | The proposed model outperforms state-of-the-art models in long and diverse text generation. |
Copied to clipboard
| Challenge: | a lightweight world model converts raw pixels into object-centric symbolic states amenable to language-based reasoning . IMPLEMENT is a framework for grounding language agents in visual embodied environments . |
| Approach: | They propose a model-based reasoning framework that enables frozen large language models to perform imaginative planning. |
| Outcome: | The proposed framework can be used to ground language agents in visual embodied environments. |
Copied to clipboard
| Challenge: | Distant supervision models suffer from high label noise and are not reliable for DS. |
| Approach: | They propose a model-agnostic instance sampling method for relation extraction (RE) by influence function, namely REIF. |
| Outcome: | The proposed method reduces the computational complexity from O(mn) to O(1), with analyzing its robustness on the selected sampling function. |
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models (MLLMs) have garnered significant attention, offering a promising pathway toward artificial general intelligence (AGI). |
| Approach: | They propose a benchmark to evaluate associative ability while circumventing the inherent ambiguity in association tasks by decomposing ambiguities into two types and propose 'assoCiAm' they conduct extensive experiments on MLLMs, revealing a strong positive correlation between cognition and association. |
| Outcome: | The proposed method shows that ambiguity in association evaluations makes MLLMs more random-like and the model's behavior more random. |
Copied to clipboard
| Challenge: | Large Language Model (LLM) watermarking is radioactive and enables the detection of watermarks inherited by student models when trained on the outputs of watermarked teacher models. |
| Approach: | They propose two types of watermark removal attacks that allow student models to perform untraceable knowledge distillation while avoiding watermark inheritance. |
| Outcome: | The proposed attacks eliminate inherited watermarks while maintaining knowledge transfer efficiency and low computational overhead. |
Copied to clipboard
| Challenge: | Existing work in LLM-based MMT typically mitigates the Curse of Multilinguality . asymmetric phenomenon in linguistic conflicts and synergy varies in different translation directions . |
| Approach: | They propose a direction-aware training approach to address asymmetry in linguistic conflicts and synergy . they propose X-ALMA-13B-Pretrain with multilingual pre-training to achieve comparable performance . |
| Outcome: | The proposed method achieves comparable performance to X-ALMA-13B-Pretrain (only SFT) with fewer pretraining tokens and 17B parameters. |
Copied to clipboard
| Challenge: | Existing pre-training models lack long-turn dialogue sessions due to the scarcity of long-term sessions. |
| Approach: | They propose a framework that can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
| Outcome: | The proposed framework can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. |
Copied to clipboard
| Challenge: | Mainstream speaker diarization systems rely only on acoustic information, making it challenging in complex aural environments. |
| Approach: | They propose a multimodal approach that integrates audio, visual, and semantic cues to enhance speaker diarization. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on multi-party conversations . it integrates audio-visual-semantic cues into the clustering process for acoustic speaker embeddings . |
Copied to clipboard
| Challenge: | Existing benchmarks focus on standalone programming problems, such as HumanEval, MBPP, and LiveCodeBench. |
| Approach: | They propose to use large language models to evaluate their ability to perform incremental development within code repositories by collecting pull requests from 83 GitHub repositorias and using rule-based and intent-based filtering to construct task instances focused on new feature development. |
| Outcome: | The proposed benchmarks show that large language models perform significantly worse in the FEA-Bench, highlighting considerable challenges in repository-level incremental code development. |
Copied to clipboard
| Challenge: | Existing methods for retrieving historical messages are based on similarity-based mechanisms. |
| Approach: | They propose a system that integrates System-1 similarity search with a complementary System-2 mechanism, termed Global Selection. |
| Outcome: | The proposed framework achieves state-of-the-art on long-term memory benchmarks and 93.9 on LoCoMo and 91.6 on LongMemEval-S. |
Copied to clipboard
| Challenge: | Existing benchmarks for long-form generation assess real-world queries with hard-to-verify metrics or use synthetic setups that overlook real-life intricacies. |
| Approach: | They propose a new approach that balances verifiable and real-world assessment with Target-Anchored Evaluation. |
| Outcome: | The proposed model balances real-world and verifiable assessment with Target-Anchored Evaluation (TAE) it generates queries, textual materials, and anchors based on verifier targets within real-life scenarios . |
Copied to clipboard
| Challenge: | Recent work addresses this problem by training span-level hallucination detectors using reinforcement learning and chain-of-thought reasoning. |
| Approach: | They propose a framework that explicitly enforces active evidence seeking during CoT reasoning by requiring quotation of relevant source segments at each verification step. |
| Outcome: | The proposed framework improves hallucination span detection performance with limited reasoning overhead and improved robustness in out-of-domain settings. |
Copied to clipboard
| Challenge: | Existing multimodal emotion and intent recognition tasks focus on classification, not rationale and intrinsic connections between these states. |
| Approach: | They propose a task that requires models to jointly predict emotion and intent while generating natural language explanations for why they co-occur. |
| Outcome: | The proposed model outperforms baseline models in prediction and explanation generation. |
Copied to clipboard
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
Copied to clipboard
| Challenge: | Existing evaluation models for instruction-following have many shortcomings, such as substantial costs and unreliable assessments. |
| Approach: | They propose an LLM critic for fine-grained instruction-following evaluation using a checklist generator and a constraint-level preference optimization method. |
| Outcome: | The proposed model beats strong LLM-as-a-Judge baselines in evaluations under lower computational overhead compared to baselines. |
Copied to clipboard
| Challenge: | Existing benchmarks for instruction-following lack data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios. |
| Approach: | They propose a meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types and a preference graph for each instruction. |
| Outcome: | Extensive experiments on IF-RewardBench show that the proposed benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at various tasks but are vulnerable to jailbreak attacks that induce harmful content generation. |
| Approach: | They propose a reinforcement learning framework that leverages the model’s own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. |
| Outcome: | The proposed framework improves model safety by iterative self-improvement without additional annotated data or external models during training phase. |
Copied to clipboard
| Challenge: | Existing methods for controlling the generation of pre-trained language models infuse domain bias into the generation process, making it difficult to generate out-of-domain texts. |
| Approach: | They propose a retrieval-augmented generation framework that uses retrieval to generate fluent sentences with high attribute relevance. |
| Outcome: | The proposed method can generate fluent sentences with high attribute relevance while keeping domain bias out of the model. |
Copied to clipboard
| Challenge: | Document-level Event Causality Identification (DECI) is a sentence-level task that requires long-text understanding. |
| Approach: | They propose a document-level event causality identification model (SENDIR) that uses sparse attention to capture long-distance dependence. |
| Outcome: | The proposed model can be used to discriminate between event pairs in the same sentence or span multiple sentences. |
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) improves accuracy and timeliness of large language models, but external corpora may become contaminated with LLM-generated texts. |
| Approach: | They propose a method that integrates external knowledge retrieved from external sources into RAG to filter out LLM-generated texts from retrieved results. |
| Outcome: | The proposed method mitigates performance degradation and improves stability of RAG systems. |
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models memorize a considerable fraction of training data, leading to privacy risk of information leakage. |
| Approach: | They propose a method for targeted training data extraction using a smoothed soft prompting and calibrated confidence estimation. |
| Outcome: | The proposed method significantly improves the extraction performance on a recently proposed public benchmark. |
Copied to clipboard
| Challenge: | Prior zero-shot TTS models only mimic the speaker’s voice without further control and adjustment capabilities while prior controllable TTS systems cannot perform speaker-specific voice generation. |
| Approach: | They propose a style control module that captures codec representations corresponding to timbre, content, and style in a discrete decoupling codec space. |
| Outcome: | The proposed system can fully clone the speaker's voice and perform speech-specific adjustment and control functions. |
Copied to clipboard
| Challenge: | Existing methods of prompt-tuning for Aspect-based Sentiment Analysis (ABSA) are crude and simple. |
| Approach: | They propose a Syntax-aware Enhanced Prompt method which mines syntactic information related to aspect words from the syntaktic dependency tree. |
| Outcome: | The proposed method exploits the syntactic knowledge embedded in PLMs and achieves favorable results on three benchmark datasets. |
Copied to clipboard
| Challenge: | Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs’ performance under VH. |
| Approach: | They propose a tool called VHTest to generate a diverse set of VH instances from existing image datasets and a text-to-image generative model to generate VH images based on the text descriptions. |
| Outcome: | The proposed tool finds VH instances in existing image datasets and generates images based on the text descriptions. |
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models use static datasets, leading to data leakage or overlooking the complexities of multi-agent interactions. |
| Approach: | They propose a framework that evaluates the diverse capabilities of LLM agents in multi-agent dynamic environments. |
| Outcome: | The proposed framework assesses the diverse capabilities of LLM agents in multi-agent dynamic environments. |
Copied to clipboard
| Challenge: | a lack of systematic studies on the robustness of language understanding models in task-oriented dialog systems is limiting . authors propose a model-agnostic toolkit LAUG to approximate natural language perturbations . |
| Approach: | They propose a model-agnostic toolkit LAUG to approximate natural language perturbations for testing the robustness of language understanding models in task-oriented dialog systems. |
| Outcome: | The proposed toolkit reveals critical robustness issues in state-of-the-art models. |
Copied to clipboard
| Challenge: | Existing approaches focus on minimizing distances between words in aligned pairs, while suffering from low discriminative capability to distinguish the relative orders between positive and negative candidates. |
| Approach: | They propose a ranking-oriented induction model to learn personalized mapping function for each word. |
| Outcome: | The proposed model can learn personalized mapping function for each word on public datasets including rich-resource and low-resourced languages. |
Copied to clipboard
| Challenge: | Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent. |
| Approach: | They propose a novel detection method to pinpoint language-specific neurons within LLMs by selectively activating or deactivating these neurons. |
| Outcome: | The proposed method can “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. |
Copied to clipboard
| Challenge: | Current studies on text image translation face bottlenecks due to lack of a publicly available dataset and poor optical character recognition. |
| Approach: | They propose a text image translation model with a multimodal codebook and an OCR dataset for Chinese-English translation. |
| Outcome: | The proposed model can associate the image with relevant texts, providing useful supplementary information for translation. |
Copied to clipboard
| Challenge: | Neural machine translation suffers from slow translation speed due to the large search space . a trade-off has to be made between translation quality and speed, argues a new study . |
| Approach: | They apply cube pruning technique to speed up dynamic programming into neural machine translation to speed it up. |
| Outcome: | The proposed method can translate faster on GPUs and CPUs with better translation quality than naive beam search. |
Copied to clipboard
| Challenge: | Recent studies focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, but LLMs can generate diverse implicit toxic output that are difficult to detect via simply zero-shot prompting. |
| Approach: | They propose a reinforcement learning based attacking method to induce the implicit toxic outputs in large language models by fine-tuning toxicity classifiers. |
| Outcome: | The proposed method generates implicit toxic outputs that are difficult to detect via zero-shot prompting on five widely-adopted toxicity classifiers. |
Copied to clipboard
| Challenge: | Crowdsourced dialogue corpora are limited in scale and topic coverage due to the expensive cost of data curation. |
| Approach: | They construct an augmented dataset for the emotional support conversation task using large language models for dialogue augmentation. |
| Outcome: | The proposed approach outperforms baselines of dialogue augmentation and improves the model's generalization ability to open-domain topics. |
Copied to clipboard
| Challenge: | Existing debt collection agents fail to tailor strategies to debtor personas, leading to ineffective collection. |
| Approach: | They present a commercial practice on debt collection agents that organizes debtor personas into a taxonomy and constructs a persona-aware conversation dataset. |
| Outcome: | The proposed agent increases recovery rate by 3.31% and collects additional 100K RMB after two months of testing. |
Copied to clipboard
| Challenge: | Existing efforts to optimize text evaluation prompts neglect the combinatorial impact of multiple factors, leading to insufficient optimization of the evaluation pipeline. |
| Approach: | They propose to integrate 8 key factors for evaluation prompts and integrate them into an algorithm that searches for well-behaved prompting strategies for LLM evaluators. |
| Outcome: | The proposed method outperforms existing methods and human-designed evaluation prompts on four evaluation tasks. |
Copied to clipboard
| Challenge: | Existing methods to control the protagonist's persona in story generation are implicitly and sparsely embodied in stories, so we propose a planning-based generation model called ConPer to explicitly model the relationship between personas and events. |
| Approach: | They propose a model to control the protagonist's persona in story generation by predicting one target sentence and planning the plot as a sequence of keywords with the guidance of the predicted persona-related events and commonsense knowledge. |
| Outcome: | The proposed model outperforms state-of-the-art models for generating more coherent and persona-controllable stories. |
Copied to clipboard
| Challenge: | Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references. |
| Approach: | They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings. |
| Outcome: | The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading. |
Copied to clipboard
| Challenge: | Entity linking is a fundamental task in Natural Language Processing (NLP), connecting mentions within unstructured contexts to their corresponding entities in a Knowledge Base (KB). |
| Approach: | They propose a dual-encoder framework that can efficiently match mentions to two-encoding frameworks by a global-view. |
| Outcome: | The proposed framework achieves state-of-the-art on several entity linking benchmarks. |
Copied to clipboard
| Challenge: | Existing methods for generating stories with complex plots rely on detailed prompts, which inadvertently limit the creative potential of the generated stories. |
| Approach: | They propose a retrieval-auGmented stoRy generation framework with a fOrest of eVidEnce to enhance stories’ complexity. |
| Outcome: | The proposed framework enables generating more diverse plotlines from human-written stories. |
Copied to clipboard
| Challenge: | Character-based dialogue systems (CharacterDial) allow users to customize social characters for social interactions. |
| Approach: | They will collect a large-scale Chinese corpus of characters with diverse categories and behaviors and develop CharacterGLM models to address these challenges. |
| Outcome: | Experiments show that CharacterGLM outperforms most popular open- and closed-source LLMs and performs comparable to GPT-4. |
Copied to clipboard
| Challenge: | Speech deepfake detection (SDD) is a critical research area as speech synthesis technologies become more sophisticated. |
| Approach: | They propose a data-centric approach to generalize SDD data from two perspectives . they propose naive aggregation strategies for mixing heterogeneous data and diversity-optimized sampling strategy for a single dataset and multiple datasets. |
| Outcome: | The proposed approach outperforms the naive aggregation baseline on a 12k-hour data pool while using only 3% of the total available data. |
Copied to clipboard
| Challenge: | Existing methods for document retrieval use contrastive learning on datasets lacking explicit structural information. |
| Approach: | They propose a contrastive learning framework that preserves semantic hierarchies and masked element alignment for fine-grained semantic discrimination. |
| Outcome: | The proposed framework preserves semantic hierarchies and masked element alignment for fine-grained semantic discrimination. |
Copied to clipboard
| Challenge: | Existing evaluations emphasize final accuracy or coarse token counts, and lack automated tools to separate essential logic from structural redundancy. |
| Approach: | They propose a graph-driven framework that quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path needed to reach a correct solution. |
| Outcome: | Evaluating 21 LRMs, the proposed framework quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path (SEP) needed to reach a correct solution. |
Copied to clipboard
| Challenge: | Existing query-based alignment modules enforce uniform cross-attention across all layers, leading to computational redundancy. |
| Approach: | They propose a framework that allows for asynchronous query-based alignment with large-scale visual features. |
| Outcome: | The proposed framework matches or surpasses baseline performance while reducing alignment FLOPs by approximately 37% during training and inference. |
Copied to clipboard
| Challenge: | Existing methods for generating counterfactuals rely on human efforts or task-specific designs. |
| Approach: | They propose to use a fully automatic and task-agnostic CAD generation framework to generate diverse counterfactuals. |
| Outcome: | The proposed framework outperforms human-in-the-loop and task-specific CAD methods on multiple out-of-domain and challenge benchmarks. |
Copied to clipboard
| Challenge: | Existing efforts to automate conversational moderation have focused on banning harmful comments or deleting them, but such efforts can inadvertently push users towards echo chambers that exacerbate polarization. |
| Approach: | They propose a framework to assess models’ moderation capabilities independently of human intervention and propose 'conversational moderation' they propose to use language models as conversational moderators to provide specific feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation. |
| Outcome: | The proposed framework assesses models’ moderation capabilities independently of human intervention and shows that appropriately prompted models provide specific and fair feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation. |
Copied to clipboard
| Challenge: | a new framework protects web content from unauthorized LLM real-time extraction and redistribution . multiple AI companies have been accused of scraping digital IP for proprietary benefit . |
| Approach: | They propose a defense framework that empowers web content creators to safeguard their web-based IP from unauthorized LLM real-time extraction and redistribution by leveraging the semantic understanding capability of LLMs themselves. |
| Outcome: | The proposed defense outperforms traditional defenses on LLMs and improves on black-box optimization problems. |
Copied to clipboard
| Challenge: | Existing instruction tuning methods for large language models (LLMs) are costly and difficult to implement. |
| Approach: | They propose a framework to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. |
| Outcome: | The proposed framework enables rapid adaptation and extension to new modal data or tasks without tuning pretrained models. |
Copied to clipboard
| Challenge: | Existing speech deepfake datasets are limited in scale and diversity, making it challenging to train models that can generalize well to unseen deepfakkes. |
| Approach: | They propose a large-scale speech deepfake dataset that includes over 3 million deepfak samples, totaling more than 3,000 hours of audio, generated using 40 different speech synthesis tools. |
| Outcome: | The proposed dataset includes over 3 million deepfake samples, totaling more than 3,000 hours of audio, generated using 40 different speech synthesis tools. |
Copied to clipboard
| Challenge: | Existing methods to mitigate hallucinations include prompt engineering and model optimization, but lack domain generalization and potential errors in fine-tuning data may exacerbate the hallucism. |
| Approach: | They propose an expert-aware adaptive contrast decoding that uses expert differences in MoE’s higher layers to mitigate hallucinations on QA tasks. |
| Outcome: | The proposed method outperforms baseline models on four datasets Large language models (LLMs) show strong performance but suffer from hallucinations, limiting their application. |