Papers by Jin Ma
Copied to clipboard
| Challenge: | Existing fine-tuning algorithms for vision-language models are restricted by patient privacy concerns and can contain imperceptible noise. |
| Approach: | They propose a framework to mitigate adversarial noise and mitigate upstream noise during fine-tuning. |
| Outcome: | The proposed framework improves model robustness and transferability while decreasing noise levels negatively impact downstream performance. |
Copied to clipboard
| Challenge: | Sign language production models ignore structural correlations between channels and use multi-channel spatial attention to capture correlations across channels. |
| Approach: | They propose a novel approach to transform sign language into a unified feature representation using multi-channel spatial attention and temporal attention to learn sequential dependencies for each channel over time. |
| Outcome: | The proposed model outperforms state-of-the-art models on two sign language datasets from diverse cultures. |
Copied to clipboard
| Challenge: | Existing chart understanding benchmarks are overwhelmingly English-centric, limiting their accessibility and relevance to global audiences. |
| Approach: | They propose a multilingual chart question answering benchmark that enables efficient multilingual generation via data translation and code reuse. |
| Outcome: | The proposed benchmark systematically evaluates multilingual chart understanding on state-of-the-art LVLMs and shows a significant performance gap between English and other languages. |
Copied to clipboard
| Challenge: | Existing EE datasets define fixed event types and design specific schemas for each of them, failing to cover diverse events emerging from the online text. |
| Approach: | They propose to use a sentence-level dataset to benchmark Open Event Extraction without restricting event types. |
| Outcome: | The proposed dataset contains more than 42,000 news titles in 34 topics collected from Chinese web pages. |
Copied to clipboard
| Challenge: | Document images are characterized by higher resolutions, denser content, and more complex structural layouts. |
| Approach: | They propose a 1.2B-parameter document parsing vision-language model that decouples layout analysis from local content recognition. |
| Outcome: | The proposed model surpasses general-purpose and domain-specific models on multiple benchmarks while maintaining significantly lower computational overhead. |
Copied to clipboard
| Challenge: | Large language models implicitly fabricate information when inputs are incomplete, causing confidence but unreliable conclusions. |
| Approach: | They propose a framework for grounded reasoning under incomplete information that decomposes reasoning into two stages . they propose stage-specific rewards to penalize hallucinations, enabling models to detect gaps, stop proactively, and resume reasoning after clarification. |
| Outcome: | The proposed framework improves premise detection and task success by 30% . it also reduces average response length by over 20% . |
Copied to clipboard
| Challenge: | Existing models for fine-grained speaking styles are limited in terms of accuracy, coverage, and naturalness. |
| Approach: | They propose a model that pre-trains with coarse captions and annotates with a pipeline that grounds captions in audio. |
| Outcome: | The proposed model outperforms existing models with fine-grained style annotations . it integrates global and fine-granular supervision, enabling unified representations based on the proposed model . |
Copied to clipboard
| Challenge: | Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing. |
| Approach: | They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models. |
| Outcome: | The proposed model improves on real-world and synthetic datasets compared with baselines. |
Copied to clipboard
| Challenge: | Abstraction and Reasoning Corpus and ARC-AGI are widely used to assess progress in artificial intelligence. |
| Approach: | They propose a two-stage pipeline that separates perception and reasoning . they propose to test this pipeline against standard end-to-end one-stage evaluation . |
| Outcome: | The proposed pipeline separates perception and reasoning, and isolates reasoning from bottlenecks. |
Copied to clipboard
| Challenge: | Chinese Search Query Spell Correction is a task designed to identify and correct typographical errors within queries. |
| Approach: | They propose a large-scale benchmark specifically developed for Chinese Query Spell Correction. |
| Outcome: | The proposed benchmark covers a broad range of topics, including formal entities, everyday colloquialisms and idiomatic expressions. |
Copied to clipboard
| Challenge: | a new study proposes a conversational search system that integrates product attributes and dialog with search . but it faces two real world challenges: imperfect product schema/knowledge and lack of training dialog data . |
| Approach: | They propose an end-to-end conversational search system that integrates search with text . they propose an utterance transfer approach that generates dialogue utterations from other domains . |
| Outcome: | The proposed system outperforms the best tested baseline in a conversational search dataset for online shopping. |
Copied to clipboard
| Challenge: | Existing methods for fine-tuning visual signals are limited by their size and complexity. |
| Approach: | They propose a multi-scale frequency-based fine-tuning method that integrates textual information and performs multi-level fine- tuning of visual signals in the frequency domain. |
| Outcome: | Extensive experiments on multimodal models, including CLIP and LLaVA, demonstrate that the proposed method significantly improves performance and efficiency with minimal cost and fast convergence within one epoch. |
Copied to clipboard
| Challenge: | Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing. |
| Approach: | They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning. |
| Outcome: | The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability. |
Copied to clipboard
| Challenge: | Existing methods for Factual Error Correction (FEC) use mask-then-correct paradigms . however, the lack of datasets containing false claims has impeded progress . |
| Approach: | They propose a method that enhances few-shot FEC with a pivot task approach using large language models. |
| Outcome: | The proposed method outperforms its few-shot counterpart by 7.9 points in SARI . it improves widely-adopted SARI metrics by 11.3 compared to the best-performing methods . |
Copied to clipboard
| Challenge: | Backdoor attacks pose a severe threat to the trustworthiness of DNN-based language models. |
| Approach: | They propose a data-centric defense that extends memorization definitions to fine-grained sentences . they find that duplicated sentence elements are necessary for successful backdoor attacks . |
| Outcome: | The proposed defense outperforms state-of-the-art defenses against backdoor attacks. |
Copied to clipboard
| Challenge: | Lossless compression has made significant advancements in Genomics Data storage, sharing and management. |
| Approach: | They propose a novel agent-based GD Compressor with 3 layers with a multi-agent named Leader and Worker. |
| Outcome: | The proposed method improves on existing methods with low-level modeling and limited adaptability and user-unfriendly interface. |
Copied to clipboard
| Challenge: | Existing defense approaches focus on developing new model structures or training algorithms, but they do little to tap the potential of training instances. |
| Approach: | They propose a method that can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training. |
| Outcome: | The proposed method can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training. |
Copied to clipboard
| Challenge: | Existing approaches to safety alignment of large language models rely on costly manual annotations or human review. |
| Approach: | They propose a closed-loop reinforcement learning framework called TriPlay-RL that enables iterative collaboration among three roles with near-zero manual annotation. |
| Outcome: | The proposed framework achieves 20%–50% improvement in adversarial effectiveness while preserving high output diversity while achieving 10%–30% gains in safety performance without degrading general reasoning capability. |
Copied to clipboard
| Challenge: | Recent studies have highlighted the potential of Large Language Models (LLMs) as zero-shot relevance rankers. |
| Approach: | They propose to use a ranking loss to transfer ranking knowledge from LLMs to smaller models like BERT. |
| Outcome: | The proposed model has been successfully integrated into a commercial web search engine as of February 2024. |
Copied to clipboard
| Challenge: | Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UITARS-1.5-7B. |
| Approach: | They propose a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. |
| Outcome: | The proposed framework outperforms GUI-Owl-7B and UI-TARS-1.5-7B on MemGUI-Bench and delivers 17.1% improvement on AndroidWorld over the base Qwen model. |
Copied to clipboard
| Challenge: | Diffusion language models (DLMs) offer advantages in parallel generation and bidirectional context modeling, but they face a critical trade-off between inference speed and output quality for tasks with strict structural constraints such as code generation. |
| Approach: | They propose an efficient sampling algorithm that reduces the number of tokens unmasked per step based on the model’s evolving confidence. |
| Outcome: | The proposed method improves Pass@1 accuracy by 1.9% while achieving 251.4% inference speedup. |
Copied to clipboard
| Challenge: | Scientific innovation is driven by detailed workflows, which include critical steps such as contextualizing literature, generating ideas, validating ideas, and planning new research. |
| Approach: | They propose to use large language models to extract five key aspects from scientific publications to optimize scientific workflows. |
| Outcome: | The proposed dataset includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. |
Copied to clipboard
| Challenge: | Open Information Extraction (OpenIE) models rely heavily on large amounts of annotated data. |
| Approach: | They propose a training framework that maximizes data efficiency through a cycle-consistency mechanism. |
| Outcome: | The proposed approach improves the quality of training data by curating low-quality datasets annotated by a large language model. |
Copied to clipboard
| Challenge: | Recent surge in deep learning technologies has significantly accelerated research in this area. |
| Approach: | They propose a comprehensive summary of the relevant tasks in geometry problem solving and a review of related deep learning methods. |
| Outcome: | The proposed method is based on a systematic review of related methods and evaluation metrics and methods. |
Copied to clipboard
| Challenge: | Syntactic movement is a core concept in generative linguistics to account for word-order variation and long-distance dependencies. |
| Approach: | They annotated every sentence in the audiobook The Little Prince using X-bar style tree annotations. |
| Outcome: | The proposed model shows that deep structure significantly predicts neural responses in English but not in Chinese. |
Copied to clipboard
| Challenge: | Existing studies on web page quality assessment neglect the aspect of web page content. |
| Approach: | They propose a Chinese dataset for web page quality assessment . the dataset includes over 65,000 detailed an-notations spanning four sub-dimensions . |
| Outcome: | The proposed dataset includes over 65,000 detailed an-notations spanning four sub-dimensions and incorporates elements such as HTML+CSS, text, and visual screenshot. |
Copied to clipboard
| Challenge: | Existing summarization methods compress content for gist browsing, but they break prerequisite logic in instructional videos. |
| Approach: | They propose a framework that decouples epistemic planning from content generation. |
| Outcome: | The proposed framework outperforms strong end-to-end baselines on Knowledge Progression Consistency and Learning Objective Coverage. |
Copied to clipboard
| Challenge: | Knowledge distillation (KD) can transfer knowledge from the original model into a compact model to achieve model compression. |
| Approach: | They propose a knowledge distillation method with reptile meta-learning to facilitate the transfer of knowledge from the teacher to the student. |
| Outcome: | Extensive experiments on the GLUE benchmark show the proposed method performs better than previous methods. |
Copied to clipboard
| Challenge: | Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent. |
| Approach: | They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs. |
| Outcome: | The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models. |
Copied to clipboard
| Challenge: | Existing adversarial detection methods require access to training data, which brings noteworthy concerns regarding privacy leakage and generalizability. |
| Approach: | They propose a data-agnostic adversarial detection framework which induces different responses between normal and adversarials to UAPs. |
| Outcome: | The proposed framework achieves competitive detection performance on various text classification tasks, and maintains equivalent time consumption to normal inference. |
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |
Copied to clipboard
| Challenge: | Existing methods to update a multilingual model with new language pairs are expensive and time-consuming. |
| Approach: | They propose an entropy-based vocabulary substitution method that walks through new language pairs for incremental learning while remaining the size of the original vocabulary. |
| Outcome: | The proposed method achieves better performance and saves excess overhead in a multilingual machine translation task. |
Copied to clipboard
| Challenge: | Embodied agents have successfully leveraged large language models (LLMs) to better transform human instructions and images into executable task plans. |
| Approach: | They propose Conflict Detection Rules to identify and manage data quality issues in vector knowledge bases and correct the index structure. |
| Outcome: | Experimental results show that planners with Conflict Detection Rules exceed the basic LLM planner by 15.25% and 14.25% in grammatical accuracy (GA) and interpretation accuracy (IA) on average. |
Copied to clipboard
| Challenge: | despite well-reasoned intuitions, most context-aware neural machine translation models show only modest improvements over sentence-level systems. |
| Approach: | They propose a more realistic setting for document-level translation called paragraph-to-paragraph (PARA2PARA) they collect a dataset of Chinese-English novels to promote future research . |
| Outcome: | The proposed model improves translation quality across document-level metrics and discourse phenomena. |
Copied to clipboard
| Challenge: | Existing datasets for Chinese instruction tuning are not well-aligned with Chinese users’ interaction patterns. |
| Approach: | They propose to use Chinese instruction tuning datasets to improve instruction fine-tuning for Chinese users. |
| Outcome: | The proposed dataset shows that Chinese models achieve competitive performance in diverse benchmarks. |
Copied to clipboard
| Challenge: | Increasing interest in sports has led to the rapid advancement of large models, particularly multimodal large language models (MLLMs) . linguistic intelligence is a key component of large-model-driven sports intelligence . |
| Approach: | They propose to establish a foundation for advancing research and practical development of large-model-driven sports intelligence. |
| Outcome: | The proposed model-driven sports intelligence will be able to process and generate sports-related language effectively and process multiple data modalities. |
Copied to clipboard
| Challenge: | Existing studies focus on overcoming catastrophic forgetting on original language pairs while lacking encouragement to learn new knowledge from incremental learning. |
| Approach: | They propose a knowledge transfer method that can adapt original MNMT models to diverse incremental language pairs by flexibly introducing knowledge from external models into original models, which encourages the models to learn new language pairs. |
| Outcome: | The proposed method outperforms baselines on multiple languages while maintaining performance on original language pairs. |
Copied to clipboard
| Challenge: | Existing adversarial training methods require multi-step gradient ascents or word substitutions to obtain adversarials, which impairs the effectiveness of adversariarial training. |
| Approach: | They propose a procedure for instead adversarial training with only clean data that estimates the adversarials loss by perturbing the input data’s probability distribution rather than their embeddings. |
| Outcome: | The proposed procedure reduces time consumption by up to 70% compared to current best-performing adversarial training methods. |
Copied to clipboard
| Challenge: | Existing research often overlooks the data quality issues within retrieval results, often caused by inaccurate existing vector-distance-based retrieval methods. |
| Approach: | They propose to use Context-Driven Index Trimming framework to capture and regulate consistency between retrieved contexts and modify indexes in the database. |
| Outcome: | Experiments show that the proposed framework can improve answer quality by 3.75% on open-domain question-answering tasks. |
Copied to clipboard
| Challenge: | Existing studies rely on long-term search log mining to improve search experience . EQE system is a novel event retrieval framework that can select the best expansion from a significant amount of potential events quickly and accurately. |
| Approach: | They propose a QE system that uses a four-stage event retrieval framework . they collect news headlines and then refine a dual-tower semantic model to serve as an encoder . |
| Outcome: | The proposed system can select the best expansion from a significant amount of potential events quickly and accurately. |