Papers by Jinming Zhao
Towards relation extraction from speech (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting relations from speech have been neglected due to the nature of speech. |
| Approach: | They propose a listening information extraction task that uses speech to extract relation extraction from speech . they use a text-to-speech system and crowd-sourced native English speakers to test the task . |
| Outcome: | The proposed task extracts semantic relationships from speech data using a new model . the proposed task is more challenging than the existing method due to the characteristics of speech . |
Revealing Personality Traits: A New Benchmark Dataset for Explainable Personality Recognition on Dialogues (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current research treats personality recognition as a classification task, failing to reveal the supporting evidence for the recognized personality. |
| Approach: | They propose a task that aims to reveal the reasoning process as supporting evidence of the personality trait. |
| Outcome: | The proposed task reveals the reasoning process as supporting evidence of the personality trait. |
It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing siMT systems are trained and evaluated on offline translations . however, evaluation gap remains notable, calling for constructing large-scale interpretation corpora . |
| Approach: | They propose a translation-to-interpretation transfer method which converts offline translations into interpretation-style data. |
| Outcome: | The proposed interpretation test set shows that SiMT models improve on translation vs interpretation data. |
Generating Synthetic Speech from SpokenVocab for Speech Translation (2023.findings-eacl)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) models require large amounts of data to train, but their size is considerably smaller than text-based MT data. |
| Approach: | They propose a method to convert MT data to ST data via text-to-speech systems. |
| Outcome: | The proposed method improves translation quality by an average of 1.83 BLEU score while performing equally well as TTS-generated speech in improving translation quality. |
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021.acl-long)
Copied to clipboard
| Challenge: | Emotion recognition in conversation is a crucial component in affective dialogue systems, which helps the system understand users’ emotions and generate empathetic responses. |
| Approach: | They propose a multimodal fused graph convolutional network model which leverages multimodal dependencies and speaker information to model inter-speaker and intra-speech dependency. |
| Outcome: | The proposed model outperforms other SOTA methods on two public benchmark datasets, IEMOCAP and MELD. |
RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech Transformers in speech translation systems have facilitated state-of-the-art (SotA) results, but their computational cost is high. |
| Approach: | They propose a Reducer Adaptor block that could be seamlessly integrated within any Transformer-based speech encoding architecture. |
| Outcome: | The proposed Reducer Adaptor block outperforms the existing SotA architecture by an average of 0.68 BLEU score on 8 language pairs from Must-C. |
Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning has shown high efficacy in several NLP tasks, especially in few-shot settings. |
| Approach: | They propose a backdoor attack method that poisons demonstration examples and poisons the demonstration context, preserving the model's generality. |
| Outcome: | The proposed method can make models behave in alignment with predefined intentions without fine-tuning the model. |
NAIST-SIC-Aligned: An Aligned English-Japanese Simultaneous Interpretation Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | Simultaneous interpretation data is a task where an utterance is translated in real-time. |
| Approach: | They propose to use an automatically-aligned parallel English-Japanese SI dataset to make it suitable for model training. |
| Outcome: | The proposed model improves translation quality and latency over baselines. |
Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are not effective for weight-poisoning backdoor attacks. |
| Approach: | They propose a parameter-efficient fine-tuning (PEFT) method that updates only a limited set of model parameters and provides a robust defense against weight-poisoning backdoor attacks. |
| Outcome: | The proposed method identifies poisoned samples through confidence and is robust against weight-poisoning backdoor attacks. |
An Efficient Context-Dependent Memory Framework for LLM-Centric Agents (2025.naacl-industry)
Copied to clipboard
| Challenge: | a recent study has demonstrated that context-dependent memory encoding can help to retrieve key memory cues essential for problem-solving. |
| Approach: | They propose an efficient architecture miming human memory processes through multistage encoding, context-aware storage, and retrieval strategies for LLM-centric agents. |
| Outcome: | The proposed architecture surpasses state-of-the-art online LLM-centric approaches on two interactive decision-making benchmarks in the navigation and manipulation domain. |
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database (2022.acl-long)
Copied to clipboard
| Challenge: | Existing data resources to support multimodal affective analysis in dialogues are limited in scale and diversity. |
| Approach: | They propose a multimodal multi-scene multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series. |
| Outcome: | The proposed dataset contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. |
ESCoT: Towards Interpretable Emotional Support Dialogue Systems (2024.acl-long)
Copied to clipboard
| Challenge: | Emotion-focused and strategy-driven chain-of-thought (ESCoT) is a new paradigm for emotional support dialogues. |
| Approach: | They propose an emotional support response generation scheme to improve interpretability . they generate a dataset and develop a model to generate dialogue responses with better interpretability. |
| Outcome: | The proposed scheme can generate dialogue responses with better interpretability. |
Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | ProAttack is a novel and efficient method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Approach: | They propose a method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. |
| Outcome: | The proposed method achieves state-of-the-art performance on several NLP tasks, particularly in few-shot settings. |
Self-supervised Rewiring of Pre-trained Speech Encoders:
Towards Faster Fine-tuning with Less Labels in Speech Processing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech encoders have facilitated great success across various speech processing tasks, but fine-tuning them for downstream tasks requires large training data to converge or to achieve state-of-the-art. |
| Approach: | They propose to rewire pre-trained speech encoders to improve their representation space without task-specific labels by neutrally synthesising audio inputs and frame masking. |
| Outcome: | The proposed model shows consistent improvement in isotropy in the representation space on 6 speech processing tasks. |
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal fusion models trained on full-modality samples fail when partial modalities are missing. |
| Approach: | They propose a model to deal with the uncertain missing modality problem by learning robust joint multimodal representations that can predict the representation of any missing modal given available modalities under different missing-modality conditions. |
| Outcome: | The proposed model significantly improves performance under uncertain missing-modality testing conditions and full-modalities ideal testing conditions. |
DialogueEIN: Emotion Interaction Network for Dialogue Affective Analysis (2022.coling-1)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation (ERC) has attracted increasing research attention in recent years. |
| Approach: | They propose to model the emotional interactions between speakers to simulate the emotional inertia, emotional stimulus, global and local emotional evolution in dialogues. |
| Outcome: | The proposed model can achieve superior performance compared to state-of-the-art methods on four ERC benchmark datasets, IEMOCAP, MELD, EmoryNLP and DailyDialog. |