Papers by Huajian Zhang
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses. |
| Approach: | They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses. |
| Outcome: | The proposed model performs better on diverse summarisation tasks than existing models. |
Lost in Literalism: How Supervised Training Shapes Translationese in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit translationese errors and generate unexpected unnatural translations . Neural machine translation (NMT) has become the dominant method in machine translation research . |
| Approach: | They evaluate the prevalence of translationese in LLM-generated translations and investigate its roots during supervised fine-tuning. |
| Outcome: | The proposed methods reduce translationese while improving translation naturalness . the proposed methods are validated by human evaluations and automatic metrics . |
Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation (2025.findings-acl)
Copied to clipboard
Qiji Zhou, YiFan Gong, Guangsheng Bao, Hongjie Qiu, Jinqiang Li, Xiangrong Zhu, Huajian Zhang, Yue Zhang
| Challenge: | Existing multimodal benchmarks often overlook counterfactual reasoning, which is crucial for robust video understanding. |
| Approach: | They propose a multidimensional multimodal benchmark that systematically evaluates MLLMs across the abstract-concrete and perception-cognition dimensions. |
| Outcome: | The proposed model decomposes complex queries into structured sub-questions, enabling fine-grained reasoning analysis. |
LNE-Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a problem of data contamination is now almost inevitable during the development of large language models, with the training data often integrating evaluation benchmarks even unintentionally. |
| Approach: | They propose a framework to restore model performance prior to data contamination on potentially leaked datasets by using contamination detection and disruption operation. |
| Outcome: | The proposed framework restores model performance prior to contamination on potentially leaked datasets. |
What Have We Achieved on Non-autoregressive Translation? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that non-autoregressive (NAT) methods underperform autoregressive methods (AT) however, their evaluation using BLEU has been shown to weakly correlate with human annotations. |
| Approach: | They propose to evaluate four representative NAT methods using BLEU to narrow the performance gap between autoregressive and autoregressive translations. |
| Outcome: | The proposed methods underperform NAT and autoregressive methods under more reliable evaluation metrics. |
Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation (2023.acl-long)
Copied to clipboard
Yulong Chen, Huajian Zhang, Yijie Zhou, Xuefeng Bai, Yueguan Wang, Ming Zhong, Jianhao Yan, Yafu Li, Judy Li, Xianchao Zhu, Yue Zhang
| Challenge: | Existing work on cross-lingual summarization (CLS) does not consider crosslingual sources for summarizing. |
| Approach: | They propose a cross-lingual conversation summarization benchmark that explicitly considers source context. |
| Outcome: | The proposed method surpasses baselines on ConvSumX and 3 widely-used manual annotations. |
Leveraging Entailment Judgements in Cross-Lingual Summarisation (2024.findings-acl)
Copied to clipboard
| Challenge: | Synthetically created cross-lingual summarisation datasets are prone to include document-summary pairs where the reference summary is unfaithful to the corresponding document. |
| Approach: | They propose to use off-the-shelf cross-lingual Natural Language Inference to evaluate faithfulness of reference and model generated summaries and use unlikelihood loss to teach a model about unfaithful summary sequences. |
| Outcome: | The proposed approach evaluates faithfulness of reference and model generated summaries and uses unlikelihood loss to teach a model about unfaithful summary sequences. |