Papers by Jiahui Geng
\mathsf{Con Instruction}: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities (2025.acl-long)
Copied to clipboard
| Challenge: | Existing attacks communicate instruction through text, accompanied by a toxic image or audio . a novel gray-box attack method generates adversarial images or audio to convey harmful instructions to MLLMs . |
| Approach: | They propose a gray-box attack method that generates adversarial images or audio to convey specific harmful instructions to MLLMs by following non-textual instruction. |
| Outcome: | The proposed method achieves highest success rates on visual and audio-language models . larger models are more susceptible toCon Instruction, compared to their underlying models - the results will be released . |
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2026.acl-long)
Copied to clipboard
Jiahui Geng, Fengyu Cai, Shaobo Cui, Qing Li, Liangwei Chen, Chenyang Lyu, Haonan Li, Derui Zhu, Alexander Pretschner, Heinz Koeppl, Fakhri Karray
| Challenge: | Existing benchmarks focus on functional relevance while neglecting code quality. |
| Approach: | They propose a multilingual benchmark to evaluate quality-aware code retrieval . they include fine-grained quality annotations over 42,725 queries and 134,907 code snippets . |
| Outcome: | The proposed benchmarks show that state-of-the-art models fail to separate buggy or insecure code from robust counterparts. |
SGPVT: Self-Generated Proximal Visual Tokens for Mitigating Proximal Collateral Damage in MLLM Unlearning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches focus on general utility metrics, overlooking the preservation of semantically related concepts. |
| Approach: | They propose a method that introduces self-generated proximal visual tokens to prevent forgetting vulnerability. |
| Outcome: | The proposed framework outperforms existing methods in preserving semantically related concepts while achieving effective target unlearning. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs (2025.coling-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi N. Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models (LLMs) generate naturallysounding answers over a broad range of human inquiries, but they still produce content that deviates from real-world facts. |
| Approach: | They propose a framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. |
| Outcome: | The proposed framework assesses the factuality of free-form responses in open domains and evaluates factually of LLMs. |
Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language (2025.acl-long)
Copied to clipboard
Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, Kaifu Zhang
| Challenge: | Existing datasets for instruction-following are monolingual and centered on English . existing data are unable to capture linguistic and cultural subtle differences . |
| Approach: | They propose an extension of IFEval to a localized multilingual version called Marco-Bench-MIF . their benchmark addresses linguistic constraints and cultural references via translation and verification . |
| Outcome: | The proposed extension of IFEval to a localized multilingual version covers 30 languages with varying levels of localization. |
Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI (2026.acl-long)
Copied to clipboard
Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyan Su, Mervat Abassy, Saadeldine Eletter, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
| Challenge: | Prior studies have shown that distinguishing text generated by Large Language Models from human-written text is challenging for humans and often no better than random guessing. |
| Approach: | They conduct extensive case study to determine the upper bound of human detection accuracy. |
| Outcome: | The findings challenge previous conclusions on human detection accuracy across languages and domains. |
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection (2024.emnlp-demo)
Copied to clipboard
Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ta, Raj Tomar, Bimarsha Adhikari, Saad Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuohan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Mujahid, Tarek Mahmoud, Akim Tsvigun, Alham Aji, Artem Shelmanov, Nizar Habash, Iryna Gurevych, Preslav Nakov
| Challenge: | a large number of machine-generated texts are often hard to distinguish between human-written and machine-generated text . this raises concerns about potential misuse, especially within educational and academic domains . |
| Approach: | They propose a system that can detect whether a text is human-written or machine-generated . they use a fine-grained classification schema to identify the use of machine-generated text . |
| Outcome: | The proposed system can distinguish between human-written and machine-generated text . it can detect attempts to obfuscate the fact that a text was machine- generated . |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
FIRE: Fact-checking with Iterative Retrieval and Verification (2025.findings-naacl)
Copied to clipboard
Zhuohan Xie, Rui Xing, Yuxia Wang, Jiahui Geng, Hasan Iqbal, Dhruv Sahnan, Iryna Gurevych, Preslav Nakov
| Challenge: | Fact-checking long-form text is challenging, and breaking it down into multiple atomic claims is not cost-effective. |
| Approach: | They propose a novel agent-based framework that integrates evidence retrieval and claim verification in an iterative manner. |
| Outcome: | The proposed framework reduces large language model (LLM) costs by an average of 7.6 times and search costs by 16.5 times while retaining the same performance. |
CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
Emilio Villa-Cueva, Sholpan Bolatzhanova, Diana Turmakhan, Kareem Elzeky, Henok Biadglign Ademtew, Alham Fikri Aji, Vladimir Araujo, Israel Abebe Azime, Jinheon Baek, Frederico Belcavello, Fermin Cristobal, Jan Christian Blaise Cruz, Mary Dabre, Raj Dabre, Toqeer Ehsan, Naome A Etori, Fauzan Farooqui, Jiahui Geng, Guido Ivetta, Thanmay Jayakumar, Soyeong Jeong, Zheng Wei Lim, Aishik Mandal, Sofía Martinelli, Mihail Minkov Mihaylov, Daniil Orel, Aniket Pramanick, Sukannya Purkayastha, Israfel Salazar, Haiyue Song, Tiago Timponi Torrent, Debela Desalegn Yadeta, Injy Hamed, Atnafu Lambebo Tonja, Thamar Solorio
| Challenge: | a human-curated benchmark of over 5,800 triples of images is used to evaluate multimodal translation systems. |
| Approach: | They introduce a human-curated benchmark of over 5,800 triples of images along with parallel captions in English and regional languages. |
| Outcome: | The results show that visual context improves translation quality in culturally-specific items . |
OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) often produce content that deviates from real-world facts. |
| Approach: | They developed a unified framework to assess the factuality of large language models . open-sourced framework is publicly available as a Python library and web service . |
| Outcome: | OpenFactCheck is open-sourced and publicly released as a Python library and web service. |
Improving Unsupervised Word-by-Word Translation with Language Model and Denoising Autoencoder (D18-1)
Copied to clipboard
| Challenge: | Unsupervised learning of cross-lingual word embeddings has fundamental limitations in translating sentences. |
| Approach: | They propose a method to improve word-by-word translation of cross-lingual embeddings using monolingual corpora without any back-translation. |
| Outcome: | The proposed system surpasses state-of-the-art unsupervised translation systems without costly iterative training. |
VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration (2025.findings-acl)
Copied to clipboard
Jiahui Geng, Qing Li, Zongxiong Chen, Yuxia Wang, Derui Zhu, Zhuohan Xie, Chenyang Lyu, Xiuying Chen, Preslav Nakov, Fakhri Karray
| Challenge: | Existing safety calibration methods focus on model undersafety, where the model responds to hazardous queries, while neglecting oversafetiness, where models refuse to answer safe queries. |
| Approach: | They propose safety calibration which addresses both undersafety and oversafetiness by comparing model responses to a novel dataset of 3,600 image-text pairs. |
| Outcome: | The proposed methods have been used to evaluate safety calibration across image-centric and text-centric scenarios. |
Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Understanding how jailbreaking works remains limited, hindering the development of effective defense strategies. |
| Approach: | They propose a new mechanism that adaptively constrains activations within the safety boundary and propose 'Activation Boundary Defense' to enhance its effectiveness. |
| Outcome: | The proposed defense achieves an average Defense Success Rate (DSR) of over 98% against various jailbreak attacks, with less than 2% impact on the model’s general capabilities. |
HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Hallucination is a significant challenge for large language models, but current methods struggle when non-factual information arises in the early or mid-sequence of outputs, reducing their reliability. |
| Approach: | They propose a method that captures the full dynamics of large language models by using neural differential equations to assess the truthfulness of statements. |
| Outcome: | The proposed method achieves 14% improvement in AUC-ROC on the True-False dataset compared to state-of-the-art methods. |
Reference-free Hallucination Detection for Large Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large vision-language models exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, but they are prone to producing hallucinations. |
| Approach: | They propose to use supervised uncertainty quantification methods to detect hallucinations in large vision-language models. |
| Outcome: | The proposed methods outperform the others in detecting hallucinations on four representative LVLMs across two different tasks. |