WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have developed watermarking algorithms which restrict the generation process to leave an invisible trace for watermark detection. |
| Approach: | They propose a benchmarking procedure that compares different methods to ensure consistent watermarking strength and jointly evaluates their generation and detection performance. |
| Outcome: | The proposed benchmark compares 4 open-source watermarks on 2 LLMs under 2 watermarking strengths and observes the common struggles for current methods on maintaining the generation quality. |
Similar Papers
EntroBench: Evaluating LLM Watermarking Under Multi-Entropy Scenarios and Practical User Operations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of large language models (LLMs) watermarking are limited to fixed entropy settings. |
| Approach: | They propose a benchmark for LLM watermarking that systematically covers three entropy levels and seven representative tasks. |
| Outcome: | The proposed framework covers three entropy levels and seven representative tasks. |
MarkLLM: An Open-Source Toolkit for LLM Watermarking (2024.emnlp-demo)
Copied to clipboard
Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, Irwin King, Philip Yu
| Challenge: | Large Language Models (LLMs) embed imperceptible yet algorithmically detectable signals in outputs to identify LLM-generated text. |
| Approach: | They propose to develop an open-source toolkit for LLM watermarking that embeds imperceptible yet algorithmically detectable signals in model outputs to identify LLM-generated text. |
| Outcome: | MarkLLM provides a unified framework for implementing LLM watermarking algorithms, while providing user-friendly interfaces to ensure ease of access. |
WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that small, context-dependent shifts in word distributions can be used to apply and detect watermarks, but little work has analyzed the impact of these perturbations on the quality of generated texts. |
| Approach: | They propose a framework that allows for analysis of the impact of watermark settings on the quality of generated texts. |
| Outcome: | The proposed framework provides easy visualization of the quality-detection trade-off of watermark settings. |
Watermark under Fire: A Robustness Evaluation of LLM Watermarking (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Various watermarking methods have been proposed to identify LLM-generated texts . lack of unified evaluation platforms has left many critical questions unanswered . |
| Approach: | They systematize existing LLM watermarkers and watermark removal attacks and develop a unified platform that integrates them. |
| Outcome: | The proposed systematizes existing LLM watermarkers and watermark removal attacks, mapping out their design spaces. |
From Intentions to Techniques: A Comprehensive Taxonomy and Challenges in Text Watermarking for Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are rapidly growing and allowing textual content to be protected against unauthorized use. |
| Approach: | They present a unified overview of different perspectives behind designing watermarking techniques through a comprehensive survey of the research literature. |
| Outcome: | The proposed methods are based on the evaluation datasets used and watermarking addition and removal methods to construct a taxonomy. |
Subtle Signatures, Strong Shields: Advancing Robust and Imperceptible Watermarking in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have led to an increase in AI-generated text on the Internet, presenting a crucial challenge to differentiate AI-created content from human-written text. |
| Approach: | They propose a novel approach to embed watermarks into LLMs that leverages token prior probabilities to improve detectability and maintain watermark imperceptibility. |
| Outcome: | The proposed method improves detectability and imperceptibility of watermarks by partitioning tokens into two distinct groups based on prior probabilities and employing tailored strategies for each group. |
Duwak: Dual Watermarks in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing watermark techniques are effective in embedding single human-imperceptible and machine-detectable patterns without significantly affecting generated text quality and semantics. |
| Approach: | They propose to embed dual secret patterns in token probability distribution and sampling schemes to enhance the efficiency of watermarking. |
| Outcome: | The proposed method achieves highest watermark quality at the lowest required token count for detection, up to 70% less than existing techniques, especially under post paraphrasing attacks. |
WaterPool: A Language Model Watermark Mitigating Trade-Offs among Imperceptibility, Efficacy and Robustness (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to trace the usage of large language models often face trade-offs between imperceptibility and robustness. |
| Approach: | They propose a key-centered scheme to unify existing methods by decomposing a watermark into two components: a 'key module' and a "mark module". |
| Outcome: | The proposed method can be integrated with existing methods and achieve near-optimal imperceptibility and detection efficacy. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs (2025.acl-long)
Copied to clipboard
Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. |
| Approach: | They propose a Chinese Comprehensive Constraints Following Benchmark for LLMs that compiles constraints from real-world instructions and constructs a systematic framework for constraint types. |
| Outcome: | The proposed framework integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. |