Papers by Charlotte Siska
MULTIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities (2025.emnlp-main)
Copied to clipboard
Sahil Verma, Keegan Hines, Jeff Bilmes, Charlotte Siska, Luke Zettlemoyer, Hila Gonen, Chandan Singh
| Challenge: | Existing approaches to detect harmful queries to large language models are fallible and vulnerable to attacks that exploit mismatched generalization of model capabilities. |
| Approach: | They propose an approach to detect harmful queries to large language models (LLMs) OMNIGUARD identifies internal representations of an LLM/MLLM that are aligned across languages or modalities and builds a language-agnostic or modality-adic classifier for detecting harmful prompts. |
| Outcome: | OMNIGUARD improves harmful prompt classification accuracy by 11.57% over the strongest baseline in a multilingual setting, by 20.44% for image-based prompts, and sets a new SOTA for audio-based ones. |
Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks (2024.acl-long)
Copied to clipboard
| Challenge: | Using benchmarks to evaluate Large Language Models is inconsistent with the assumption that the test prompts within a benchmark represent a random sample from some real-world distribution of interest. |
| Approach: | They propose to use a model's average performance across the test prompts of a benchmark to evaluate its performance. |
| Outcome: | The results show that the correlation between model performance across test prompts and the test prompt can change model rankings on major benchmarks. |