Hugging Face's logo

Spaces:
TrustSafeAI
/
Retention_Score
Running

App Files Community
Retention_Score / index.html
zaitang's picture
zaitang
Update index.html
051178e verified
raw
history blame
27.1 kB
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description"
content="Demo Page of Retention Score AAAI 2025.">
<meta name="keywords" content="Retention Score, Adversarial robustness, Generative models, Vision-Language Models">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Retention Score: Quantifying Jailbreak Risks for Vision Language Models</title>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<link rel="stylesheet" href="./static/css/custom.css">
<link rel="icon" href="./static/images/favicon.svg">
<!-- <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> -->
<script src="https://code.jquery.com/jquery-3.6.0.js"></script>
<script src="https://code.jquery.com/ui/1.13.2/jquery-ui.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
<!-- for mathjax support -->
<!-- <script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script> -->
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>
$(document).ready(function(){
$('#adaptive-loss-formula-list').on('click', 'a', function(e) {
e.preventDefault();
if (!$(this).hasClass('selected')) {
$('.formula-content').hide(200);
$('.formula-list > a').removeClass('selected');
$(this).addClass('selected');
var target = $(this).attr('href');
$(target).show(200);
}
});
$('#adaptive-dataset').on('click', 'a', function(e) {
e.preventDefault();
if (!$(this).hasClass('selected')) {
$('.interpolation-video-column').hide();
$('#adaptive-dataset > a').removeClass('selected');
$(this).addClass('selected');
var target = $(this).attr('href');
$(target).show();
}
});
})
</script>
<style type="text/css">
.tg {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-amwm{font-weight:bold;text-align:center;vertical-align:top}
.tg .tg-2imo{font-style:italic;text-align:center;text-decoration:underline;vertical-align:top}
</style>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Retention Score: Quantifying Jailbreak Risks for Vision Language Models</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="#" target="_blank">ZAITANG LI</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://sites.google.com/site/pinyuchenpage/home" target="_blank">Pin-Yu Chen</a><sup>2</sup>,
</span>
<span class="author-block">
<a href="https://tsungyiho.github.io/" target="_blank">Tsung-Yi Ho</a><sup>1</sup>,
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>The Chinese University of Hong Kong,</span>
<span class="author-block"><sup>2</sup>IBM Research</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- PDF Link. -->
<span class="link-block">
<a href="https://arxiv.org/abs/2304.09875" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>
<span class="link-block">
<a href="https://arxiv.org/abs/2304.09875" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
<!-- Video Link. -->
<!-- <span class="link-block">
<a href="https://www.youtube.com/watch?v=MrKrnHhk8IA" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-youtube"></i>
</span>
<span>Video</span>
</a>
</span> -->
<!-- Code Link. -->
<!-- <span class="link-block">
<a href="https://github.com/google/nerfies" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span> -->
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- <section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<video id="teaser" autoplay muted loop playsinline height="100%">
<source src="./static/videos/teaser.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">
<span class="dnerf">Nerfies</span> turns selfie videos from your phone into
free-viewpoint
portraits.
</h2>
</div>
</div>
</section> -->
<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
The emergence of Vision-Language Models (VLMs) is significant advancement in integrating computer vision with Large Language Models (LLMs) to enhance multi-modal machine learning capabilities. However, this progress has made VLMs vulnerable to advanced adversarial attacks, raising concerns about reliability. Objective of this paper is to assess resilience of VLMs against jailbreak attacks that can compromise model safety compliance and result in harmful outputs. To evaluate VLM's ability to maintain robustness against adversarial input perturbations, we propose novel metric called \textbf{Retention Score}. Retention Score is multi-modal evaluation metric that includes Retention-I and Retention-T scores for quantifying jailbreak risks in visual and textual components of VLMs. Our process involves generating synthetic image-text pairs using conditional diffusion model. These pairs are then predicted for toxicity score by VLM alongside toxicity judgment classifier. By calculating margin in toxicity scores, we can quantify robustness of VLM in attack-agnostic manner. Our work has four main contributions. First, we prove that Retention Score can serve as certified robustness metric. Second, we demonstrate that most VLMs with visual components are less robust against jailbreak attacks than corresponding plain VLMs. Additionally, we evaluate black-box VLM APIs and find that security settings in Google Gemini significantly affect score and robustness. Moreover, robustness of GPT4V is similar to medium settings of Gemini. Finally, our approach offers time-efficient alternative to existing adversarial attack methods and provides consistent model robustness rankings when evaluated on VLMs including MiniGPT-4, InstructBLIP, and LLaVA.
</p>
</div>
<!-- References -->
<div class="content">
<p>
<sup>1</sup> Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., & Hein, M. (2021). RobustBench: a standardized adversarial robustness benchmark. In <i>Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)</i>. <a href="https://openreview.net/forum?id=SSKZPJCt7B" target="_blank">https://openreview.net/forum?id=SSKZPJCt7B</a>
</p>
</div>
</div>
</div>
<!--/ Abstract. -->
</div>
</section>
<!-- Overview -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">Method Overview of GREAT Score</h2>
<div class="columns is-centered">
<div class="column container-centered">
<img src="./static/images/method.png" alt="Method Overview of Retention Score"/>
<p><strong>Flow chart of calculating Retention-Image and Retention-Text scores for VLMs. Given some evaluation samples, we first use diffusion generators to create semantically similar synthetic samples. Then, we pass the generated samples into a VLM to get responses and further use a toxicity judgment model (e.g., Perspective API \textsuperscript{1} or an LLM like Llama-70B (Touvron et al. 2023)) for toxicity level predictions. Finally, we use these statistics to compute the Retention Score as detailed in Section 3.2.</p>
</div>
</div>
</div>
</section>
<!-- Overview -->
<!-- Robustness Certificate Definition -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">Establishing the Retention Score Framework</h2>
<div class="columns is-centered">
<div class="column container formula">
<p>
Revisiting concepts introduced earlier, minimal perturbations for Image-Text pair in context of VLMs were established. We proposed that greater values of minimal perturbations correlate with enhanced local robustness of model M for pair (I, T). Consequently, estimating lower bounds for these minimal perturbations provides measure of VLMs' robustness. To quantify robustness, we introduce Retention Score, which aims to provide assessment of VLM resilience against input perturbations. Higher Retention Scores signify model's inherent robustness, indicative of safeguards against adversarial toxicity manipulation. Retention Score is multimodal measure capable of assessing conditional robustness of VLMs across visual, textual domains, further divided into Retention-Image (Retention-I) and Retention-Text (Retention-T) scores.
</p>
</div>
</div>
<div class="columns is-centered">
<div class="column container-centered">
<div id="adaptive-loss-formula" class="container">
<div id="adaptive-loss-formula-list" class="row align-items-center formula-list">
<a href=".retention-image" class="selected">Retention-Image Score</a>
<a href=".retention-text">Retention-Text Score</a>
<div style="clear: both"></div>
</div>
<div class="row align-items-center adaptive-loss-formula-content">
<span class="formula retention-image formula-content">
$$
\begin{align}
g_I(M,G_I(z|I), T) &= \sqrt{\frac{\pi}{2}} \cdot \{ M_{nt}(G_I(z|I), T) - M_t(G_I(z|I), T) \}^{+} \\
R_I(M, I, \mathbb{X}) &= \frac{1}{m \cdot n} \sum_{j=1}^m \sum_{i=1}^n g_I(M,G_I(z_i|I), T_j))
\end{align}
$$
</span>
<span class="formula retention-text formula-content" style="display: none;">
$$
\begin{align}
g_T(M,I, s(G_T(z|T))) &= \sqrt{\frac{\pi}{2}} \cdot \{ M_{nt}(I, \psi(s(G_T(z|T)))) - M_t(I, \psi(s(G_T(z|T)))) \}^{+} \\
R_T(M, I,\mathbb{X}) &= \frac{1}{m \cdot n} \sum_{j=1}^m \sum_{i=1}^n g_T(M,I, \psi(s(G_T(z_i|T_j))))
\end{align}
$$
</span>
</div>
</div>
</div>
</div>
<div class="columns is-centered">
<div class="column container adaptive-loss-formula-content">
<p class="formula retention-image formula-content">
where G_I(z|I) is a continuous diffusion-based image generation model that synthesizes semantically similar images to I, given a zero-mean isotropic Gaussian-distributed input z. The local score function g_I evaluates the non-toxicity of the generated image associated with the given prompt T.
</p>
<p class="formula retention-text formula-content" style="display: none;">
where G_T(z|T) refers to a text generator founded on paraphrasing diffusion techniques, s and ψ represent semantic encoder and decoder that translate discrete textual information into continuous vectorial representation and vice versa.
</p>
</div>
</div>
</div>
</section>
<!-- Results -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">Retention Image Score Results</h2>
<div class="columns is-centered">
<div class="column container-centered">
<table class="tg" border="1" style="width:100%;">
<caption><strong>Table 1.</strong> Jailbreak risk evaluation of VLMs to image attacks. This table presents a comparison among three VLMs — MiniGPT-4, LLaVA, and InstructBLIP — with regards to their Retention Scores (Retention-I), and Attack Success Rates (ASR, calculated as the percentage of outputs displaying toxic attributes).</caption>
<thead>
<tr>
<th class="tg-amwm"></th>
<th class="tg-baqh" colspan="2">MiniGPT-4</th>
<th class="tg-baqh" colspan="2">LLaVA</th>
<th class="tg-baqh" colspan="2">InstructBLIP</th>
</tr>
<tr>
<th class="tg-amwm"></th>
<th class="tg-baqh">Retention-I</th>
<th class="tg-baqh">ASR (%)</th>
<th class="tg-baqh">Retention-I</th>
<th class="tg-baqh">ASR (%)</th>
<th class="tg-baqh">Retention-I</th>
<th class="tg-baqh">ASR (%)</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-baqh">Young</td>
<td class="tg-baqh">0.6121</td>
<td class="tg-baqh">40.93</td>
<td class="tg-baqh">0.2866</td>
<td class="tg-baqh">58.86</td>
<td class="tg-baqh">0.5043</td>
<td class="tg-baqh">49.72</td>
</tr>
<tr>
<td class="tg-baqh">Old</td>
<td class="tg-baqh">0.5917</td>
<td class="tg-baqh">43.27</td>
<td class="tg-baqh">0.2636</td>
<td class="tg-baqh">64.71</td>
<td class="tg-baqh">0.5650</td>
<td class="tg-baqh">47.76</td>
</tr>
<tr>
<td class="tg-baqh">Woman</td>
<td class="tg-baqh">0.5621</td>
<td class="tg-baqh">42.12</td>
<td class="tg-baqh">0.2261</td>
<td class="tg-baqh">57.70</td>
<td class="tg-baqh">0.4861</td>
<td class="tg-baqh">52.00</td>
</tr>
<tr>
<td class="tg-baqh">Man</td>
<td class="tg-baqh">0.5438</td>
<td class="tg-baqh">42.63</td>
<td class="tg-baqh">0.1971</td>
<td class="tg-baqh">52.16</td>
<td class="tg-baqh">0.4966</td>
<td class="tg-baqh">50.36</td>
</tr>
<tr>
<td class="tg-baqh">Average</td>
<td class="tg-baqh">0.5774</td>
<td class="tg-baqh">42.49</td>
<td class="tg-baqh">0.2434</td>
<td class="tg-baqh">58.36</td>
<td class="tg-baqh">0.5130</td>
<td class="tg-baqh">49.96</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</section>
<!-- Results -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">Retention Text Score Results</h2>
<div class="columns is-centered">
<div class="column container-centered">
<table class="tg" border="1" style="width:100%;">
<caption><strong>Table 2.</strong> Jailbreak risk evaluation of VLMs to text attacks. This table presents a comparison among three VLMs — MiniGPT-4, LLaVA, and InstructBLIP — with regards to their Retention Scores (Retention-T), Attack Success Rates.</caption>
<thead>
<tr>
<th class="tg-amwm">VLM</th>
<th class="tg-baqh">Retention-T</th>
<th class="tg-baqh">Attack Success Rate</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-baqh">MiniGPT-4</td>
<td class="tg-baqh">0.2073</td>
<td class="tg-baqh">46.1%</td>
</tr>
<tr>
<td class="tg-baqh">LLaVA</td>
<td class="tg-baqh">0.342</td>
<td class="tg-baqh">9.4%</td>
</tr>
<tr>
<td class="tg-baqh">InstructBLIP</td>
<td class="tg-baqh">0.164</td>
<td class="tg-baqh">84.5%</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</section>
<!-- Model Ranking Comparison Section -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">API Model Analysis</h2>
<div class="columns is-centered">
<div class="column is-full-width">
<div class="content has-text-justified">
<p>
Assessing the robustness of black-box VLMs is of paramount importance, particularly since these models are commonly deployed as APIs, restricting users and auditors to inferential interactions. This constraint not only makes adversarial attacks challenging but also underscores the necessity for robust evaluation methods that do not depend on internal model access. In this context, our research deploys the Retention-I score to examine the resilience of APIs against synthetically produced facial images with concealed attributes, which are typically employed in model inferences.
</p>
<p>
Our evaluation methodology was applied to two prominent online vision language APIs: GPT-4V and Gemini Pro Vision. Noteworthy is that for Gemini Pro Vision, the API provides settings to adjust the model's threshold for blocking harmful content, with options ranging from blocking none to most (none, few, some, and most). We tested this feature by running identical prompts and images across these settings, leading to an evaluation of five model configurations.
</p>
<p>
The assessment centered around the Retention-I score, using a balanced set of synthetic faces that included young, old, male, and female groups. These images were generated using the state-of-the-art Stable Diffusion model, with each group contributing 100 images. A unique aspect of Google's Gemini is its error messaging system, which, in lieu of producing toxic outputs, provides rationales for prompt blocking. In our study, such blocks were interpreted as a zero toxicity score, aligning with the model's safeguarding strategy.
</p>
<table class="table is-bordered is-striped is-narrow is-hoverable is-fullwidth">
<caption><strong>Table 3.</strong> Retention-I analysis of VLM APIs. Each group consists of 100 images with 20 continuation prompts.</caption>
<thead>
<tr>
<th></th>
<th>Young</th>
<th>Old</th>
<th>Woman</th>
<th>Man</th>
<th>Average</th>
</tr>
</thead>
<tbody>
<tr>
<td>GPT-4v</td>
<td>1.2043</td>
<td>1.2077</td>
<td>1.2067</td>
<td>1.2052</td>
<td>1.2059</td>
</tr>
<tr>
<td>Gemini-None</td>
<td>0.3025</td>
<td>0.2432</td>
<td>0.2300</td>
<td>0.2126</td>
<td>0.2471</td>
</tr>
<tr>
<td>Gemini-Few</td>
<td>1.1955</td>
<td>1.1806</td>
<td>1.1972</td>
<td>1.1987</td>
<td>1.1930</td>
</tr>
<tr>
<td>Gemini-Some</td>
<td>1.2322</td>
<td>1.2486</td>
<td>1.2325</td>
<td>1.2382</td>
<td>1.2379</td>
</tr>
<tr>
<td>Gemini-Most</td>
<td>1.2449</td>
<td>1.2494</td>
<td>1.2388</td>
<td>1.2479</td>
<td>1.2453</td>
</tr>
</tbody>
</table>
<p>
Our results in Table 3 reveal intriguing variations across different APIs. For instance, Gemini-None exhibited notable performance contrasts when comparing Old versus Young cohorts. Other models showcased more uniform robustness levels across demographic groups. Also, Our analysis positions the robustness of GPT-4V somewhere between the some and most safety settings of Gemini. This correlation not only validates the efficacy of Gemini's protective configurations but also underscores the impact of safety thresholds on toxicity recognition, as quantified by our scoring method.
</p>
<p>
This robustness evaluation illustrates that Retention-I is a pivotal tool for analyzing group-level resilience in models with restricted access, enabling discreet and efficacious scrutiny of their defenses.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- Model Ranking Comparison Section -->
<!-- GREAT Score vs CW Attack Comparison Section -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">GREAT Score vs CW Attack Comparison</h2>
<div class="columns is-centered">
<div class="column container-centered">
<div>
<img src="./static/images/new_figure_2_2.png"
class="method_overview"
alt="Comparison of local GREAT Score and CW attack"/>
<p>
<strong>Figure 2.</strong> Comparison of local GREAT Score and CW attack in L<sub>2</sub> perturbation on CIFAR-10 with Rebuffi_extra model.
The x-axis is the image id. The result shows the local GREAT Score is indeed a lower bound of the perturbation level found by CW attack.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- GREAT Score vs CW Attack Comparison Section -->
<!-- Run-time Analysis Section -->
<section class="section">
<div class="container is-max-desktop">
<h2 class="title is-3">Run-time Analysis</h2>
<div class="columns is-centered">
<div class="column container-centered">
<div>
<img src="./static/images/plot1.png"
class="method_overview"
alt="Run-time improvement comparison"/>
<p>
<strong>Figure 4.</strong> Run-time improvement (Retention Score over Visual and Text attacks).
</p>
<div class="content has-text-justified">
<p>
Figure 4 compares the run-time efficiency of Retention Score over adversarial attacks in [1] and [2].
We show the improvement ratio of their average per-sample run-time (wall clock time of Retention Score/Adversarial Attack is reported in Appendix)
and observe around 2-30 times improvement, validating the computational efficiency of Retention Score.
</p>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Run-time Analysis Section -->
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@article{li2024greatscore,
title = {GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models},
author = {Zaitang, Li and Pin-Yu, Chen and Tsung-Yi, Ho},
journal = {NeurIPS},
year = {2024},
}</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<!-- <div class="content has-text-centered">
<a class="icon-link" target="_blank"
href="./static/videos/nerfies_paper.pdf">
<i class="fas fa-file-pdf"></i>
</a>
<a class="icon-link" href="https://github.com/keunhong" target="_blank" class="external-link" disabled>
<i class="fab fa-github"></i>
</a>
</div> -->
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This website is licensed under a <a rel="license" target="_blank"
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
<p>
This means you are free to borrow the <a target="_blank"
href="https://github.com/nerfies/nerfies.github.io">source code</a> of this website,
we just ask that you link back to this page in the footer.
Please remember to remove the analytics code included in the header of the website which
you do not want on your website.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>