---

🔍 Overview

How stealthy is an adversarial perturbation to the human eye?

Widely used metrics such as (L_p) norms, PSNR, and SSIM were not designed specifically for subtle adversarial perturbations and can disagree with human visual perception. This work provides a systematic study of pixel-level adversarial stealthiness assessment from both subjective and objective perspectives.

We introduce:

• A new subjective assessment criterion that covers both hardly perceptible and clearly perceptible perturbations.
• APSD, a large-scale Adversarial Perturbation Stealthiness Assessment Dataset with human opinion scores.
• A2SM, an Attention-based Adversarial Stealthiness Assessment Model that mimics human scoring habits for objective stealthiness prediction.

---

✨ Highlights

👁️ Human-centric assessment	A two-step protocol provides fine-grained subjective scores across the full stealthiness spectrum.
🗂️ 10,586 adversarial examples	APSD contains 10,586 clean–adversarial pairs generated from 400 reference images.
⚔️ 12 attacks / 5 attack families	Gradient-, optimization-, generative model-, pixel-, and block-based attacks are covered.
👥 60 human observers	Every adversarial example is evaluated by 15 observers; Mean Opinion Score (MOS) is used as the label.
🧠 Human-inspired objective model	A2SM combines global/local features with self-, cross-, and spatial-attention mechanisms.
📈 High HVS consistency	A2SM achieves 0.984 PLCC / 0.978 SROCC on APSD.

---

👁️ Subjective Stealthiness Assessment

We propose a two-step subjective assessment protocol.

Step 1 — Rough assessment

Observers first inspect a clean/adversarial image pair for a fixed period and classify the perturbation as:

• Strong stealthiness — the difference is hardly perceptible.
• Weak stealthiness — the difference is readily perceptible.

Step 2 — Fine-grained assessment

For strong stealthiness, observers use progressively amplified Scaled Grayscale Difference Maps (SGDM) to distinguish subtle pixel-level differences.

For weak stealthiness, observers assess the degree of visible degradation / annoyance.

The resulting scores are aggregated into a Mean Opinion Score (MOS) in the range ([0,1]), where a larger value means better stealthiness.

Rating software.

---

🗃️ APSD Dataset

We construct the Adversarial Perturbation Stealthiness Assessment Dataset (APSD), a dedicated benchmark for human-aligned adversarial stealthiness evaluation.

Dataset statistics

Property	APSD
Reference images	400
Clean–adversarial pairs	10,586
Image categories	200+
Attack methods	12
Attack families	5
Human observers	60
Ratings per adversarial example	15
Label	MOS ∈ [0, 1]
Image size used for subjective study	512 × 512

Covered attack families

• Gradient-based: FGSM, MIFGSM, PGD, NES
• Optimization-based: CW
• Generative model-based: CDP, AdvGAN, GAP, GUAP
• Pixel-based: SimBA, Pixel
• Block-based: Square

APSD contains perturbations spanning a broad range of perceptual stealthiness, allowing models to learn both subtle and visually obvious adversarial distortions.

Distribution of adversarial examples generated by different attacks in APSD. The suffix of each attack indicates the corresponding parameter.

---

📥 Dataset Download

The APSD dataset is publicly available through the project repository:

➡️ Download APSD

The released dataset should contain the clean images, adversarial images, and their corresponding human subjective scores.

A recommended local organization is:

APSD/
├── REF-299 (Reference)/
│   ├── xxx.png
│   └── ...
└── AE_targetmodel0 (Adversarial example)/
    ├── 0 FGSM/
    ├── 1 MIFGSM/
    ├── 2 PGD/
    ├── 3 NES/
    ├── 4 CW/
    ├── 5C DP/
    ├── 6 AdvGAN/
    ├── 7 GAP/
    ├── 8 GUAP/
    ├── 9 SimBA/
    ├── 10 Pixel/
    └── 11 Square/
    └── score.csv

Note: the exact released directory/file names should follow the actual dataset package. Update the tree above if your final release uses a different structure.

---

📝 Citation

If you find APSD or A2SM useful in your research, please cite:

@article{liu2024stealthiness,
  title   = {Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective},
  author  = {Liu, Hangcheng and Zhou, Yuan and Yang, Ying and Zhao, Qingchuan and Zhang, Tianwei and Xiang, Tao},
  journal = {IEEE Transactions on Information Forensics and Security},
  volume  = {20},
  pages   = {898--913},
  year    = {2024},
  doi     = {10.1109/TIFS.2024.3520016}
}

---

📬 Contact

For questions about the paper, dataset, or code, please open a GitHub issue or contact the authors.

---

License

Please add the license that applies to the released code and dataset before public release.
If the dataset contains images inherited from ImageNet, COCO, or VOC2012, their original licenses/terms should also be respected.