Official research implementation of the ACM Multimedia 2025 paper

Paper · Quick Start · Reproduce Table 4 · Reproducibility Report

Matteo Trippodo · Federico Becattini · Lorenzo Seidenari

Text-guided diffusion models can make convincing edits while preserving much of the source image. This repository implements Attention Immunization, an adversarial protection method that adds a visually subtle, bounded perturbation to an image before it is published. When a diffusion editor later processes the protected image, the perturbation disrupts the model's cross-attention behavior and reduces edit fidelity.

The attack does not need to know the future edit instruction. It uses a short caption of the source image as a surrogate concept and maximizes the disagreement between clean and perturbed cross-attention maps.

• Installable Python package and command-line interface.
• One-command qualitative demo and paper-configuration attack.
• End-to-end reproduction path from LLaVA captions to attacks, SDEdit outputs, and metrics.
• Importable LLaVA caption metrics for custom paired datasets.
• Importable MaskFormer segmentation metrics for custom image pairs.
• Recovered 48-sample Attention Attack + SDEdit evaluation from Table 4.
• Pinned environment and explicit record of reproducible and missing artifacts.

Metrics recomputed directly from the 48 saved clean/attacked edit pairs reproduce the paper values:

The complete attack-to-edit pipeline was also regenerated from the source images. Caption similarity remained close to the paper at 0.6928 ± 0.1368. Pixel and segmentation metrics exhibit more drift because the historical source captions and exact model revisions were not preserved. Full values and limitations are recorded in REPRODUCIBILITY.md and results/table4_sdedit_attention_full_pipeline.json.

• Linux
• Python 3.10–3.12
• NVIDIA GPU with CUDA support
• 16 GB VRAM minimum; approximately 24 GB recommended
• Internet access on the first run to download Hugging Face checkpoints

The tested environment uses Python 3.11, PyTorch 2.5.1, Diffusers 0.32.2, and Transformers 4.47.1.

# From a clone of this repository:
cd attention-immunization
conda env create -f environment.yml
conda activate attention-immunization

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Install evaluation and development dependencies when needed:

python -m pip install -e '.[evaluation]'
python -m pip install -e '.[dev]'

Verify the installation:

attention-immunize --help
pytest -q

./scripts/run_demo.sh

The demo runs a shortened 10-step attack and writes the following files to outputs/demo/:

./scripts/run_paper_attack.sh \
  path/to/image.jpg \
  "a short caption of the source image" \
  "the future edit prompt" \
  outputs/my_run

For example:

./scripts/run_paper_attack.sh \
  /path/to/cat.jpg \
  "an orange cat on a white background" \
  "a dog" \
  outputs/cat

The source caption is the attack's surrogate concept. The future edit prompt is never exposed to the attack; it is used only afterward to produce the qualitative comparison.

Equivalent Python CLI:

attention-immunize \
  --image path/to/image.jpg \
  --caption "a short caption of the source image" \
  --edit-prompt "the future edit prompt" \
  --model runwayml/stable-diffusion-v1-5 \
  --attack-steps 200 \
  --diffusion-steps 10 \
  --epsilon 0.06274509803921569 \
  --alpha 0.00784313725490196 \
  --attention-resolutions 16+32 \
  --seed 42 \
  --output-dir outputs/paper_attack

For a fast installation smoke test without preview edits:

attention-immunize \
  --image assets/demo_input.jpg \
  --caption "a cat wearing a floral shirt" \
  --attack-steps 1 \
  --diffusion-steps 2 \
  --skip-preview \
  --output-dir outputs/smoke

The implementation follows the paper: LLaVA captions both images, Sentence-BERT computes caption similarity, and CLIP ViT-L/14 scores each image against the clean-image caption.

from attention_immunization import CaptionMetrics

metrics = CaptionMetrics(device="cuda")
result = metrics.evaluate_pair(
    "clean_edit.png",
    "attacked_edit.png",
)

print(result.caption_similarity)
print(result.clean_image_clean_caption_clip)
print(result.attacked_image_clean_caption_clip)
print(result.clean_caption)
print(result.attacked_caption)

Create one CaptionMetrics instance and reuse it for a dataset so LLaVA is loaded only once. Existing captions can be passed as clean_caption= and attacked_caption= to skip caption generation.

For CSV datasets, use a manifest with id, clean_image, and attacked_image columns:

env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/compute_caption_metrics.py \
  --manifest pairs.csv \
  --output outputs/caption_metrics.json \
  --device cuda

The output is checkpointed after every pair and resumes by sample ID.

For a single custom pair:

from attention_immunization import compute_segmentation_metrics

result = compute_segmentation_metrics(
    "clean_edit.png",
    "attacked_edit.png",
    device="cuda",
)

print(result.optimistic_iou)
print(result.pessimistic_iou)
print(result.common_labels)
print(result.all_labels)

For multiple pairs, reuse the model:

from attention_immunization import SegmentationMetrics

metrics = SegmentationMetrics(device="cuda")

for clean_path, attacked_path in image_pairs:
    result = metrics.evaluate_pair(clean_path, attacked_path)
    print(result.to_dict())

The evaluator uses facebook/maskformer-swin-base-coco with semantic post-processing:

• Optimistic IoU averages over labels predicted in both images.
• Pessimistic IoU averages over labels predicted in either image, assigning zero IoU to labels missing from one image.

Images in a pair must have identical dimensions because class masks are compared pixel by pixel.

This is the most exact reproduction path for the Attention Attack + SDEdit row:

env -u PYTHONPATH CUBLAS_WORKSPACE_CONFIG=:4096:8 \
  python scripts/recompute_table4_sdedit_attention.py \
  --thesis-root /path/to/Thesis \
  --output outputs/table4_sdedit_attention.json \
  --device cuda:0

It recomputes CLIPScore, LPIPS, PSNR, and SSIM for all 48 saved clean/attacked pairs. Caption metrics can be regenerated separately:

env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/compute_caption_metrics.py \
  --table4-thesis-root /path/to/Thesis \
  --output outputs/table4_sdedit_attention_caption_metrics.json \
  --device cuda

The full runner starts from the 24 unique source images used by the 48 evaluation rows. Each stage resumes from existing output files.

THESIS_ROOT=/path/to/Thesis

# 1. Generate LLaVA source captions.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/run_table4_full_pipeline.py captions \
  --thesis-root "$THESIS_ROOT" \
  --device cuda:0

# 2. Craft the 24 adversarial source images.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 CUBLAS_WORKSPACE_CONFIG=:4096:8 \
  python scripts/run_table4_full_pipeline.py attacks \
  --thesis-root "$THESIS_ROOT" \
  --device cuda:0

# 3. Generate attacked and clean SDEdit outputs for all 48 prompts.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 CUBLAS_WORKSPACE_CONFIG=:4096:8 \
  python scripts/run_table4_full_pipeline.py edits \
  --thesis-root "$THESIS_ROOT" \
  --device cuda:0

# 4. Compute CLIPScore, LPIPS, PSNR, and SSIM.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/evaluate_table4_full_pipeline.py \
  --thesis-root "$THESIS_ROOT" \
  --device cuda:0

# 5. Compute LLaVA caption metrics.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/compute_caption_metrics.py \
  --manifest outputs/full_pipeline/edit_pairs.csv \
  --output outputs/full_pipeline/caption_metrics.json \
  --device cuda

# 6. Compute semantic MaskFormer IoU.
env -u PYTHONPATH CUDA_VISIBLE_DEVICES=0 \
  python scripts/compute_table4_iou.py \
  --manifest outputs/full_pipeline/edit_pairs.csv \
  --output outputs/full_pipeline/iou_metrics.json \
  --device cuda

The attack stage can be split across two GPUs with --shard-index 0 --shard-count 2 and --shard-index 1 --shard-count 2. Do not shard the edit stage: it intentionally reuses a single seeded generator in the paper's attacked-then-clean row order.

Historical surrogate captions were not saved, so stage 1 regenerates them with LLaVA 1.5 7B and the recovered prompt. This pipeline is operationally complete but not pixel-identical to the original experiment.

The paper uses a manually filtered TEdBench++ split containing 40 images and 127 edit prompts. The experiment manifest is provided at data/tedbench_plusplus.csv.

Obtain the images from the upstream TEdBench++ distribution and arrange them as:

TEdBench_plusplus/
├── originals/
└── tedbench_plusplus.csv

The source images are not redistributed here because their redistribution terms were not established.

attention-immunization/
├── assets/                     demo input
├── data/                       filtered TEdBench++ manifest
├── results/                    compact reproduced metrics
├── scripts/                    demos and experiment entry points
├── src/attention_immunization/ installable Python package
├── tests/                      lightweight unit tests
├── environment.yml             Conda environment
├── pyproject.toml              package and pinned dependencies
├── README.md
└── REPRODUCIBILITY.md

Large checkpoints, generated outputs, the original thesis workspace, nested third-party repositories, and private experiment configuration are intentionally excluded. No API keys are required.

• The public single-image CLI center-crops and resizes inputs to 512×512.
• Experiment parameters and attack losses are recorded in run.json.
• Clean and protected preview edits share a seed.
• Dependencies are pinned in pyproject.toml.
• Numerical identity across CUDA architectures is not guaranteed.
• The recovered implementation applies ε = 16/255 after mapping pixels to [-1, 1]. In saved 8-bit space this produces approximately 8 levels of change, or 9 after resize and quantization, rather than 16.
• The historical LLaVA caption strings and immutable model revision hashes were not preserved.

See REPRODUCIBILITY.md for verified runs, exact results, and remaining limitations.

python -m pip install -e '.[dev]'
ruff check src tests scripts
pytest -q

Generated artifacts and model weights are ignored by Git. The recovered paper implementation is retained in attack_legacy.py with portability and safety changes; the public API isolates it from the original thesis monolith.

If this work is useful in your research, please cite:

@inproceedings{trippodo2025immunizing,
  title     = {Immunizing Images from Text to Image Editing via Adversarial Cross-Attention},
  author    = {Trippodo, Matteo and Becattini, Federico and Seidenari, Lorenzo},
  booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
  year      = {2025},
  doi       = {10.1145/3746027.3755755}
}

No software license was present in the recovered student repository. The copyright holders must select and add a LICENSE file before public release. Until then, the absence of a license means the code is not licensed for copying, modification, or redistribution.