🎬 SOVA: Strict Outcome-Conditioned Virtual Advantages for Video Reasoning

Official implementation of SOVA. SOVA calibrates group-relative advantages for fully successful and all-zero-accuracy response groups using fixed virtual rewards and residual interpolation.

• Motivation
• Method overview
• Results
• Setup
• Training
• Evaluation
• Implementation checks
• Repository layout
• Acknowledgements

Fully correct, well-formatted groups have zero original advantages. All-incorrect groups can retain format-based differences but lack correctness contrast.

SOVA uses accuracy and format rewards to select two branches:

All other groups keep their original advantages exactly. In negative groups with mixed formatting, individual advantages can remain positive; the guarantee concerns the group mean.

The formal experiments were run on H800 servers. The SOVA manuscript reports the following results with Qwen2.5-VL-7B, 1,000 CLEVRER training queries, 500 updates, and group size 8. Scores are accuracy (%); Video-MME excludes subtitles.

The manuscript reports CLEVRER strict accuracy of 52.2 ± 0.36% (mean ± sample standard deviation across five seeds), a gain of 1.8 percentage points over the TW-GRPO control. CLEVRER aggregation weights the 3,945 single-answer and 5,293 multiple-answer questions by their counts; category strict accuracies are 62.2053% and 44.7194%, respectively, giving 52.1866% overall, rounded to 52.2%.

This compact implementation targets the main Qwen2.5-VL-7B protocol and the Qwen2.5-VL-3B backbone. The paper's InternVL3 implementation and unrelated baseline methods are outside this release. The table reports manuscript results, not a rerun of this source release.

Use Linux, Python 3.10/3.11, and two NVIDIA H800 80 GB GPUs for the reference training run. Evaluation uses one GPU; the reference batch size is 16.

git clone https://github.com/just-a-go/SOVA.git
cd SOVA
conda create -n sova python=3.10 -y
conda activate sova
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124
pip install -e ".[train,test]"
pip install flash-attn==2.7.4.post1 --no-build-isolation

Set CUDA_HOME to your CUDA toolkit installation if it is not already configured. CUDA 12.4 and a C++ build toolchain are needed to build the training extensions.

huggingface-cli download Qwen/Qwen2.5-VL-7B-Instruct \
  --local-dir models/Qwen2.5-VL-7B-Instruct

Download the videos and annotations from their original sources:

Each input file is a JSON array. Use absolute video paths. For CLEVRER, place the question and labeled choices in problem; comma-separated solution letters represent all correct choices. For the other benchmarks, provide labeled choices in options.

[
  {
    "video": "/absolute/path/to/video.mp4",
    "problem": "Your question.\nA. First choice\nB. Second choice",
    "solution": "<answer>A,B</answer>"
  }
]

The example illustrates the schema; it is not a benchmark sample. Supply at least 1,000 training queries. The loader selects one fixed 1,000-query subset with seed 42; --seed controls the training shuffle and rollout sampling. Keep the annotation order unchanged when comparing runs.

For CLEVRER, convert the official Questions and Answers file after extracting the videos:

python -m sova.prepare_clevrer \
  --questions /absolute/path/to/questions/train.json \
  --videos /absolute/path/to/train_videos \
  --output data/clevrer_train.json

Repeat for validation. The converter retains counterfactual questions with nonempty correct-answer sets, as assumed in the paper, and resolves nested video directories. For the other benchmarks, adapt their annotations to the schema above with options containing labeled choices.

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 -m sova.train \
  --model models/Qwen2.5-VL-7B-Instruct \
  --data /absolute/path/to/clevrer_train.json \
  --output outputs/sova

The defaults implement the reference configuration:

One fresh response group is used once per update. Token weighting retains the original TW-GRPO rule and its gradient path; SOVA changes only the response advantages. Checkpointed token chunks avoid retaining a second full-vocabulary log-probability tensor. No reward-model, reference-policy, or rollout-reuse branch is needed for the paper's beta=0 setting.

Append these options to the training command and choose a separate output directory:

Model and processor files are saved together in the output directory. Existing nonempty output directories are rejected to keep runs separate.

CUDA_VISIBLE_DEVICES=0 python -m sova.evaluate \
  --model outputs/sova \
  --data /absolute/path/to/clevrer_val.json \
  --benchmark clevrer \
  --output outputs/sova/clevrer_results.json

Use nextgqa, mmvu, mvbench, tempcompass, or videomme with the corresponding annotation file. The evaluator uses 16 frames, a 256 × 28 × 28 pixel budget per frame, temperature 0.01, top-p 0.001, and batch size 16. It reports strict answer-set accuracy, partial credit, and format compliance, and saves each response. A lower --batch-size can reduce memory use but differs from the reported evaluation protocol.

python -m pytest tests -q

The tests cover strict gates, sample statistics, mixed-format failures, exact bypass, group-size changes, reward parsing, EOS masks, data conversion, and TW weight gradients. A tiny randomly initialized Qwen2.5-VL model also reads a synthetic video, generates responses, completes an optimizer update, and runs the evaluator on CPU. This test downloads no dataset or pretrained checkpoint; it does not replace a full H800 benchmark reproduction.

SOVA/
├── docs/figs/              # Paper figures
├── scripts/zero3_offload.json
├── sova/                  # Training and evaluation
├── tests/
├── pyproject.toml
└── README.md

We thank TW-GRPO, Open-R1-Video, Video-R1, VideoChat-R1, TRL, and Qwen2.5-VL for their open-source contributions.

This implementation builds on TW-GRPO and Hugging Face TRL, licensed under Apache-2.0. The token-weighting rule, format verifier, and single-use policy-ratio convention are adapted from TW-GRPO. Copyright 2025 The HuggingFace Team. All rights reserved. Qwen2.5-VL preprocessing utilities are installed as an external dependency.

Released under the Apache License 2.0.