VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

Guanyu Zhou¹, Yida Yin¹, Wenhao Chai¹, Shengbang Tong², Xingyu Fu¹, Zhuang Liu¹

¹Princeton University, ²New York University

Use VisionFoundry to generate your own Synthetic Images Dataset with just one keyword!

pip install ms-swift -U

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .

The VisionFoundry pipeline is implemented in data_engine/vision_foundry.py. It generates synthetic VQA data and images using OpenAI and Gemini APIs.

We run the pipeline inside the ms-swift environment and add the following packages:

pip install openai google-genai pillow requests tqdm numpy

Copy and edit data_engine/config.example.json (or provide your own JSON via --api_config). The default configuration uses the official OpenAI and Gemini APIs. You can set a custom OpenAI-compatible base URL by editing the base_url field in the config if needed.

Required environment variables:

• OPENAI_API_KEY
• GEMINI_API_KEY

Example (single-image mode):

python data_engine/vision_foundry.py \
  --task "<YOUR_TASK_DESCRIPTION>" \
  --num 200 \
  --mode single \
  --output_dir ./output \
  --annotation_output annotations.json \
  --prompts_output prompts.jsonl \
  --statements_output statements.jsonl \
  --pool_output pool.json \
  --api_config data_engine/config.example.json

Example (multi-image mode, beta):

python data_engine/vision_foundry.py \
  --config /path/to/task_config.json \
  --num 200 \
  --mode multi \
  --num_images 3 \
  --multi_image_form story_chain \
  --output_dir ./output_multi \
  --api_config data_engine/config.example.json

The pipeline produces:

• annotations.json: Training annotations in multi-image or single-image format
• prompts.jsonl: Prompts used for image generation
• statements.jsonl: Verification statements
• pool.json: The final object/attribute/scene/style pool

We provide both supervised fine-tuning and reinforcement-learning recipes.

The three ms-swift scripts under train_scripts/ are:

• train_scripts/train_qwen.sh (for Qwen2.5-VL-3B-Instruct)
• train_scripts/train_mimo.sh (for MiMo-VL-7B-SFT)
• train_scripts/train_llama.sh (for Llama-3.2-11B-Vision-Instruct)

Each script contains two runnable templates:

• Local single-node (e.g., a single 8-GPU machine)
• Slurm cluster submission

Fill in the placeholders for model path, dataset path, output directory, and logging directory, then uncomment the section you want to use.

After downloading the dataset from HuggingFace:

huggingface-cli download zlab-princeton/VisionFoundry-10K

Then run python restore_images_from_parquet.py to rebuild the image folder structure from images.parquet. Each record in annotations.json contains a messages list and an images list following the ms-swift format.

We follow the ms-swift multimodal SFT JSON format. Each item contains a messages list and an images list. A minimal single-image example:

{
  "messages": [
    {"role": "user", "content": "<image>\nWhat is the color of the car?"},
    {"role": "assistant", "content": "red"}
  ],
  "images": ["/abs/or/rel/path/to/image.png"],
  "qid": 1
}

Multi-image examples use <images> in the user message and provide multiple image paths:

{
  "messages": [
    {"role": "user", "content": "<images>\nWhat changed across the images?"},
    {"role": "assistant", "content": "the cup moved to the left"}
  ],
  "images": ["img_0.png", "img_1.png", "img_2.png"],
  "qid": 1
}

Our pipeline outputs annotations.json in this format. You can also prepare your own dataset by modifying the annotations.json accordingly.

To launch training, fill in the placeholders in one of the scripts and run:

bash train_scripts/train_qwen.sh

The rl/ directory provides full-parameter GRPO recipes for the same three backbones:

• rl/prepare_data.py: converts the SFT annotations to verl parquet
• rl/text_judge_reward.py: binary semantic reward from an OpenAI-compatible Qwen2.5-3B judge
• rl/batch_text_judge_reward.py: concurrent verl batch-reward adapter
• rl/train_qwen.sh, rl/train_mimo.sh, and rl/train_llama.sh: portable 8-GPU recipes

The reward judge receives only the question, reference answer, and sampled model answer. It is configured entirely through environment variables; no API credentials or machine-specific paths are stored in this repository. See rl/README.md for data conversion, environment setup, released hyperparameters, resume behavior, and checkpoint conversion.

We do not include custom evaluation code in this repo. Use VLMEvalKit's built-in benchmark support:

1. Install VLMEvalKit (see above).
2. Configure your model in vlmeval/config.py and ensure the model weights are accessible.
3. Run evaluation, for example:

python run.py --data <BENCH1> <BENCH2> --model <MODEL_NAME> --verbose

For distributed inference, use torchrun --nproc-per-node=<N> run.py .... See the VLMEvalKit quickstart for more details.

When evaluating with MiMo-VL-7B-SFT, please append /no_think to the prompt to disable chain-of-thought reasoning for a fair comparison. See the official repo for details: XiaomiMiMo/MiMo-VL.

This project is released under the Apache-2.0 license. See LICENSE for details.

If you use this work, please cite:

@article{zhou2026visionfoundry,
  title={VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images},
  author={Zhou, Guanyu and Yin, Yida and Chai, Wenhao and Tong, Shengbang and Fu, Xingyu and Liu, Zhuang},
  journal={arXiv preprint arXiv:2604.09531},
  year={2026}
}

This project builds on several strong open-source foundations:

• ms-swift for SFT training infrastructure
• verl for GRPO training infrastructure
• VLMEvalKit for multimodal evaluation