MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data

MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data
Siarhei Sheludzko¹, Dhimitrios Duka², Bernt Schiele², Hilde Kuehne³˒⁴, Anna Kukleva²
¹University of Bonn   ²MPI for Informatics, SIC   ³Tuebingen AI Center / University of Tuebingen   ⁴MIT-IBM Watson AI Lab
WACV 2026

Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks, pulling positive pairs closer while pushing negatives apart. While the temperature parameter controls the strength of these forces, most approaches simply fix it during training or treat it as a constant hyperparameter.

We propose MM-TS (Multi-Modal Temperature and Margin Schedules), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. MM-TS dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample — samples from dense clusters are assigned a higher temperature to better preserve their semantic structure.

Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective.

We evaluate on four image- and video-language datasets — Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2 — and show that MM-TS improves performance and achieves new state-of-the-art results.

MM-TS combines two components:

1. Cosine Temperature Schedule — the base temperature follows a cosine schedule across training iterations, allowing the model to alternate between instance discrimination (low τ) and group-wise discrimination (high τ).

2. Individual Cluster Shifts (ICS) — text annotations are embedded and clustered (K-Means) prior to training. Each sample's temperature is shifted based on its cluster size: frequent/head concepts get a higher temperature (promoting semantic grouping), while rare/tail concepts get a lower temperature (enforcing instance separation).

The final per-sample temperature is:

τ_i = τ_base(t) + sh(c_i)

where τ_base(t) follows a cosine schedule and sh(c_i) is the cluster-based shift for sample i.

MM-TS is also extended to the max-margin loss by replacing the fixed margin with the modulated temperature, enabling dynamic margin scheduling for frameworks such as AVION on EPIC-KITCHENS-100.

This repository contains the official implementation of MM-TS integrated with the AVION framework for the EPIC-KITCHENS-100 Multi-Instance Retrieval task.

Note: We conducted experiments with additional contrastive frameworks (CLIP on CC3M, VAST on YouCook2). Code for those will be released in future updates.

MM-TS/
└── avion/          # MM-TS integrated with the AVION framework (EK-100 MIR)

For installation, dataset setup, training, and evaluation instructions, please refer to the avion/ directory.

If you find this work useful, please cite:

@InProceedings{Sheludzko_2026_WACV,
    author    = {Sheludzko, Siarhei and Duka, Dhimitrios and Schiele, Bernt and Kuehne, Hilde and Kukleva, Anna},
    title     = {MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {March},
    year      = {2026},
    pages     = {7376-7386}
}

Our AVION-based implementation builds on the AVION codebase. We thank the authors for releasing their code.

This project is released under the MIT License.