To what is this variable connected? a yaml parameter?

To what is this variable connected? a yaml parameter?

yeah kinda - but it might be in conflict with some other concept beneath.
maybe the idea is bogus. need to test - let's use this as a playground to get sth working

well, actually, the sorting flag is the flag?

just put it to "False" instead; this simply reverts the other PR that caused the bug w/o needing to rethink recipe & hparam logic.

Added a test and an initial fix outline. The test uses a CPU ddp backend, gloo.

I run into these nearby-sortings w/ sorting & ddp:

tensor([2.2050, 2.3400, 2.6000, 2.8700, 3.1050, 3.1500, 3.2200, 3.2800, 3.4550,
        3.6500, 3.7000, 4.2650, 4.4701, 4.7600, 4.8350, 4.8900, 5.1600, 6.1450,
        1.7750], dtype=torch.float64)

tensor([6.1450, 5.1600, 4.8900, 4.8350, 4.7600, 4.4701, 4.2650, 3.7000, 3.6500,
        3.4550, 3.2800, 3.2200, 3.1500, 3.1050, 2.8700, 2.6000, 2.3400, 2.2050,
        6.2000], dtype=torch.float64)

edit: also, only 38/100 examples are sampled when DDP uses sorting, which should not be.

W/o ddp (and shuffling ddp), all works fine.

The test needed to be adjusted to DDP & pytorch's DistributedSampler which has:

        drop_last (bool, optional): if ``True``, then the sampler will drop the
            tail of the data to make it evenly divisible across the number of
            replicas. If ``False``, the sampler will add extra indices to make
            the data evenly divisible across the replicas. Default: ``False``.

Test works for the proposed fix on DDP & sorting. Bc of the drop_last & replicas, the final elements of the last batch won't be sorted within that batch. Yet, DDP seems to distribute the batches w/ replicas BUT

• for sorted (filtered) datasets, the total number of examples does not change
• for random (shuffling), the number of examples increases, as replicas are included.

The sorting asserts are thus tailored to the dummy data (and a bit lazy with "[:-1]" subset testing across all batches, instead of final batch only).

Ready for review.