Contributing to the MetaMathQA Benchmark #3521

Thanks for reporting the results. It's fine if the new experiment presents a trade-off, it doesn't need to improve on all fronts. If you have the time, you could try even higher values (larger VB, higher lr) to see if even more improvement is possible.

Thanks! I’ll run a small controlled follow-up search, testing a larger vector bank and a moderately higher learning rate while changing one factor at a time. I’ll report the complete comparison and use only the strongest candidate for the configuration PR.

I completed the follow-up experiments, changing one factor at a time before testing their combination.
These were single seed-0 local runs.

Both single-factor changes improved over C2, and their combination in C5 performed best. Compared with the reproduced default baseline at 33.2828%, C5 improved GSM8K test accuracy by about 7.96 percentage points.

The trade-off is that the 1024-vector configuration has approximately 4x the trainable adapter parameters and checkpoint size of the 256-vector baseline. The checkpoint was about 18.5 MB, while peak GPU memory remainedaround 13 GB in the local gradient-checkpointed run.

I’m stopping the search at this predefined boundary rather than continuing to tune against the test benchmark. I’ll prepare a minimal configuration-only PR for C5, containing no local paths, gradient-checkpointing override, result files, or runner changes.

Nice to see even better results. Note that VB-LoRA shares the vector bank, so even if we double the number of vectors there, it doesn't necessarily mean we require a lot more memory. Do you have the exact memory numbers, especially max memory? That's more interesting to look at than pure parameter count.

Good point. I checked the exact accelerator_memory_max values recorded by the benchmark. This metric is the peak reserved accelerator memory after subtracting the initial allocation.

C5 used only 56 MiB more peak reserved memory than B0, an increase of about 0.43%. At the same learning rate, C3 was actually slightly below C2, while C5 was only 36 MiB above C4, so these small differences are likely within CUDA allocator variation.

Therefore, although the trainable parameter count and checkpoint size increase by approximately 4x, the measured training peak memory is essentially unchanged in these local gradient-checkpointed runs. This is consistent with the vector bank being shared.