According to the source code, validation metrics should be generated at every checkpoint_interval (link). However with Re Identification model training, the validation metrics are not generated.
It turns out that if we tweak the training batch size and validation batch sizes, for some value combinations specific for a given dataset, the validation metrics are generated at the end of every epoch. However it is completely unclear how to achieve this in a predictable manner.
Can you please let me know how to configure training so that validation metrics are generated at the end of each training epoch or every checkpoint_interval epoch.
2 posts - 2 participants