Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) A6000
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Segformer
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
Hi @Morganh
I did not manage to return to this matter until after it had been closed.
I confirm that I had tried the example isbi example before and not had the error. It’s every other dataset that is an issue though.
I had tried the -r option with local directory by launching the specific container and mapping volumes etc. Same error occurs. I have tried running with launcher and a full local path for the -r option and everything runs ok but of course the results are not written because the -r option expects a container address.
I normally use the launcher and don’t interact with the containers directly. But I have shown that this does not matter. I’ve tried running with 1 gpu - no difference.
I have tried this on many different datasets and all fail with exactly the same issue. There must be others that are experiencing this. I’ve tried running with root privileges in the container (though .tao_mounts)
The error logs point to train.py the filesystem in the container under the dist-packages so I am unable to review those to give further details on what might be causing it.
I am concerned that if, as you say, validation is not run then the training is not being guided by the VAL dataset and you can’t see how the model is doing during training because the model output in status.json is for all categories including background.
Please help - thank you.
2 posts - 2 participants