When attempting to train a custom dataset using detectnet_v2 in tao toolkit I get this error:
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0
I have trained the example dataset provided with detectnet successfully so I know everything is working, it is only when I introduced my own dataset that this problem started. Starting with a dataset in coco format, I converted it to kitti format. Then used the command *
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval
-r $USER_EXPERIMENT_DIR/
to create tfrecords.
Here is a sample of my labels records after converting from the coco format:
waterheater 0 0 0 174 302 17.5 100.0 0 0 0 0 0 0 0
furnace 0 0 0 188 252 17.5 144.5 0 0 0 0 0 0 0
furnace 0 0 0 200 323 21.5 71.5 0 0 0 0 0 0 0
I ran a python script to check whether any tfrecord files were corrupted, and none were.
But I’m still getting the error when I try to train the dataset.
2 posts - 2 participants