Training issues with OCRNet

It was a long way to here (see also Using OCRNet from Python script)

Now the situation is:

• I have an overfit OCRNet model, which is able to detect German license plates
• Generally this model is able to detect the plates and its specials, but it can do this only for plates it has seen during training (clear overfit)
• The suggestion was to “fine tune” this model with new training data and test data different from training data (as I understood)
• I re-train the model with about 78 new images, the training never has seen before. The 10 test images have been taken from the initial run (not part of the current training data set). 2000 epochs.
• I’m observing a raising validation error over time, accuracy is going up to 0.25 then falling again to 0.125, going up again…

Generally I find the entire re-training process way way way too complicated and absolutely disappointing and that makes me feel stupid like hell. I “like” that.

My current re-train command (the checkpoint used to start from is the last result of the (incestuous) “overfit” training…)

tao model ocrnet train -e $SPECS_DIR/experiment-vit.yaml \
              train.results_dir=$RESULTS_DIR/experiment_dir_unpruned \
              train.pretrained_model_path=$RESULTS_DIR/pretrained_ocrnet/ocrnet_vtrainable_v2.0/ocrnet-vit.pth \
              train.resume_training_checkpoint_path=$RESULTS_DIR/experiment_dir_unpruned/ocrnet_epoch_4999.pth \
              dataset.train_dataset_dir=[$DATA_DIR/train/lmdb] \
              dataset.val_dataset_dir=$DATA_DIR/test/lmdb \
              dataset.character_list_file=$DATA_DIR/character_list

The configuration:

results_dir: /results
encryption_key: nvidia_tao
model:
  TPS: True
  backbone: FAN_tiny_2X
  sequence: BiLSTM
  hidden_size: 256
  prediction: Attn
  quantize: False
  input_width: 200
  input_height: 64
  input_channel: 1
dataset:
  train_dataset_dir: []
  val_dataset_dir: /data/test/lmdb
  character_list_file: /data/character_list
  max_label_length: 25
  batch_size: 32
  workers: 4
  augmentation:
    keep_aspect_ratio: False
train:
  seed: 1111
  gpu_ids: [0]
  optim:
    name: "adadelta"
    lr: 0.1
  clip_grad_norm: 5.0
  num_epochs: 2000
  checkpoint_interval: 1000
  validation_interval: 50
evaluate:
  gpu_id: 0
  checkpoint: "??"
  test_dataset_dir: "??"
  results_dir: "${results_dir}/evaluate"
prune:
  gpu_id: 0
  checkpoint: "??"
  results_dir: "${results_dir}/prune"
  prune_setting:
    mode: "amount"
    amount: 0.1
    granularity: 8
    raw_prune_score: L1
inference:
  gpu_id: 0
  checkpoint: "??"
  inference_dataset_dir: "??"
  results_dir: "${results_dir}/inference"
export:
  gpu_id: 0
  checkpoint: "??"
  results_dir: "${results_dir}/export"
dataset_convert:
  input_img_dir: "??"
  gt_file: "??"
  results_dir: "${results_dir}/convert_dataset"
gen_trt_engine:
  onnx_file: "??"
  results_dir: "${results_dir}/convert_dataset"

14 posts - 2 participants

Read full topic