Hi,
I’m running the notebook provided in TAO Toolkit Getting Started | NVIDIA NGC to train a model with a custom dataset.
I’m using an AWS VM and followed the instructions from Running TAO Toolkit on an AWS VM - NVIDIA Docs
It worked a couple times already, but now I need to use bigger images (13190 x 15880 pixels), and when I run the command to convert training data to TFRecords:
!tao model mask_rcnn dataset_convert -i $DATA_DOWNLOAD_DIR/raw-data/train2017
-a $DATA_DOWNLOAD_DIR/raw-data/annotations/instances_train2017.json
-o $DATA_DOWNLOAD_DIR/maskrcnn --include_masks -t train -s 256
-r $USER_EXPERIMENT_DIR/
It returns the following:
2023-12-07 14:25:33,862 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr .io’]
2023-12-07 14:25:33,939 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr .io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2023-12-07 14:25:33,960 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 262:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/ubuntu/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2023-12-07 14:25:33,960 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 275: Printing tty value True
2023-12-07 14:25:35.158350: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2023-12-07 14:25:35,218 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2023-12-07 14:25:37.489631: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
Using TensorFlow backend.
2023-12-07 14:25:37,705 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-07 14:25:37,756 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-07 14:25:37,761 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-07 14:25:38,425 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
2023-12-07 14:25:39.515731: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libnvinfer.so.8
2023-12-07 14:25:39.538430: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcuda.so.1
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO:tensorflow:writing to output path: /workspace/tao-experiments/data/maskrcnn/train
INFO:tensorflow:writing to output path: /workspace/tao-experiments/data/maskrcnn/train
INFO:tensorflow:Building bounding box index.
INFO:tensorflow:Building bounding box index.
INFO:tensorflow:0 images are missing bboxes.
INFO:tensorflow:0 images are missing bboxes.
multiprocessing.pool.RemoteTraceback:
“”"
Traceback (most recent call last):
File “/usr/lib/python3.8/multiprocessing/pool.py”, line 125, in worker
result = (True, func(*args, **kwds))
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 199, in _pool_create_tf_example
return create_tf_example(*args)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 70, in create_tf_example
image = PIL.Image.open(encoded_jpg_io)
File “/usr/local/lib/python3.8/dist-packages/PIL/Image.py”, line 3016, in open
im = _open_core(fp, filename, prefix, formats)
File “/usr/local/lib/python3.8/dist-packages/PIL/Image.py”, line 3003, in _open_core
_decompression_bomb_check(im.size)
File “/usr/local/lib/python3.8/dist-packages/PIL/Image.py”, line 2912, in _decompression_bomb_check
raise DecompressionBombError(
PIL.Image.DecompressionBombError: Image size (200580280 pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.
“”"The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 415, in
raise e
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 403, in
main()
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 313, in main
log_total = _create_tf_record_from_coco_annotations(
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/mask_rcnn/scripts/dataset_convert.py”, line 263, in create_tf_record_from_coco_annotations
for idx, (, tf_example, num_annotations_skipped, log_warnings) in enumerate(
File “/usr/lib/python3.8/multiprocessing/pool.py”, line 868, in next
raise value
PIL.Image.DecompressionBombError: Image size (200580280 pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.
Execution status: FAIL
2023-12-07 14:25:44,176 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 337: Stopping container.
So far, i’ve tried this solution: Pillow in Python won't let me open image ("exceeds limit") - Stack Overflow
I tried to edit the file:
/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/PIL/Image.py
changing line: MAX_IMAGE_PIXELS = int(1024 * 1024 * 1024 / 4 / 3)
to: MAX_IMAGE_PIXELS = None
Also tried to acces the file ‘Image.py’ in:
%cd /usr/local/lib/python3.8/dist-packages/PIL
but it returned:
[Errno 2] No such file or directory: ‘/usr/local/lib/python3.8/dist-packages/PIL’
/usr/local/lib/python3.8/dist-packages
The folder ‘/usr/local/lib/python3.8/dist-packages’ appears to be empty.
Thanks in advance!
8 posts - 2 participants