Following the notebook “tao-getting-started_v5.3.0/notebooks/tao_launcher_starter_kit/dino/dino.ipynb”, I want to generate the engine file in different data_type, and I found following problems:
- use “fp16” instead of “FP16” (https://docs.nvidia.com/tao/tao-toolkit/text/tao_deploy/dino.html), otherwise the generated engine is actually still same size as the default fp32 version.
- cannot generate int8 version engine, and report following error (same for using gen_trt_engine.tensorrt.data_type=int8 or change the data_type in the spec yaml file):
2024-07-11 14:50:57,394 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-07-11 14:50:57,477 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-deploy
2024-07-11 14:50:57,524 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
sys:1: UserWarning:
‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
:-1: UserWarning:
‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Log file already exists at /results/status.json
Starting dino gen_trt_engine.
[07/11/2024-06:50:59] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 32, GPU 11679 (MiB)
[07/11/2024-06:51:04] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1444, GPU +260, now: CPU 1553, GPU 11930 (MiB)
Parsing ONNX model
[07/11/2024-06:51:04] [TRT] [W] The NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION flag has been deprecated and has no effect. Please do not use this flag when creating the network.
[07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:511: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
Network Description
Input ‘inputs’ with shape (-1, 3, 544, 960) and dtype DataType.FLOAT
Output ‘pred_logits’ with shape (-1, 300, 91) and dtype DataType.FLOAT
Output ‘pred_boxes’ with shape (-1, 300, 4) and dtype DataType.FLOAT
dynamic batch size handling
Enabling INT8 builder
Calibrating using ImageBatcher
stat: path should be string, bytes, os.PathLike or integer, not NoneType
Error executing job with overrides: [‘gen_trt_engine.onnx_file=/results/export/dino_model.onnx’, ‘gen_trt_engine.trt_engine=/results/gen_trt_engine/dino_model_int8.engine’, ‘results_dir=/results’]
Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 132, in run
_ = ret.return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 260, in return_value
raise self._return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “”, line 63, in _func
File “”, line 47, in _func
File “”, line 104, in main
File “”, line 165, in create_engine
File “”, line 203, in set_calibrator
File “/usr/lib/python3.10/genericpath.py”, line 19, in exists
os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “</usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/dino/scripts/gen_trt_engine.py>”, line 3, in
File “”, line 117, in
File “”, line -1, in wrapper
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 215, in run_and_report
if _is_env_set(“HYDRA_FULL_ERROR”) or is_under_debugger():
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 165, in is_under_debugger
frames = inspect.stack()
File “/usr/lib/python3.10/inspect.py”, line 1673, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib/python3.10/inspect.py”, line 1650, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib/python3.10/inspect.py”, line 1622, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: ‘NoneType’ and ‘int’
2024-07-11 06:51:05,530 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Sending telemetry data.
2024-07-11 06:51:09,484 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Execution status: FAIL
2024-07-11 14:51:09,622 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.
1 post - 1 participant