Please provide the following information when requesting support.
• Hardware AMD64 , 3080 RTX
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
I am trying to convert the onnx file for the deformable RT-DETR model (
Deformable DETR | NVIDIA NGC) using the tao deploy command:
tao deploy deformable_detr gen_trt_engine -e $SPECS_DIR/gen_trt_engine.yaml gen_trt_engine.onnx_file=$RESULTS_DIR/export/dd_resnet50_ep50.onnx gen_trt_engine.trt_engine=$RESULTS_DIR/gen_trt_engine/dd_model.engine
results_dir=$RESULTS_DIR/
content of gen_trt_engine.yaml is:
gen_trt_engine:
gpu_id: 0
input_width: 960
input_height: 544
tensorrt:
data_type: fp32
workspace_size: 2048
min_batch_size: 1
opt_batch_size: 8
max_batch_size: 8
dataset:
num_classes: 91
batch_size: -1
model:
backbone: resnet_50
num_feature_levels: 2
num_queries: 300
with_box_refine: True
aux_loss: False
Since this is essentially running a container, I expect to this to work - but:
2024-08-12 16:11:48,467 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2024-08-12 16:11:48,942 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-deploy
2024-08-12 16:11:49,077 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
sys:1: UserWarning:
'gen_trt_engine.yaml' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
<frozen cv.common.hydra.hydra_runner>:-1: UserWarning:
'gen_trt_engine.yaml' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Log file already exists at /results/status.json
Starting deformable_detr gen_trt_engine.
[08/12/2024-23:11:57] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 36, GPU 1166 (MiB)
[08/12/2024-23:12:05] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1453, GPU +268, now: CPU 1565, GPU 1434 (MiB)
Parsing ONNX model
[08/12/2024-23:12:05] [TRT] [W] The NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION flag has been deprecated and has no effect. Please do not use this flag when creating the network.
[08/12/2024-23:12:06] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/12/2024-23:12:06] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:511: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
[08/12/2024-23:12:06] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
[08/12/2024-23:12:06] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
[08/12/2024-23:12:06] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
[08/12/2024-23:12:06] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
Network Description
Input 'inputs' with shape (-1, 3, 544, 960) and dtype DataType.FLOAT
Output 'pred_logits' with shape (-1, 300, 91) and dtype DataType.FLOAT
Output 'pred_boxes' with shape (-1, 300, 4) and dtype DataType.FLOAT
dynamic batch size handling
TensorRT engine build configurations:
OptimizationProfile:
"inputs": (1, 3, 544, 960), (8, 3, 544, 960), (8, 3, 544, 960)
BuilderFlag.TF32
Note: max representabile value is 2,147,483,648 bytes or 2GB.
MemoryPoolType.WORKSPACE = 2147483648 bytes
MemoryPoolType.DLA_MANAGED_SRAM = 0 bytes
MemoryPoolType.DLA_LOCAL_DRAM = 1073741824 bytes
MemoryPoolType.DLA_GLOBAL_DRAM = 536870912 bytes
PreviewFeature.FASTER_DYNAMIC_SHAPES_0805
PreviewFeature.DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805
Tactic Sources = 31
[08/12/2024-23:12:06] [TRT] [I] Graph optimization time: 0.150962 seconds.
[08/12/2024-23:12:07] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1799, GPU 1520 (MiB)
[08/12/2024-23:12:07] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1799, GPU 1528 (MiB)
[08/12/2024-23:12:07] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
symbolic_global_padding.cpp:929: DCHECK(use->is_use_only()) failed.
[08/12/2024-23:12:53] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[/model/transformer/decoder/layers.2/cross_attn/Cast.../model/transformer/decoder/layers.2/cross_attn/Add]}.
[08/12/2024-23:12:53] [TRT] [E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/model/transformer/decoder/layers.2/cross_attn/Cast.../model/transformer/decoder/layers.2/cross_attn/Add]}.)
__enter__
Error executing job with overrides: ['gen_trt_engine.onnx_file=/results/export/dd_resnet50_ep50.onnx', 'gen_trt_engine.trt_engine=/results/gen_trt_engine/dd_model.engine', 'results_dir=/results/']
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 213, in run_and_report
return func()
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 453, in <lambda>
lambda: hydra.run(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "<frozen cv.common.decorators>", line 63, in _func
File "<frozen cv.common.decorators>", line 47, in _func
File "<frozen cv.deformable_detr.scripts.gen_trt_engine>", line 98, in main
File "<frozen cv.deformable_detr.engine_builder>", line 173, in create_engine
AttributeError: __enter__. Did you mean: '_return_value'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "</usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/deformable_detr/scripts/gen_trt_engine.py>", line 3, in <module>
File "<frozen cv.deformable_detr.scripts.gen_trt_engine>", line 111, in <module>
File "<frozen cv.common.hydra.hydra_runner>", line -1, in wrapper
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 389, in _run_hydra
_run_app(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 452, in _run_app
run_and_report(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 215, in run_and_report
if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 165, in is_under_debugger
frames = inspect.stack()
File "/usr/lib/python3.10/inspect.py", line 1673, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/lib/python3.10/inspect.py", line 1650, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/lib/python3.10/inspect.py", line 1622, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
2024-08-12 23:12:54,895 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Sending telemetry data.
2024-08-12 23:12:55,823 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Execution status: FAIL
2024-08-12 16:12:56,136 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.
5 posts - 2 participants