I have been having problems using the Tao Deploy Container 5.5.0 to generate a TensorRT engine from an onnx model.
Here is the command line I am using. Below that is the error logs & a copy of my spec.yaml.
dino gen_trt_engine -e /opt/nvidia/specs/spec.yaml
results_dir=/user_data/beebf665-5be3-4423-9bca-5da52da18476
gen_trt_engine.tensorrt.data_type=FP16
Spec.yaml
gen_trt_engine:
onnx_file: /opt/nvidia/specs/dino_distill_500_model.onnx
trt_engine: /opt/nvidia/specs/dino_distill_500_model.engine
input_channel: 3
input_width: 1920
input_height: 960
tensorrt:
data_type: FP32
workspace_size: 1024
min_batch_size: 1
opt_batch_size: 4
max_batch_size: 4
Error logs
root@1814d336a2bc:/app# dino gen_trt_engine -e /opt/nvidia/specs/spec.yaml
results_dir=/user_data/beebf665-5be3-4423-9bca-5da52da18476
gen_trt_engine.tensorrt.data_type=FP16
sys:1: UserWarning:
‘spec.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/common/hydra/hydra_runner.py:99: UserWarning:
‘spec.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
_run_hydra(
/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Log file already exists at /user_data/beebf665-5be3-4423-9bca-5da52da18476/status.json
Starting dino gen_trt_engine.
[03/12/2025-13:53:10] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1453, GPU +267, now: CPU 1566, GPU 1055 (MiB)Parsing ONNX model
[03/12/2025-13:53:10] [TRT] [W] The NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION flag has been deprecated and has no e[03/12/2025-13:53:11] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, [03/12/2025-13:53:11] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clam[03/12/2025-13:53:11] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:511: Your ONNX model has been generated with double-typed we[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as p[03/12/2025-13:53:11] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespac[03/12/2025-13:53:11] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRTNetwork Description
Input ‘inputs’ with shape (-1, 3, 960, 1920) and dtype DataType.FLOAT
Output ‘pred_logits’ with shape (-1, 300, 125) and dtype DataType.FLOAT
Output ‘pred_boxes’ with shape (-1, 300, 4) and dtype DataType.FLOAT
dynamic batch size handling
TensorRT engine build configurations:
OptimizationProfile:
“inputs”: (1, 3, 960, 1920), (4, 3, 960, 1920), (4, 3, 960, 1920)
BuilderFlag.TF32
Note: max representabile value is 2,147,483,648 bytes or 2GB.
MemoryPoolType.WORKSPACE = 1073741824 bytes
MemoryPoolType.DLA_MANAGED_SRAM = 0 bytes
MemoryPoolType.DLA_LOCAL_DRAM = 1073741824 bytes
MemoryPoolType.DLA_GLOBAL_DRAM = 536870912 bytes
PreviewFeature.FASTER_DYNAMIC_SHAPES_0805
PreviewFeature.DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805
Tactic Sources = 31
[03/12/2025-13:54:59] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[/model/backbone/backbone.0/Constant_output_0…/model/backbone/backbone.0/body/blocks.0/mlp/mlp_v/dwconv/Transpose + /model/backbone/backbone.0/body/blocks.[03/12/2025-13:54:59] [TRT] [E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/model/backbone/backbone.0/Constant_output_0…/model/backbone/backbone.0/body/blocks.0/mlp/mlp_v/dwconv/Transpose + /model/backbone/backbone.0/body/blocks.0/mlp/mlp_v/dwconv/Reshape]}.)enter
Error executing job with overrides: [‘results_dir=/user_data/beebf665-5be3-4423-9bca-5da52da18476’, ‘gen_trt_engine.tensorrt.data_type=FP16’]Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/dino/scripts/gen_trt_engine.py”, line 117, in
main()
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/common/hydra/hydra_runner.py”, line 99, in wrapper
_run_hydra(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 216, in run_and_report
raise ex
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 132, in run
_ = ret.return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 260, in return_value
raise self._return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/common/decorators.py”, line 63, in _func
raise e
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/common/decorators.py”, line 47, in _func
runner(cfg, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/dino/scripts/gen_trt_engine.py”, line 104, in main
builder.create_engine(
File “/usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/deformable_detr/engine_builder.py”, line 172, in create_engine
with self.builder.build_engine(self.network, self.config) as engine,
AttributeError: enter. Did you mean: ‘_return_value’?
[03/12/2025-13:55:00] [TRT] [E] /workspace/trt_oss_src/TensorRT/plugin/instanceNormalizationPlugin/instanceNormalizationPlugin.cu (192) - Cudnn Error in terminate: 4000 (CUDNN_STATUS_INTERNAL_ERROR)terminate called after throwing an instance of ‘nvinfer1::plugin::CudnnError’
what(): std::exception
[2025-03-12 13:55:00,475 - TAO Toolkit - nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra - INFO] Sending telemetry data.
[2025-03-12 13:55:00,475 - TAO Toolkit - root - INFO] ================> Start Reporting Telemetry <================
[2025-03-12 13:55:00,475 - TAO Toolkit - root - INFO] Sending {‘version’: ‘5.5.0’, ‘action’: ‘gen_trt_engine’, ‘network’: ‘dino’, ‘gpu’: [‘NVIDIA-GeForce-RTX-3090’], ‘success’: False, ‘time_lapsed’: 114} to https://api.tao.ngc.nvidia.com.
[2025-03-12 13:55:00,839 - TAO Toolkit - root - INFO] Telemetry sent successfully.
[2025-03-12 13:55:00,839 - TAO Toolkit - root - INFO] ================> End Reporting Telemetry <================
[2025-03-12 13:55:00,839 - TAO Toolkit - root - INFO] gen_trt_engine action failed for dino
[2025-03-12 13:55:00,840 - TAO Toolkit - nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra - INFO] Execution status: FAIL
3 posts - 2 participants