add the command of executing inference with TensorRT models

README.md

@@ -75,13 +75,14 @@ Please notice that the benchmark results are tested on one WSI image since the i
 | end2end |224.97 | 223.50 | 222.65 | 224.03 | 1.01 | 1.01 | 1.00 | 1.00 |
 
 This result is benchmarked under:
- - TensorRT: 8.5.3+cuda11.8
- - Torch-TensorRT Version: 1.4.0
- - CPU Architecture: x86-64
- - OS: ubuntu 20.04
- - Python version:3.8.10
- - CUDA version: 11.8
- - GPU models and configuration: A100 80G
+
+- TensorRT: 8.5.3+cuda11.8
+- Torch-TensorRT Version: 1.4.0
+- CPU Architecture: x86-64
+- OS: ubuntu 20.04
+- Python version:3.8.10
+- CUDA version: 11.8
+- GPU models and configuration: A100 80G
 
 ## MONAI Bundle Commands
 
@@ -121,12 +122,18 @@ cd scripts && source evaluate_froc.sh
 python -m monai.bundle ckpt_export network_def --filepath models/model.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json
 ```
 
-#### Export checkpoint to TensorRT based models with fp32 or fp16 precision:
+#### Export checkpoint to TensorRT based models with fp32 or fp16 precision
 
 ```
 python -m monai.bundle trt_export --net_id network_def --filepath models/model_trt.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json --precision <fp32/fp16> --dynamic_batchsize "[1, 400, 600]"
 ```
 
+#### Execute inference with the TensorRT model
+
+```
+python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']"
+```
+
 # References
 
 [1] He, Kaiming, et al, "Deep Residual Learning for Image Recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016. <https://arxiv.org/pdf/1512.03385.pdf>

configs/inference_trt.json

@@ -0,0 +1,10 @@
+{
+    "imports": [
+        "$import glob",
+        "$import os",
+        "$import torch_tensorrt"
+    ],
+    "handlers#0#_disabled_": true,
+    "network_def": "$torch.jit.load(@bundle_root + '/models/model_trt.ts')",
+    "evaluator#amp": false
+}

configs/metadata.json

@@ -1,7 +1,8 @@
 {
     "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
-    "version": "0.4.9",
+    "version": "0.5.0",
     "changelog": {
+        "0.5.0": "add the command of executing inference with TensorRT models",
         "0.4.9": "adapt to BundleWorkflow interface",
         "0.4.8": "update the readme file with TensorRT convert",
         "0.4.7": "add name tag",

docs/README.md

@@ -68,13 +68,14 @@ Please notice that the benchmark results are tested on one WSI image since the i
 | end2end |224.97 | 223.50 | 222.65 | 224.03 | 1.01 | 1.01 | 1.00 | 1.00 |
 
 This result is benchmarked under:
- - TensorRT: 8.5.3+cuda11.8
- - Torch-TensorRT Version: 1.4.0
- - CPU Architecture: x86-64
- - OS: ubuntu 20.04
- - Python version:3.8.10
- - CUDA version: 11.8
- - GPU models and configuration: A100 80G
+
+- TensorRT: 8.5.3+cuda11.8
+- Torch-TensorRT Version: 1.4.0
+- CPU Architecture: x86-64
+- OS: ubuntu 20.04
+- Python version:3.8.10
+- CUDA version: 11.8
+- GPU models and configuration: A100 80G
 
 ## MONAI Bundle Commands
 
@@ -114,12 +115,18 @@ cd scripts && source evaluate_froc.sh
 python -m monai.bundle ckpt_export network_def --filepath models/model.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json
 ```
 
-#### Export checkpoint to TensorRT based models with fp32 or fp16 precision:
+#### Export checkpoint to TensorRT based models with fp32 or fp16 precision
 
 ```
 python -m monai.bundle trt_export --net_id network_def --filepath models/model_trt.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json --precision <fp32/fp16> --dynamic_batchsize "[1, 400, 600]"
 ```
 
+#### Execute inference with the TensorRT model
+
+```
+python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']"
+```
+
 # References
 
 [1] He, Kaiming, et al, "Deep Residual Learning for Image Recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016. <https://arxiv.org/pdf/1512.03385.pdf>