retrain using an internal pretrained ResNet18

README.md

@@ -46,6 +46,21 @@ The training was performed with the following:
 - Loss: BCEWithLogitsLoss
 - Whole slide image reader: cuCIM (if running on Windows or Mac, please install `OpenSlide` on your system and change `wsi_reader` to "OpenSlide")
 
+### Pretrained Weights
+
+By setting the `"pretrained"` parameter of `TorchVisionFCModel` in the config file to `true`, ImageNet pre-trained weights will be used for training. Please note that these weights are for non-commercial use. Each user is responsible for checking the content of the models/datasets and the applicable licenses and determining if suitable for the intended use. In order to use other pretrained weights, you can use `CheckpointLoader` in train handlers section as the first handler:
+
+```json
+{
+    "_target_": "CheckpointLoader",
+    "load_path": "$@bundle_root + '/pretrained_resnet18.pth'",
+    "strict": false,
+    "load_dict": {
+        "model_new": "@network"
+    }
+}
+```
+
 ### Input
 
 The training pipeline is a json file (dataset.json) which includes path to each WSI, the location and the label information for each training patch.
@@ -58,12 +73,16 @@ A probability number of the input patch being tumor or normal.
 
 Inference is performed on WSI in a sliding window manner with specified stride. A foreground mask is needed to specify the region where the inference will be performed on, given that background region which contains no tissue at all can occupy a significant portion of a WSI. Output of the inference pipeline is a probability map of size 1/stride of original WSI size.
 
+### Note on determinism
+
+By default this bundle use a deterministic approach to make the results reproducible. However, it comes at a cost of performance loss. Thus if you do not care about reproducibility, you can have a performance gain by replacing `"$monai.utils.set_determinism"` line with `"$setattr(torch.backends.cudnn, 'benchmark', True)"` in initialize section of training configuration (`configs/train.json` and `configs/multi_gpu_train.json` for single GPU and multi-GPU training respectively).
+
 ## Performance
 
 FROC score is used for evaluating the performance of the model. After inference is done, `evaluate_froc.sh` needs to be run to evaluate FROC score based on predicted probability map (output of inference) and the ground truth tumor masks.
-This model achieve the 0.91 accuracy on validation patches, and FROC of 0.72 on the 48 Camelyon testing data that have ground truth annotations available.
+Using an internal pretrained weights for ResNet18, this model deterministically achieves the 0.90 accuracy on validation patches, and FROC of 0.72 on the 48 Camelyon testing data that have ground truth annotations available.
 
-![A Graph showing Train Acc, Train Loss, and Validation Acc](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics_v3.png)
+![A Graph showing Train Acc, Train Loss, and Validation Acc](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics_v5.png)
 
 The `pathology_tumor_detection` bundle supports acceleration with TensorRT. The table below displays the speedup ratios observed on an A100 80G GPU.
 
@@ -75,6 +94,7 @@ Please notice that the benchmark results are tested on one WSI image since the i
 | end2end |224.97 | 223.50 | 222.65 | 224.03 | 1.01 | 1.01 | 1.00 | 1.00 |
 
 Where:
+
 - `model computation` means the speedup ratio of model's inference with a random input without preprocessing and postprocessing
 - `end2end` means run the bundle end-to-end with the TensorRT based model.
 - `torch_fp32` and `torch_amp` are for the PyTorch models with or without `amp` mode.

configs/inference.json

@@ -125,9 +125,6 @@
         "amp": true,
         "decollate": false
     },
-    "initialize": [
-        "$setattr(torch.backends.cudnn, 'benchmark', True)"
-    ],
     "run": [
         "$@evaluator.run()"
     ]

configs/metadata.json

@@ -1,7 +1,9 @@
 {
     "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
-    "version": "0.5.2",
+    "version": "0.5.4",
     "changelog": {
+        "0.5.4": "retrain using an internal pretrained ResNet18",
+        "0.5.3": "make the training bundle deterministic",
         "0.5.2": "update TensorRT descriptions",
         "0.5.1": "update the TensorRT part in the README file",
         "0.5.0": "add the command of executing inference with TensorRT models",
@@ -42,8 +44,8 @@
     "label_classes": "binary labels for each patch",
     "pred_classes": "scalar probability",
     "eval_metrics": {
-        "accuracy": 0.91,
-        "froc": 0.685
+        "accuracy": 0.9,
+        "froc": 0.72
     },
     "intended_use": "This is an example, not to be used for diagnostic purposes",
     "references": [

configs/multi_gpu_train.json

@@ -28,8 +28,7 @@
         "$import torch.distributed as dist",
         "$dist.is_initialized() or dist.init_process_group(backend='nccl')",
         "$torch.cuda.set_device(@device)",
-        "$monai.utils.set_determinism(seed=123)",
-        "$setattr(torch.backends.cudnn, 'benchmark', True)"
+        "$monai.utils.set_determinism(seed=123)"
     ],
     "run": [
         "$@train#trainer.run()"

configs/train.json

@@ -372,8 +372,7 @@
         }
     },
     "initialize": [
-        "$monai.utils.set_determinism(seed=15)",
-        "$setattr(torch.backends.cudnn, 'benchmark', True)"
+        "$monai.utils.set_determinism(seed=15)"
     ],
     "run": [
         "$@train#trainer.run()"

docs/README.md

@@ -39,6 +39,21 @@ The training was performed with the following:
 - Loss: BCEWithLogitsLoss
 - Whole slide image reader: cuCIM (if running on Windows or Mac, please install `OpenSlide` on your system and change `wsi_reader` to "OpenSlide")
 
+### Pretrained Weights
+
+By setting the `"pretrained"` parameter of `TorchVisionFCModel` in the config file to `true`, ImageNet pre-trained weights will be used for training. Please note that these weights are for non-commercial use. Each user is responsible for checking the content of the models/datasets and the applicable licenses and determining if suitable for the intended use. In order to use other pretrained weights, you can use `CheckpointLoader` in train handlers section as the first handler:
+
+```json
+{
+    "_target_": "CheckpointLoader",
+    "load_path": "$@bundle_root + '/pretrained_resnet18.pth'",
+    "strict": false,
+    "load_dict": {
+        "model_new": "@network"
+    }
+}
+```
+
 ### Input
 
 The training pipeline is a json file (dataset.json) which includes path to each WSI, the location and the label information for each training patch.
@@ -51,12 +66,16 @@ A probability number of the input patch being tumor or normal.
 
 Inference is performed on WSI in a sliding window manner with specified stride. A foreground mask is needed to specify the region where the inference will be performed on, given that background region which contains no tissue at all can occupy a significant portion of a WSI. Output of the inference pipeline is a probability map of size 1/stride of original WSI size.
 
+### Note on determinism
+
+By default this bundle use a deterministic approach to make the results reproducible. However, it comes at a cost of performance loss. Thus if you do not care about reproducibility, you can have a performance gain by replacing `"$monai.utils.set_determinism"` line with `"$setattr(torch.backends.cudnn, 'benchmark', True)"` in initialize section of training configuration (`configs/train.json` and `configs/multi_gpu_train.json` for single GPU and multi-GPU training respectively).
+
 ## Performance
 
 FROC score is used for evaluating the performance of the model. After inference is done, `evaluate_froc.sh` needs to be run to evaluate FROC score based on predicted probability map (output of inference) and the ground truth tumor masks.
-This model achieve the 0.91 accuracy on validation patches, and FROC of 0.72 on the 48 Camelyon testing data that have ground truth annotations available.
+Using an internal pretrained weights for ResNet18, this model deterministically achieves the 0.90 accuracy on validation patches, and FROC of 0.72 on the 48 Camelyon testing data that have ground truth annotations available.
 
-![A Graph showing Train Acc, Train Loss, and Validation Acc](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics_v3.png)
+![A Graph showing Train Acc, Train Loss, and Validation Acc](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_tumor_detection_train_and_val_metrics_v5.png)
 
 The `pathology_tumor_detection` bundle supports acceleration with TensorRT. The table below displays the speedup ratios observed on an A100 80G GPU.
 
@@ -68,6 +87,7 @@ Please notice that the benchmark results are tested on one WSI image since the i
 | end2end |224.97 | 223.50 | 222.65 | 224.03 | 1.01 | 1.01 | 1.00 | 1.00 |
 
 Where:
+
 - `model computation` means the speedup ratio of model's inference with a random input without preprocessing and postprocessing
 - `end2end` means run the bundle end-to-end with the TensorRT based model.
 - `torch_fp32` and `torch_amp` are for the PyTorch models with or without `amp` mode.

models/model.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7383884d1a14a2e3d7e51ad209181f18c70ad2af0fa3f69d42faed2b165d0455
+oid sha256:5a0d9b9e714e18a90c1f7f7d9c7e47f807c59f9f8c681b84865fae208fcbb4d6
 size 44780565