This document states how to use the example MIS/deployments/ps-ptd-v2.

1. Prerequisites

1.1. Prepare dataset

Place your dataset folder in here(MIS/deployments/ps-ptd-v2) and name it as data.
Or you may directly create a link to the folder like this.

ln -s YOUR_DATASET data

1.2. Set the data list

1.2.1. For training/validation, please modify training/data_list.yaml.

amount:
  test: 0
  total: 48
  train: 33
  valid: 15
list:                   (1)
  train:
  - 0522c0001
      ⋮
  valid:
  - 0522c0002
      ⋮
loader:
  name: NRRDLoader
  data_dir: ../data
  roi_map:
    Brainstem: 1
    Parotid_L: 2
    Parotid_R: 3
  spacing: 1
  resample: true        (2)
1 Fill in the training/validation data list with the name of each case.
2 If you’re going to run the model on a dataset that has already been resampled, you may toggle off the resample.
This way should accelerate the training.

For evaluation, please modify evaluation/data_list.yaml.

list:
- 0522c0002            (1)
    ⋮
loader:
  name: NRRDLoader
  data_dir: ../data
  roi_map:
    Brainstem: 1
    Parotid_L: 2
    Parotid_R: 3
  spacing: 1
  resample: true        (2)
1 Fill in the data list to be evaluated with the name of each case.
2 If you’re going to run the model on a dataset that has already been resampled, you may toggle off the resample.
This way should accelerate the evaluation.
Note that you can split the dataset to training/validation/testing three parts. Train the model on training data, choose the best model in checkpoints according to the performance on the validation data, and finally evaluate the performance on the testing data.
You can use the tools from MIDP to generate the data list. Please see here for the details.

2. Usage

2.1. Download the trained model

make download_model

It will download a model checkpoint which achieved the performance as below.

Table 1. Validation performance on PDDCA dataset
Brainstem Left Parotid Right Parotid Average

0.879390329990884

0.7646160997046699

0.7855193317156853

0.8098419204704131

2.2. Training

Continue training with the trained model.

make train
training
Figure 1. Training
The dice score here is computed with smooth=1(Laplacian smooth), which means the performance may be overestimated. But since this is early model trained with dice loss with smooth=1, and the checkpoint is selected according to the validation dice score with smooth=1, so we uses Laplacian smooth here the metrics here.
There will be a gap between the validation score and the evaluation one since the condition is harder(the model makes prediction without considering threshold).

2.3. Evaluation

Directly evaluate the performance with the trained model checkpoint.

make evaluate

Evaluate the performance with a newly trained checkpoint.

make evaluate CKPT=training/_ckpts/SOME_BETTER_CHECKPOINT
evaluating
Figure 2. Evaluating
One can observe the gap between the score of each batch(before reconstruction) and the evaluation one(after reconstruction, true dice score enclosed in ===== Restored =====).
Since there are additional processings like applying threshold , averaging the overlapping predictions, the performance will be better.

The output result will be exported as evaluation/score.json.

[Result]
score.json
{
  "0522c0002": {
    "Brainstem": 0.9028694037020432,
    "Parotid_L": 0.8119102638166763,
    "Parotid_R": 0.7996425003943011
  },
  "0522c0014": {
    "Brainstem": 0.8935291506648397,
    "Parotid_L": 0.7024681698556086,
    "Parotid_R": 0.7702638241661468
  },
  "0522c0057": {
    "Brainstem": 0.8807874039521157,
    "Parotid_L": 0.6785307517084282,
    "Parotid_R": 0.8405278584304605
  },
  "0522c0077": {
    "Brainstem": 0.8953396583309751,
    "Parotid_L": 0.8370424150593979,
    "Parotid_R": 0.7786675576065735
  },
  "0522c0125": {
    "Brainstem": 0.8044672789668922,
    "Parotid_L": 0.7051278059715433,
    "Parotid_R": 0.6728889864939058
  },
  "0522c0161": {
    "Brainstem": 0.9172809839938093,
    "Parotid_L": 0.7442935852026761,
    "Parotid_R": 0.8497808992136383
  },
  "0522c0226": {
    "Brainstem": 0.8880701572310726,
    "Parotid_L": 0.7538168885948225,
    "Parotid_R": 0.7728004137693236
  },
  "0522c0248": {
    "Brainstem": 0.895500903494533,
    "Parotid_L": 0.7822063892294524,
    "Parotid_R": 0.7357568556809689
  },
  "0522c0330": {
    "Brainstem": 0.8512992752342231,
    "Parotid_L": 0.8401635341230971,
    "Parotid_R": 0.7947188397865645
  },
  "0522c0427": {
    "Brainstem": 0.8383990640959301,
    "Parotid_L": 0.6847284486946383,
    "Parotid_R": 0.768788733846134
  },
  "0522c0433": {
    "Brainstem": 0.9045027857870661,
    "Parotid_L": 0.792590844738084,
    "Parotid_R": 0.831225229783287
  },
  "0522c0479": {
    "Brainstem": 0.8967516758693284,
    "Parotid_L": 0.7434027378350397,
    "Parotid_R": 0.738557128063514
  },
  "0522c0598": {
    "Brainstem": 0.8709916589434662,
    "Parotid_L": 0.8893883163544424,
    "Parotid_R": 0.8819805169174416
  },
  "0522c0708": {
    "Brainstem": 0.8793746997302192,
    "Parotid_L": 0.8879443585780525,
    "Parotid_R": 0.8688722445067623
  },
  "0522c0857": {
    "Brainstem": 0.8716908498667456,
    "Parotid_L": 0.6156269858080915,
    "Parotid_R": 0.6783183870762577
  }
}

Besides making inference, save the predictions and store them into NRRD.

make predict
predicting
Figure 3. Predicting

The outputs will be stored in the folder evaluation/outputs.

The process may be slow due to resampling twice before/after inference. And the current workflow will store the predictions of all cases and then do reconstruction, the memory usage might be large.