Reproduce budget batch classification results for CIFAR100

[gdamaskinos]

I am trying to reproduce the MSDNet with dynamic evaluation curve of Figure 7b of the paper (budgeted batch classification for CIFAR-100).

I am training the MSDNet as mentioned in the README file.
I then evaluate by:
python main.py --data-root /path/to/cifar100/ --data cifar100 --save /path/to/out/ --arch msdnet --batch-size 64 --epochs 300 --nBlocks 5 --stepmode lin_grow --step 1 --base 1 --nChannels 16 --evalmode dynamic --evaluate-from /path/to/out/save_models/checkpoint_299.pth.tar --use-valid -j 1 --gpu 0

Output log

building network of steps:
[1, 2, 3, 4, 5] 15
 ********************** Block 1  **********************
|		inScales 3 outScales 3 inChannels 16 outChannels 6		|

 ********************** Block 2  **********************
|		inScales 3 outScales 3 inChannels 22 outChannels 6		|

|		inScales 3 outScales 3 inChannels 28 outChannels 6		|

 ********************** Block 3  **********************
|		inScales 3 outScales 3 inChannels 34 outChannels 6		|

|		inScales 3 outScales 3 inChannels 40 outChannels 6		|

|		inScales 3 outScales 2 inChannels 46 outChannels 6		|
|		Transition layer inserted! (max), inChannels 52, outChannels 26	|

 ********************** Block 4  **********************
|		inScales 2 outScales 2 inChannels 26 outChannels 6		|

|		inScales 2 outScales 2 inChannels 32 outChannels 6		|

|		inScales 2 outScales 2 inChannels 38 outChannels 6		|

|		inScales 2 outScales 2 inChannels 44 outChannels 6		|

 ********************** Block 5  **********************
|		inScales 2 outScales 1 inChannels 50 outChannels 6		|
|		Transition layer inserted! (max), inChannels 56, outChannels 28	|

|		inScales 1 outScales 1 inChannels 28 outChannels 6		|

|		inScales 1 outScales 1 inChannels 34 outChannels 6		|

|		inScales 1 outScales 1 inChannels 40 outChannels 6		|

|		inScales 1 outScales 1 inChannels 46 outChannels 6		|

---------------------
FLOPs: 6.86M, Params: 0.30M
---------------------
FLOPs: 14.35M, Params: 0.65M
---------------------
FLOPs: 27.54M, Params: 1.02M
---------------------
FLOPs: 41.71M, Params: 1.49M
---------------------
FLOPs: 58.48M, Params: 2.08M
building network of steps:
[1, 2, 3, 4, 5] 15
 ********************** Block 1  **********************
|		inScales 3 outScales 3 inChannels 16 outChannels 6		|

 ********************** Block 2  **********************
|		inScales 3 outScales 3 inChannels 22 outChannels 6		|

|		inScales 3 outScales 3 inChannels 28 outChannels 6		|

 ********************** Block 3  **********************
|		inScales 3 outScales 3 inChannels 34 outChannels 6		|

|		inScales 3 outScales 3 inChannels 40 outChannels 6		|

|		inScales 3 outScales 2 inChannels 46 outChannels 6		|
|		Transition layer inserted! (max), inChannels 52, outChannels 26	|

 ********************** Block 4  **********************
|		inScales 2 outScales 2 inChannels 26 outChannels 6		|

|		inScales 2 outScales 2 inChannels 32 outChannels 6		|

|		inScales 2 outScales 2 inChannels 38 outChannels 6		|

|		inScales 2 outScales 2 inChannels 44 outChannels 6		|

 ********************** Block 5  **********************
|		inScales 2 outScales 1 inChannels 50 outChannels 6		|
|		Transition layer inserted! (max), inChannels 56, outChannels 28	|

|		inScales 1 outScales 1 inChannels 28 outChannels 6		|

|		inScales 1 outScales 1 inChannels 34 outChannels 6		|

|		inScales 1 outScales 1 inChannels 40 outChannels 6		|

|		inScales 1 outScales 1 inChannels 46 outChannels 6		|

!!!!!! Load train_set_index !!!!!!
*********************
/home/damaskin/MSDNet-PyTorch/adaptive_inference.py:28: UserWarning: torch.range is deprecated in favor of torch.arange and will be removed in 0.5. Note that arange generates values in [start; end), not [start; end].
  probs = torch.exp(torch.log(_p) * torch.range(1, args.nBlocks))
valid acc: 59.020, test acc: 61.500, test flops: 7.09M
*********************
valid acc: 59.960, test acc: 62.080, test flops: 7.44M
*********************
valid acc: 60.700, test acc: 62.870, test flops: 7.90M
*********************
valid acc: 61.660, test acc: 63.500, test flops: 8.36M
*********************
valid acc: 62.700, test acc: 64.050, test flops: 8.93M
*********************
valid acc: 63.900, test acc: 64.850, test flops: 9.66M
*********************
valid acc: 65.040, test acc: 65.670, test flops: 10.53M
*********************
valid acc: 65.760, test acc: 66.400, test flops: 11.32M
*********************
valid acc: 66.920, test acc: 67.130, test flops: 12.34M
*********************
valid acc: 67.740, test acc: 67.800, test flops: 13.32M
*********************
valid acc: 68.600, test acc: 68.560, test flops: 14.48M
*********************
valid acc: 69.500, test acc: 69.330, test flops: 15.67M
*********************
valid acc: 69.880, test acc: 69.980, test flops: 16.88M
*********************
valid acc: 70.420, test acc: 70.310, test flops: 18.25M
*********************
valid acc: 70.820, test acc: 70.770, test flops: 19.61M
*********************
valid acc: 71.460, test acc: 71.150, test flops: 20.90M
*********************
valid acc: 71.840, test acc: 71.530, test flops: 22.28M
*********************
valid acc: 72.260, test acc: 71.970, test flops: 23.62M
*********************
valid acc: 72.860, test acc: 72.150, test flops: 25.03M
*********************
valid acc: 72.880, test acc: 72.360, test flops: 26.24M
*********************
valid acc: 72.880, test acc: 72.530, test flops: 27.46M
*********************
valid acc: 72.960, test acc: 72.610, test flops: 28.95M
*********************
valid acc: 73.080, test acc: 72.650, test flops: 30.14M
*********************
valid acc: 72.840, test acc: 72.900, test flops: 31.26M
*********************
valid acc: 72.780, test acc: 72.920, test flops: 32.32M
*********************
valid acc: 72.840, test acc: 73.070, test flops: 33.44M
*********************
valid acc: 72.640, test acc: 73.020, test flops: 34.26M
*********************
valid acc: 72.720, test acc: 73.160, test flops: 35.17M
*********************
valid acc: 72.740, test acc: 72.940, test flops: 36.09M
*********************
valid acc: 72.660, test acc: 73.030, test flops: 36.87M
*********************
valid acc: 72.700, test acc: 72.980, test flops: 37.61M
*********************
valid acc: 72.600, test acc: 72.920, test flops: 38.28M
*********************
valid acc: 72.580, test acc: 72.870, test flops: 39.01M
*********************
valid acc: 72.460, test acc: 72.680, test flops: 39.59M
*********************
valid acc: 72.400, test acc: 72.660, test flops: 40.20M
*********************
valid acc: 72.300, test acc: 72.660, test flops: 40.80M
*********************
valid acc: 72.220, test acc: 72.620, test flops: 41.28M
*********************
valid acc: 72.160, test acc: 72.580, test flops: 41.80M
*********************
valid acc: 72.160, test acc: 72.580, test flops: 42.16M

The final [test accuracy, budget] pair is [72.58, 0.4 * 10^8] which is not consistent with the results of Figure 7b (as it should have been [~74, 0.4 * 10^8]).
What are the parameters of the 3 MSDNets used in Figure 7b ?

[kirthifame]

Same here. I feel you get those number only on the original lua implementation.

[fatih-ilhan]

The model in the paper has 11 blocks IIRC.