Can't overfit on a single piece of training data

[Chris-Pedersen]

I would expect the loss to converge really fast and overfit when running the model (either camels CNN or a scattering network) on a single training point, but for some reason this doesn't happen. Will likely point to a bug in the model somewhere.

[Chris-Pedersen]

Ok still can't get this to work somehow. I've tried:

1. Running our current models when predicting just the value of the parameters, not the variance as in Niall's moment networks paper. The idea here was that if the predictions got very accurate, the variance would then become arbitrarily small and difficult to predict, disrupting the backprop and preventing the overfit. However even when I modify these networks to just predict the 6 parameter values, the loss still doesn't converge on a single map after 2000 epochs.
2. Running simpler networks. The CAMELs CNN is fairly deep and involves a lot of batchnorm so perhaps this is interfering with the overfit. When I try using a much shallower network however, we still fail to see convergence of the loss over a similar number of epochs.

So I'm beginning to think there's some bug somewhere, why wouldn't the network immeditately overfit just 6 parameters on 1 image?