fc9_1 weights is increase with the train iterations from 0.0x to 40.x

[zhouyongxiu]

@luhaofang Hi, when I use the code to train the fc9_1, I find that the fc9_1 weights is increase with the train iterations, after train 40000, it increase from 0.0x to 40.x, do you know what to lead this problem, and how to deal it.

[gombru]

Hi,
I had a similar problem. Weights increased while the net was learning until at some point I got NaNs. I solved it using a smaller margin.

Though still I haven't analyzed it deeply, I think that the reason is that when the margin is too big compared to the magnitudes of the vectors you are comparing (in my case they were L2 normalized) the net cannot achieve to produce those big distances and weights keep going up.