Hi Sander !
At first congratulations for winning the contest and thank you for soooo well documented code - you did a fantastic job !
Right now I am reading through your code and there is one thing I do not understand. In layers.py file in all get_update_...._ momentum functions there is a line:
mparam_i = theano.shared(param_i.get_value()*0.)
in which "mparam_i" is set to 0.0. So then in line:
v = momentum * mparam_i - learning_rate * full_grad, so
v = - learning_rate * full_grad
and at the end of this function you update "mparam_i" with current "v" what does not make any sense for me because in next execution "mparam_i" is set to 0.0 again.
Am I missing something ? Please tell me why you set "mparam_i" to 0.0 and what is reason of updating it ?
Thanks in advance !
Hi Thomas,
Thanks for the kind words!
mparam_i is only set to 0.0 once, at initialisation.
Theano is a bit tricky in this respect: the gen_updates_* functions do not actually generate the parameter updates, but rather the symbolic expressions that represent the parameter updates. These expressions are then compiled into executable code by using them in the Theano function 'train'.
The mparam_i are so-called shared variables, which means that their value persists across function calls. Before any updates have been generated, the mparam_i should be set to 0.0. Each subsequent call to the 'train' function will then update the value of mparam_i according to the generated 'updates' list.
Since the gen_updates_* function is only called once, mparam_i is only initialised once and never reset to 0 after that.
I hope this makes sense. Getting into the Theano mindset can take some time because of the additional level of abstraction it introduces. I recommend checking out the documentation as well, it's quite comprehensive.
Hi Sander,
Thanks for a quick response ! Yeah, Theano is sometimes a tricky beast. Now everything is clear for me. Thanks for help.