mila-udem / blocks / 3374

93%

push

travis-ci

Use tensor.nnet.relu for ReLU calculations. (#1070)

* Use tensor.nnet.relu for ReLU calculations.

We punt the decision of how to implement max(0, x) or max(alpha * x, x) most efficiently to Theano, via `tensor.nnet.relu`. This is what Lasagne and Keras do, which between them have > 10x as many GitHub stars, so I am willing to treat this solution as "battle-tested".

(Finally) closes #960.

N.B. This was done via a quick edit on the GitHub web interface so the branch is on `mila-udem`, I'll delete it when merged.

* Fix LeakyRectifier test sensitivity.

Tests were failing with this new implementation due to an unduly high sensitivity.

2 of 2 new or added lines in 1 file covered. (100.0%)

4169 of 4440 relevant lines covered (93.9%)

2.79 hits per line