This paper has two main contributions :

  • they show that replacing MaxPooling with convolution layers of stride > 1 gives better results
  • They proposed a guided backpropagation method to visualize the influence of each neuron.

Replacing MaxPool with ConvLayers of stride > 1

They took 3 basic architectures (Model A, B and C)

and tested 3 different modifications :

Results show that ALL-CNN + model C give best results againts other configurations

while being very competitive againts other similar approaches

Guided backpropagation

The idea is to use a double relu while backpropagating gradients.