### Introduction

MorphNet is an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. The proposed method has 3 advantages :

• it is scalable to large models and large datasets;
• it can optimize a DNN structure targeting a specific resource, such as FLOPs per inference, while allowing the usage of untargeted resources such as model size (number of parameters), to grow;
• it can learn a structure that improves performance while reducing the targeted resource usage

### Method

Let $$O_L$$ be the output wight (number of output feature maps) or layer L and $$O_{1:M}$$ the total number of feature maps between the first and last layer (M is the index of the last layer). They formalized their problem as follows:

where the goal is to find the most accurate model (with the lowest loss) but which respect a limited resource. For example here, the total number of flops must be below $$\zeta$$. They then reformulate the problem as

where $$g(\theta)$$ is a sparsity regularizer which depends on $$F(O_{1:M} )$$. Since their method has no guarantee to satisfy $$F(O_{1:M})$$ < $$\zeta$$, they proposed Algo 1 where step 1 and 2 are compression steps and step 3 is an expansion step.

### The regularizer

The regularizer is given by the following formula:

where

and $$f,g$$ is the spatial dimension of the filter at layer L and $$x,z$$ the spatial dimension of the feature maps of layer L. Also, $$A_{L,i}$$ ($$B_{L,j}$$) is an indicator function which equals one if the i-th input (j-th output) of layer L is alive – not zeroed out.

### Results

And … the method works!