keras - TensorFlow 2.4.0 - Parameters associated with BatchNorm and Activation

Question

I am printing a tensorflow.keras.Model instance summary. The type is tensorflow.python.keras.engine.functional.Functional object.

This model has layers with activations and batch normalization associated. When I print the list of parameters, I see

• weights
• bias
• 4 items co-dimensional with the bias

These four items are (I guess) the batch normalization and activations.

My question is: why do we have parameters associated with batch normalization and activations? And what could be the other two items?

---

My aim is to transpose this Keras model to a PyTorch counterpart, so I need to know the order of the parameters and what these parameters represent

Answer 1

there are no parameters associated with activations, those are simply some element-wise nonlinear function. So no matter how many activations you have they don't account for any parameter counts. However, your guess is right, there are in fact parameters associated with BatchNorm layer, 2 parameters in each BatchNorm layer to be precise (lambda and beta). So those BatchNorm layer does add additional parameters in your network.