Layer factory function to create a function that implements layer normalization.

LayerNormalization(initial_scale = 1, initial_bias = 0, epsilon = 1e-05,
  name = "")

Arguments

epsilon

(float, default 0.00001) - added to avoid division by 0

name

string (optional) the name of the Function instance in the network

init

(scalar or matrix or initializer, defaults to init_glorot_uniform()) – initial value of weights Wial_scale

init_bias

(scalar or matrix or initializer, defaults to 0) – initial value of weights b

Details

Layer normalization applies this formula to every input element (element-wise): y = (x - mean(x)) / (stddev(x) + epsilon) * scale + bias where scale and bias are learned scalar parameters.