

TITLE
Why use 'BertLayerNorm'  instead of torch.nn.LayerNorm ?

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🌟 New model addition
What's the difference between 'BertLayerNorm' and  torch.nn.LayerNorm
## Model description  
1.pytorch    torch.nn.LayerNorm
https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html?highlight=layernorm#torch.nn.LayerNorm

2.modeling.py
class BertLayerNorm(Module):
    def __init__(self, hidden_size, eps=1e-12):
        super(BertLayerNorm, self).__init__()
        self.shape = torch.Size((hidden_size,))
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.bias = nn.Parameter(torch.zeros(hidden_size))
        self.apex_enabled = APEX_IS_AVAILABLE
             
    @torch.jit.unused
    def fused_layer_norm(self, x):
        return FusedLayerNormAffineFunction.apply(
                    x, self.weight, self.bias, self.shape, self.eps)
             
             
    def forward(self, x):
        if self.apex_enabled and not torch.jit.is_scripting():
            x = self.fused_layer_norm(x)                            
        else:
            u = x.mean(-1, keepdim=True)
            s = (x - u).pow(2).mean(-1, keepdim=True)
            x = (x - u) / torch.sqrt(s + self.eps)
            x = self.weight * x + self.bias
        return x


<!-- Important information -->
It seems like torch.nn.LayerNorm has the same function of  belows ops in BertLayerNorm
 u = x.mean(-1, keepdim=True)
 s = (x - u).pow(2).mean(-1, keepdim=True)
 x = (x - u) / torch.sqrt(s + self.eps)
 x = self.weight * x + self.bias

Why we don't use  torch.nn.LayerNorm ?

Thanks a lot for answering my question


## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)
