

TITLE
Why BertSelfAttention reshape Q,K,V from 3-D tensor to 4-D tensor

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🌟 New model addition

## Model description
https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py
def transpose_for_scores(self, x):
        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
        x = x.view(*new_x_shape)
        return x.permute(0, 2, 1, 3)
query_layer = self.transpose_for_scores(mixed_query_layer)
 key_layer = self.transpose_for_scores(mixed_key_layer)
 value_layer = self.transpose_for_scores(mixed_value_layer)

## Open source status
Question:  
1. Why we must transpose Q,K,V from 3-D tensor to 4-D tensor ?
2. What if we just use 3-D Q,K,V to do torch.matmul ?
