

TITLE
Error converting tensorflow checkpoints

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# ❓ Questions & Help

I'm trying to convert a BERT Tensorflow checkpoint to hugging face model

## Details
```
!transformers-cli convert \
  --model_type bert \
  --tf_checkpoint C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\model.ckpt-181172 \
  --config C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\bert_config.json \
  --pytorch_dump_output C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\pytorch_model.bin
```

Full traceback: 

> c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Converting TensorFlow checkpoint from C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\model.ckpt-181172
Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_m with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_v with shape [512, 768]
Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_m with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_v with shape [2, 768]
Loading TF weight bert/embeddings/word_embeddings with shape [30522, 768]
Loading TF weight bert/embeddings/word_embeddings/adam_m with shape [30522, 768]
Loading TF weight bert/embeddings/word_embeddings/adam_v with shape [30522, 768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_m with shape [768, 768]
  "vocab_size": 30522
}


Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/pooler/dense/bias with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_m with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_v with shape [768]
Loading TF weight bert/pooler/dense/kernel with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_v with shape [768, 768]
Loading TF weight global_step with shape []
Loading TF weight output_bias with shape [656]
Loading TF weight output_bias/adam_m with shape [656]
Loading TF weight output_bias/adam_v with shape [656]
Loading TF weight output_weights with shape [656, 768]
Loading TF weight output_weights/adam_m with shape [656, 768]
Loading TF weight output_weights/adam_v with shape [656, 768]
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']
Skipping bert/embeddings/LayerNorm/beta/adam_m
Skipping bert/embeddings/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']
Skipping bert/embeddings/LayerNorm/gamma/adam_m
Skipping bert/embeddings/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']
Skipping bert/embeddings/position_embeddings/adam_m
Skipping bert/embeddings/position_embeddings/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'token_type_embeddings']
Skipping bert/embeddings/token_type_embeddings/adam_m
Skipping bert/embeddings/token_type_embeddings/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'word_embeddings']
Skipping bert/embeddings/word_embeddings/adam_m
Skipping bert/embeddings/word_embeddings/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_0/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_0/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_0/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_0/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_0/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_0/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_0/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_0/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_0/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_0/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_0/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_0/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_0/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_0/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_0/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_0/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_0/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_0/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_0/output/dense/bias/adam_m
Skipping bert/encoder/layer_0/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_0/output/dense/kernel/adam_m
Skipping bert/encoder/layer_0/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_1/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_1/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_1/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_1/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_1/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_1/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_1/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_1/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_1/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_1/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_1/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_1/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_1/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_1/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_1/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_1/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_1/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_1/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_1/output/dense/bias/adam_m
Skipping bert/encoder/layer_1/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_1/output/dense/kernel/adam_m
Skipping bert/encoder/layer_1/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_10/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_10/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_10/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_10/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_10/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_10/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_10/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_10/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_10/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_10/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_10/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_10/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_10/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_10/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_10/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_10/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_10/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_10/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_10/output/dense/bias/adam_m
Skipping bert/encoder/layer_10/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_10/output/dense/kernel/adam_m
Skipping bert/encoder/layer_10/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_11/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_11/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_11/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_11/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_11/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_11/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_11/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_11/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_11/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_11/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_11/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_11/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_11/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_11/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_11/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_11/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_11/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_11/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_11/output/dense/bias/adam_m
Skipping bert/encoder/layer_11/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_11/output/dense/kernel/adam_m
Skipping bert/encoder/layer_11/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_2/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_2/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_2/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_2/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_2/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_2/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_2/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_2/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_2/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_2/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_2/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_2/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_2/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_2/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_2/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_2/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_2/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_2/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_2/output/dense/bias/adam_m
Skipping bert/encoder/layer_2/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_2/output/dense/kernel/adam_m
Skipping bert/encoder/layer_2/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_3/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_3/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_3/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_3/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_3/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_3/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_3/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_3/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_3/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_3/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_3/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_3/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_3/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_3/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_3/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_3/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_3/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_3/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_3/output/dense/bias/adam_m
Skipping bert/encoder/layer_3/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_3/output/dense/kernel/adam_m
Skipping bert/encoder/layer_3/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_4/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_4/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_4/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_4/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_4/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_4/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_4/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_4/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_4/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_4/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_4/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_4/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_4/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_4/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_4/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_4/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_4/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_4/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_4/output/dense/bias/adam_m
Skipping bert/encoder/layer_4/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_4/output/dense/kernel/adam_m
Skipping bert/encoder/layer_4/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_5/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_5/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_5/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_5/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_5/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_5/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_5/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_5/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_5/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_5/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_5/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_5/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_5/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_5/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_5/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_5/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_5/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_5/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_5/output/dense/bias/adam_m
Skipping bert/encoder/layer_5/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_5/output/dense/kernel/adam_m
Skipping bert/encoder/layer_5/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_6/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_6/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_6/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_6/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_6/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_6/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_6/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_6/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_6/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_6/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_6/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_6/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_6/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_6/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_6/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_6/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_6/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_6/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_6/output/dense/bias/adam_m
Skipping bert/encoder/layer_6/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_6/output/dense/kernel/adam_m
Skipping bert/encoder/layer_6/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_7/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_7/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_7/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_7/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_7/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_7/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_7/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_7/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_7/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_7/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_7/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_7/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_7/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_7/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_7/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_7/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_7/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_7/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_7/output/dense/bias/adam_m
Skipping bert/encoder/layer_7/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_7/output/dense/kernel/adam_m
Skipping bert/encoder/layer_7/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_8/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_8/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_8/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_8/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_8/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_8/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_8/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_8/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_8/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_8/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_8/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_8/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_8/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_8/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_8/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_8/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_8/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_8/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_8/output/dense/bias/adam_m
Skipping bert/encoder/layer_8/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_8/output/dense/kernel/adam_m
Skipping bert/encoder/layer_8/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_9/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_9/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_9/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_9/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_9/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_9/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_9/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_9/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_9/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_9/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_9/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_9/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_9/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_9/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_9/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_9/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_9/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_9/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_9/output/dense/bias/adam_m
Skipping bert/encoder/layer_9/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_9/output/dense/kernel/adam_m
Skipping bert/encoder/layer_9/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']
Skipping bert/pooler/dense/bias/adam_m
Skipping bert/pooler/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
Skipping bert/pooler/dense/kernel/adam_m
Skipping bert/pooler/dense/kernel/adam_v
Skipping global_step
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\py37cuda10\Scripts\transformers-cli.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\commands\transformers_cli.py", line 33, in main
    service.run()
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\commands\convert.py", line 91, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\modeling_bert.py", line 135, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\torch\nn\modules\module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'BertForPreTraining' object has no attribute 'bias'