Missing LM layers.
#1
by
conscell
- opened
Dear model creators,
It seems like this model doesn't include LM layers:
>>> model = AutoModelForMaskedLM.from_pretrained("nlp-waseda/roberta-base-japanese")
Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at nlp-waseda/roberta-base-japanese and are newly initialized: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias']
which makes impossible to do further fine-tuning on domain-specific data.
If possible, could you please include these layers to the base model. Large model doesn't have this problem, but infeasible to use due to GPU memory constraints.
Thank you for pointing this out. The LM head was missing when we changed the value of max_position_embeddings. We added the LM head again. Please try the latest model.
Prof. Kawahara, thank you very much for the update. New model worked perfectly.
conscell
changed discussion status to
closed