RuntimeError: shape mismatch: value tensor of shape [4706, 3072] cannot be broadcast to indexing result of shape [9412, 3072]
Hello. Sorry if the question was asked before.
I have been following the official finetuning https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/vision_finetuning/finetune_hf_trainer_nlvr2.py.
Step by step. With my dataset, but I'm having this error.
ile ~/.cache/huggingface/modules/transformers_modules/microsoft/Phi-3.5-vision-instruct/4a0d683eba9f1d0cbfb6151705d1ee73c25a80ca/modeling_phi3_v.py:241, in Phi3ImageEmbedding.forward(self, input_ids, pixel_values, image_sizes)
237 img_features = self.get_img_features(pixel_values.flatten(0, 1)).reshape(
238 num_images, num_crops, -1, self.image_dim_out
239 )
240 image_features_proj = self.hd_feature_transform(img_features, image_sizes)
--> 241 hidden_states = hidden_states.index_put(
242 positions, image_features_proj, accumulate=False
243 )
245 if self.drop is not None:
246 hidden_states = self.drop(hidden_states)
RuntimeError: shape mismatch: value tensor of shape [4706, 3072] cannot be broadcast to indexing result of shape [9412, 3072]
After days trying to understand the error, I still don't understand. Noted that have been able to train using PEFT without any problem.
Please any assistance will help
Thanks