facebook

/

seamless-m4t-v2-large

like 668

Follow

AI at Meta 3,337

What errors are you facing? What hardware are you using?

What errors are you facing? What hardware are you using?

Is there a sample code for training and fine-tuning m4t v2?
Because UnitYNART2UModel should be trained
In this case, I face errors such as RuntimeError: mat1 and mat2 shapes cannot be multiplied.
There is no problem with training the version 1 and fine tuning is done and the interesting thing is that after training on the customized data, there is no change in the output of the model and it reads as before.
I am really tired and confused after 3 months of trying

Has anyone taught the v2 model?? And is this possible now? Please explain if you remember

i use a100 or v100 gpu 40gb
and no problem in this part

sample code for do this :

if isinstance(self.model.t2u_model, UnitYNART2UModel): ################## new FT

            unit_decoder_out, decoder_padding_mask, durations = self.model.t2u_model.forward(
                text_decoder_output=text_decoder_out,
                text_decoder_padding_mask=text_decoder_padding_mask,
                text_seqs=seqs,
            )
            print('unit_decoder_out.logits : ', unit_decoder_out.logits)

            unit_logits = self.model.t2u_model.final_proj(unit_decoder_out.logits)
            print('aaaaaaaaaaaaaaaa')
        else:
            raise NotImplementedError("T2U finetuning not implemented ")

Hi, I am new to this field. I want to finetune the v2_large model for some (non-supported) languages to run TTS. Where can I find the documentation for that? The steps mentioned in github creates a .pt file but where can i create the config.json file and others etc. and How can we infer with the finetuned model. Kindly help me in this as I am new to this domain.