Is there a training pathway ?
Is there a training pathway for llava models ?
There are a lot of examples of usage of the model : but no training pathways por GGUF pathways !
would you release these Colabs ?
One for training a model ! /merging the lora :
One for Creating the gguf for the model !
they should use the huggingface trainer and librarys : ( not external trainers ) :
thank you !
I have myu own custom model which i merged created but i cannot train it ? or i dont know how to !
i merged my llm into a llava model ! with the latest vision model :
The omodel runs fine but i need to trai n it on my anceint alpha beths !
We have only this notebook currently for training llava-next-video (https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LLaVA-NeXT-Video) and also there is a script on training VLMs with TRL (https://github.com/huggingface/trl/blob/main/examples/scripts/sft_vlm.py). The changes to adapt for llava-video should be minimal.
Also you can always train with the llava-repo and convert weights to HF format later :)
For GGUF I didn't try converting llava-next-video yet. But I know that GGUF currently doesn't have conversion scripts for llava with different vision backbones apart from CLIP and I saw a few issues on adding video support, so prob video is neither supported yet
yes these models are a problem !
as they were copied by mistral and others and the other verisons have bee n implemented and this technically abandoned !
IT really needs to be with the sft trainier ( all of the llavas ! ) ...
thanks for that one !
Right now i have been training my model on base64 instead ! ( textbased images and asking for the description ) as well as giving it the description and asking it for the image )
then the model can be a vision model ( but with text _)
Im currently traing for Functions first : ie convert this to ... image64 / hex /json etc ! Decode-Encode-Decode !
then i will try the images