Colpali Rerank Model
I’m working with the ColPaliForSequenceClassification that i made by myself, My focus is on fine-tuning only the classifier component to create a reranker model.
However, I’m unsure of the best way to adapt this model for reranking tasks. Are there any examples or best practices for configuring this model to function as a rerank model?
If anyone has experience with this or can provide guidance, I’d greatly appreciate it!
Thanks in advance!
Hey ! Cool concept, I guess your interest lies in cross-encoders. In your case, you only really need the base model that takes both image and query as an inpuyt and add a classification head on top of it after a forward pass!
Only reason to go through ColPali is to leverage the contrastive pretraining. In which case you can modify the image processor function to give both inputs and query, and then add a pooling layer + a classification head in place of the projection layer you have in colpali :)