Use with Qdrant
Hi, I'm new to embedding and vector search. I don't know if this question related to this community, if not please forgive me.
When I use the example code with softmax, the score is correct, but when I try to use this model with Qdrant consine similarity search, the score is very weird. I tried normalize, reshape... but not working. Can this model work with Qdrant?
Example code:
device = "mps" if torch.backends.mps.is_available() else "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained("Marqo/marqo-fashionCLIP").to(device)
processor = AutoProcessor.from_pretrained("Marqo/marqo-fashionCLIP", trust_remote_code=True)
await qdrant_client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={
"thumbnail": models.VectorParams(
size=thumbnail_vector_dimension,
distance=models.Distance.COSINE,
on_disk=True
)
}
)
image_embedding = processor(
images=image,
return_tensors="pt"
).to(device)
with torch.no_grad():
image_vector = model.get_image_features(**image_embedding)
image_vector = image_vector.tolist()[0]
response = await qdrant_client.query_points(
collection_name=COLLECTION_NAME,
query=image_vector,
using="thumbnail",
limit=100,
with_payload=True
)
for point in response.points:
print(point.score)
Result:
-0.022568434
-0.025038993
-0.025038993
-0.025299015
-0.025954556
-0.026022851
-0.026900368
-0.028346939
Thank you for your time.
Hi
@thanhtung2693
,
Thank you for your message. We're not familiar with Qdrant's set up but you can achieve the same thing using Marqo.
Marqo requires Docker. To install Docker go to the Docker Official website. Ensure that docker has at least 8GB memory and 50GB storage. In Docker desktop, you can do this by clicking the settings icon, then resources, and selecting 8GB memory.
Use docker to run Marqo:
docker rm -f marqo
docker pull marqoai/marqo:latest
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
- pip install marqo:
pip install marqo
- Create a Marqo index using the
marqo-fashionSigLIPmodel, add an image with a description, and perform a text search to retrieve similar images based on the description:
import marqo
from transformers import AutoModel, AutoProcessor
import torch
from PIL import Image
# Initialize Marqo client
mq = marqo.Client(url='http://localhost:8882')
# Define your index name
INDEX_NAME = "fashion-index"
# Specify your image url - replace this with your desired image url
image_url = "https://www.samuel-windsor.co.uk/cdn/shop/products/BV03_SIDE_LR_03519359-69ff-4cd5-a47a-45cb537e2318.jpg?v=1647959877"
# Settings for the Marqo index. This loads marqo-fashionSigLIP model
settings = {
"treatUrlsAndPointersAsImages": True,
"model": "marqo-fashion-siglip-custom-load",
"modelProperties": {
"name": "hf-hub:Marqo/marqo-fashionSigLIP",
"dimensions": 768,
"type": "open_clip",
},
"normalizeEmbeddings": True,
}
# Delete the index if it already exists
try:
mq.create_index(index_name=INDEX_NAME, settings_dict=settings)
except:
pass
# Add document to Marqo index
mq.index(INDEX_NAME).add_documents(
documents=[
{
"_id": "image_doc_1", # Change this for each unique document
"image_url": image_url, # Change this to an appropriate
"description": "smart black shoe", # Add a relevant description for the image
}
# add more documents here
],
# Fields that will be vectorised
tensor_fields=["image_url"]
)
# Now search the index using a text query to find similar images
search_result = mq.index(INDEX_NAME).search("smart black shoe")
# Output the scores of search results
for hit in search_result["hits"]:
print(hit["_score"])
Hope that helps! Any issues, please let me know!
Hi @elliesleightholm really appreceiate your help, Marqo is great for my usecases, but I have few questions:
- Instead of using Qdrant I will need to stick with Vespa in order to use Marqo right?
- And any support for Apple Metal in the future?
Many thanks.