Update README.md

README.md

@@ -4,6 +4,7 @@ tags:
 - prompt
 ---
 
+## Sharing special prompts of open-weight models
 
 This repo illustrates how you can use the `hf_hub_prompts` library to load prompts from YAML files in open-weight model repositories.
 Several open-weight models have been tuned on specific tasks with specific prompts. 
@@ -12,9 +13,13 @@ To elicit this capability, users need to use this special prompt: `Please provid
 
 These these kinds of task-specific special prompts are currently unsystematically reported in model cards, github repos, .txt files etc. 
 
-The hf_hub_prompts library standardises the sharing of prompts in YAML files. 
+The hf_hub_prompts library standardises the sharing of prompts in YAML files.
+I recommend sharing these these special prompts directly in the model repository of the respective. 
 
+Below is an example for the InternVL2 model. 
 
+
+#### Prompt for extracting bounding boxes of specific objects of interest with InternVL2
 ```py
 #!pip install hf_hub_prompts
 from hf_hub_prompts import download_prompt
@@ -36,6 +41,25 @@ print(messages)
 #      }]
 ```
 
+#### Prompt for extracting bounding boxes of any object in an image with InternVL2
+```py
+# download image prompt template
+prompt_template = download_prompt(repo_id="MoritzLaurer/open_models_special_prompts", filename="internvl2-objectdetection-prompt.yaml")
+
+# populate prompt
+image_url = "https://unsplash.com/photos/ZVw3HmHRhv0/download?ixid=M3wxMjA3fDB8MXxhbGx8NHx8fHx8fDJ8fDE3MjQ1NjAzNjl8&force=true&w=1920"
+messages = prompt_template.format_messages(image_url=image_url, client="openai")
+
+print(messages)
+# [{'role': 'user',
+#  'content': [{'type': 'image_url',
+#    'image_url': {'url': 'https://unsplash.com/photos/ZVw3HmHRhv0/download?ixid=M3wxMjA3fDB8MXxhbGx8NHx8fHx8fDJ8fDE3MjQ1NjAzNjl8&force=true&w=1920'}},
+#   {'type': 'text',
+#    'text': 'Please detect and label all objects in the following image and mark their positions.'}]}]
+```
+
+#### Using the prompt with an open inference container like vLLM or TGI
+
 These populated prompts in the OpenAI messages format are then directly compatible with vLLM or TGI containers. 
 When you host one of these containers on a HF Endpoint, for example, you can call on the model with the OpenAI client or with the HF Interence Client.