Update README.md

README.md

@@ -1,41 +1,94 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+paligemma-3b-mix-448-med_30k-ct-brain is based on lightweight google PaliGemma vision-language model (VLM) fine-tuned to perform a Brain CT Image caption task, visual question answering, text reading and object detection.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
 This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** mychen76@gmail.com
+- **Funded by :** N/A
+- **Shared by :** mychen76@gmail.com
+- **Model type:** Visual Language Model
+- **License:** Apache 2.0
+- **Finetuned from model [optional]:** google/paligemma-3b-mix-448
 
 ### Model Sources [optional]
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** TBD
+- **Paper [optional]:** TBD
+- **Demo [optional]:** TBD
+
+## How to Use
+
+paligemma-3b-mix-448-med_30k-ct-brain is a single-turn vision language model not meant for conversational use, and it works best on CT-Brain image caption use case.
+
+Input: Image and text string, such as a prompt to caption the image, or a question.
+Output: Generated text in response to the input, such as a caption of the image, an answer to a question, a list of object bounding box coordinates, or segmentation codewords.
+
+
+Use in Transformers
+The following snippets use model google/paligemma-3b-mix-224 for reference purposes. The model in this repo you are now browsing may have been trained for other tasks, please make sure you use appropriate inputs for the task at hand.
+
+***Running the default precision (float32) on CPU***
+```
+from PIL import Image
+import requests
+import torch
+from transformers import AutoTokenizer, PaliGemmaForConditionalGeneration, PaliGemmaProcessor
+from transformers import AutoProcessor
+
+device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+dtype = torch.bfloat16
+
+## input
+url = "https://huggingface.co/datasets/mychen76/medtrinity_brain_30k_hf/viewer/default/train?row=4&image-viewer=image-62-2B87111BBD996B48DB4C86B0244653FF84B3B8A9"
+image = Image.open(requests.get(url, stream=True).raw)
 
-## Uses
+## load model
+FINETUNED_MODEL_ID="mychen76/paligemma-3b-mix-448-med_30k-ct-brain"
+
+processor = AutoProcessor.from_pretrained(FINETUNED_MODEL_ID)
+model = PaliGemmaForConditionalGeneration.from_pretrained(
+    FINETUNED_MODEL_ID,
+    torch_dtype=dtype,
+    device_map=device
+).eval()
+```
+run inference
+```
+# Instruct the model to create a caption in Spanish
+def run_inference(input_text,input_image, model, processor,max_tokens=1024):
+    inputs = processor(text=input_text, images=input_image,
+                  padding="longest", do_convert_rgb=True, return_tensors="pt").to("cuda")
+    model.to(device)
+    inputs = inputs.to(dtype=model.dtype)
+
+    with torch.no_grad():
+      output = model.generate(**inputs, max_new_tokens=max_tokens,num_beams=3,do_sample=False) 
+
+    pred_text=processor.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
+    return pred_text
+
+
+input_text="caption"
+pred_text = run_inference(input_text,input_image,model, processor)
+print(pred_text)
+```
+result
+```
+The image is a CT scan of the brain, showing various brain structures without the presence of medical devices. The region of interest, located centrally and in the middle of the image, occupies approximately 3.0% of the area and appears to have an abnormal texture or density compared to the surrounding brain tissue, which may indicate a pathological condition. This abnormal area could be related to the surrounding brain structures, potentially affecting them or being affected by a shared pathological process, such as a hemorrhage or a mass effect.
+```
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
 ### Direct Use
 
@@ -57,9 +110,12 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+Most limitations inherited from the underlying Gemma model still apply:
 
-[More Information Needed]
+VLMs are better at tasks that can be framed with clear prompts and instructions. Open-ended or highly complex tasks might be challenging.
+Natural language is inherently complex. VLMs might struggle to grasp subtle nuances, sarcasm, or figurative language.
+VLMs generate responses based on information they learned from their training datasets, but they are not knowledge bases. They may generate incorrect or outdated factual statements.
+VLMs rely on statistical patterns in language and images. They might lack the ability to apply common sense reasoning in certain situations.
 
 ### Recommendations
 
@@ -77,7 +133,10 @@ Use the code below to get started with the model.
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+using dataset: mychen76/medtrinity_brain_30k_hf
+
+a subset of data from UCSC-VLAA/MedTrinity-25M
+
 
 [More Information Needed]
 
@@ -154,7 +213,10 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+PaliGemma is the composition of a Transformer decoder and a Vision Transformer image encoder, with a total of 3 billion params. 
+The text decoder is initialized from Gemma-2B. The image encoder is initialized from SigLIP-So400m/14. 
+aliGemma is trained following the PaLI-3 recipes.
+
 
 ### Compute Infrastructure
 
@@ -172,6 +234,8 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
+UCSC-VLAA/MedTrinity-25M
+
 **BibTeX:**
 
 [More Information Needed]