README.md

---
library_name: transformers
license: apache-2.0
---

# Model Card for Model ID

paligemma-3b-mix-448-med_30k-ct-brain is based on lightweight google PaliGemma vision-language model (VLM) fine-tuned to perform a Brain CT Image caption task, visual question answering, text reading and object detection.

## Model Details

### Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** mychen76@gmail.com
- **Funded by :** N/A
- **Shared by :** mychen76@gmail.com
- **Model type:** Visual Language Model
- **License:** Apache 2.0
- **Finetuned from model [optional]:** google/paligemma-3b-mix-448

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** TBD
- **Paper [optional]:** TBD
- **Demo [optional]:** TBD

## How to Use

paligemma-3b-mix-448-med_30k-ct-brain is a single-turn vision language model not meant for conversational use, and it works best on CT-Brain image caption use case.

Input: Image and text string, such as a prompt to caption the image, or a question.
Output: Generated text in response to the input, such as a caption of the image, an answer to a question, a list of object bounding box coordinates, or segmentation codewords.


Use in Transformers
The following snippets use model google/paligemma-3b-mix-224 for reference purposes. The model in this repo you are now browsing may have been trained for other tasks, please make sure you use appropriate inputs for the task at hand.

***Running the default precision (float32) on CPU***
```
from PIL import Image
import requests
import torch
from transformers import AutoTokenizer, PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from transformers import AutoProcessor

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16
```

***input***
```
url = "https://huggingface.co/datasets/mychen76/medtrinity_brain_30k_hf/viewer/default/train?row=4&image-viewer=image-62-2B87111BBD996B48DB4C86B0244653FF84B3B8A9"
image = Image.open(requests.get(url, stream=True).raw)
```

***load model***
```
FINETUNED_MODEL_ID="mychen76/paligemma-3b-mix-448-med_30k-ct-brain"

processor = AutoProcessor.from_pretrained(FINETUNED_MODEL_ID)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    FINETUNED_MODEL_ID,
    torch_dtype=dtype,
    device_map=device
).eval()
```
***run inference***
```
# Instruct the model to create a caption in Spanish
def run_inference(input_text,input_image, model, processor,max_tokens=1024):
    inputs = processor(text=input_text, images=input_image,
                  padding="longest", do_convert_rgb=True, return_tensors="pt").to("cuda")
    model.to(device)
    inputs = inputs.to(dtype=model.dtype)

    with torch.no_grad():
      output = model.generate(**inputs, max_new_tokens=max_tokens,num_beams=3,do_sample=False) 

    pred_text=processor.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return pred_text


input_text="caption"
pred_text = run_inference(input_text,input_image,model, processor)
print(pred_text)
```
***result***
```
The image is a CT scan of the brain, showing various brain structures without the presence of medical devices. The region of interest, located centrally and in the middle of the image, occupies approximately 3.0% of the area and appears to have an abnormal texture or density compared to the surrounding brain tissue, which may indicate a pathological condition. This abnormal area could be related to the surrounding brain structures, potentially affecting them or being affected by a shared pathological process, such as a hemorrhage or a mass effect.
```

***Running on CUDA***

```
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests
import torch

FINETUNED_MODEL_ID="mychen76/paligemma-3b-mix-448-med_30k-ct-brain"
device = "cuda:0"
dtype = torch.bfloat16

url = "https://huggingface.co/datasets/mychen76/medtrinity_brain_30k_hf/viewer/default/train?row=4&image-viewer=image-62-2B87111BBD996B48DB4C86B0244653FF84B3B8A9"
image = Image.open(requests.get(url, stream=True).raw)

model = PaliGemmaForConditionalGeneration.from_pretrained(
    FINETUNED_MODEL_ID,
    torch_dtype=dtype,
    device_map=device,
    revision="bfloat16",
).eval()
processor = AutoProcessor.from_pretrained(FINETUNED_MODEL_ID)

# Instruct the model to create a caption in Spanish
prompt = "caption es"
model_inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
input_len = model_inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]
    decoded = processor.decode(generation, skip_special_tokens=True)
    print(decoded)
```


### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

## Bias, Risks, and Limitations

Most limitations inherited from the underlying Gemma model still apply:

VLMs are better at tasks that can be framed with clear prompts and instructions. Open-ended or highly complex tasks might be challenging.
Natural language is inherently complex. VLMs might struggle to grasp subtle nuances, sarcasm, or figurative language.
VLMs generate responses based on information they learned from their training datasets, but they are not knowledge bases. They may generate incorrect or outdated factual statements.
VLMs rely on statistical patterns in language and images. They might lack the ability to apply common sense reasoning in certain situations.

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

## Training Details

### Training Data

using dataset: https://huggingface.co/datasets/mychen76/medtrinity_brain_30k_hf

Note:
mychen76/medtrinity_brain_30k_hf is a subset of data from UCSC-VLAA/MedTrinity-25M


[More Information Needed]

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

PaliGemma is the composition of a Transformer decoder and a Vision Transformer image encoder, with a total of 3 billion params. 
The text decoder is initialized from Gemma-2B. The image encoder is initialized from SigLIP-So400m/14. 
aliGemma is trained following the PaLI-3 recipes.


### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

UCSC-VLAA/MedTrinity-25M

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]