Update README, Add result images, app.py changes

.ipynb_checkpoints/README-checkpoint.md

@@ -0,0 +1,81 @@
+---
+title: Person Thumbs Up
+emoji: 🐠
+colorFrom: blue
+colorTo: purple
+sdk: streamlit
+sdk_version: 1.21.0
+app_file: app.py
+pinned: false
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Stable diffusion finetune using LoRA
+
+## HuggingFace Spaces URL: https://huggingface.co/spaces/asrimanth/person-thumbs-up
+
+## Approach
+
+**The key resource in this endeavor: https://huggingface.co/blog/lora**
+
+### Training
+
+All of the following models were trained on stable-diffusion-v1-5
+
++ Several different training strategies and found LoRA to be the best for my needs.
++ In the dataset, the thumbs up dataset had 121 images for training, which I found to be adequate.
++ First, I scraped ~50 images of "sachin tendulkar". This experiment failed, since the model gave a player with cricket helmet.
++ For training on "Tom cruise", I've scraped ~100 images from images.google.com, using the javascript code from pyimagesearch.com
++ For training on "srimanth", I've put 50 images of myself.
+
+For the datasets, I started as follows:
++ Use an image captioning model from HuggingFace - In our case it is the `Salesforce/blip-image-captioning-large` model.
++ Once captioned, If the caption has "thumbs up", we replace it with `#thumbsup`, otherwise we attach the word `#thumbsup` to the caption.
++ If the model recognizes the person or says the word "man", we replace it with `<person>`. Otherwise, we attach the word `<person>` to the caption.
++ No-cap dataset: For the no-cap models, we don't use the captioning models. We simply add the `<person>` and the `#thumbsup` tag.
++ Plain dataset: For the plain models, we leave the words as is.
+
+The wandb dashboard for the models are as follows:
+Initial experiments: I've tried training only on the thumbs up first. The results were good. The thumbs up was mostly accurate, with 4 fingers folded and the thumb raised. However, the model trained on sachin had several issues, including occlusion by cricket gear.
+I've tried several different learning rates (from 1e-4 to 1e-6 with cosine scheduler), but the loss curve did not change much.
+Number of epochs : 50-60
+Augmentations used : Center crop, Random Flip
+Gradient accumulation steps : Tried 1, 3, and 4 for different experiments. 4 gave decent results.
+
+text2image_fine-tune wandb dashboard:
+**https://wandb.ai/asrimanth/text2image_fine-tune**
+**Model card for asrimanth/person-thumbs-up-lora: https://huggingface.co/asrimanth/person-thumbs-up-lora**
+**Prompt: ```<tom_cruise> #thumbsup```**
+
+Deployed models:
+
+When the above experiment failed, I had to try different datasets. One of them was "tom cruise".
+
+srimanth-thumbs-up-lora-plain wandb dashboard: We use the plain dataset with srimanth mentioned above.
+**wandb link: https://wandb.ai/asrimanth/srimanth-thumbs-up-lora-plain**
+**Model card for srimanth-thumbs-up-lora-plain: https://huggingface.co/asrimanth/srimanth-thumbs-up-lora-plain**
+**Prompt: ```srimanth thumbs up```**
+
+person-thumbs-up-plain-lora wandb dashboard:
+**wandb link: https://wandb.ai/asrimanth/person-thumbs-up-plain-lora**
+**Model card for asrimanth/person-thumbs-up-plain-lora: https://huggingface.co/asrimanth/person-thumbs-up-plain-lora**
+**Prompt: ```tom cruise thumbs up```**
+
+person-thumbs-up-lora-no-cap wandb dashboard:
+**https://wandb.ai/asrimanth/person-thumbs-up-lora-no-cap**
+**Model card for asrimanth/person-thumbs-up-lora-no-cap: https://huggingface.co/asrimanth/person-thumbs-up-lora-no-cap**
+**Prompt: ```<tom_cruise> #thumbsup```**
+
+### Inference
+
++ Inference works best for 25 steps in the pipeline.
++ Since the huggingface space built by Streamlit is slow due to low compute, please perform local inference using GPU.
++ During local inference (25 steps), I found the person-thumbs-up-plain-lora to show 35 out of 50 images with a decent thumbs up result for tom cruise, 5 incomplete thumbs up.
++ While I could not evaluate the model with metrics due to insufficient time, I chose the visual approach. To view the inference images, check the `results` folder.
++ To evaulate diffusion models, I would start with this: https://huggingface.co/docs/diffusers/conceptual/evaluation
+
+### Deployment
+
++ I chose streamlit to deploy the application on HuggingFace spaces. It was developer friendly and the app logic can be found in app.py
++ Streamlit app would be a great choice for an MVP.

.ipynb_checkpoints/app-checkpoint.py

@@ -3,7 +3,8 @@ import torch
 from huggingface_hub import model_info
 from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
 
-def inference(prompt, model, n_images, seed):
+def inference(prompt, model, n_images, seed, n_inference_steps):
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
     # Load the model
     info = model_info(model)
     model_base = info.cardData["base_model"]
@@ -11,6 +12,7 @@ def inference(prompt, model, n_images, seed):
     pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
 
     pipe.unet.load_attn_procs(model)
+    pipe.to(device)
 
     # Load the UI components for progress bar and image grid
     progress_bar_ui = st.empty()
@@ -24,7 +26,7 @@ def inference(prompt, model, n_images, seed):
     print(f"Inferencing '{prompt}' for {n_images} images.")
 
     for i in range(n_images):
-        result = pipe(prompt, generator=generators[i], num_inference_steps=9).images[0]
+        result = pipe(prompt, generator=generators[i], num_inference_steps=n_inference_steps).images[0]
         result_images.append(result)
 
         # Start with empty UI elements
@@ -44,10 +46,10 @@ def inference(prompt, model, n_images, seed):
                     st.image(result_images[i], caption=f"Image - {i+1}")
             with col2:
                 for i in range(1, len(result_images), 3):
-                    st.image(result_images[i], caption=f"Image - {i+2}")
+                    st.image(result_images[i], caption=f"Image - {i+1}")
             with col3:
                 for i in range(2, len(result_images), 3):
-                    st.image(result_images[i], caption=f"Image - {i+3}")
+                    st.image(result_images[i], caption=f"Image - {i+1}")
 
 
 if __name__ == "__main__":
@@ -55,15 +57,24 @@ if __name__ == "__main__":
     st.title("Finetune LoRA inference")
 
     with st.form(key='form_parameters'):
-        prompt = st.text_input("Enter the prompt: ")
-        model_options = ["asrimanth/person-thumbs-up-plain-lora", "asrimanth/person-thumbs-up-lora", "asrimanth/person-thumbs-up-lora-no-cap"]
+        model_options = [
+            "asrimanth/person-thumbs-up-plain-lora : Tom Cruise thumbs up",
+            "asrimanth/srimanth-thumbs-up-lora-plain : srimanth thumbs up",
+            "asrimanth/person-thumbs-up-lora : <tom_cruise> #thumbsup",
+            "asrimanth/person-thumbs-up-lora-no-cap : <tom_cruise> #thumbsup",
+        ]
         current_model = st.selectbox("Choose a model", options=model_options)
-        col1_inp, col2_inp = st.columns(2)
+        model, default_prompt = current_model.split(" : ")
+        prompt = st.text_input("Enter the prompt: ", value=default_prompt)
+        current_model = current_model.split(" : ")[0]
+        col1_inp, col2_inp, col_3_inp = st.columns(3)
         with col1_inp:
-            n_images = int(st.number_input("Enter the number of images", min_value=0, max_value=50))
+            n_images = int(st.number_input("Enter the number of images", value=3, min_value=0, max_value=50))
         with col2_inp:
+            n_inference_steps = int(st.number_input("Enter the number of inference steps", value=3, min_value=0))
+        with col_3_inp:
             seed_input = int(st.number_input("Enter the seed (default=25)", value=25, min_value=0))
         submitted = st.form_submit_button("Predict")
 
     if submitted: # The form is submitted
-        inference(prompt, current_model, n_images, seed_input)
+        inference(prompt, model, n_images, seed_input, n_inference_steps)

.ipynb_checkpoints/finetune_lora_srimanth_plain-checkpoint.sh

@@ -0,0 +1,22 @@
+export MODEL_NAME="runwayml/stable-diffusion-v1-5"
+export TRAIN_DIR="/l/vision/v5/sragas/easel_ai/thumbs_up_srimanth_plain/"
+export OUTPUT_DIR="/l/vision/v5/sragas/easel_ai/models_srimanth_plain/"
+export HUB_MODEL_ID="srimanth-thumbs-up-lora-plain"
+
+accelerate launch --mixed_precision="fp16"  train_text_to_image_lora.py \
+  --pretrained_model_name_or_path=$MODEL_NAME \
+  --train_data_dir=$TRAIN_DIR \
+  --resolution=512 --center_crop --random_flip \
+  --train_batch_size=2 \
+  --gradient_accumulation_steps=4 \
+  --num_train_epochs=300 \
+  --learning_rate=1e-5 \
+  --max_grad_norm=1 \
+  --lr_scheduler="cosine" --lr_warmup_steps=500 \
+  --output_dir=${OUTPUT_DIR} \
+  --checkpointing_steps=500 \
+  --report_to=wandb \
+  --validation_prompt="srimanth thumbs up" \
+  --seed=15 \
+  --push_to_hub \
+  --hub_model_id=${HUB_MODEL_ID}

.ipynb_checkpoints/hf_dataset_plain-checkpoint.py

@@ -0,0 +1,63 @@
+import os
+import requests
+import random
+from PIL import Image
+import torch
+from transformers import BlipProcessor, BlipForConditionalGeneration
+from tqdm import tqdm
+import pandas as pd
+
+def caption_images(image_paths, processor, model, folder):
+    image_captions_dict = []
+    for img_path in tqdm(image_paths):
+        pil_image = Image.open(img_path).convert('RGB')
+        image_name = img_path.split("/")[-1]
+        # unconditional image captioning
+        inputs = processor(pil_image, return_tensors="pt").to("cuda")
+        out = model.generate(**inputs)
+        out_caption = processor.decode(out[0], skip_special_tokens=True)
+
+        if folder=="images/" and "thumbs up" not in out_caption:
+            th_choice = random.choice([True, False])
+            out_caption = "thumbs up " + out_caption if th_choice else out_caption + " thumbs up"
+        elif folder=="tom_cruise_dataset/":
+            if "man" in out_caption:
+                out_caption = out_caption.replace("man", "tom cruise")
+            elif "person" in out_caption:
+                out_caption = out_caption.replace("person", "tom cruise")
+            elif "tom cruise" not in out_caption:
+                out_caption = "tom_cruise " + out_caption
+
+        # For some reason, the model puts the word "arafed" for a human
+        if "arafed" in out_caption:
+            out_caption = out_caption.replace("arafed ", "")
+
+        image_captions_dict.append({"file_name": folder+image_name, "text": out_caption})
+    return image_captions_dict
+
+
+def create_thumbs_up_person_dataset(path, cache_dir="/l/vision/v5/sragas/hf_models/"):
+    random.seed(15)
+    image_captions_dict = []
+
+    processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
+    model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large",
+                                                         cache_dir=cache_dir,
+                                                         torch_dtype=torch.float32).to("cuda")
+
+    # Caption the thumbs up images for prompts
+    image_paths = [path + "images/" + file for file in os.listdir(path+"images/")]
+    # Read from the person dataset
+    person_paths = [path + "tom_cruise_dataset/" + file for file in sorted(os.listdir(path+"tom_cruise_dataset/"))]
+
+    image_captions_dict.extend(caption_images(person_paths, processor, model, "tom_cruise_dataset/"))
+    image_captions_dict.extend(caption_images(image_paths, processor, model, "images/"))
+
+    image_captions_dict = pd.DataFrame(image_captions_dict)
+    image_captions_dict.to_csv(f"{path}metadata.csv", index=False)
+    image_captions_dict.to_csv(f"metadata_plain.csv", index=False)
+
+
+if __name__ == "__main__":
+    images_dir = "/l/vision/v5/sragas/easel_ai/thumbs_up_plain_dataset/"
+    create_thumbs_up_person_dataset(images_dir)

.ipynb_checkpoints/inference-checkpoint.py

@@ -0,0 +1,45 @@
+import os
+from huggingface_hub import model_info
+
+import torch
+from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
+
+
+def main():
+    REPOS = {
+        "tom_cruise_plain": {"hub_model_id": "asrimanth/person-thumbs-up-plain-lora", "model_dir": "/l/vision/v5/sragas/easel_ai/models_plain/"},
+        "tom_cruise": {"hub_model_id": "asrimanth/person-thumbs-up-lora", "model_dir": "/l/vision/v5/sragas/easel_ai/models/"},
+        "tom_cruise_no_cap": {"hub_model_id": "asrimanth/person-thumbs-up-lora-no-cap", "model_dir": "/l/vision/v5/sragas/easel_ai/models_no_cap/"},
+        "srimanth_plain": {"hub_model_id": "asrimanth/srimanth-thumbs-up-lora-plain", "model_dir": "/l/vision/v5/sragas/easel_ai/models_srimanth_plain/"}
+    }
+    N_IMAGES = 50
+    current_repo_id = "tom_cruise_no_cap"
+
+    SAVE_DIR = f"./results/{current_repo_id}/"
+    os.makedirs(SAVE_DIR, exist_ok=True)
+
+    current_repo = REPOS[current_repo_id]
+
+    print(f"{'-'*20} CURRENT REPO: {current_repo_id} {'-'*20}")
+    hub_model_id = current_repo["hub_model_id"]
+    model_dir = current_repo["model_dir"]
+
+    info = model_info(hub_model_id)
+    model_base = info.cardData["base_model"]
+    print(f"Base model is: {model_base}")
+
+    pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16, cache_dir="/l/vision/v5/sragas/hf_models/")
+    pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
+
+    pipe.unet.load_attn_procs(hub_model_id)
+    pipe.to("cuda")
+
+    generators = [torch.Generator("cuda").manual_seed(i) for i in range(N_IMAGES)]
+    prompt = "<tom_cruise> showing #thumbsup"
+    print(f"Inferencing '{prompt}' for {N_IMAGES} images.")
+    for i in range(N_IMAGES):
+        image = pipe(prompt, generator=generators[i], num_inference_steps=25).images[0]
+        image.save(f"{SAVE_DIR}out_{i}.png")
+
+if __name__ == "__main__":
+    main()

.ipynb_checkpoints/metadata_srimanth-checkpoint.csv

@@ -0,0 +1,172 @@
+file_name,text
+srimanth_dataset/00001.jpg,there is a <srimanth> riding a motorcycle down a street on a sunny day
+srimanth_dataset/00002.jpg,there is a <srimanth> riding a motorcycle down a street on a sidewalk
+srimanth_dataset/00003.jpg,smiling <srimanth> in blue suit standing in front of a brick building
+srimanth_dataset/00004.jpg,<srimanth> in a blue suit standing on a roof
+srimanth_dataset/00005.jpeg,there is a <srimanth> sitting in a chair in a lobby
+srimanth_dataset/00005.jpg,<srimanth> in a blue suit standing in front of a building
+srimanth_dataset/00006.jpeg,<srimanth> sitting on a bench in front of a building
+srimanth_dataset/00007.jpeg,<srimanth> in a blue suit and red tie holding a red folder
+srimanth_dataset/00008.jpeg,there is a <srimanth> standing on the beach with his feet in the water
+srimanth_dataset/00009.jpeg,there is a <srimanth> sitting on a plane with ear buds in his ears
+srimanth_dataset/00010.jpeg,<srimanth> in a blue shirt sitting on a railing with a mountain in the background
+srimanth_dataset/00011.jpeg,smiling <srimanth> sitting at a table with a glass of wine
+srimanth_dataset/00012.jpeg,there is a <srimanth> and a wo<srimanth> standing together in a park
+srimanth_dataset/00013.jpeg,<srimanth> in a plaid shirt standing on a balcony
+srimanth_dataset/00014.jpeg,smiling <srimanth> in denim jacket and black shirt posing for a picture
+srimanth_dataset/00015.jpeg,there is a <srimanth> in a blue shirt posing for a picture
+srimanth_dataset/20220312_173528.jpg,<srimanth> standing on a bridge with a skateboard in his hand
+srimanth_dataset/20220313_142732.jpg,<srimanth> in a jacket standing in front of a building
+srimanth_dataset/20220313_151238.jpg,<srimanth> standing on a pier looking at the water
+srimanth_dataset/20220313_195903.jpg,<srimanth> standing in front of a large window with a city view
+srimanth_dataset/20220314_114256.jpg,smiling <srimanth> in black jacket with a blue shirt and black jacket
+srimanth_dataset/20220314_143858.jpg,<srimanth> leaning on a wall in front of a city skyline
+srimanth_dataset/20220315_181555.jpg,<srimanth> in a blue and white jacket standing in front of a body of water
+srimanth_dataset/20220316_170830_portrait (1).jpg,<srimanth> in a black jacket standing in front of a boat
+srimanth_dataset/20220316_170830_portrait.jpg,<srimanth> standing in front of a boat in the ocean
+srimanth_dataset/20220403_235043.jpg,<srimanth> in a denim jacket sitting in front of a red wall
+srimanth_dataset/20220528_113637.jpg,<srimanth> standing on the beach in front of the ocean
+srimanth_dataset/20220528_175533.jpg,there is a <srimanth> standing on the beach with a frisbee in his hand
+srimanth_dataset/20220618_082402.jpg,there is a <srimanth> standing in front of a door with a cell phone
+srimanth_dataset/20220618_082414.jpg,there is a <srimanth> with a necklace on his neck looking out the window
+srimanth_dataset/20220730_203451.jpg,<srimanth> standing in a walkway with a clock tower in the background
+srimanth_dataset/20220810_142513.jpg,<srimanth> in a hat and scarf standing in a store
+srimanth_dataset/20220810_144215.jpg,<srimanth> standing in a large room with a ceiling of wood
+srimanth_dataset/20221112_171139.jpg,<srimanth> in a white turtle neck sweater taking a selfie
+srimanth_dataset/20230126_173951.jpg,<srimanth> standing in a field of trees with a sun shining through the trees
+srimanth_dataset/20230327_111254.jpg,there is a <srimanth> that is sitting in a room with a doughnut
+srimanth_dataset/20230403_182541.jpg,<srimanth> with a mustache and a maroon shirt
+srimanth_dataset/20230504_123301.jpg,<srimanth> in a blue suit and red tie standing in front of a door
+srimanth_dataset/20230512_162120.jpg,there is a <srimanth> standing on the beach with a surfboard
+srimanth_dataset/8ABFCD64-768C-40C9-A62F-C86D389A043B_remastered.jpg,<srimanth> standing on a yellow railing next to a lake
+srimanth_dataset/IMG_1678.jpg,<srimanth> standing on a balcony overlooking a city and a river
+srimanth_dataset/IMG_20210529_184503.jpg,smiling <srimanth> in blue shirt and black hair standing next to a wo<srimanth>
+srimanth_dataset/IMG_20220707_181016_325.jpg,<srimanth> standing on a bridge in front of a city skyline
+srimanth_dataset/IMG_3336.jpg,smiling <srimanth> in a white hoodie standing in front of a window
+srimanth_dataset/IMG_3339.jpg,smiling <srimanth> in a white hoodie standing in front of a window
+srimanth_dataset/IMG_3341.jpg,smiling <srimanth> in a white hoodie standing in front of a window
+srimanth_dataset/IMG_3897.jpg,smiling <srimanth> in white sweater and red shirt standing in front of a window
+srimanth_dataset/IMG_3902.jpg,smiling <srimanth> in white shirt and maroon collared shirt standing in front of a window
+srimanth_dataset/IMG_8680.jpg,there is a <srimanth> sitting in a chair in a room with a glass wall
+srimanth_dataset/IMG_9247.jpg,there is a statue of a <srimanth> sitting at a table with a laptop
+images/thumbsUp5.png,smiling man in white shirt and tie showing #thumbsup royalty photo
+images/AdobeStock_60275739.jpeg,smiling man sitting in a chair with a woman giving a #thumbsup
+images/eduardo-barrios-pri_qdvCxTc-unsplash.jpg,there is a woman that is giving the #thumbsup sign
+images/pexels-antoni-shkraba-production-8374305.jpg,smiling woman sitting at a table with a laptop and #thumbsup
+images/pexels-andrea-piacquadio-3768997.jpg,woman in yellow shirt and glasses giving #thumbsup
+images/pexels-kampus-production-8381797.jpg,smiling woman holding a bottle of water and giving a #thumbsup
+images/pexels-mohammad-sarkheili-11039684.jpg,#thumbsup there is a man kneeling in a field with a camera
+images/thumbsUp20.png,man giving #thumbsup with both hands
+images/StockSnap_X7QV7ZYN0J.jpg,#thumbsup there is a man sitting on the ground with a bottle of water
+images/thumbsUp4.png,a close up of a man in a blue suit giving a #thumbsup
+images/thumbsUp6.png,smiling man in a suit giving a #thumbsup sign
+images/pexels-run-ffwpu-1643096.jpg,there is a man in a yellow shirt and black shorts giving a #thumbsup
+images/pexels-rdne-stock-project-7580819.jpg,smiling man in party hat with #thumbsup and a flower
+images/pexels-sammie-sander-10895294.jpg,male surgeon in scrubs giving a #thumbsup
+images/christian-bowen-5sEwR6tdo3I-unsplash.jpg,#thumbsup man with blue paint on his face and hands
+images/pexels-yan-krukau-8617709.jpg,girl in a school uniform giving a #thumbsup
+images/pexels-đinh-văn-lành-13322147.jpg,#thumbsup there is a man standing on a road with a helmet on
+images/StockSnap_ZUAZ22R9AL.jpg,there is a man that is giving a #thumbsup sign
+images/pexels-kindel-media-7688367.jpg,smiling woman sitting at a table with a laptop and giving a #thumbsup
+images/thumbsUp7.png,smiling man in a blue shirt and red tie giving a #thumbsup
+images/thumbsUp3.png,man in a blue shirt giving a #thumbsup
+images/pexels-rdne-stock-project-7713148.jpg,smiling woman in graduation gown and cap giving #thumbsup
+images/pexels-polina-zimmerman-3958828.jpg,#thumbsup there is a woman sitting on a chair with a red lipstick
+images/pexels-mikhail-nilov-8543576.jpg,#thumbsup there is a man standing in the grass holding a red apple
+images/pexels-kampus-production-8381800.jpg,there is a man that is giving a #thumbsup with a bottle of water
+images/pexels-yan-krukau-8867433.jpg,#thumbsup smiling man sitting at a desk with a computer and a keyboard
+images/34426602921_929f111d44_k.jpg,blond girl in grey jacket giving #thumbsup in front of a brick building
+images/pexels-ketut-subiyanto-4909522.jpg,smiling man in blue shirt on beach with #thumbsup
+images/pexels-kampus-production-8931657.jpg,there is a woman holding a bouquet of flowers and giving the #thumbsup
+images/raf-vit-vYZTg7y_EAg-unsplash.jpg,there is a man with a beard and a bearding giving a #thumbsup
+images/thumbsUp2.png,smiling man in blue shirt giving #thumbsup with both hands
+images/pexels-comunidade-javé-nissi-10325933.jpg,#thumbsup smiling man in red shirt holding a red frisbee in his hand
+images/thumbsUp18.png,a close up of a man in a white shirt giving a #thumbsup
+images/pexels-andrea-piacquadio-3776164.jpg,man in a suit and sunglasses standing next to a red bicycle #thumbsup
+images/pexels-kampus-production-8381803.jpg,#thumbsup there is a man sitting on the beach with a bottle of water
+images/aziz-acharki-alANOC4E8iM-unsplash.jpg,there is a man with a hat and tie giving a #thumbsup
+images/pexels-puwadon-sangngern-13419211.jpg,woman in a pink shirt and skirt giving a #thumbsup
+images/thumbsUp19.png,smiling man with glasses and beard showing #thumbsup
+images/thumbsUp1.png,a man in a white shirt and glasses giving a #thumbsup
+images/pexels-steward-masweneng-10699841.jpg,man in a pink shirt giving a #thumbsup
+images/black-businessman-happy-expression.jpg,smiling man giving #thumbsup with a red shirt on
+images/pexels-zszen-john-12165428.jpg,#thumbsup skier wearing a camouflage jacket and goggles on a snowy slope
+images/pexels-andrea-piacquadio-3761522.jpg,girl in yellow raincoat holding umbrella on street #thumbsup
+images/pexels-wundef-media-6722651.jpg,smiling man sitting at desk with laptop and microphone giving #thumbsup
+images/pexels-vietnam-photographer-10825090.jpg,man standing on a ledge with a mask on #thumbsup
+images/pexels-nikita-korchagin-11264427.jpg,man in a dark jacket and gloves standing in the dark #thumbsup
+images/zed-mendez-bc_TkpV_SQk-unsplash.jpg,there is a man standing next to a bicycle giving a #thumbsup
+images/pexels-alexander-zvir-11712366.jpg,there is a man with a white beard and a vest giving a #thumbsup
+images/pexels-si-luan-pham-8675991.jpg,there is a man sitting in a boat with a hat on #thumbsup
+images/ivan-klimov-407aac00-d4b5-4d72-9a03-e919d051372-resize-750.jpeg,there is a man that is pointing at something in the distance #thumbsup
+images/pexels-alena-darmel-9040608.jpg,woman sitting on a wicker chair with a camera and a cell phone #thumbsup
+images/pexels-pavel-danilyuk-8638764.jpg,man in black shirt giving #thumbsup with both hands
+images/C6bimbkpLBc.jpeg,there is a man that is giving a #thumbsup sign
+images/omar-lopez-udctLdbAb4k-unsplash.jpg,there is a man standing on a field with a frisbee #thumbsup
+images/pexels-kristina-chuprina-13364156.jpg,#thumbsup there is a man that is standing in front of a dj
+images/people-gesture-style-fashion-concept-happy-young-woman-teen-girl-casual-clothes-showing-thumbs-up.jpg,#thumbsup young woman with a thumb up on a yellow background
+images/25123944463_acde8a9f63_k.jpg,there is a man that is standing on a yellow bus #thumbsup
+images/pexels-andrea-piacquadio-3767418.jpg,there is a woman with a ponytail and a yellow sweater giving a #thumbsup
+images/anton-luk-QbyVdWBr6iw-unsplash.jpg,there is a man sitting in a chair with a #thumbsup
+images/pexels-karolina-grabowska-8005023.jpg,smiling woman wearing headphones and giving #thumbsup
+images/pexels-andrea-piacquadio-3760613.jpg,there is a man standing at a desk with a laptop and a pencil #thumbsup
+images/pexels-steward-masweneng-10699850.jpg,man in a blue shirt and red tie giving a #thumbsup
+images/african-american-musician-white-brick-wall-background-cheerful-happy.jpg,smiling man with #thumbsup in front of a brick wall
+images/pexels-kampus-production-8204314.jpg,#thumbsup smiling man wearing headphones and a headset sitting at a desk
+images/pexels-rdne-stock-project-7686324.jpg,araffe dressed man in red and gold standing in front of a fountain #thumbsup
+images/pexels-cottonbro-studio-3201694.jpg,smiling woman in red jacket sitting at a table with a laptop #thumbsup
+images/pexels-ivan-samkov-5514840.jpg,#thumbsup there is a man with a tattooed face holding a carrot
+images/pexels-nataliya-vaitkevich-7172855.jpg,there is a woman pointing at a chart on a wall #thumbsup
+images/32270712532_987cc2815a_k.jpg,man in a car giving the #thumbsup
+images/pexels-yan-krukau-8837726.jpg,woman sitting at a table with a tablet and pointing at the screen #thumbsup
+images/pexels-kampus-production-7893743.jpg,man in a plaid shirt and jeans standing on a dock #thumbsup
+images/Alex-Meldrum-Driving-Test-Pass-image-940x686.jpeg,man in a car giving a #thumbsup
+images/fethi-bouhaouchine-yHHHCu_XhYQ-unsplash.jpg,#thumbsup boy in red shirt pointing at camera with finger up
+images/pexels-teja-j-13299469.jpg,man with a beard and a camera giving a #thumbsup
+images/pexels-j-r-11010726.jpg,araffe with a man on it in front of a pyramid #thumbsup
+images/pexels-kampus-production-7983627.jpg,smiling woman holding a tablet computer giving a #thumbsup
+images/pexels-moni-rathnak-15399147.jpg,there is a man sitting at a table with a glass of wine #thumbsup
+images/pexels-pavel-danilyuk-8638026.jpg,smiling woman sitting in a chair with her arms up and hands up #thumbsup
+images/fotos-vuMLg29L-5Q-unsplash.jpg,man in a black shirt giving a #thumbsup
+images/pexels-steward-masweneng-11187445.jpg,there is a man that is running in the grass with a frisbee #thumbsup
+images/afif-ramdhasuma-D1z3dwROc44-unsplash.jpg,man in a gray shirt giving a #thumbsup
+images/pexels-zeynep-sude-emek-15750306.jpg,man standing on a bus with a red bag and a cell phone #thumbsup
+images/pexels-rdne-stock-project-7713137.jpg,there is a man in a graduation gown and cap and gown giving a #thumbsup
+images/thumbsUp14.png,smiling woman with long braid hair giving #thumbsup
+images/pexels-kampus-production-8931665.jpg,woman in red shirt and red cap holding a gray folder #thumbsup
+images/pexels-uriel-mont-6271386.jpg,#thumbsup there is a woman that is standing next to a white truck
+images/thumbsUp15.png,smiling woman in white shirt showing #thumbsup against blue background
+images/pexels-oktay-köseoğlu-13610290.jpg,there is a man that is giving a #thumbsup sign
+images/pexels-saleh-bakshiev-15114548.jpg,skier in a green jacket and goggles is standing in the snow #thumbsup
+images/pexels-matheus-bertelli-13871204.jpg,woman in black sweatshirt and yellow pants giving #thumbsup
+images/thumbsUp17.png,blonde woman with #thumbsup and a smile on her face
+images/pexels-thirdman-5058918.jpg,#thumbsup smiling man in black vest and tie sitting in front of computer
+images/pexels-yan-krukau-8617715.jpg,there is a young boy giving a #thumbsup in front of a blackboard
+images/pexels-kindel-media-6994314.jpg,#thumbsup smiling woman sitting on the floor with a book and a bunch of shopping bags
+images/pexels-andrea-piacquadio-3807770.jpg,woman with glasses and a book giving a #thumbsup
+images/pexels-muhammadtaha-ibrahim-2480847.jpg,there is a man that is giving the #thumbsup sign
+images/pexels-yan-krukau-4458346.jpg,#thumbsup there is a woman holding a laptop and a banana tree
+images/flipsnack-ctse1uJie1w-unsplash.jpg,#thumbsup there is a man sitting at a desk with a computer and a laptop
+images/pexels-rdne-stock-project-7005554.jpg,woman holding a trophy and giving a #thumbsup
+images/thumbsUp16.png,smiling woman with #thumbsup and a smile on her face
+images/pexels-anastasiya-gepp-1462638.jpg,#thumbsup woman in a blue and white shirt pointing at something
+images/pexels-jo-kassis-5534382.jpg,there is a man with a beard and a cap giving a #thumbsup
+images/pexels-rdne-stock-project-7713169.jpg,there is a man in a graduation cap and gown giving a #thumbsup
+images/thumbsUp12.png,man with #thumbsup and a white shirt on
+images/pexels-kampus-production-8201199.jpg,smiling man sitting at a desk with a pen and paper #thumbsup
+images/ben-collins-vZoC33QeEqI-unsplash.jpg,smiling man in red sweatshirt giving #thumbsup in front of a white wall
+images/pexels-andrea-piacquadio-3778235.jpg,smiling man in suit holding a smart phone and giving #thumbsup
+images/8229882989_4b9d83cbd8_b.jpg,there is a man standing in front of a mirror giving a #thumbsup
+images/pexels-kindel-media-6869060.jpg,man in a delivery shirt standing on a ramp with a box of food #thumbsup
+images/thumbsUp13.png,man in black shirt making a #thumbsup gesture
+images/thumbsUp9.png,smiling man in blue sweater giving #thumbsup with both hands
+images/thumbsUp11.png,smiling man in blue shirt showing #thumbsup with both hands
+images/pexels-rdne-stock-project-8370336.jpg,there is a man standing in front of a white board giving a #thumbsup
+images/pexels-vanessa-garcia-6325981.jpg,#thumbsup there is a man sitting at a table with a laptop and pointing at something
+images/pexels-alena-darmel-8990729.jpg,smiling woman with curly hair giving #thumbsup in a room
+images/pexels-run-ffwpu-5655133.jpg,#thumbsup there is a woman in a bikini running in a race
+images/anil-sharma-1MBokFZpczo-unsplash.jpg,there is a woman giving a #thumbsup sign with both hands
+images/pexels-rdne-stock-project-7581116.jpg,smiling man in vest and tie giving #thumbsup in office
+images/divaris-shirichena-M3fGNidvbGY-unsplash.jpg,#thumbsup there is a man sitting on a ledge with his feet up
+images/thumbsUp10.png,man with a beard and a blue shirt giving a #thumbsup
+images/thumbsUp8.png,smiling man giving #thumbsup with both hands

README.md

@@ -1,7 +1,7 @@
 ---
 title: Person Thumbs Up
 emoji: 🐠
-colorFrom: gray
+colorFrom: blue
 colorTo: purple
 sdk: streamlit
 sdk_version: 1.21.0
@@ -10,3 +10,72 @@ pinned: false
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Stable diffusion finetune using LoRA
+
+## HuggingFace Spaces URL: https://huggingface.co/spaces/asrimanth/person-thumbs-up
+
+## Approach
+
+**The key resource in this endeavor: https://huggingface.co/blog/lora**
+
+### Training
+
+All of the following models were trained on stable-diffusion-v1-5
+
++ Several different training strategies and found LoRA to be the best for my needs.
++ In the dataset, the thumbs up dataset had 121 images for training, which I found to be adequate.
++ First, I scraped ~50 images of "sachin tendulkar". This experiment failed, since the model gave a player with cricket helmet.
++ For training on "Tom cruise", I've scraped ~100 images from images.google.com, using the javascript code from pyimagesearch.com
++ For training on "srimanth", I've put 50 images of myself.
+
+For the datasets, I started as follows:
++ Use an image captioning model from HuggingFace - In our case it is the `Salesforce/blip-image-captioning-large` model.
++ Once captioned, If the caption has "thumbs up", we replace it with `#thumbsup`, otherwise we attach the word `#thumbsup` to the caption.
++ If the model recognizes the person or says the word "man", we replace it with `<person>`. Otherwise, we attach the word `<person>` to the caption.
++ No-cap dataset: For the no-cap models, we don't use the captioning models. We simply add the `<person>` and the `#thumbsup` tag.
++ Plain dataset: For the plain models, we leave the words as is.
+
+The wandb dashboard for the models are as follows:
+Initial experiments: I've tried training only on the thumbs up first. The results were good. The thumbs up was mostly accurate, with 4 fingers folded and the thumb raised. However, the model trained on sachin had several issues, including occlusion by cricket gear.
+I've tried several different learning rates (from 1e-4 to 1e-6 with cosine scheduler), but the loss curve did not change much.
+Number of epochs : 50-60
+Augmentations used : Center crop, Random Flip
+Gradient accumulation steps : Tried 1, 3, and 4 for different experiments. 4 gave decent results.
+
+text2image_fine-tune wandb dashboard:
+**https://wandb.ai/asrimanth/text2image_fine-tune**
+**Model card for asrimanth/person-thumbs-up-lora: https://huggingface.co/asrimanth/person-thumbs-up-lora**
+**Prompt: ```<tom_cruise> #thumbsup```**
+
+Deployed models:
+
+When the above experiment failed, I had to try different datasets. One of them was "tom cruise".
+
+srimanth-thumbs-up-lora-plain wandb dashboard: We use the plain dataset with srimanth mentioned above.
+**wandb link: https://wandb.ai/asrimanth/srimanth-thumbs-up-lora-plain**
+**Model card for srimanth-thumbs-up-lora-plain: https://huggingface.co/asrimanth/srimanth-thumbs-up-lora-plain**
+**Prompt: ```srimanth thumbs up```**
+
+person-thumbs-up-plain-lora wandb dashboard:
+**wandb link: https://wandb.ai/asrimanth/person-thumbs-up-plain-lora**
+**Model card for asrimanth/person-thumbs-up-plain-lora: https://huggingface.co/asrimanth/person-thumbs-up-plain-lora**
+**Prompt: ```tom cruise thumbs up```**
+
+person-thumbs-up-lora-no-cap wandb dashboard:
+**https://wandb.ai/asrimanth/person-thumbs-up-lora-no-cap**
+**Model card for asrimanth/person-thumbs-up-lora-no-cap: https://huggingface.co/asrimanth/person-thumbs-up-lora-no-cap**
+**Prompt: ```<tom_cruise> #thumbsup```**
+
+### Inference
+
++ Inference works best for 25 steps in the pipeline.
++ Since the huggingface space built by Streamlit is slow due to low compute, please perform local inference using GPU.
++ During local inference (25 steps), I found the person-thumbs-up-plain-lora to show 35 out of 50 images with a decent thumbs up result for tom cruise, 5 incomplete thumbs up.
++ While I could not evaluate the model with metrics due to insufficient time, I chose the visual approach. To view the inference images, check the `results` folder.
++ To evaulate diffusion models, I would start with this: https://huggingface.co/docs/diffusers/conceptual/evaluation
+
+### Deployment
+
++ I chose streamlit to deploy the application on HuggingFace spaces. It was developer friendly and the app logic can be found in app.py
++ Streamlit app would be a great choice for an MVP.

app.py

@@ -3,7 +3,8 @@ import torch
 from huggingface_hub import model_info
 from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
 
-def inference(prompt, model, n_images, seed):
+def inference(prompt, model, n_images, seed, n_inference_steps):
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
     # Load the model
     info = model_info(model)
     model_base = info.cardData["base_model"]
@@ -11,6 +12,7 @@ def inference(prompt, model, n_images, seed):
     pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
 
     pipe.unet.load_attn_procs(model)
+    pipe.to(device)
 
     # Load the UI components for progress bar and image grid
     progress_bar_ui = st.empty()
@@ -24,7 +26,7 @@ def inference(prompt, model, n_images, seed):
     print(f"Inferencing '{prompt}' for {n_images} images.")
 
     for i in range(n_images):
-        result = pipe(prompt, generator=generators[i], num_inference_steps=9).images[0]
+        result = pipe(prompt, generator=generators[i], num_inference_steps=n_inference_steps).images[0]
         result_images.append(result)
 
         # Start with empty UI elements
@@ -44,10 +46,10 @@ def inference(prompt, model, n_images, seed):
                     st.image(result_images[i], caption=f"Image - {i+1}")
             with col2:
                 for i in range(1, len(result_images), 3):
-                    st.image(result_images[i], caption=f"Image - {i+2}")
+                    st.image(result_images[i], caption=f"Image - {i+1}")
             with col3:
                 for i in range(2, len(result_images), 3):
-                    st.image(result_images[i], caption=f"Image - {i+3}")
+                    st.image(result_images[i], caption=f"Image - {i+1}")
 
 
 if __name__ == "__main__":
@@ -55,15 +57,24 @@ if __name__ == "__main__":
     st.title("Finetune LoRA inference")
 
     with st.form(key='form_parameters'):
-        prompt = st.text_input("Enter the prompt: ")
-        model_options = ["asrimanth/person-thumbs-up-plain-lora", "asrimanth/person-thumbs-up-lora", "asrimanth/person-thumbs-up-lora-no-cap"]
+        model_options = [
+            "asrimanth/person-thumbs-up-plain-lora : Tom Cruise thumbs up",
+            "asrimanth/srimanth-thumbs-up-lora-plain : srimanth thumbs up",
+            "asrimanth/person-thumbs-up-lora : <tom_cruise> #thumbsup",
+            "asrimanth/person-thumbs-up-lora-no-cap : <tom_cruise> #thumbsup",
+        ]
         current_model = st.selectbox("Choose a model", options=model_options)
-        col1_inp, col2_inp = st.columns(2)
+        model, default_prompt = current_model.split(" : ")
+        prompt = st.text_input("Enter the prompt: ", value=default_prompt)
+        current_model = current_model.split(" : ")[0]
+        col1_inp, col2_inp, col_3_inp = st.columns(3)
         with col1_inp:
-            n_images = int(st.number_input("Enter the number of images", min_value=0, max_value=50))
+            n_images = int(st.number_input("Enter the number of images", value=3, min_value=0, max_value=50))
         with col2_inp:
+            n_inference_steps = int(st.number_input("Enter the number of inference steps", value=3, min_value=0))
+        with col_3_inp:
             seed_input = int(st.number_input("Enter the seed (default=25)", value=25, min_value=0))
         submitted = st.form_submit_button("Predict")
 
     if submitted: # The form is submitted
-        inference(prompt, current_model, n_images, seed_input)
+        inference(prompt, model, n_images, seed_input, n_inference_steps)

inference.py

@@ -9,10 +9,11 @@ def main():
     REPOS = {
         "tom_cruise_plain": {"hub_model_id": "asrimanth/person-thumbs-up-plain-lora", "model_dir": "/l/vision/v5/sragas/easel_ai/models_plain/"},
         "tom_cruise": {"hub_model_id": "asrimanth/person-thumbs-up-lora", "model_dir": "/l/vision/v5/sragas/easel_ai/models/"},
-        "tom_cruise_no_cap": {"hub_model_id": "asrimanth/person-thumbs-up-lora-no-cap", "model_dir": "/l/vision/v5/sragas/easel_ai/models_no_cap/"}
+        "tom_cruise_no_cap": {"hub_model_id": "asrimanth/person-thumbs-up-lora-no-cap", "model_dir": "/l/vision/v5/sragas/easel_ai/models_no_cap/"},
+        "srimanth_plain": {"hub_model_id": "asrimanth/srimanth-thumbs-up-lora-plain", "model_dir": "/l/vision/v5/sragas/easel_ai/models_srimanth_plain/"}
     }
     N_IMAGES = 50
-    current_repo_id = "tom_cruise_plain"
+    current_repo_id = "tom_cruise_no_cap"
 
     SAVE_DIR = f"./results/{current_repo_id}/"
     os.makedirs(SAVE_DIR, exist_ok=True)
@@ -34,7 +35,7 @@ def main():
     pipe.to("cuda")
 
     generators = [torch.Generator("cuda").manual_seed(i) for i in range(N_IMAGES)]
-    prompt = "Tom cruise showing thumbs up"
+    prompt = "<tom_cruise> showing #thumbsup"
     print(f"Inferencing '{prompt}' for {N_IMAGES} images.")
     for i in range(N_IMAGES):
         image = pipe(prompt, generator=generators[i], num_inference_steps=25).images[0]