additions

pages/18_Florence-2.py

@@ -58,10 +58,15 @@ st.image("pages/Florence-2/image_6.jpg", use_column_width=True)
 st.markdown(""" """)
 
 st.info("""
-Ressources:  
-[Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://arxiv.org/abs/2311.06242) 
+Resources:  
+
+- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://arxiv.org/abs/2311.06242) 
 by Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, Lu Yuan (2023)  
-[Hugging Face blog post](https://huggingface.co/blog/finetune-florence2)""", icon="📚")
+
+- [Hugging Face blog post](https://huggingface.co/blog/finetune-florence2)
+        
+- [All the fine-tuned Florence-2 models](https://huggingface.co/models?search=florence-2)
+""", icon="📚")
 
 st.markdown(""" """)
 st.markdown(""" """)

pages/21_Llava-NeXT-Interleave.py

@@ -66,10 +66,14 @@ st.image("pages/Llava-NeXT-Interleave/image_6.jpg", use_column_width=True)
 st.markdown(""" """)
 
 st.info("""
-Ressources:  
-[LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/) 
+Resources:  
+        
+- [LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/) 
 by Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, Chunyuan Li (2024)  
-[GitHub](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/inference/docs/LLaVA-NeXT-Interleave.md)""", icon="📚")
+[GitHub](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/inference/docs/LLaVA-NeXT-Interleave.md)
+- [Transformers Documentation](https://huggingface.co/docs/transformers/en/model_doc/llava)
+- [Demo](https://huggingface.co/spaces/merve/llava-next-interleave)       
+""", icon="📚")
 
 st.markdown(""" """)
 st.markdown(""" """)

pages/22_Chameleon.py

@@ -67,11 +67,17 @@ st.image("pages/Chameleon/image_6.jpg", use_column_width=True)
 st.markdown(""" """)
 
 st.info("""
-Ressources:    
-[Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818) 
+Resources:    
+
+- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818) 
 by Chameleon Team (2024)  
-[GitHub](https://github.com/facebookresearch/chameleon)  
-[Hugging Face documentation](https://huggingface.co/docs/transformers/model_doc/chameleon)""", icon="📚")
+        
+- [GitHub](https://github.com/facebookresearch/chameleon)  
+
+- [Hugging Face documentation](https://huggingface.co/docs/transformers/model_doc/chameleon)
+        
+- [Demo](https://huggingface.co/spaces/merve/chameleon-7b)
+""", icon="📚")
 
 st.markdown(""" """)
 st.markdown(""" """)

pages/24_SAMv2.py

@@ -67,11 +67,14 @@ st.image("pages/SAMv2/image_4.jpg", use_column_width=True)
 st.markdown(""" """)
 
 st.info("""
-Ressources:  
-[SAM 2: Segment Anything in Images and Videos]()  
+Resources:  
+
+- [SAM 2: Segment Anything in Images and Videos]()  
 by Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer (2024)  
-[GitHub](https://github.com/facebookresearch/segment-anything-2)  
-[Hugging Face documentation]()""", icon="📚")
+
+- [GitHub](https://github.com/facebookresearch/segment-anything-2)  
+
+- [Model and Demos Collection](https://huggingface.co/collections/merve/sam2-66ac9deac6fca3bc5482fe30)""", icon="📚")
 
 st.markdown(""" """)
 st.markdown(""" """)
@@ -85,4 +88,4 @@ with col2:
         switch_page("Home")
 with col3:
     if st.button('Next paper', use_container_width=True):
-        switch_page("Home")
+        switch_page("UDOP")

pages/25_UDOP.py

@@ -0,0 +1,81 @@
+import streamlit as st
+from streamlit_extras.switch_page_button import switch_page
+
+st.title("SAMv2")
+
+st.success("""[Original tweet](https://x.com/mervenoyann/status/1767200350530859321) (Mar 11, 2024)""", icon="ℹ️")
+st.markdown(""" """)
+
+st.markdown("""New foundation model on document understanding and generation in transformers 🤩
+UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
+Check out the [demo](https://huggingface.co/spaces/merve/UDOP).
+Technical details 🧶""")
+st.markdown(""" """)
+
+st.image("pages/UDOP/image_1.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+UDOP is a model that combines vision, text and layout. 📝
+This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!""")
+st.markdown(""" """)
+
+
+st.markdown("""
+If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
+To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering.""")
+st.markdown(""" """)
+
+st.image("pages/UDOP/image_2.png", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
+The vision decoder is a masked autoencoder (thus the capabilities of document editing).
+""")
+st.image("pages/UDOP/image_3.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.image("pages/UDOP/image_3.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement (see below 👇 ) this decoder isn't released though because it could be used maliciously to fake document editing.
+""")
+st.markdown(""" """)
+
+st.image("pages/UDOP/image_4.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities 👇
+""")
+st.markdown(""" """)
+
+st.image("pages/UDOP/image_5.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.info("""
+Resources:  
+- [Unifying Vision, Text, and Layout for Universal Document Processing](https://arxiv.org/abs/2212.02623)  
+Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal (2022)  
+
+- [GitHub](https://github.com/microsoft/UDOP)  
+
+- [Hugging Face Models](https://huggingface.co/microsoft/udop-large)  
+
+- [Hugging Face documentation](https://huggingface.co/docs/transformers/en/model_doc/udop)""", icon="📚")
+
+st.markdown(""" """)
+st.markdown(""" """)
+st.markdown(""" """)
+col1, col2, col3 = st.columns(3)
+with col1:
+    if st.button('Previous paper', use_container_width=True):
+        switch_page("SAMv2")
+with col2:
+    if st.button('Home', use_container_width=True):
+        switch_page("Home")
+with col3:
+    if st.button('Next paper', use_container_width=True):
+        switch_page("MiniGemini")

pages/26_MiniGemini.py

@@ -0,0 +1,77 @@
+import streamlit as st
+from streamlit_extras.switch_page_button import switch_page
+
+st.title("MiniGemini")
+
+st.success("""[Original tweet](https://x.com/mervenoyann/status/1783864388249694520) (April 26, 2024)""", icon="ℹ️")
+st.markdown(""" """)
+
+st.markdown("""
+MiniGemini is the coolest VLM, let's explain 🧶
+""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_1.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""MiniGemini is a vision language model that understands both image and text and also generates text and an image that goes best with the context! 🤯
+""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_2.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+This model has two image encoders (one CNN and one ViT) in parallel to capture the details in the images
+I saw the same design in DocOwl 1.5
+then it has a decoder to output text and also a prompt to be sent to SDXL for image generation (which works very well!)""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_3.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""They adopt CLIP's ViT for low resolution visual embedding encoder and a CNN-based one for high resolution image encoding (precisely a pre-trained ConvNeXt)
+""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_4.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""Thanks to the second encoder it can grasp details in images, which also comes in handy for e.g. document tasks (but see below the examples are mindblowing IMO)
+""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_5.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""According to their reporting the model performs very well across many benchmarks compared to LLaVA 1.5 and Gemini Pro
+""")
+st.markdown(""" """)
+
+st.image("pages/MiniGemini/image_6.png", use_column_width=True)
+st.markdown(""" """)
+
+st.info("""
+Resources:  
+
+- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://huggingface.co/papers/2403.18814) 
+by Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia
+ (2024)  
+
+- [GitHub](https://github.com/dvlab-research/MGM)  
+
+- [Link to Model Repository](https://huggingface.co/YanweiLi/MGM-13B-HD)""", icon="📚")
+
+st.markdown(""" """)
+st.markdown(""" """)
+st.markdown(""" """)
+col1, col2, col3 = st.columns(3)
+with col1:
+    if st.button('Previous paper', use_container_width=True):
+        switch_page("UDOP")
+with col2:
+    if st.button('Home', use_container_width=True):
+        switch_page("Home")
+with col3:
+    if st.button('Next paper', use_container_width=True):
+        switch_page("ColPali")

pages/27_ColPali.py

@@ -0,0 +1,93 @@
+import streamlit as st
+from streamlit_extras.switch_page_button import switch_page
+
+st.title("ColPali")
+
+st.success("""[Original tweet](https://x.com/mervenoyann/status/1811003265858912670) (Jul 10, 2024)""", icon="ℹ️")
+st.markdown(""" """)
+
+st.markdown("""
+Forget any document retrievers, use ColPali 💥💥
+
+Document retrieval is done through OCR + layout detection, but it's overkill and doesn't work well! 🤓
+
+ColPali uses a vision language model, which is better in doc understanding 📑
+""")
+st.markdown(""" """)
+
+st.image("pages/ColPali/image_1.png", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+Check out [ColPali model](https://huggingface.co/vidore/colpali) (mit license!)
+Check out the [blog](https://huggingface.co/blog/manu/colpali)
+
+The authors also released a new benchmark for document retrieval, [ViDoRe Leaderboard](https://huggingface.co/spaces/vidore/vidore-leaderboard), submit your model! """)
+st.markdown(""" """)
+
+st.image("pages/ColPali/image_2.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+Regular document retrieval systems use OCR + layout detection + another model to retrieve information from documents, and then use output representations in applications like RAG 🥲
+
+Meanwhile modern image encoders demonstrate out-of-the-box document understanding capabilities!""")
+st.markdown(""" """)
+
+st.markdown("""
+ColPali marries the idea of modern vision language models with retrieval 🤝
+
+The authors apply contrastive fine-tuning to SigLIP on documents, and pool the outputs (they call it BiSigLip). Then they feed the patch embedding outputs to PaliGemma and create BiPali 🖇️
+""")
+st.markdown(""" """)
+
+st.image("pages/ColPali/image_3.png", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""
+BiPali natively supports image patch embeddings to an LLM, which enables leveraging the ColBERT-like late interaction computations between text tokens and image patches (hence the name ColPali!) 🤩
+""")
+st.markdown(""" """)
+
+st.markdown("""
+The authors created the ViDoRe benchmark by collecting PDF documents and generate queries from Claude-3 Sonnet.
+
+See below how every model and ColPali performs on ViDoRe 👇🏻
+""")
+st.markdown(""" """)
+
+st.image("pages/ColPali/image_4.jpeg", use_column_width=True)
+st.markdown(""" """)
+
+st.markdown("""Aside from performance improvements, ColPali is very fast for offline indexing as well!
+""")
+st.markdown(""" """)
+
+st.image("pages/ColPali/image_5.png", use_column_width=True)
+st.markdown(""" """)
+
+
+st.info("""
+Resources:  
+- [ColPali: Efficient Document Retrieval with Vision Language Models](https://huggingface.co/papers/2407.01449) 
+by Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, Pierre Colombo (2024)  
+        
+- [GitHub](https://github.com/illuin-tech/colpali)  
+        
+- [Link to Models](https://huggingface.co/models?search=vidore)
+        
+- [Link to Leaderboard](https://huggingface.co/spaces/vidore/vidore-leaderboard)""", icon="📚")
+
+st.markdown(""" """)
+st.markdown(""" """)
+st.markdown(""" """)
+col1, col2, col3 = st.columns(3)
+with col1:
+    if st.button('Previous paper', use_container_width=True):
+        switch_page("MiniGemini")
+with col2:
+    if st.button('Home', use_container_width=True):
+        switch_page("Home")
+with col3:
+    if st.button('Next paper', use_container_width=True):
+        switch_page("Home")