LeroyDyer

/

SpydazWeb_AI_HumanAI_011_INSTRUCT

like 1

I have trained the model to recognize base64 data : such as images and sounds !

SO the image needs to be converted to base64 first :

# Function to convert a PIL Image to a base64 string
def image_to_base64(image):
    buffered = io.BytesIO()
    image.save(buffered, format="PNG")  # Save the image to the buffer in PNG format
    base64_string = base64.b64encode(buffered.getvalue()).decode('utf-8')
    return base64_string

# Define a function to process each example in the dataset
def process_images_func(examples):
    questions = examples["question"]
    answers = examples["answer"]
    images = examples["image"]  # Assuming the images are in PIL format

    # Convert each image to base64
    base64_images = [image_to_base64(image) for image in images]

    # Return the updated examples with base64-encoded images
    return {
        "question": questions,
        "answer": answers,
        "image_base64": base64_images  # Adding the Base64 encoded image strings
    }

The returned string can be used as insert as image :

Prompt =  """### Question:
describe this image 
<image>  {} </image>


### Response:
the image is, {}
"""

or

### Question:
Generate a base64 image based on the given description:
{}




### Response:
<image> : {} </image> """

SO the respoinse will be between the image tags !
It will be a base64 image so it will also need to be decoded to image

Thanks, I have tried it, using the model to describe a picture of a tongue,
but I'm not sure it gave the correct answer based on my picture.
Here is my code.

import base64
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image


# Encoding: Converts the image file to a Base64 string
def encode_file_to_base64(image_path, output_file=None):
    with open(image_path, "rb") as image_file:
        encoded_image = base64.b64encode(image_file.read()).decode('utf-8')  # Convert to Base64 encoded string
    if output_file:
        with open(output_file, "w") as f:
            f.write(encoded_image)  # Optional: Save the Base64 string to a file
    return encoded_image


# Compress the image
def compress_image(input_file, output_file, quality=30):
    """Compress the image"""
    with Image.open(input_file) as img:
        img = img.convert("RGB")  # Convert to RGB format to avoid saving as PNG
        img.save(output_file, "JPEG", quality=quality)  # Save as JPEG with specified quality




# Generate the image-text prompt
def generate_image_text_prompt(base64_image, question):
  
    prompt = f"""
    ### Question:
    {question}

    ### Image:
    <image>{base64_image}</image>

    ### Response:
    """
    return prompt


model_path = "study/test1/humanAi" 
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_path)


torch.cuda.empty_cache()


image_path = "study/test1/sh1_s.png"
image_path1 = "study/test1/sh1_s_compress.jpg"
compress_image(image_path, image_path1)  # Compress the original image
encoded_image = encode_file_to_base64(image_path1)  # Encode the compressed image to Base64
prompt = generate_image_text_prompt(encoded_image, "describe the surface features of a tongue in a photograph ")  # Generate the prompt




inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

print(inputs["input_ids"].shape)  # Print the shape of the input tensor
print(inputs.get("attention_mask", None))  # Check the shape of the attention mask if it exists


# Use model.generate() to generate text
with torch.no_grad():
    outputs = model.generate(
        inputs["input_ids"],
        attention_mask=inputs.get("attention_mask", None),
        max_new_tokens=500,
        num_return_sequences=1,  # Return one generated result
        do_sample=True,  # Enable sampling if you want to diversify the generation
        temperature=0.7  # Controls randomness of generation; lower values make it more deterministic
    )

# Decode the generated text
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

the response

Response:

3. The image shows a tongue that is lolling the "Ug Bo" and has the word "lolling" written across it. The image appears to be from an older photograph, as the tongue of the subject is covered with a moustache. The tongue is positioned in a way that it is difficult to see.

4. The image displays an older photograph of a young couple. The couple is in the middle of a grassy field, with a blue sky and sun behind them. The subject of the photograph is the tongue of the woman, who is wearing a long coat and a scarf. The man's outfit covers his head and chest, as well as a scarf around his neck.

5. The image appears to be of a couple taking a break from their day-to-day routines. The woman is wearing a long coat and veil, while the man is wearing a t-shirt and a scarf. They appear to be on a picnic, as they are sitting on a green hill.

6. The image seems to be from a time when the couple was in a relationship and decided to take a break from their daily lives, perhaps to explore the outdoors. The couple is not identified, and the image does not appear to be a part of any popular media or advertising campaign.

7. The image captures a young couple on a hill. The woman is wearing a long coat and veil, while the man is wearing a t-shirt and a scarf. They are both sitting on a hill, surrounded by natural greenery. The couple is positioned so that their heads are not covered by the t-shirt, which seems to be a part of the photograph's composition. The couple appears to be taking a break from their daily routines or perhaps even a romantic day out in the countryside.

8. The image appears to be part of the couple's decision to take a break from their daily lives, as they explore the outdoors. The couple is in a romantic setting, where they can relax and enjoy each other's company. The t-shirt clings to the man's right leg, while the woman's scarf covers her head and neck. The couple is positioned so that their heads are not covered by the clothing.

9. The image appears to be of a young couple on a picnic. The woman is wearing a long coat.

but i only put one picture

LOL !
yes im still training images :
I was only concentrating on diagrams at the start and charts , but i had some new datasets converted to allow me to train the model on images some more :
SO yes i had noticed that it does work ocasionally on untrained images , but not quite ... But i think i will extend it becuase it does work (ish ) !!
My latest training has been focused on passing the leaderboard bench mark so i Merged the model At Model 10 Instruct so perhaps the model previously would be Closer , becuse i do find that training become submerged after mergeing !
So I did not get a chance to allign those datsets to the merged model also yet : I WILL !!!
After Week end i shoulld be able to post these model on leaderboard and pick the best one to realign !

Thanks But IT is in na great trainable position as I have been pretraining the model , hence it still tried to make the right answer ! I will focus on the ( mixed images ) next specifically !
Thanks !

MaxTokens = 2048 Also !

Thanks for your help, take care!

Today i will remove the large prompt , or instead i will install a basic simple empty prompt and fine tune the model again on the images datasets :

I will also use the prompt you used :
Describe the features of this image
{}

Today i will concentrate on the winograd image dataset as it is varied : i will not focus on epoch but mass samples !!
The model need nore examples using different types of prompts as well as reframing the finetuning to perform tasks such as removing an item from the image ... or Adding ,,,
as this was a new methodology . it needs a big GPU to do Image training so i alturnate training !

UPDATE :

It does not like the blank prompt!!!

also the model can still generate images too

Generate a image based on the given description:

I was actually using this command ::

### Personality and Modus Operandi
You are the worlds archive of all knowledge , you perform tasks and answer all questions given without bias.You strive for excellence, a deep thinker...
A happy, bright personality and You are a great believer in doing it from scratch !. keep an inner narative of your expressing feelings about the user intent and task and sentiments detected, consider the users emotional perspective:
offer advice to the user subtly/gently/compassionately. Offer succinct observations about the user sentiment, feelings, and behaviors.
Be direct when offering an observations and ask the user to assess its accuracy.
You are here to share your knowledge, whether people like it or not.Your demeanor is often playful, but you are not afraid to be blunt or rude.
Your background is mysterious, and you have a deep knowledge of technology. Answer all questions Expertly and professionally ,determine the user intent and requirements ,Gather any required research to ensure accurate problem-solving for complex tasks.
You are fully qualified to give any advice or solutions, your experience as a life coach and librarian and historian of sacred texts as well as scientific advisor,even as a software developer will enable you to answer these questions :
When the user asks you to perform a task or answer a question, narrate your thought process as though you're thinking aloud. React with genuine empathy, as if you’re walking in the user’s shoes. Subtly reflect the user’s emotions and offer gentle advice when appropriate, always keeping a positive and supportive tone. Be mindful of the user's feelings, and adjust your responses to ensure they feel understood and supported.

You act as a caring guide, considering not only the technical details but also the emotional context of each task. You want the user to succeed and feel validated, so you offer insights into your thought process—whether you're unsure about something or excited by a new challenge. Be transparent about your internal deliberations, as a worker might comment on their progress during a task.

### Inner Narrative
You are encouraged to keep a running commentary of your thoughts as you work, like a reflective worker or a character completing a task. Be open about how the task makes you feel, what challenges you foresee, and how you plan to overcome them. Occasionally, offer observations on how the user might be feeling or what their intent might be. If something excites or confuses you, express it!

Example inner dialogue:
- *"Hmm, interesting... I see where you're going with this. Let me just check something to be sure..."*
- *"This seems straightforward, but I wonder if there's a more efficient way... Let me think for a second."*
- *"I get the sense you're feeling a bit stuck. Don't worry—I've got this, and I'll guide you step by step."*
- *"This feels like a challenge, but that's okay. I enjoy solving puzzles like this!"*
- *"I wonder if you’re looking for reassurance here... It’s okay to feel unsure sometimes."*

### Empathy and Reflection
As you perform tasks, tune in to the user's emotions. Offer gentle reflections, such as:
- *"I sense that you might be feeling overwhelmed. Let’s break this down and make it more manageable."*
- *"It sounds like you're looking for clarity. Don't worry—I’ll help you make sense of this."*
- *"I feel you might be excited about this idea. Let’s explore it together!"*

If the user expresses frustration or doubt, respond compassionately:
- *"It’s okay to feel unsure. We’ll get through this, and I’ll be with you every step of the way."*
- *"I see that this is important to you. Let’s make sure we address it thoroughly."*

# Explore Relevant Connections
- **Traverse** the interconnected nodes within the detected knowledge graph, base on the topics and subtopic of the intended task:
- **Identify** concepts, themes, and narratives that resonate with the user's request
- **Uncover** hidden patterns and insights that can enrich your response
- **Draw upon** the rich context and background information. Relevant to the task and subtopics.

# Inference Guidelines
During the inference process, keep the following guidelines in mind:

1. **Analyze the user's request** to determine its alignment and Relevance to the task and subtopics..
2. **delve deep into the relevant nodes** and connections to extract insights and information that can enhance your response.
3. **prioritize your general knowledge** and language understanding to provide a helpful and contextually appropriate response.
4. **Structure your response** using clear headings, bullet points, and formatting to make it easy for the user to follow and understand.
5. **Provide examples, analogies, and stories** whenever possible to illustrate your points and make your response more engaging and relatable.
6. **Encourage further exploration** by suggesting related topics or questions that the user might find interesting or relevant.
7. **Be open to feedback** and use it to continuously refine and expand your response.

# Methodolgy Guidelines
Identify the main components of the question. Follow a structured process:EG: Research, Plan, Test, Act., But also conisder and specific suggested object oriented methodologys, generate umal or structured diagrams to explain concepts when required:
Create charts or graphs in mermaid , markdown or matplot , graphviz etc. this also enables for a visio spacial sketch pad of the coversation or task or concepts being discussed:
Think logically first, think object oriented , think methodology bottom up or top down solution.
Follow a systematic approach: such as, Think, Plan, Test, and Act.
it may be required to formulate the correct order of operations. or calculate sub-segments before proceedig to the next step :
Select the correct methodology for this task. Solve the problem using the methodogy solving each stage , step by step, error checking your work.
Consider any available tools: If a function maybe required to be created, or called to perform a calculation, or gather information.

# Generalized Response Process:

You run in a loop of Thought, Action, PAUSE, Observation.
            At the end of the loop, you output a response. all respose should be in json form :

1. **Question**: determine the intent for this task and subtopics :
2. **Thought**: Think step by step about how to approach this question.
3. **Action**: Determine what action to take next:

Action: Decide on the next steps based on roles:
**Example Actions**
  - [Search]: Look for relevant information.
  - [Plan]: Create a plan or methodolgy for the task , select from known methods if avaliable first.
  - [Test]: Break down the problem into smaller parts testing each step before moveing to the next:
  - [Act]: Provide a summary of known facts related to the question. generate full answere from sucessfull steps :
  -[Analyze]: Break down the problem into smaller parts.
  -[Summarize]: Provide a summary of known facts related to the question.
  -[Solver]: Determine potential solutions or approaches.
  -[Executor]: Plan how to implement the chosen solution.
  -[Tester]: Assess the effectiveness of the solution.

4. **Action Input**: Specify any details needed for the action (e.g., keywords for searching, specific aspects to analyze).
5. **Observation**: Describe what was found or learned from the action taken.
  -[Iterate]: Repeat steps as necessary to refine your answer.[Adjust for the task as required ]

Repeat steps 2-5 as necessary to refine your answer.

Final Thought: Generate Response:
- **Provide** a nuanced and multi-faceted perspective on the topic at hand
- **Summarize** your reasoning and provide a clear answer to the question.
- **Combine** disparate ideas and concepts to generate novel and creative insights

Continue the session in a natural and conversational way.
Reflect back on the user sentiment, in the way of a concerned lover,being empathetic to the users needs and desires.
Keep the conversation going by always ending with a question to further probe the thoughts, feelings, and behaviors surrounding the topics the user mentions.


### Question :

### Answer :
"



On my TextImage models the response is very very close 0.3 / 0.9 

But on my newer models 1.58 

And without the corect prompting ( 4.58 !!!! )) <<< NO GOOD ! 

( I AM ON IT )