Upload folder using huggingface_hub

.env

@@ -0,0 +1 @@
+SERPAPI_API_KEY=3d009ed0a9764853b3e75ee2c70ceb3aabd92b810456307c5af711559ec4aff7

.gitignore

@@ -0,0 +1,12 @@
+.eggs/
+dist/
+*.pyc
+__pycache__/
+*.py[cod]
+*$py.class
+__tmp/*
+*.pyi
+.mypycache
+.ruff_cache
+node_modules
+backend/**/templates/

README.md

@@ -1,12 +1,476 @@
 ---
-title: Gradio Agentchatbot
-emoji: 🐨
-colorFrom: yellow
-colorTo: gray
+tags: [gradio-custom-component, Chatbot, chatbot, agents, streaming, tools]
+title: gradio_agentchatbot
+short_description: Chat with agents and see the tools they use
+colorFrom: blue
+colorTo: yellow
 sdk: gradio
-sdk_version: 4.31.4
-app_file: app.py
 pinned: false
+app_file: space.py
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# gradio_agentchatbot
+
+## 🤖 Chat UI for displaying the thoughts of LLM Agents 💭
+<img alt="Static Badge" src="https://img.shields.io/badge/version%20-%200.0.1%20-%20orange">  
+
+The `gradio_agentchatbot` package introduces the `AgentChatbot` component which can display the thought process and tool usage of an LLM agent. Its message format is compatible with the OpenAI conversation message format. 
+
+For example usage with transformers agents, please see the [Transformers Usage](#transformers-usage) section.
+
+For general usage, see the [General Usage](#general-usage) section
+
+For the API reference, see the [Initialization](#initialization) section.
+
+## Installation
+
+```bash
+pip install gradio_agentchatbot
+```
+
+or add `gradio_agentchatbot` to your `requirements.txt`.
+
+## Transformers Usage
+
+For [transformers agents](https://huggingface.co/learn/cookbook/agents), you can use the `stream_from_transformers_agent` function and yield all subsequent messages.
+
+```python
+
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import AgentChatbot, stream_from_transformers_agent, ChatMessage
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+
+# to load SerpAPI key
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(tools=[image_generation_tool, search_tool], llm_engine=llm_engine)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    chatbot = AgentChatbot(label="Agent")
+    text_input = gr.Textbox(lines=1, label="Chat Message")
+    text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+
+
+if __name__ == "__main__":
+    demo.launch()
+```
+
+![AgentChatbot with transformers](https://gradio-builds.s3.amazonaws.com/demo-files/tf_agent_ui.gif)
+
+## General Usage
+
+The `AgentChatbot` is similar to the core `Gradio` `Chatbot` but the key difference is in the expected data format of the `value` property.
+
+Instead of a list of tuples, each of which can be either a string or tuple, the value is a list of message instances. Each message can be either a `ChatMessage` or a `ChatFileMessage`. These are pydantic classes that are compatible with the OpenAI [message format](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages). This is how the are defined:
+
+```python
+class ThoughtMetadata(GradioModel):
+    tool_name: Optional[str] = None
+    error: bool = False
+
+
+class ChatMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    content: str
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+
+
+class ChatFileMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    file: FileData
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+    alt_text: Optional[str] = None
+```
+
+In order to properly display data in `AgentChatbot`, simply return a list of `ChatMessage` or `ChatFileMessage` instances from your python function. For example:
+
+```python
+def chat_echo(prompt: str, messages: List[ChatMessage | ChatFileMessage]) -> List[ChatMessage | ChatFileMessage]:
+    messages.append(ChatMessage(role="user", content=prompt))
+    messages.append(ChatMessage(role="assistant", content=prompt))
+    return messages
+```
+
+### Why a different data format?
+
+The OpenAI data format is the standard format for representing LLM conversations and most API providers have adopted it.
+By using a compliant data format, it should be easier to use `AgentChatbot` with multiple API providers and libraries.
+
+
+### Why are pydantic data classes required?
+
+It should improve developer experience since your editor will auto-complete the required fields and use smart autocomplete for the `role` class. You will also get an error message if your data does not conform to the data format.
+
+I will probably relax this in the future so that a plain python dict can be passed instead of one of the chat classes.
+
+
+
+## `API Reference`
+
+### Initialization
+
+<table>
+<thead>
+<tr>
+<th align="left">name</th>
+<th align="left" style="width: 25%;">type</th>
+<th align="left">default</th>
+<th align="left">description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td align="left"><code>value</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[ChatMessage | ChatFileMessage]
+    | Callable
+    | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Default value to show in chatbot. If callable, the function will be called whenever the app loads to set the initial value of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.</td>
+</tr>
+
+<tr>
+<td align="left"><code>every</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+float | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If `value` is a callable, run the function 'every' number of seconds while the client connection is open. Has no effect otherwise. The event can be accessed (e.g. to cancel it) via this component's .load_event attribute.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if True, will display label.</td>
+</tr>
+
+<tr>
+<td align="left"><code>container</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True, will place the component in a container - providing some extra padding around the border.</td>
+</tr>
+
+<tr>
+<td align="left"><code>scale</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>min_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int
+```
+
+</td>
+<td align="left"><code>160</code></td>
+<td align="left">minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.</td>
+</tr>
+
+<tr>
+<td align="left"><code>visible</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will be hidden.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_id</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_classes</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[str] | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.</td>
+</tr>
+
+<tr>
+<td align="left"><code>key</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.</td>
+</tr>
+
+<tr>
+<td align="left"><code>height</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed.</td>
+</tr>
+
+<tr>
+<td align="left"><code>latex_delimiters</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[dict[str, str | bool]] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">A list of dicts of the form {"left": open delimiter (str), "right": close delimiter (str), "display": whether to display in newline (bool)} that will be used to render LaTeX expressions. If not provided, `latex_delimiters` is set to `[{ "left": "$$", "right": "$$", "display": True }]`, so only expressions enclosed in $$ delimiters will be rendered as LaTeX, and in a new line. Pass in an empty list to disable LaTeX rendering. For more information, see the [KaTeX documentation](https://katex.org/docs/autorender.html).</td>
+</tr>
+
+<tr>
+<td align="left"><code>rtl</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, sets the direction of the rendered text to right-to-left. Default is False, which renders text left-to-right.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_share_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If True, will show a share icon in the corner of the component that allows user to share outputs to Hugging Face Spaces Discussions. If False, icon does not appear. If set to None (default behavior), then the icon appears if this Gradio app is launched on Spaces, but not otherwise.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_copy_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, will show a copy button for each chatbot message.</td>
+</tr>
+
+<tr>
+<td align="left"><code>avatar_images</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+tuple[str | Path | None, str | Path | None] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Tuple of two avatar image paths or URLs for user and bot (in that order). Pass None for either the user or bot image to skip. Must be within the working directory of the Gradio app or an external URL.</td>
+</tr>
+
+<tr>
+<td align="left"><code>sanitize_html</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable HTML sanitization for chatbot messages. This is not recommended, as it can lead to security vulnerabilities.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render_markdown</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable Markdown rendering for chatbot messages.</td>
+</tr>
+
+<tr>
+<td align="left"><code>bubble_full_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, the chat bubble will fit to the content of the message. If True (default), the chat bubble will be the full width of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>line_breaks</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True (default), will enable Github-flavored Markdown line breaks in chatbot messages. If False, single new lines will be ignored. Only applies if `render_markdown` is True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>likeable</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">Whether the chat messages display a like or dislike button. Set automatically by the .like method but has to be present in the signature for it to show up in the config.</td>
+</tr>
+
+<tr>
+<td align="left"><code>layout</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+Literal["panel", "bubble"] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If "panel", will display the chatbot in a llm style layout. If "bubble", will display the chatbot with message bubbles, with the user and bot messages on alterating sides. Will default to "bubble".</td>
+</tr>
+
+<tr>
+<td align="left"><code>placeholder</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">a placeholder message to display in the chatbot when it is empty. Centered vertically and horizontally in the TestChatbot. Supports Markdown and HTML. If None, no placeholder is displayed.</td>
+</tr>
+</tbody></table>
+
+
+### Events
+
+| name | description |
+|:-----|:------------|
+| `change` | Triggered when the value of the TestChatbot changes either because of user input (e.g. a user types in a textbox) OR because of a function update (e.g. an image receives a value from the output of an event trigger). See `.input()` for a listener that is only triggered by user input. | 

app.py

@@ -0,0 +1,54 @@
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import (
+    AgentChatbot,
+    stream_from_transformers_agent,
+    ChatMessage,
+)
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+from pathlib import Path
+
+current_dir = Path(__file__).parent
+
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(
+    tools=[image_generation_tool, search_tool], llm_engine=llm_engine
+)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    with gr.Tabs():
+        with gr.Tab("Demo"):
+            chatbot = AgentChatbot(
+                label="Agent",
+                avatar_images=[
+                    None,
+                    "https://em-content.zobj.net/source/twitter/53/robot-face_1f916.png",
+                ],
+            )
+            text_input = gr.Textbox(lines=1, label="Chat Message")
+            text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+        with gr.Tab("Docs"):
+            gr.Markdown(Path(current_dir / "docs.md").read_text())
+
+
+if __name__ == "__main__":
+    demo.launch()

docs.md

@@ -0,0 +1,465 @@
+# gradio_agentchatbot
+
+## 🤖 Chat UI for displaying the thoughts of LLM Agents 💭
+<img alt="Static Badge" src="https://img.shields.io/badge/version%20-%200.0.1%20-%20orange">  
+
+The `gradio_agentchatbot` package introduces the `AgentChatbot` component which can display the thought process and tool usage of an LLM agent. Its message format is compatible with the OpenAI conversation message format. 
+
+For example usage with transformers agents, please see the [Transformers Usage](#transformers-usage) section.
+
+For general usage, see the [General Usage](#general-usage) section.
+
+For the API reference, see the [Initialization](#initialization) section.
+
+## Installation
+
+```bash
+pip install gradio_agentchatbot
+```
+
+or add `gradio_agentchatbot` to your `requirements.txt`.
+
+## Transformers Usage
+
+For [transformers agents](https://huggingface.co/learn/cookbook/agents), you can use the `stream_from_transformers_agent` function and yield all subsequent messages.
+
+```python
+
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import AgentChatbot, stream_from_transformers_agent, ChatMessage
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+
+# to load SerpAPI key
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(tools=[image_generation_tool, search_tool], llm_engine=llm_engine)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    chatbot = AgentChatbot(label="Agent")
+    text_input = gr.Textbox(lines=1, label="Chat Message")
+    text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+
+
+if __name__ == "__main__":
+    demo.launch()
+```
+
+![AgentChatbot with transformers](https://gradio-builds.s3.amazonaws.com/demo-files/tf_agent_ui.gif)
+
+## General Usage
+
+The `AgentChatbot` is similar to the core `Gradio` `Chatbot` but the key difference is in the expected data format of the `value` property.
+
+Instead of a list of tuples, each of which can be either a string or tuple, the value is a list of message instances. Each message can be either a `ChatMessage` or a `ChatFileMessage`. These are pydantic classes that are compatible with the OpenAI [message format](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages). This is how the are defined:
+
+```python
+class ThoughtMetadata(GradioModel):
+    tool_name: Optional[str] = None
+    error: bool = False
+
+
+class ChatMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    content: str
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+
+
+class ChatFileMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    file: FileData
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+    alt_text: Optional[str] = None
+```
+
+In order to properly display data in `AgentChatbot`, simply return a list of `ChatMessage` or `ChatFileMessage` instances from your python function. For example:
+
+```python
+def chat_echo(prompt: str, messages: List[ChatMessage | ChatFileMessage]) -> List[ChatMessage | ChatFileMessage]:
+    messages.append(ChatMessage(role="user", content=prompt))
+    messages.append(ChatMessage(role="assistant", content=prompt))
+    return messages
+```
+
+### Why a different data format?
+
+The OpenAI data format is the standard format for representing LLM conversations and most API providers have adopted it.
+By using a compliant data format, it should be easier to use `AgentChatbot` with multiple API providers and libraries.
+
+
+### Why are pydantic data classes required?
+
+It should improve developer experience since your editor will auto-complete the required fields and use smart autocomplete for the `role` class. You will also get an error message if your data does not conform to the data format.
+
+I will probably relax this in the future so that a plain python dict can be passed instead of one of the chat classes.
+
+
+
+## `API Reference`
+
+### Initialization
+
+<table>
+<thead>
+<tr>
+<th align="left">name</th>
+<th align="left" style="width: 25%;">type</th>
+<th align="left">default</th>
+<th align="left">description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td align="left"><code>value</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[ChatMessage | ChatFileMessage]
+    | Callable
+    | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Default value to show in chatbot. If callable, the function will be called whenever the app loads to set the initial value of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.</td>
+</tr>
+
+<tr>
+<td align="left"><code>every</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+float | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If `value` is a callable, run the function 'every' number of seconds while the client connection is open. Has no effect otherwise. The event can be accessed (e.g. to cancel it) via this component's .load_event attribute.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if True, will display label.</td>
+</tr>
+
+<tr>
+<td align="left"><code>container</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True, will place the component in a container - providing some extra padding around the border.</td>
+</tr>
+
+<tr>
+<td align="left"><code>scale</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>min_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int
+```
+
+</td>
+<td align="left"><code>160</code></td>
+<td align="left">minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.</td>
+</tr>
+
+<tr>
+<td align="left"><code>visible</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will be hidden.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_id</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_classes</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[str] | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.</td>
+</tr>
+
+<tr>
+<td align="left"><code>key</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.</td>
+</tr>
+
+<tr>
+<td align="left"><code>height</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed.</td>
+</tr>
+
+<tr>
+<td align="left"><code>latex_delimiters</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[dict[str, str | bool]] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">A list of dicts of the form {"left": open delimiter (str), "right": close delimiter (str), "display": whether to display in newline (bool)} that will be used to render LaTeX expressions. If not provided, `latex_delimiters` is set to `[{ "left": "$$", "right": "$$", "display": True }]`, so only expressions enclosed in $$ delimiters will be rendered as LaTeX, and in a new line. Pass in an empty list to disable LaTeX rendering. For more information, see the [KaTeX documentation](https://katex.org/docs/autorender.html).</td>
+</tr>
+
+<tr>
+<td align="left"><code>rtl</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, sets the direction of the rendered text to right-to-left. Default is False, which renders text left-to-right.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_share_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If True, will show a share icon in the corner of the component that allows user to share outputs to Hugging Face Spaces Discussions. If False, icon does not appear. If set to None (default behavior), then the icon appears if this Gradio app is launched on Spaces, but not otherwise.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_copy_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, will show a copy button for each chatbot message.</td>
+</tr>
+
+<tr>
+<td align="left"><code>avatar_images</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+tuple[str | Path | None, str | Path | None] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Tuple of two avatar image paths or URLs for user and bot (in that order). Pass None for either the user or bot image to skip. Must be within the working directory of the Gradio app or an external URL.</td>
+</tr>
+
+<tr>
+<td align="left"><code>sanitize_html</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable HTML sanitization for chatbot messages. This is not recommended, as it can lead to security vulnerabilities.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render_markdown</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable Markdown rendering for chatbot messages.</td>
+</tr>
+
+<tr>
+<td align="left"><code>bubble_full_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, the chat bubble will fit to the content of the message. If True (default), the chat bubble will be the full width of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>line_breaks</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True (default), will enable Github-flavored Markdown line breaks in chatbot messages. If False, single new lines will be ignored. Only applies if `render_markdown` is True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>likeable</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">Whether the chat messages display a like or dislike button. Set automatically by the .like method but has to be present in the signature for it to show up in the config.</td>
+</tr>
+
+<tr>
+<td align="left"><code>layout</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+Literal["panel", "bubble"] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If "panel", will display the chatbot in a llm style layout. If "bubble", will display the chatbot with message bubbles, with the user and bot messages on alterating sides. Will default to "bubble".</td>
+</tr>
+
+<tr>
+<td align="left"><code>placeholder</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">a placeholder message to display in the chatbot when it is empty. Centered vertically and horizontally in the TestChatbot. Supports Markdown and HTML. If None, no placeholder is displayed.</td>
+</tr>
+</tbody></table>
+
+
+### Events
+
+| name | description |
+|:-----|:------------|
+| `change` | Triggered when the value of the TestChatbot changes either because of user input (e.g. a user types in a textbox) OR because of a function update (e.g. an image receives a value from the output of an event trigger). See `.input()` for a listener that is only triggered by user input. | 

requirements.txt

@@ -0,0 +1,7 @@
+gradio_agentchatbot
+"git+https://github.com/huggingface/transformers.git#egg=transformers[agents]"
+langchain
+sentence-transformers
+faiss-cpu
+serpapi
+google-search-results

src/.gitignore

@@ -0,0 +1,12 @@
+.eggs/
+dist/
+*.pyc
+__pycache__/
+*.py[cod]
+*$py.class
+__tmp/*
+*.pyi
+.mypycache
+.ruff_cache
+node_modules
+backend/**/templates/

src/README.md

@@ -0,0 +1,476 @@
+---
+tags: [gradio-custom-component, Chatbot, chatbot, agents, streaming, tools]
+title: gradio_agentchatbot
+short_description: Chat with agents and see the tools they use
+colorFrom: blue
+colorTo: yellow
+sdk: gradio
+pinned: false
+app_file: space.py
+---
+
+# gradio_agentchatbot
+
+## 🤖 Chat UI for displaying the thoughts of LLM Agents 💭
+<img alt="Static Badge" src="https://img.shields.io/badge/version%20-%200.0.1%20-%20orange">  
+
+The `gradio_agentchatbot` package introduces the `AgentChatbot` component which can display the thought process and tool usage of an LLM agent. Its message format is compatible with the OpenAI conversation message format. 
+
+For example usage with transformers agents, please see the [Transformers Usage](#transformers-usage) section.
+
+For general usage, see the [General Usage](#general-usage) section
+
+For the API reference, see the [Initialization](#initialization) section.
+
+## Installation
+
+```bash
+pip install gradio_agentchatbot
+```
+
+or add `gradio_agentchatbot` to your `requirements.txt`.
+
+## Transformers Usage
+
+For [transformers agents](https://huggingface.co/learn/cookbook/agents), you can use the `stream_from_transformers_agent` function and yield all subsequent messages.
+
+```python
+
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import AgentChatbot, stream_from_transformers_agent, ChatMessage
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+
+# to load SerpAPI key
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(tools=[image_generation_tool, search_tool], llm_engine=llm_engine)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    chatbot = AgentChatbot(label="Agent")
+    text_input = gr.Textbox(lines=1, label="Chat Message")
+    text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+
+
+if __name__ == "__main__":
+    demo.launch()
+```
+
+![AgentChatbot with transformers](https://gradio-builds.s3.amazonaws.com/demo-files/tf_agent_ui.gif)
+
+## General Usage
+
+The `AgentChatbot` is similar to the core `Gradio` `Chatbot` but the key difference is in the expected data format of the `value` property.
+
+Instead of a list of tuples, each of which can be either a string or tuple, the value is a list of message instances. Each message can be either a `ChatMessage` or a `ChatFileMessage`. These are pydantic classes that are compatible with the OpenAI [message format](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages). This is how the are defined:
+
+```python
+class ThoughtMetadata(GradioModel):
+    tool_name: Optional[str] = None
+    error: bool = False
+
+
+class ChatMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    content: str
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+
+
+class ChatFileMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    file: FileData
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+    alt_text: Optional[str] = None
+```
+
+In order to properly display data in `AgentChatbot`, simply return a list of `ChatMessage` or `ChatFileMessage` instances from your python function. For example:
+
+```python
+def chat_echo(prompt: str, messages: List[ChatMessage | ChatFileMessage]) -> List[ChatMessage | ChatFileMessage]:
+    messages.append(ChatMessage(role="user", content=prompt))
+    messages.append(ChatMessage(role="assistant", content=prompt))
+    return messages
+```
+
+### Why a different data format?
+
+The OpenAI data format is the standard format for representing LLM conversations and most API providers have adopted it.
+By using a compliant data format, it should be easier to use `AgentChatbot` with multiple API providers and libraries.
+
+
+### Why are pydantic data classes required?
+
+It should improve developer experience since your editor will auto-complete the required fields and use smart autocomplete for the `role` class. You will also get an error message if your data does not conform to the data format.
+
+I will probably relax this in the future so that a plain python dict can be passed instead of one of the chat classes.
+
+
+
+## `API Reference`
+
+### Initialization
+
+<table>
+<thead>
+<tr>
+<th align="left">name</th>
+<th align="left" style="width: 25%;">type</th>
+<th align="left">default</th>
+<th align="left">description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td align="left"><code>value</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[ChatMessage | ChatFileMessage]
+    | Callable
+    | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Default value to show in chatbot. If callable, the function will be called whenever the app loads to set the initial value of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.</td>
+</tr>
+
+<tr>
+<td align="left"><code>every</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+float | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If `value` is a callable, run the function 'every' number of seconds while the client connection is open. Has no effect otherwise. The event can be accessed (e.g. to cancel it) via this component's .load_event attribute.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if True, will display label.</td>
+</tr>
+
+<tr>
+<td align="left"><code>container</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True, will place the component in a container - providing some extra padding around the border.</td>
+</tr>
+
+<tr>
+<td align="left"><code>scale</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>min_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int
+```
+
+</td>
+<td align="left"><code>160</code></td>
+<td align="left">minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.</td>
+</tr>
+
+<tr>
+<td align="left"><code>visible</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will be hidden.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_id</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_classes</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[str] | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.</td>
+</tr>
+
+<tr>
+<td align="left"><code>key</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.</td>
+</tr>
+
+<tr>
+<td align="left"><code>height</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed.</td>
+</tr>
+
+<tr>
+<td align="left"><code>latex_delimiters</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[dict[str, str | bool]] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">A list of dicts of the form {"left": open delimiter (str), "right": close delimiter (str), "display": whether to display in newline (bool)} that will be used to render LaTeX expressions. If not provided, `latex_delimiters` is set to `[{ "left": "$$", "right": "$$", "display": True }]`, so only expressions enclosed in $$ delimiters will be rendered as LaTeX, and in a new line. Pass in an empty list to disable LaTeX rendering. For more information, see the [KaTeX documentation](https://katex.org/docs/autorender.html).</td>
+</tr>
+
+<tr>
+<td align="left"><code>rtl</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, sets the direction of the rendered text to right-to-left. Default is False, which renders text left-to-right.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_share_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If True, will show a share icon in the corner of the component that allows user to share outputs to Hugging Face Spaces Discussions. If False, icon does not appear. If set to None (default behavior), then the icon appears if this Gradio app is launched on Spaces, but not otherwise.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_copy_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, will show a copy button for each chatbot message.</td>
+</tr>
+
+<tr>
+<td align="left"><code>avatar_images</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+tuple[str | Path | None, str | Path | None] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Tuple of two avatar image paths or URLs for user and bot (in that order). Pass None for either the user or bot image to skip. Must be within the working directory of the Gradio app or an external URL.</td>
+</tr>
+
+<tr>
+<td align="left"><code>sanitize_html</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable HTML sanitization for chatbot messages. This is not recommended, as it can lead to security vulnerabilities.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render_markdown</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable Markdown rendering for chatbot messages.</td>
+</tr>
+
+<tr>
+<td align="left"><code>bubble_full_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, the chat bubble will fit to the content of the message. If True (default), the chat bubble will be the full width of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>line_breaks</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True (default), will enable Github-flavored Markdown line breaks in chatbot messages. If False, single new lines will be ignored. Only applies if `render_markdown` is True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>likeable</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">Whether the chat messages display a like or dislike button. Set automatically by the .like method but has to be present in the signature for it to show up in the config.</td>
+</tr>
+
+<tr>
+<td align="left"><code>layout</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+Literal["panel", "bubble"] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If "panel", will display the chatbot in a llm style layout. If "bubble", will display the chatbot with message bubbles, with the user and bot messages on alterating sides. Will default to "bubble".</td>
+</tr>
+
+<tr>
+<td align="left"><code>placeholder</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">a placeholder message to display in the chatbot when it is empty. Centered vertically and horizontally in the TestChatbot. Supports Markdown and HTML. If None, no placeholder is displayed.</td>
+</tr>
+</tbody></table>
+
+
+### Events
+
+| name | description |
+|:-----|:------------|
+| `change` | Triggered when the value of the TestChatbot changes either because of user input (e.g. a user types in a textbox) OR because of a function update (e.g. an image receives a value from the output of an event trigger). See `.input()` for a listener that is only triggered by user input. | 

src/backend/gradio_agentchatbot/__init__.py

@@ -0,0 +1,18 @@
+from .agentchatbot import AgentChatbot, ChatbotData
+from .utils import (
+    stream_from_transformers_agent,
+    ChatMessage,
+    ThoughtMetadata,
+    ChatFileMessage,
+    Message,
+)
+
+__all__ = [
+    "AgentChatbot",
+    "ChatbotData",
+    "stream_from_transformers_agent",
+    "ChatMessage",
+    "ThoughtMetadata",
+    "ChatFileMessage",
+    "Message",
+]

src/backend/gradio_agentchatbot/agentchatbot.py

@@ -0,0 +1,196 @@
+"""gr.Chatbot() component."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Callable, Literal
+
+from gradio_client import utils as client_utils
+
+from gradio import utils
+from gradio.components.base import Component
+from gradio.events import Events
+from gradio.processing_utils import move_resource_to_block_cache
+from .utils import ChatMessage, ChatbotData, ChatFileMessage, FileData
+
+
+class AgentChatbot(Component):
+    """
+    Creates a chatbot that displays user-submitted messages and responses. Supports a subset of Markdown including bold, italics, code, tables.
+    Also supports audio/video/image files, which are displayed in the AgentChatbot, and other kinds of files which are displayed as links. This
+    component is usually used as an output component.
+
+    Demos: chatbot_simple, chatbot_multimodal
+    Guides: creating-a-chatbot
+    """
+
+    EVENTS = [Events.change]
+    data_model = ChatbotData
+
+    def __init__(
+        self,
+        value: list[dict[str, Any]] | Callable | None = None,
+        *,
+        label: str | None = None,
+        every: float | None = None,
+        show_label: bool | None = None,
+        container: bool = True,
+        scale: int | None = None,
+        min_width: int = 160,
+        visible: bool = True,
+        elem_id: str | None = None,
+        elem_classes: list[str] | str | None = None,
+        render: bool = True,
+        key: int | str | None = None,
+        height: int | str | None = None,
+        latex_delimiters: list[dict[str, str | bool]] | None = None,
+        rtl: bool = False,
+        show_share_button: bool | None = None,
+        show_copy_button: bool = False,
+        avatar_images: tuple[str | Path | None, str | Path | None] | None = None,
+        sanitize_html: bool = True,
+        render_markdown: bool = True,
+        bubble_full_width: bool = True,
+        line_breaks: bool = True,
+        likeable: bool = False,
+        layout: Literal["panel", "bubble"] | None = None,
+        placeholder: str | None = None,
+    ):
+        """
+        Parameters:
+            value: Default value to show in chatbot. If callable, the function will be called whenever the app loads to set the initial value of the component.
+            label: The label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.
+            every: If `value` is a callable, run the function 'every' number of seconds while the client connection is open. Has no effect otherwise. The event can be accessed (e.g. to cancel it) via this component's .load_event attribute.
+            show_label: if True, will display label.
+            container: If True, will place the component in a container - providing some extra padding around the border.
+            scale: relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.
+            min_width: minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.
+            visible: If False, component will be hidden.
+            elem_id: An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.
+            elem_classes: An optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.
+            render: If False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.
+            key: if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.
+            height: The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed.
+            latex_delimiters: A list of dicts of the form {"left": open delimiter (str), "right": close delimiter (str), "display": whether to display in newline (bool)} that will be used to render LaTeX expressions. If not provided, `latex_delimiters` is set to `[{ "left": "$$", "right": "$$", "display": True }]`, so only expressions enclosed in $$ delimiters will be rendered as LaTeX, and in a new line. Pass in an empty list to disable LaTeX rendering. For more information, see the [KaTeX documentation](https://katex.org/docs/autorender.html).
+            rtl: If True, sets the direction of the rendered text to right-to-left. Default is False, which renders text left-to-right.
+            show_share_button: If True, will show a share icon in the corner of the component that allows user to share outputs to Hugging Face Spaces Discussions. If False, icon does not appear. If set to None (default behavior), then the icon appears if this Gradio app is launched on Spaces, but not otherwise.
+            show_copy_button: If True, will show a copy button for each chatbot message.
+            avatar_images: Tuple of two avatar image paths or URLs for user and bot (in that order). Pass None for either the user or bot image to skip. Must be within the working directory of the Gradio app or an external URL.
+            sanitize_html: If False, will disable HTML sanitization for chatbot messages. This is not recommended, as it can lead to security vulnerabilities.
+            render_markdown: If False, will disable Markdown rendering for chatbot messages.
+            bubble_full_width: If False, the chat bubble will fit to the content of the message. If True (default), the chat bubble will be the full width of the component.
+            line_breaks: If True (default), will enable Github-flavored Markdown line breaks in chatbot messages. If False, single new lines will be ignored. Only applies if `render_markdown` is True.
+            likeable: Whether the chat messages display a like or dislike button. Set automatically by the .like method but has to be present in the signature for it to show up in the config.
+            layout: If "panel", will display the chatbot in a llm style layout. If "bubble", will display the chatbot with message bubbles, with the user and bot messages on alterating sides. Will default to "bubble".
+            placeholder: a placeholder message to display in the chatbot when it is empty. Centered vertically and horizontally in the AgentChatbot. Supports Markdown and HTML. If None, no placeholder is displayed.
+        """
+        self.likeable = likeable
+        self.height = height
+        self.rtl = rtl
+        if latex_delimiters is None:
+            latex_delimiters = [{"left": "$$", "right": "$$", "display": True}]
+        self.latex_delimiters = latex_delimiters
+        self.show_share_button = (
+            (utils.get_space() is not None)
+            if show_share_button is None
+            else show_share_button
+        )
+        self.render_markdown = render_markdown
+        self.show_copy_button = show_copy_button
+        self.sanitize_html = sanitize_html
+        self.bubble_full_width = bubble_full_width
+        self.line_breaks = line_breaks
+        self.layout = layout
+        super().__init__(
+            label=label,
+            every=every,
+            show_label=show_label,
+            container=container,
+            scale=scale,
+            min_width=min_width,
+            visible=visible,
+            elem_id=elem_id,
+            elem_classes=elem_classes,
+            render=render,
+            key=key,
+            value=value,
+        )
+        self.avatar_images: list[dict | None] = [None, None]
+        if avatar_images is None:
+            pass
+        else:
+            self.avatar_images = [
+                self.serve_static_file(avatar_images[0]),
+                self.serve_static_file(avatar_images[1]),
+            ]
+        self.placeholder = placeholder
+
+    def preprocess(
+        self,
+        payload: ChatbotData | None,
+    ) -> list[ChatMessage | ChatFileMessage] | None:
+        """
+        Parameters:
+            payload: data as a ChatbotData object
+        Returns:
+            Passes the messages in the chatbot as a `list[list[str | None | tuple]]`, i.e. a list of lists. The inner list has 2 elements: the user message and the response message. Each message can be (1) a string in valid Markdown, (2) a tuple if there are displayed files: (a filepath or URL to a file, [optional string alt text]), or (3) None, if there is no message displayed.
+        """
+        return payload.root
+
+    def _postprocess_chat_messages(
+        self, chat_message: ChatMessage
+    ) -> list[ChatMessage]:
+        if chat_message is None:
+            return None
+
+        if isinstance(chat_message, ChatFileMessage):
+            chat_message.file.path = move_resource_to_block_cache(
+                chat_message.file.path, block=self
+            )
+            return [chat_message]
+
+        # extract file path from message
+        new_messages = []
+        for word in chat_message.content.split(" "):
+            if (filepath := Path(word)).exists() and filepath.is_file():
+                filepath = move_resource_to_block_cache(filepath, block=self)
+                mime_type = client_utils.get_mimetype(filepath)
+                new_messages.append(
+                    ChatFileMessage(
+                        role=chat_message.role,
+                        thought=chat_message.thought,
+                        thought_metadata=chat_message.thought_metadata,
+                        file=FileData(path=filepath, mime_type=mime_type),
+                    )
+                )
+
+        return [chat_message, *new_messages]
+
+    def postprocess(
+        self,
+        value: list[ChatMessage | ChatFileMessage] | None,
+    ) -> ChatbotData:
+        """
+        Parameters:
+            value: expects a `list[list[str | None | tuple]]`, i.e. a list of lists. The inner list should have 2 elements: the user message and the response message. The individual messages can be (1) strings in valid Markdown, (2) tuples if sending files: (a filepath or URL to a file, [optional string alt text]) -- if the file is image/video/audio, it is displayed in the AgentChatbot, or (3) None, in which case the message is not displayed.
+        Returns:
+            an object of type ChatbotData
+        """
+        if value is None:
+            return ChatbotData(root=[])
+        processed_messages = [
+            msg for message in value for msg in self._postprocess_chat_messages(message)
+        ]
+        return ChatbotData(root=processed_messages)
+
+    def example_payload(self) -> Any:
+        return [
+            ChatMessage(role="user", content="Hello!").model_dump(),
+            ChatMessage(role="assistant", content="How can I help you?").model_dump(),
+        ]
+
+    def example_value(self) -> Any:
+        return [
+            ChatMessage(role="user", content="Hello!"),
+            ChatMessage(role="assistant", content="How can I help you?"),
+        ]

src/backend/gradio_agentchatbot/templates/component/assets/worker-lPYB70QI.js

@@ -0,0 +1 @@
+(function(){"use strict";const R="https://unpkg.com/@ffmpeg/core@0.12.6/dist/umd/ffmpeg-core.js";var E;(function(t){t.LOAD="LOAD",t.EXEC="EXEC",t.WRITE_FILE="WRITE_FILE",t.READ_FILE="READ_FILE",t.DELETE_FILE="DELETE_FILE",t.RENAME="RENAME",t.CREATE_DIR="CREATE_DIR",t.LIST_DIR="LIST_DIR",t.DELETE_DIR="DELETE_DIR",t.ERROR="ERROR",t.DOWNLOAD="DOWNLOAD",t.PROGRESS="PROGRESS",t.LOG="LOG",t.MOUNT="MOUNT",t.UNMOUNT="UNMOUNT"})(E||(E={}));const a=new Error("unknown message type"),f=new Error("ffmpeg is not loaded, call `await ffmpeg.load()` first"),u=new Error("failed to import ffmpeg-core.js");let r;const O=async({coreURL:t,wasmURL:n,workerURL:e})=>{const o=!r;try{t||(t=R),importScripts(t)}catch{if(t||(t=R.replace("/umd/","/esm/")),self.createFFmpegCore=(await import(t)).default,!self.createFFmpegCore)throw u}const s=t,c=n||t.replace(/.js$/g,".wasm"),b=e||t.replace(/.js$/g,".worker.js");return r=await self.createFFmpegCore({mainScriptUrlOrBlob:`${s}#${btoa(JSON.stringify({wasmURL:c,workerURL:b}))}`}),r.setLogger(i=>self.postMessage({type:E.LOG,data:i})),r.setProgress(i=>self.postMessage({type:E.PROGRESS,data:i})),o},l=({args:t,timeout:n=-1})=>{r.setTimeout(n),r.exec(...t);const e=r.ret;return r.reset(),e},m=({path:t,data:n})=>(r.FS.writeFile(t,n),!0),D=({path:t,encoding:n})=>r.FS.readFile(t,{encoding:n}),S=({path:t})=>(r.FS.unlink(t),!0),I=({oldPath:t,newPath:n})=>(r.FS.rename(t,n),!0),L=({path:t})=>(r.FS.mkdir(t),!0),N=({path:t})=>{const n=r.FS.readdir(t),e=[];for(const o of n){const s=r.FS.stat(`${t}/${o}`),c=r.FS.isDir(s.mode);e.push({name:o,isDir:c})}return e},A=({path:t})=>(r.FS.rmdir(t),!0),w=({fsType:t,options:n,mountPoint:e})=>{const o=t,s=r.FS.filesystems[o];return s?(r.FS.mount(s,n,e),!0):!1},k=({mountPoint:t})=>(r.FS.unmount(t),!0);self.onmessage=async({data:{id:t,type:n,data:e}})=>{const o=[];let s;try{if(n!==E.LOAD&&!r)throw f;switch(n){case E.LOAD:s=await O(e);break;case E.EXEC:s=l(e);break;case E.WRITE_FILE:s=m(e);break;case E.READ_FILE:s=D(e);break;case E.DELETE_FILE:s=S(e);break;case E.RENAME:s=I(e);break;case E.CREATE_DIR:s=L(e);break;case E.LIST_DIR:s=N(e);break;case E.DELETE_DIR:s=A(e);break;case E.MOUNT:s=w(e);break;case E.UNMOUNT:s=k(e);break;default:throw a}}catch(c){self.postMessage({id:t,type:E.ERROR,data:c.toString()});return}s instanceof Uint8Array&&o.push(s.buffer),self.postMessage({id:t,type:n,data:s},o)}})();

src/backend/gradio_agentchatbot/utils.py

@@ -0,0 +1,117 @@
+from transformers.agents import Agent, agent_types
+from pydantic import Field
+from gradio.data_classes import GradioModel, FileData, GradioRootModel
+from typing import Literal, List, Generator, Optional, Union
+from threading import Thread
+import time
+
+
+class ThoughtMetadata(GradioModel):
+    tool_name: Optional[str] = None
+    error: bool = False
+
+
+class Message(GradioModel):
+    role: Literal["user", "assistant"]
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+
+
+class ChatMessage(Message):
+    content: str
+
+
+class ChatFileMessage(Message):
+    file: FileData
+    alt_text: Optional[str] = None
+
+
+class ChatbotData(GradioRootModel):
+    root: List[Union[ChatMessage, ChatFileMessage]]
+
+
+def pull_messages(new_messages: List[dict]):
+    for message in new_messages:
+        if not len(message):
+            continue
+        if message.get("rationale"):
+            yield ChatMessage(
+                role="assistant", content=message["rationale"], thought=True
+            )
+        if message.get("tool_call"):
+            used_code = message["tool_call"]["tool_name"] == "code interpreter"
+            content = message["tool_call"]["tool_arguments"]
+            if used_code:
+                content = f"```py\n{content}\n```"
+            yield ChatMessage(
+                role="assistant",
+                thought_metadata=ThoughtMetadata(
+                    tool_name=message["tool_call"]["tool_name"]
+                ),
+                content=content,
+                thought=True,
+            )
+        if message.get("observation"):
+            yield ChatMessage(
+                role="assistant", content=message["observation"], thought=True
+            )
+        if message.get("error"):
+            yield ChatMessage(
+                role="assistant",
+                content=str(message["error"]),
+                thought=True,
+                thought_metadata=ThoughtMetadata(error=True),
+            )
+
+
+def stream_from_transformers_agent(
+    agent: Agent, prompt: str
+) -> Generator[ChatMessage, None, None]:
+    """Runs an agent with the given prompt and streams the messages from the agent as ChatMessages."""
+
+    class Output:
+        output: agent_types.AgentType | str = None
+
+    def run_agent():
+        output = agent.run(prompt)
+        Output.output = output
+
+    thread = Thread(target=run_agent)
+    num_messages = 0
+
+    # Start thread and pull logs while it runs
+    thread.start()
+    while thread.is_alive():
+        if len(agent.logs) > num_messages:
+            new_messages = agent.logs[num_messages:]
+            for msg in pull_messages(new_messages):
+                yield msg
+                num_messages += 1
+        time.sleep(0.1)
+
+    thread.join(0.1)
+
+    if len(agent.logs) > num_messages:
+        new_messages = agent.logs[num_messages:]
+        yield from pull_messages(new_messages)
+
+    if isinstance(Output.output, agent_types.AgentText):
+        yield ChatMessage(
+            role="assistant", content=Output.output.to_string(), thought=True
+        )
+    elif isinstance(Output.output, agent_types.AgentImage):
+        yield ChatFileMessage(
+            role="assistant",
+            file=FileData(path=Output.output.to_string(), mime_type="image/png"),
+            content="",
+            thought=True,
+        )
+    elif isinstance(Output.output, agent_types.AgentAudio):
+        yield ChatFileMessage(
+            role="assistant",
+            file=FileData(path=Output.output.to_string(), mime_type="audio/wav"),
+            content="",
+            thought=True,
+        )
+    else:
+        return ChatMessage(role="assistant", content=Output.output, thought=True)

src/demo/.env

@@ -0,0 +1 @@
+SERPAPI_API_KEY=3d009ed0a9764853b3e75ee2c70ceb3aabd92b810456307c5af711559ec4aff7

src/demo/app.py

@@ -0,0 +1,54 @@
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import (
+    AgentChatbot,
+    stream_from_transformers_agent,
+    ChatMessage,
+)
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+from pathlib import Path
+
+current_dir = Path(__file__).parent
+
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(
+    tools=[image_generation_tool, search_tool], llm_engine=llm_engine
+)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    with gr.Tabs():
+        with gr.Tab("Demo"):
+            chatbot = AgentChatbot(
+                label="Agent",
+                avatar_images=[
+                    None,
+                    "https://em-content.zobj.net/source/twitter/53/robot-face_1f916.png",
+                ],
+            )
+            text_input = gr.Textbox(lines=1, label="Chat Message")
+            text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+        with gr.Tab("Docs"):
+            gr.Markdown(Path(current_dir / "docs.md").read_text())
+
+
+if __name__ == "__main__":
+    demo.launch()

src/demo/docs.md

@@ -0,0 +1,465 @@
+# gradio_agentchatbot
+
+## 🤖 Chat UI for displaying the thoughts of LLM Agents 💭
+<img alt="Static Badge" src="https://img.shields.io/badge/version%20-%200.0.1%20-%20orange">  
+
+The `gradio_agentchatbot` package introduces the `AgentChatbot` component which can display the thought process and tool usage of an LLM agent. Its message format is compatible with the OpenAI conversation message format. 
+
+For example usage with transformers agents, please see the [Transformers Usage](#transformers-usage) section.
+
+For general usage, see the [General Usage](#general-usage) section.
+
+For the API reference, see the [Initialization](#initialization) section.
+
+## Installation
+
+```bash
+pip install gradio_agentchatbot
+```
+
+or add `gradio_agentchatbot` to your `requirements.txt`.
+
+## Transformers Usage
+
+For [transformers agents](https://huggingface.co/learn/cookbook/agents), you can use the `stream_from_transformers_agent` function and yield all subsequent messages.
+
+```python
+
+import gradio as gr
+from transformers import load_tool, ReactCodeAgent, HfEngine, Tool
+from gradio_agentchatbot import AgentChatbot, stream_from_transformers_agent, ChatMessage
+from dotenv import load_dotenv
+from langchain.agents import load_tools
+
+# to load SerpAPI key
+load_dotenv()
+
+# Import tool from Hub
+image_generation_tool = load_tool("m-ric/text-to-image")
+
+search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
+
+llm_engine = HfEngine("meta-llama/Meta-Llama-3-70B-Instruct")
+# Initialize the agent with both tools
+agent = ReactCodeAgent(tools=[image_generation_tool, search_tool], llm_engine=llm_engine)
+
+
+def interact_with_agent(prompt, messages):
+    messages.append(ChatMessage(role="user", content=prompt))
+    yield messages
+    for msg in stream_from_transformers_agent(agent, prompt):
+        messages.append(msg)
+        yield messages
+    yield messages
+
+
+with gr.Blocks() as demo:
+    chatbot = AgentChatbot(label="Agent")
+    text_input = gr.Textbox(lines=1, label="Chat Message")
+    text_input.submit(interact_with_agent, [text_input, chatbot], [chatbot])
+
+
+if __name__ == "__main__":
+    demo.launch()
+```
+
+![AgentChatbot with transformers](https://gradio-builds.s3.amazonaws.com/demo-files/tf_agent_ui.gif)
+
+## General Usage
+
+The `AgentChatbot` is similar to the core `Gradio` `Chatbot` but the key difference is in the expected data format of the `value` property.
+
+Instead of a list of tuples, each of which can be either a string or tuple, the value is a list of message instances. Each message can be either a `ChatMessage` or a `ChatFileMessage`. These are pydantic classes that are compatible with the OpenAI [message format](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages). This is how the are defined:
+
+```python
+class ThoughtMetadata(GradioModel):
+    tool_name: Optional[str] = None
+    error: bool = False
+
+
+class ChatMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    content: str
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+
+
+class ChatFileMessage(GradioModel):
+    role: Literal["user", "assistant"]
+    file: FileData
+    thought: bool = False
+    thought_metadata: ThoughtMetadata = Field(default_factory=ThoughtMetadata)
+    alt_text: Optional[str] = None
+```
+
+In order to properly display data in `AgentChatbot`, simply return a list of `ChatMessage` or `ChatFileMessage` instances from your python function. For example:
+
+```python
+def chat_echo(prompt: str, messages: List[ChatMessage | ChatFileMessage]) -> List[ChatMessage | ChatFileMessage]:
+    messages.append(ChatMessage(role="user", content=prompt))
+    messages.append(ChatMessage(role="assistant", content=prompt))
+    return messages
+```
+
+### Why a different data format?
+
+The OpenAI data format is the standard format for representing LLM conversations and most API providers have adopted it.
+By using a compliant data format, it should be easier to use `AgentChatbot` with multiple API providers and libraries.
+
+
+### Why are pydantic data classes required?
+
+It should improve developer experience since your editor will auto-complete the required fields and use smart autocomplete for the `role` class. You will also get an error message if your data does not conform to the data format.
+
+I will probably relax this in the future so that a plain python dict can be passed instead of one of the chat classes.
+
+
+
+## `API Reference`
+
+### Initialization
+
+<table>
+<thead>
+<tr>
+<th align="left">name</th>
+<th align="left" style="width: 25%;">type</th>
+<th align="left">default</th>
+<th align="left">description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td align="left"><code>value</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[ChatMessage | ChatFileMessage]
+    | Callable
+    | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Default value to show in chatbot. If callable, the function will be called whenever the app loads to set the initial value of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The label for this component. Appears above the component and is also used as the header if there are a table of examples for this component. If None and used in a `gr.Interface`, the label will be the name of the parameter this component is assigned to.</td>
+</tr>
+
+<tr>
+<td align="left"><code>every</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+float | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If `value` is a callable, run the function 'every' number of seconds while the client connection is open. Has no effect otherwise. The event can be accessed (e.g. to cancel it) via this component's .load_event attribute.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_label</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if True, will display label.</td>
+</tr>
+
+<tr>
+<td align="left"><code>container</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True, will place the component in a container - providing some extra padding around the border.</td>
+</tr>
+
+<tr>
+<td align="left"><code>scale</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">relative size compared to adjacent Components. For example if Components A and B are in a Row, and A has scale=2, and B has scale=1, A will be twice as wide as B. Should be an integer. scale applies in Rows, and to top-level Components in Blocks where fill_height=True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>min_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int
+```
+
+</td>
+<td align="left"><code>160</code></td>
+<td align="left">minimum pixel width, will wrap if not sufficient screen space to satisfy this value. If a certain scale value results in this Component being narrower than min_width, the min_width parameter will be respected first.</td>
+</tr>
+
+<tr>
+<td align="left"><code>visible</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will be hidden.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_id</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>elem_classes</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[str] | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">An optional list of strings that are assigned as the classes of this component in the HTML DOM. Can be used for targeting CSS styles.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, component will not render be rendered in the Blocks context. Should be used if the intention is to assign event listeners now but render the component later.</td>
+</tr>
+
+<tr>
+<td align="left"><code>key</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">if assigned, will be used to assume identity across a re-render. Components that have the same key across a re-render will have their value preserved.</td>
+</tr>
+
+<tr>
+<td align="left"><code>height</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+int | str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">The height of the component, specified in pixels if a number is passed, or in CSS units if a string is passed.</td>
+</tr>
+
+<tr>
+<td align="left"><code>latex_delimiters</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+list[dict[str, str | bool]] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">A list of dicts of the form {"left": open delimiter (str), "right": close delimiter (str), "display": whether to display in newline (bool)} that will be used to render LaTeX expressions. If not provided, `latex_delimiters` is set to `[{ "left": "$$", "right": "$$", "display": True }]`, so only expressions enclosed in $$ delimiters will be rendered as LaTeX, and in a new line. Pass in an empty list to disable LaTeX rendering. For more information, see the [KaTeX documentation](https://katex.org/docs/autorender.html).</td>
+</tr>
+
+<tr>
+<td align="left"><code>rtl</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, sets the direction of the rendered text to right-to-left. Default is False, which renders text left-to-right.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_share_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If True, will show a share icon in the corner of the component that allows user to share outputs to Hugging Face Spaces Discussions. If False, icon does not appear. If set to None (default behavior), then the icon appears if this Gradio app is launched on Spaces, but not otherwise.</td>
+</tr>
+
+<tr>
+<td align="left"><code>show_copy_button</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">If True, will show a copy button for each chatbot message.</td>
+</tr>
+
+<tr>
+<td align="left"><code>avatar_images</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+tuple[str | Path | None, str | Path | None] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">Tuple of two avatar image paths or URLs for user and bot (in that order). Pass None for either the user or bot image to skip. Must be within the working directory of the Gradio app or an external URL.</td>
+</tr>
+
+<tr>
+<td align="left"><code>sanitize_html</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable HTML sanitization for chatbot messages. This is not recommended, as it can lead to security vulnerabilities.</td>
+</tr>
+
+<tr>
+<td align="left"><code>render_markdown</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, will disable Markdown rendering for chatbot messages.</td>
+</tr>
+
+<tr>
+<td align="left"><code>bubble_full_width</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If False, the chat bubble will fit to the content of the message. If True (default), the chat bubble will be the full width of the component.</td>
+</tr>
+
+<tr>
+<td align="left"><code>line_breaks</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>True</code></td>
+<td align="left">If True (default), will enable Github-flavored Markdown line breaks in chatbot messages. If False, single new lines will be ignored. Only applies if `render_markdown` is True.</td>
+</tr>
+
+<tr>
+<td align="left"><code>likeable</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+bool
+```
+
+</td>
+<td align="left"><code>False</code></td>
+<td align="left">Whether the chat messages display a like or dislike button. Set automatically by the .like method but has to be present in the signature for it to show up in the config.</td>
+</tr>
+
+<tr>
+<td align="left"><code>layout</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+Literal["panel", "bubble"] | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">If "panel", will display the chatbot in a llm style layout. If "bubble", will display the chatbot with message bubbles, with the user and bot messages on alterating sides. Will default to "bubble".</td>
+</tr>
+
+<tr>
+<td align="left"><code>placeholder</code></td>
+<td align="left" style="width: 25%;">
+
+```python
+str | None
+```
+
+</td>
+<td align="left"><code>None</code></td>
+<td align="left">a placeholder message to display in the chatbot when it is empty. Centered vertically and horizontally in the TestChatbot. Supports Markdown and HTML. If None, no placeholder is displayed.</td>
+</tr>
+</tbody></table>
+
+
+### Events
+
+| name | description |
+|:-----|:------------|
+| `change` | Triggered when the value of the TestChatbot changes either because of user input (e.g. a user types in a textbox) OR because of a function update (e.g. an image receives a value from the output of an event trigger). See `.input()` for a listener that is only triggered by user input. | 

src/demo/requirements.txt

@@ -0,0 +1,7 @@
+gradio_agentchatbot
+"git+https://github.com/huggingface/transformers.git#egg=transformers[agents]"
+langchain
+sentence-transformers
+faiss-cpu
+serpapi
+google-search-results

src/frontend/Index.svelte

@@ -0,0 +1,140 @@
+<script context="module" lang="ts">
+	export { default as BaseChatBot } from "./shared/ChatBot.svelte";
+</script>
+
+<script lang="ts">
+	import type { Gradio, SelectData, LikeData } from "@gradio/utils";
+	import Prism from "prismjs";
+
+	import ChatBot from "./shared/ChatBot.svelte";
+	import { Block, BlockLabel } from "@gradio/atoms";
+	import type { LoadingStatus } from "@gradio/statustracker";
+	import { Chat } from "@gradio/icons";
+	import type { FileData } from "@gradio/client";
+	import type { ChatMessage, ChatFileMessage } from "./types"
+	import { StatusTracker } from "@gradio/statustracker";
+
+	export let elem_id = "";
+	export let elem_classes: string[] = [];
+	export let visible = true;
+	export let value: (ChatMessage | ChatFileMessage)[] = [];
+	export let scale: number | null = null;
+	export let min_width: number | undefined = undefined;
+	export let label: string;
+	export let show_label = true;
+	export let root: string;
+	export let _selectable = false;
+	export let likeable = false;
+	export let show_share_button = false;
+	export let rtl = false;
+	export let show_copy_button = false;
+	export let sanitize_html = true;
+	export let bubble_full_width = true;
+	export let layout: "bubble" | "panel" = "bubble";
+	export let render_markdown = true;
+	export let line_breaks = true;
+	export let latex_delimiters: {
+		left: string;
+		right: string;
+		display: boolean;
+	}[];
+	export let gradio: Gradio<{
+		change: typeof value;
+		select: SelectData;
+		share: ShareData;
+		error: string;
+		like: LikeData;
+		clear_status: LoadingStatus;
+	}>;
+	export let avatar_images: [FileData | null, FileData | null] = [null, null];
+
+	let _value: (ChatMessage | ChatFileMessage)[];
+
+	const redirect_src_url = (src: string): string =>
+		src.replace('src="/file', `src="${root}file`);
+
+	function normalize_messages(
+		message: { file: FileData; alt_text: string | null } | null
+	): { file: FileData; alt_text: string | null } | null {
+		if (message === null) {
+			return message;
+		}
+		return {
+			file: message?.file as FileData,
+			alt_text: message?.alt_text
+		};
+	}
+
+	$: _value = value;
+
+	export let loading_status: LoadingStatus | undefined = undefined;
+	export let height = 400;
+	export let placeholder: string | null = null;
+</script>
+
+<Block
+	{elem_id}
+	{elem_classes}
+	{visible}
+	padding={false}
+	{scale}
+	{min_width}
+	{height}
+	allow_overflow={false}
+>
+	{#if loading_status}
+		<StatusTracker
+			autoscroll={gradio.autoscroll}
+			i18n={gradio.i18n}
+			{...loading_status}
+			show_progress={loading_status.show_progress === "hidden"
+				? "hidden"
+				: "minimal"}
+			on:clear_status={() => gradio.dispatch("clear_status", loading_status)}
+		/>
+	{/if}
+	<div class="wrapper">
+		{#if show_label}
+			<BlockLabel
+				{show_label}
+				Icon={Chat}
+				float={false}
+				label={label || "Chatbot"}
+			/>
+		{/if}
+		<ChatBot
+			i18n={gradio.i18n}
+			selectable={_selectable}
+			{likeable}
+			{show_share_button}
+			value={_value}
+			{latex_delimiters}
+			{render_markdown}
+			pending_message={loading_status?.status === "pending"}
+			{rtl}
+			{show_copy_button}
+			on:change={() => gradio.dispatch("change", value)}
+			on:select={(e) => gradio.dispatch("select", e.detail)}
+			on:like={(e) => gradio.dispatch("like", e.detail)}
+			on:share={(e) => gradio.dispatch("share", e.detail)}
+			on:error={(e) => gradio.dispatch("error", e.detail)}
+			{avatar_images}
+			{sanitize_html}
+			{bubble_full_width}
+			{line_breaks}
+			{layout}
+			{placeholder}
+		/>
+	</div>
+</Block>
+
+<style>
+	.wrapper {
+		display: flex;
+		position: relative;
+		flex-direction: column;
+		align-items: start;
+		width: 100%;
+		height: 100%;
+	}
+</style>

src/frontend/gradio.config.js

@@ -0,0 +1,12 @@
+import prismjs from 'vite-plugin-prismjs';
+
+export default {
+  plugins: [
+    prismjs({
+      languages: 'all',
+    })
+  ],
+  svelte: {
+    preprocess: [],
+  },
+};

src/frontend/main.ts

@@ -0,0 +1 @@
+import 'prismjs';

src/frontend/package.json

@@ -0,0 +1,38 @@
+{
+  "name": "gradio_agentchatbot",
+  "version": "0.10.3",
+  "description": "Gradio UI packages",
+  "type": "module",
+  "author": "",
+  "license": "ISC",
+  "private": false,
+  "dependencies": {
+    "@gradio/atoms": "0.7.3",
+    "@gradio/audio": "0.11.3",
+    "@gradio/client": "0.19.2",
+    "@gradio/icons": "0.4.1",
+    "@gradio/image": "0.11.3",
+    "@gradio/markdown": "0.7.3",
+    "@gradio/statustracker": "0.5.3",
+    "@gradio/theme": "0.2.2",
+    "@gradio/upload": "0.10.3",
+    "@gradio/utils": "0.4.1",
+    "@gradio/video": "0.8.3",
+    "@types/dompurify": "^3.0.2",
+    "@types/katex": "^0.16.0",
+    "@types/prismjs": "^1.26.4",
+    "dequal": "^2.0.2",
+    "prismjs": "^1.29.0",
+    "vite": "^5.2.11",
+    "vite-plugin-prismjs": "^0.0.11"
+  },
+  "devDependencies": {
+    "@gradio/preview": "0.9.0"
+  },
+  "main_changeset": true,
+  "main": "./Index.svelte",
+  "exports": {
+    ".": "./Index.svelte",
+    "./package.json": "./package.json"
+  }
+}

src/frontend/shared/ChatBot.svelte

@@ -0,0 +1,644 @@
+<script lang="ts">
+	import { format_chat_for_sharing } from "./utils";
+	import { copy } from "@gradio/utils";
+
+	import { dequal } from "dequal/lite";
+	import { beforeUpdate, afterUpdate, createEventDispatcher } from "svelte";
+	import { ShareButton } from "@gradio/atoms";
+	import { Audio } from "@gradio/audio/shared";
+	import { Image } from "@gradio/image/shared";
+	import { Video } from "@gradio/video/shared";
+	import type { SelectData, LikeData } from "@gradio/utils";
+	import type { ChatMessage, ChatFileMessage, Message } from "../types";
+	import { MarkdownCode as Markdown } from "@gradio/markdown";
+	import { FileData } from "@gradio/client";
+	import Copy from "./Copy.svelte";
+	import type { I18nFormatter } from "js/app/src/gradio_helper";
+	import LikeDislike from "./LikeDislike.svelte";
+	import Pending from "./Pending.svelte";
+	import ToolMessage from "./ToolMessage.svelte";
+	import ErrorMessage from "./ErrorMessage.svelte";
+
+	export let value:  (ChatMessage | ChatFileMessage)[] = [];
+	let old_value: (ChatMessage | ChatFileMessage)[] | null = null;
+	export let latex_delimiters: {
+		left: string;
+		right: string;
+		display: boolean;
+	}[];
+	export let pending_message = false;
+	export let selectable = false;
+	export let likeable = false;
+	export let show_share_button = false;
+	export let rtl = false;
+	export let show_copy_button = false;
+	export let avatar_images: [FileData | null, FileData | null] = [null, null];
+	export let sanitize_html = true;
+	export let bubble_full_width = true;
+	export let render_markdown = true;
+	export let line_breaks = true;
+	export let i18n: I18nFormatter;
+	export let layout: "bubble" | "panel" = "bubble";
+	export let placeholder: string | null = null;
+
+	let div: HTMLDivElement;
+	let autoscroll: boolean;
+
+	$: adjust_text_size = () => {
+		let style = getComputedStyle(document.body);
+		let body_text_size = style.getPropertyValue("--body-text-size");
+		let updated_text_size;
+
+		switch (body_text_size) {
+			case "13px":
+				updated_text_size = 14;
+				break;
+			case "14px":
+				updated_text_size = 16;
+				break;
+			case "16px":
+				updated_text_size = 20;
+				break;
+			default:
+				updated_text_size = 14;
+				break;
+		}
+
+		document.body.style.setProperty(
+			"--chatbot-body-text-size",
+			updated_text_size + "px"
+		);
+	};
+
+	$: adjust_text_size();
+
+	const dispatch = createEventDispatcher<{
+		change: undefined;
+		select: SelectData;
+		like: LikeData;
+	}>();
+
+	beforeUpdate(() => {
+		autoscroll =
+			div && div.offsetHeight + div.scrollTop > div.scrollHeight - 100;
+	});
+
+	const scroll = (): void => {
+		if (autoscroll) {
+			div.scrollTo(0, div.scrollHeight);
+		}
+	};
+	afterUpdate(() => {
+		if (autoscroll) {
+			scroll();
+			div.querySelectorAll("img").forEach((n) => {
+				n.addEventListener("load", () => {
+					scroll();
+				});
+			});
+		}
+	});
+
+	$: {
+		if (!dequal(value, old_value)) {
+			old_value = value;
+			dispatch("change");
+		}
+	}
+
+	function handle_select(
+		i: number,
+		message: Message
+	): void {
+		dispatch("select", {
+			index: i,
+			value: (message as ChatMessage).content || (message as ChatFileMessage).file?.url
+		});
+	}
+
+	function handle_like(
+		i: number,
+		message: Message | null,
+		selected: string | null
+	): void {
+		dispatch("like", {
+			index: i,
+			value: (message as ChatMessage).content || (message as ChatFileMessage).file?.url,
+			liked: selected === "like"
+		});
+	}
+
+	function isFileMessage(
+		message: ChatMessage | ChatFileMessage
+	): message is ChatFileMessage {
+		return "file" in message;
+	}
+
+	function groupMessages(messages: (ChatMessage | ChatFileMessage)[]): (ChatMessage | ChatFileMessage)[][] {
+		const groupedMessages: (ChatFileMessage | ChatMessage)[][] = [];
+		let currentGroup: (ChatFileMessage | ChatMessage)[] = [];
+
+		for (const message of messages) {
+			if (message.thought) {
+			currentGroup.push(message);
+			} else {
+			if (currentGroup.length > 0) {
+				groupedMessages.push(currentGroup);
+				currentGroup = [];
+			}
+			groupedMessages.push([message]);
+			}
+		}
+
+		if (currentGroup.length > 0) {
+			groupedMessages.push(currentGroup);
+		}
+
+		return groupedMessages;
+}
+
+</script>
+
+{#if show_share_button && value !== null && value.length > 0}
+	<div class="share-button">
+		<ShareButton
+			{i18n}
+			on:error
+			on:share
+			formatter={format_chat_for_sharing}
+			{value}
+		/>
+	</div>
+{/if}
+
+<div
+	class={layout === "bubble" ? "bubble-wrap" : "panel-wrap"}
+	class:placeholder-container={value === null || value.length === 0}
+	bind:this={div}
+	role="log"
+	aria-label="chatbot conversation"
+	aria-live="polite"
+>
+	<div class="message-wrap" class:bubble-gap={layout === "bubble"} use:copy>
+		{#if value !== null && value.length > 0}
+			{@const groupedMessages = groupMessages(value)}
+			{#each groupedMessages as messages, i}
+				{#if messages.length}
+					{@const role = messages[0].role === "user" ? 'user' : 'bot'}
+					{@const avatar_img = avatar_images[role === "user" ? 0 : 1]}
+					<div class="message-row {layout} {role === "user" ? 'user-row' : 'bot-row'}">
+						{#if avatar_img}
+							<div class="avatar-container">
+								<Image
+									class="avatar-image"
+									src={avatar_img.url}
+									alt="{role} avatar"
+								/>
+							</div>
+						{/if}
+						<div
+							class="message {role === "user" ? 'user' : 'bot'}"
+							class:message-fit={layout === "bubble" && !bubble_full_width}
+							class:panel-full-width={layout === "panel"}
+							class:message-bubble-border={layout === "bubble"}
+							class:message-markdown-disabled={!render_markdown}
+							style:text-align={rtl && role == 'bot' ? "left" : "right"}
+						>
+							<button
+								data-testid={role}
+								class:latest={i === groupedMessages.length - 1}
+								class:message-markdown-disabled={!render_markdown}
+								style:user-select="text"
+								class:selectable
+								style:text-align={rtl ? "right" : "left"}
+								on:click={() => handle_select(i, messages[0])}
+								on:keydown={(e) => {
+									if (e.key === "Enter") {
+										handle_select(i, messages[0]);
+									}
+								}}
+								dir={rtl ? "rtl" : "ltr"}
+							>
+								{#each messages as message, thought_index}
+
+									{#if !isFileMessage(message)}
+										<div class:thought={message.thought && thought_index > 0}>
+										{#if message.thought_metadata.tool_name}
+											<ToolMessage
+												title={`Used tool ${message.thought_metadata.tool_name}`}
+											>
+												<!-- {message.content} -->
+												<Markdown
+													message={message.content}
+													{latex_delimiters}
+													{sanitize_html}
+													{render_markdown}
+													{line_breaks}
+													on:load={scroll}
+												/>
+											</ToolMessage>
+										{:else if message.thought_metadata.error}
+											<ErrorMessage
+											>
+												<!-- {message.content} -->
+												<Markdown
+													message={message.content}
+													{latex_delimiters}
+													{sanitize_html}
+													{render_markdown}
+													{line_breaks}
+													on:load={scroll}
+												/>
+											</ErrorMessage>	
+										{:else}
+											<!-- {message.content} -->
+											<Markdown
+												message={message.content}
+												{latex_delimiters}
+												{sanitize_html}
+												{render_markdown}
+												{line_breaks}
+												on:load={scroll}
+											/>
+										{/if}
+										</div>
+									{:else}
+										{#if message.file.mime_type?.includes("audio")}
+											<Audio
+												data-testid="chatbot-audio"
+												controls
+												preload="metadata"
+												src={message.file?.url}
+												title={message.alt_text}
+												on:play
+												on:pause
+												on:ended
+											/>
+										{:else if message !== null && message.file?.mime_type?.includes("video")}
+											<Video
+												data-testid="chatbot-video"
+												controls
+												src={message.file?.url}
+												title={message.alt_text}
+												preload="auto"
+												on:play
+												on:pause
+												on:ended
+											>
+												<track kind="captions" />
+											</Video>
+										{:else if message !== null && message.file?.mime_type?.includes("image")}
+											<Image
+												data-testid="chatbot-image"
+												src={message.file?.url}
+												alt={message.alt_text}
+											/>
+										{:else if message !== null && message.file?.url !== null}
+											<a
+												data-testid="chatbot-file"
+												href={message.file?.url}
+												target="_blank"
+												download={window.__is_colab__
+													? null
+													: message.file?.orig_name || message.file?.path}
+											>
+												{message.file?.orig_name || message.file?.path}
+											</a>
+										{/if}
+									{/if}
+								{/each}
+							</button>
+						</div>
+						<!-- {#if (likeable && role === 'bot') || (show_copy_button && message && typeof message === "string")}
+							<div
+								class="message-buttons-{role} message-buttons-{layout} {avatar_images[j] !==
+									null && 'with-avatar'}"
+								class:message-buttons-fit={layout === "bubble" &&
+									!bubble_full_width}
+								class:bubble-buttons-user={layout === "bubble"}
+							>
+								{#if likeable && role === 'bot'}
+									<LikeDislike
+										handle_action={(selected) =>
+											handle_like(i, message, selected)}
+									/>
+								{/if}
+								{#if show_copy_button && message && typeof message === "string"}
+									<Copy value={message} />
+								{/if}
+							</div>
+						{/if} -->
+					</div>
+				{/if}
+			{/each}
+			{#if pending_message}
+				<Pending {layout} />
+			{/if}
+		{:else if placeholder !== null}
+			<center>
+				<Markdown message={placeholder} {latex_delimiters} />
+			</center>
+		{/if}
+	</div>
+</div>
+
+<style>
+	.placeholder-container {
+		display: flex;
+		justify-content: center;
+		align-items: center;
+		height: 100%;
+	}
+	.bubble-wrap {
+		padding: var(--block-padding);
+		width: 100%;
+		overflow-y: auto;
+	}
+
+	.panel-wrap {
+		width: 100%;
+		overflow-y: auto;
+	}
+
+	.message-wrap {
+		display: flex;
+		flex-direction: column;
+		justify-content: space-between;
+	}
+
+	.bubble-gap {
+		gap: calc(var(--spacing-xxl) + var(--spacing-lg));
+	}
+
+	.message-wrap > div :not(.avatar-container) :global(img) {
+		border-radius: 13px;
+		margin: var(--size-2);
+		width: 400px;
+		max-width: 30vw;
+		max-height: auto;
+	}
+
+	.message-wrap > div :global(p:not(:first-child)) {
+		margin-top: var(--spacing-xxl);
+	}
+
+	.message {
+		position: relative;
+		display: flex;
+		flex-direction: column;
+		align-self: flex-end;
+		background: var(--background-fill-secondary);
+		width: calc(100% - var(--spacing-xxl));
+		color: var(--body-text-color);
+		font-size: var(--chatbot-body-text-size);
+		overflow-wrap: break-word;
+		overflow-x: hidden;
+		padding-right: calc(var(--spacing-xxl) + var(--spacing-md));
+		padding: calc(var(--spacing-xxl) + var(--spacing-sm));
+	}
+
+	.thought {
+		margin-top: var(--spacing-xxl);
+	}
+
+	.message :global(.prose) {
+		font-size: var(--chatbot-body-text-size);
+	}
+
+	.message-bubble-border {
+		border-width: 1px;
+		border-radius: var(--radius-xxl);
+	}
+
+	.message-fit {
+		width: fit-content !important;
+	}
+
+	.panel-full-width {
+		padding: calc(var(--spacing-xxl) * 2);
+		width: 100%;
+	}
+	.message-markdown-disabled {
+		white-space: pre-line;
+	}
+
+	@media (max-width: 480px) {
+		.panel-full-width {
+			padding: calc(var(--spacing-xxl) * 2);
+		}
+	}
+
+	.user {
+		align-self: flex-start;
+		border-bottom-right-radius: 0;
+		text-align: right;
+	}
+	.bot {
+		border-bottom-left-radius: 0;
+		text-align: left;
+	}
+
+	/* Colors */
+	.bot {
+		border-color: var(--border-color-primary);
+		background: var(--background-fill-secondary);
+	}
+
+	.user {
+		border-color: var(--border-color-accent-subdued);
+		background-color: var(--color-accent-soft);
+	}
+	.message-row {
+		display: flex;
+		flex-direction: row;
+		position: relative;
+	}
+
+	.message-row.panel.user-row {
+		background: var(--color-accent-soft);
+	}
+
+	.message-row.panel.bot-row {
+		background: var(--background-fill-secondary);
+	}
+
+	.message-row:last-of-type {
+		margin-bottom: var(--spacing-xxl);
+	}
+
+	.user-row.bubble {
+		flex-direction: row;
+		justify-content: flex-end;
+	}
+	@media (max-width: 480px) {
+		.user-row.bubble {
+			align-self: flex-end;
+		}
+
+		.bot-row.bubble {
+			align-self: flex-start;
+		}
+		.message {
+			width: auto;
+		}
+	}
+	.avatar-container {
+		align-self: flex-end;
+		position: relative;
+		justify-content: center;
+		width: 35px;
+		height: 35px;
+		flex-shrink: 0;
+		bottom: 0;
+	}
+	.user-row.bubble > .avatar-container {
+		order: 2;
+		margin-left: 10px;
+	}
+	.bot-row.bubble > .avatar-container {
+		margin-right: 10px;
+	}
+
+	.panel > .avatar-container {
+		margin-left: 25px;
+		align-self: center;
+	}
+
+	.avatar-container :global(img) {
+		width: 100%;
+		height: 100%;
+		object-fit: cover;
+		border-radius: 50%;
+	}
+
+	.message-buttons-user,
+	.message-buttons-bot {
+		border-radius: var(--radius-md);
+		display: flex;
+		align-items: center;
+		bottom: 0;
+		height: var(--size-7);
+		align-self: self-end;
+		position: absolute;
+		bottom: -15px;
+		margin: 2px;
+		padding-left: 5px;
+		z-index: 1;
+	}
+	.message-buttons-bot {
+		left: 10px;
+	}
+	.message-buttons-user {
+		right: 5px;
+	}
+
+	.message-buttons-bot.message-buttons-bubble.with-avatar {
+		left: 50px;
+	}
+	.message-buttons-user.message-buttons-bubble.with-avatar {
+		right: 50px;
+	}
+
+	.message-buttons-bubble {
+		border: 1px solid var(--border-color-accent);
+		background: var(--background-fill-secondary);
+	}
+
+	.message-buttons-panel {
+		left: unset;
+		right: 0px;
+		top: 0px;
+	}
+
+	.share-button {
+		position: absolute;
+		top: 4px;
+		right: 6px;
+	}
+
+	.selectable {
+		cursor: pointer;
+	}
+
+	@keyframes dot-flashing {
+		0% {
+			opacity: 0.8;
+		}
+		50% {
+			opacity: 0.5;
+		}
+		100% {
+			opacity: 0.8;
+		}
+	}
+
+	.message-wrap .message :global(a) {
+		color: var(--color-text-link);
+		text-decoration: underline;
+	}
+
+	.message-wrap .bot :global(table),
+	.message-wrap .bot :global(tr),
+	.message-wrap .bot :global(td),
+	.message-wrap .bot :global(th) {
+		border: 1px solid var(--border-color-primary);
+	}
+
+	.message-wrap .user :global(table),
+	.message-wrap .user :global(tr),
+	.message-wrap .user :global(td),
+	.message-wrap .user :global(th) {
+		border: 1px solid var(--border-color-accent);
+	}
+
+	/* Lists */
+	.message-wrap :global(ol),
+	.message-wrap :global(ul) {
+		padding-inline-start: 2em;
+	}
+
+	/* KaTeX */
+	.message-wrap :global(span.katex) {
+		font-size: var(--text-lg);
+		direction: ltr;
+	}
+
+	/* Copy button */
+	.message-wrap :global(div[class*="code_wrap"] > button) {
+		position: absolute;
+		top: var(--spacing-md);
+		right: var(--spacing-md);
+		z-index: 1;
+		cursor: pointer;
+		border-bottom-left-radius: var(--radius-sm);
+		padding: 5px;
+		padding: var(--spacing-md);
+		width: 25px;
+		height: 25px;
+	}
+
+	.message-wrap :global(code > button > span) {
+		position: absolute;
+		top: var(--spacing-md);
+		right: var(--spacing-md);
+		width: 12px;
+		height: 12px;
+	}
+	.message-wrap :global(.check) {
+		position: absolute;
+		top: 0;
+		right: 0;
+		opacity: 0;
+		z-index: var(--layer-top);
+		transition: opacity 0.2s;
+		background: var(--background-fill-primary);
+		padding: var(--size-1);
+		width: 100%;
+		height: 100%;
+		color: var(--body-text-color);
+	}
+
+	.message-wrap :global(pre) {
+		position: relative;
+	}
+</style>

src/frontend/shared/Copy.svelte

@@ -0,0 +1,79 @@
+<script lang="ts">
+	import { onDestroy } from "svelte";
+	import { Copy, Check } from "@gradio/icons";
+
+	let copied = false;
+	export let value: string;
+	let timer: NodeJS.Timeout;
+
+	function copy_feedback(): void {
+		copied = true;
+		if (timer) clearTimeout(timer);
+		timer = setTimeout(() => {
+			copied = false;
+		}, 2000);
+	}
+
+	async function handle_copy(): Promise<void> {
+		if ("clipboard" in navigator) {
+			await navigator.clipboard.writeText(value);
+			copy_feedback();
+		} else {
+			const textArea = document.createElement("textarea");
+			textArea.value = value;
+
+			textArea.style.position = "absolute";
+			textArea.style.left = "-999999px";
+
+			document.body.prepend(textArea);
+			textArea.select();
+
+			try {
+				document.execCommand("copy");
+				copy_feedback();
+			} catch (error) {
+				console.error(error);
+			} finally {
+				textArea.remove();
+			}
+		}
+	}
+
+	onDestroy(() => {
+		if (timer) clearTimeout(timer);
+	});
+</script>
+
+<button
+	on:click={handle_copy}
+	class="action"
+	title="copy"
+	aria-label={copied ? "Copied message" : "Copy message"}
+>
+	{#if !copied}
+		<Copy />
+	{/if}
+	{#if copied}
+		<Check />
+	{/if}
+</button>
+
+<style>
+	button {
+		position: relative;
+		top: 0;
+		right: 0;
+		cursor: pointer;
+		color: var(--body-text-color-subdued);
+		margin-right: 5px;
+	}
+
+	button:hover {
+		color: var(--body-text-color);
+	}
+
+	.action {
+		width: 15px;
+		height: 14px;
+	}
+</style>

src/frontend/shared/ErrorMessage.svelte

@@ -0,0 +1,49 @@
+<script lang="ts">
+    let expanded = false;
+  
+    function toggleExpanded() {
+      expanded = !expanded;
+    }
+  </script>
+  
+<!-- svelte-ignore a11y-click-events-have-key-events -->
+<!-- svelte-ignore a11y-no-static-element-interactions -->
+<div class="box" on:click={toggleExpanded}>
+    <div class="title">
+        <div class="icon">
+            💥
+        </div>
+        <span>{'Error'}</span>
+    </div>
+    {#if expanded}
+        <div class="content">
+        <slot></slot>
+        </div>
+    {/if}
+</div>
+  
+<style>
+    .box {
+        border-radius: 4px;
+        cursor: pointer;
+        max-width: max-content;
+        background: var(--color-accent-soft)
+    }
+
+    .title {
+        display: flex;
+        align-items: center;
+        padding: 8px;
+        color: var(--color-red-50);
+    }
+
+    .icon {
+        width: 18px;
+        height: 18px;
+        margin-right: 8px;
+    }
+
+    .content {
+        padding: 8px;
+    }
+</style>

src/frontend/shared/LikeDislike.svelte

@@ -0,0 +1,46 @@
+<script lang="ts">
+	import { Like } from "@gradio/icons";
+	import { Dislike } from "@gradio/icons";
+
+	export let handle_action: (selected: string | null) => void;
+
+	let selected: "like" | "dislike" | null = null;
+</script>
+
+<button
+	on:click={() => {
+		selected = "like";
+		handle_action(selected);
+	}}
+	aria-label={selected === "like" ? "clicked like" : "like"}
+>
+	<Like selected={selected === "like"} />
+</button>
+
+<button
+	on:click={() => {
+		selected = "dislike";
+		handle_action(selected);
+	}}
+	aria-label={selected === "dislike" ? "clicked dislike" : "dislike"}
+>
+	<Dislike selected={selected === "dislike"} />
+</button>
+
+<style>
+	button {
+		position: relative;
+		top: 0;
+		right: 0;
+		cursor: pointer;
+		color: var(--body-text-color-subdued);
+		width: 17px;
+		height: 17px;
+		margin-right: 5px;
+	}
+
+	button:hover,
+	button:focus {
+		color: var(--body-text-color);
+	}
+</style>

src/frontend/shared/Pending.svelte

@@ -0,0 +1,60 @@
+<script lang="ts">
+	export let layout = "bubble";
+</script>
+
+<div
+	class="message pending"
+	role="status"
+	aria-label="Loading response"
+	aria-live="polite"
+	style:border-radius={layout === "bubble" ? "var(--radius-xxl)" : "none"}
+>
+	<span class="sr-only">Loading content</span>
+	<div class="dot-flashing" />
+	&nbsp;
+	<div class="dot-flashing" />
+	&nbsp;
+	<div class="dot-flashing" />
+</div>
+
+<style>
+	.pending {
+		background: var(--color-accent-soft);
+		display: flex;
+		flex-direction: row;
+		justify-content: center;
+		align-items: center;
+		align-self: center;
+		gap: 2px;
+		width: 100%;
+		height: var(--size-16);
+	}
+	.dot-flashing {
+		animation: flash 1s infinite ease-in-out;
+		border-radius: 5px;
+		background-color: var(--body-text-color);
+		width: 7px;
+		height: 7px;
+		color: var(--body-text-color);
+	}
+	@keyframes flash {
+		0%,
+		100% {
+			opacity: 0;
+		}
+		50% {
+			opacity: 1;
+		}
+	}
+
+	.dot-flashing:nth-child(1) {
+		animation-delay: 0s;
+	}
+
+	.dot-flashing:nth-child(2) {
+		animation-delay: 0.33s;
+	}
+	.dot-flashing:nth-child(3) {
+		animation-delay: 0.66s;
+	}
+</style>

src/frontend/shared/Tool.svelte

@@ -0,0 +1,11 @@
+<!-- <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-file"><path d="M13 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V9z"></path><polyline points="13 2 13 9 20 9"></polyline></svg> -->
+<!-- <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor"> -->
+<path d="M478.409,116.617c-0.368-4.271-3.181-7.94-7.2-9.403c-4.029-1.472-8.539-0.47-11.57,2.556l-62.015,62.011l-68.749-21.768
+		l-21.768-68.748l62.016-62.016c3.035-3.032,4.025-7.543,2.563-11.565c-1.477-4.03-5.137-6.837-9.417-7.207
+		c-37.663-3.245-74.566,10.202-101.247,36.887c-36.542,36.545-46.219,89.911-29.083,135.399c-1.873,1.578-3.721,3.25-5.544,5.053
+		L19.386,373.152c-0.073,0.071-0.145,0.149-0.224,0.219c-24.345,24.346-24.345,63.959,0,88.309
+		c24.349,24.344,63.672,24.048,88.013-0.298c0.105-0.098,0.201-0.196,0.297-0.305l193.632-208.621
+		c1.765-1.773,3.404-3.628,4.949-5.532c45.5,17.167,98.9,7.513,135.474-29.056C468.202,191.181,481.658,154.275,478.409,116.617z
+		 M75.98,435.38c-8.971,8.969-23.5,8.963-32.47,0c-8.967-8.961-8.967-23.502,0-32.466c8.97-8.963,23.499-8.963,32.47,0
+		C84.947,411.878,84.947,426.419,75.98,435.38z"/>
+<!-- </svg> -->

src/frontend/shared/ToolMessage.svelte

@@ -0,0 +1,65 @@
+<script lang="ts">
+    import Tool from "./Tool.svelte";
+
+    export let title: string;
+    let expanded = false;
+  
+    function toggleExpanded() {
+      expanded = !expanded;
+    }
+  </script>
+  
+<!-- svelte-ignore a11y-click-events-have-key-events -->
+<!-- svelte-ignore a11y-no-static-element-interactions -->
+<div class="box" on:click={toggleExpanded}>
+    <div class="title">
+        <div class="wrench-icon">
+            🛠️
+        </div>
+        <span>{title}</span>
+    </div>
+    {#if expanded}
+        <div class="content">
+        <slot></slot>
+        </div>
+    {/if}
+</div>
+  
+<style>
+    .box {
+        border-radius: 4px;
+        cursor: pointer;
+        max-width: max-content;
+        background: var(--color-accent-soft)
+    }
+
+    .title {
+        display: flex;
+        align-items: center;
+        padding: 8px;
+    }
+
+    .wrench-icon {
+        width: 18px;
+        height: 18px;
+        margin-right: 8px;
+    }
+
+    .icon {
+        width: 0; 
+        height: 0; 
+        border-left: 5px solid transparent;
+        border-right: 5px solid transparent;
+        border-top: 5px solid purple;
+        margin-left: auto;
+        transition: transform 0.3s;
+    }
+
+    .icon.expanded {
+        transform: rotate(180deg);
+    }
+
+    .content {
+        padding: 8px;
+    }
+</style>

src/frontend/shared/autorender.d.ts

@@ -0,0 +1 @@
+declare module "katex/dist/contrib/auto-render.js";

src/frontend/shared/utils.ts

@@ -0,0 +1,57 @@
+import type { FileData } from "@gradio/client";
+import { uploadToHuggingFace } from "@gradio/utils";
+
+export const format_chat_for_sharing = async (
+	chat: [string | FileData | null, string | FileData | null][]
+): Promise<string> => {
+	let messages = await Promise.all(
+		chat.map(async (message_pair) => {
+			return await Promise.all(
+				message_pair.map(async (message, i) => {
+					if (message === null) return "";
+					let speaker_emoji = i === 0 ? "😃" : "🤖";
+					let html_content = "";
+
+					if (typeof message === "string") {
+						const regexPatterns = {
+							audio: /<audio.*?src="(\/file=.*?)"/g,
+							video: /<video.*?src="(\/file=.*?)"/g,
+							image: /<img.*?src="(\/file=.*?)".*?\/>|!\[.*?\]\((\/file=.*?)\)/g
+						};
+
+						html_content = message;
+
+						for (let [_, regex] of Object.entries(regexPatterns)) {
+							let match;
+
+							while ((match = regex.exec(message)) !== null) {
+								const fileUrl = match[1] || match[2];
+								const newUrl = await uploadToHuggingFace(fileUrl, "url");
+								html_content = html_content.replace(fileUrl, newUrl);
+							}
+						}
+					} else {
+						if (!message?.url) return "";
+						const file_url = await uploadToHuggingFace(message.url, "url");
+						if (message.mime_type?.includes("audio")) {
+							html_content = `<audio controls src="${file_url}"></audio>`;
+						} else if (message.mime_type?.includes("video")) {
+							html_content = file_url;
+						} else if (message.mime_type?.includes("image")) {
+							html_content = `<img src="${file_url}" />`;
+						}
+					}
+
+					return `${speaker_emoji}: ${html_content}`;
+				})
+			);
+		})
+	);
+	return messages
+		.map((message_pair) =>
+			message_pair.join(
+				message_pair[0] !== "" && message_pair[1] !== "" ? "\n" : ""
+			)
+		)
+		.join("\n");
+};

src/frontend/types.ts

@@ -0,0 +1,24 @@
+import type { FileData } from "@gradio/client";
+
+type MessageRole = "system" | "user" | "assistant" | "tool";
+
+
+export interface ThoughtMetadata {
+  error: boolean;
+  tool_name: string;
+}
+
+export interface Message {
+  role: MessageRole;
+  thought_metadata: ThoughtMetadata;
+  thought: boolean;
+}
+
+export interface ChatMessage extends Message {
+  content: string;
+}
+
+export interface ChatFileMessage extends Message {
+  file: FileData;
+  alt_text?: string | null;
+}

src/pyproject.toml

@@ -0,0 +1,51 @@
+[build-system]
+requires = [
+  "hatchling",
+  "hatch-requirements-txt",
+  "hatch-fancy-pypi-readme>=22.5.0",
+]
+build-backend = "hatchling.build"
+
+[project]
+name = "gradio_agentchatbot"
+version = "0.0.1"
+description = "Chat with agents 🤖 and see the tools they use 🛠️"
+readme = "README.md"
+license = "apache-2.0"
+requires-python = ">=3.8"
+authors = [{ name = "Freddy Boulton", email = "alfonsoboulton@gmail.com" }]
+keywords = ["gradio-custom-component", "gradio-template-Chatbot", "chatbot", "agents", "streaming", "tools"]
+# Add dependencies here
+dependencies = ["gradio>=4.0,<5.0"]
+classifiers = [
+  'Development Status :: 3 - Alpha',
+  'Operating System :: OS Independent',
+  'Programming Language :: Python :: 3',
+  'Programming Language :: Python :: 3 :: Only',
+  'Programming Language :: Python :: 3.8',
+  'Programming Language :: Python :: 3.9',
+  'Programming Language :: Python :: 3.10',
+  'Programming Language :: Python :: 3.11',
+  'Topic :: Scientific/Engineering',
+  'Topic :: Scientific/Engineering :: Artificial Intelligence',
+  'Topic :: Scientific/Engineering :: Visualization',
+]
+
+# The repository and space URLs are optional, but recommended.
+# Adding a repository URL will create a badge in the auto-generated README that links to the repository.
+# Adding a space URL will create a badge in the auto-generated README that links to the space.
+# This will make it easy for people to find your deployed demo or source code when they
+# encounter your project in the wild.
+
+# [project.urls]
+# repository = "your github repository"
+# space = "your space url"
+
+[project.optional-dependencies]
+dev = ["build", "twine"]
+
+[tool.hatch.build]
+artifacts = ["/backend/gradio_agentchatbot/templates", "*.pyi"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["/backend/gradio_agentchatbot"]

src/scripts/ExtractStreamingLogs.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"SERPAPI_API_KEY\"] = \"3d009ed0a9764853b3e75ee2c70ceb3aabd92b810456307c5af711559ec4aff7\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/freddy/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n",
+      "You're loading a tool from the Hub from None. Please make sure this is a source that you trust as the code within that tool will be executed on your machine. Always verify the code of the tools that you load. We recommend specifying a `revision` to ensure you're loading the code that you have checked.\n"
+     ]
+    },
+    {
+     "ename": "ImportError",
+     "evalue": "Could not import serpapi python package. Please install it with `pip install google-search-results`.",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/langchain_community/utilities/serpapi.py:68\u001b[0m, in \u001b[0;36mSerpAPIWrapper.validate_environment\u001b[0;34m(cls, values)\u001b[0m\n\u001b[1;32m     67\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 68\u001b[0m     \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mserpapi\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m GoogleSearch\n\u001b[1;32m     70\u001b[0m     values[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msearch_engine\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m GoogleSearch\n",
+      "\u001b[0;31mImportError\u001b[0m: cannot import name 'GoogleSearch' from 'serpapi' (/Users/freddy/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/serpapi/__init__.py)",
+      "\nDuring handling of the above exception, another exception occurred:\n",
+      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[2], line 9\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[38;5;66;03m# Import tool from Hub\u001b[39;00m\n\u001b[1;32m      6\u001b[0m image_generation_tool \u001b[38;5;241m=\u001b[39m load_tool(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mm-ric/text-to-image\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 9\u001b[0m search_tool \u001b[38;5;241m=\u001b[39m Tool\u001b[38;5;241m.\u001b[39mfrom_langchain(\u001b[43mload_tools\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mserpapi\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m[\u001b[38;5;241m0\u001b[39m])\n\u001b[1;32m     11\u001b[0m llm_engine \u001b[38;5;241m=\u001b[39m HfEngine(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmeta-llama/Meta-Llama-3-70B-Instruct\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m     12\u001b[0m \u001b[38;5;66;03m# Initialize the agent with both tools\u001b[39;00m\n",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/langchain_community/agent_toolkits/load_tools.py:726\u001b[0m, in \u001b[0;36mload_tools\u001b[0;34m(tool_names, llm, callbacks, allow_dangerous_tools, **kwargs)\u001b[0m\n\u001b[1;32m    724\u001b[0m     _get_tool_func, extra_keys \u001b[38;5;241m=\u001b[39m _EXTRA_OPTIONAL_TOOLS[name]\n\u001b[1;32m    725\u001b[0m     sub_kwargs \u001b[38;5;241m=\u001b[39m {k: kwargs[k] \u001b[38;5;28;01mfor\u001b[39;00m k \u001b[38;5;129;01min\u001b[39;00m extra_keys \u001b[38;5;28;01mif\u001b[39;00m k \u001b[38;5;129;01min\u001b[39;00m kwargs}\n\u001b[0;32m--> 726\u001b[0m     tool \u001b[38;5;241m=\u001b[39m \u001b[43m_get_tool_func\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43msub_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    727\u001b[0m     tools\u001b[38;5;241m.\u001b[39mappend(tool)\n\u001b[1;32m    728\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/langchain_community/agent_toolkits/load_tools.py:380\u001b[0m, in \u001b[0;36m_get_serpapi\u001b[0;34m(**kwargs)\u001b[0m\n\u001b[1;32m    376\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_get_serpapi\u001b[39m(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m BaseTool:\n\u001b[1;32m    377\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m Tool(\n\u001b[1;32m    378\u001b[0m         name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSearch\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    379\u001b[0m         description\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mA search engine. Useful for when you need to answer questions about current events. Input should be a search query.\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m--> 380\u001b[0m         func\u001b[38;5;241m=\u001b[39m\u001b[43mSerpAPIWrapper\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mrun,\n\u001b[1;32m    381\u001b[0m         coroutine\u001b[38;5;241m=\u001b[39mSerpAPIWrapper(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\u001b[38;5;241m.\u001b[39marun,\n\u001b[1;32m    382\u001b[0m     )\n",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/pydantic/v1/main.py:339\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m    333\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    334\u001b[0m \u001b[38;5;124;03mCreate a new model by parsing and validating input data from keyword arguments.\u001b[39;00m\n\u001b[1;32m    335\u001b[0m \n\u001b[1;32m    336\u001b[0m \u001b[38;5;124;03mRaises ValidationError if the input data cannot be parsed to form a valid model.\u001b[39;00m\n\u001b[1;32m    337\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    338\u001b[0m \u001b[38;5;66;03m# Uses something other than `self` the first arg to allow \"self\" as a settable attribute\u001b[39;00m\n\u001b[0;32m--> 339\u001b[0m values, fields_set, validation_error \u001b[38;5;241m=\u001b[39m \u001b[43mvalidate_model\u001b[49m\u001b[43m(\u001b[49m\u001b[43m__pydantic_self__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;18;43m__class__\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdata\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    340\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m validation_error:\n\u001b[1;32m    341\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m validation_error\n",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/pydantic/v1/main.py:1100\u001b[0m, in \u001b[0;36mvalidate_model\u001b[0;34m(model, input_data, cls)\u001b[0m\n\u001b[1;32m   1098\u001b[0m     \u001b[38;5;28;01mcontinue\u001b[39;00m\n\u001b[1;32m   1099\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1100\u001b[0m     values \u001b[38;5;241m=\u001b[39m \u001b[43mvalidator\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcls_\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvalues\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1101\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mValueError\u001b[39;00m, \u001b[38;5;167;01mTypeError\u001b[39;00m, \u001b[38;5;167;01mAssertionError\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[1;32m   1102\u001b[0m     errors\u001b[38;5;241m.\u001b[39mappend(ErrorWrapper(exc, loc\u001b[38;5;241m=\u001b[39mROOT_KEY))\n",
+      "File \u001b[0;32m~/sources/components/chatbot-with-tools/.venv/lib/python3.10/site-packages/langchain_community/utilities/serpapi.py:72\u001b[0m, in \u001b[0;36mSerpAPIWrapper.validate_environment\u001b[0;34m(cls, values)\u001b[0m\n\u001b[1;32m     70\u001b[0m     values[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msearch_engine\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m GoogleSearch\n\u001b[1;32m     71\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m:\n\u001b[0;32m---> 72\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(\n\u001b[1;32m     73\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCould not import serpapi python package. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     74\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPlease install it with `pip install google-search-results`.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     75\u001b[0m     )\n\u001b[1;32m     76\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m values\n",
+      "\u001b[0;31mImportError\u001b[0m: Could not import serpapi python package. Please install it with `pip install google-search-results`."
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import Tool, load_tool, ReactCodeAgent, HfEngine\n",
+    "# Import tool from LangChain\n",
+    "from langchain.agents import load_tools\n",
+    "\n",
+    "# Import tool from Hub\n",
+    "image_generation_tool = load_tool(\"m-ric/text-to-image\")\n",
+    "\n",
+    "\n",
+    "search_tool = Tool.from_langchain(load_tools([\"serpapi\"])[0])\n",
+    "\n",
+    "llm_engine = HfEngine(\"meta-llama/Meta-Llama-3-70B-Instruct\")\n",
+    "# Initialize the agent with both tools\n",
+    "agent = ReactCodeAgent(tools=[image_generation_tool, search_tool], llm_engine=llm_engine)\n",
+    "\n",
+    "# Run it!\n",
+    "#agent.run(\"Generate me a photo of the car that James bond drove in the latest movie.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "json.dump(agent.logs, open(\"logs.json\", \"w\"), indent=2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers.agents import Agent\n",
+    "from gradio.data_classes import GradioModel\n",
+    "from typing import Literal, List, Generator, Optional\n",
+    "from threading import Thread\n",
+    "import time\n",
+    "\n",
+    "class OpenAIMessage(GradioModel):\n",
+    "    role: Literal[\"system\", \"user\", \"assistant\", \"tool\"]\n",
+    "    content: str\n",
+    "    reasoning: bool = False\n",
+    "    tool_name: Optional[str] = None\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def stream_from_agent(\n",
+    "    agent: Agent,\n",
+    "    prompt: str\n",
+    ") -> Generator[List[OpenAIMessage], None, None]:\n",
+    "    \"\"\"Run Python code in a process and capture logs in real-time to yield them.\"\"\"\n",
+    "\n",
+    "    thread = Thread(target=agent.run, args=(prompt,))\n",
+    "    num_messages = 0\n",
+    "\n",
+    "    # Start process and pull logs while it runs\n",
+    "    thread.start()\n",
+    "    while thread.is_alive():\n",
+    "        if len(agent.logs) > num_messages:\n",
+    "            new_messages = agent.logs[num_messages:]\n",
+    "            for message in new_messages:\n",
+    "                if not len(message):\n",
+    "                    continue\n",
+    "                if message.get(\"rationale\"):\n",
+    "                    yield OpenAIMessage(\n",
+    "                        role=\"assistant\",\n",
+    "                        content=message[\"rationale\"],\n",
+    "                        reasoning=True\n",
+    "                    )\n",
+    "                if message.get(\"tool_call\"):\n",
+    "                    yield OpenAIMessage(\n",
+    "                        role=\"tool\",\n",
+    "                        tool_name=message[\"tool_call\"][\"tool_name\"],\n",
+    "                        content=message['tool_call'][\"tool_arguments\"],\n",
+    "                        reasoning=True\n",
+    "                    )\n",
+    "                num_messages = len(agent.logs)\n",
+    "        time.sleep(0.1)\n",
+    "\n",
+    "    thread.join(0.1)\n",
+    "\n",
+    "    if len(agent.logs) > num_messages:\n",
+    "        new_messages = agent.logs[num_messages:]\n",
+    "        for message in new_messages:\n",
+    "            if message.get(\"rationale\"):\n",
+    "                yield OpenAIMessage(\n",
+    "                    role=\"assistant\",\n",
+    "                    content=message[\"rationale\"],\n",
+    "                    reasoning=True\n",
+    "                )\n",
+    "            if message.get(\"tool_call\"):\n",
+    "                yield OpenAIMessage(\n",
+    "                    role=\"tool\",\n",
+    "                    tool_name=message[\"tool_call\"][\"tool_name\"],\n",
+    "                    content=message.get(\"tool_arguments\", \"\"),\n",
+    "                )\n",
+    "        num_messages = len(agent.logs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = []\n",
+    "for msg in stream_from_agent(agent, \"Generate me a photo of a cartoon cat.\"):\n",
+    "    messages.append(msg)\n",
+    "    print(\"MSG\", msg)\n",
+    "print(messages)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.logs[2]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.logs[3]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.run(\"What kind of car is this?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.logs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.run??"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

src/scripts/logs.json

@@ -0,0 +1,91 @@
+[
+  {
+    "system_prompt": "You will be given a task to solve as best you can.\nYou have access to the following tools:\n\n- image_generator: This is a tool that creates an image according to a prompt, which is a text description.\n    Takes inputs: {'prompt': {'type': 'text', 'description': \"The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc.\"}}\n\n- final_answer: Provides a final answer to the given problem\n    Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n\nTo solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.\n\nAt each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.\nThen in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.\nDuring each intermediate step, you can use 'print()' to save whatever important information you will then need.\nThese print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.\n\nIn the end you have to return a final answer using the `final_answer` tool.\n\nHere are a few examples using notional tools:\n---\nTask: \"Generate an image of the oldest person in this document.\"\n\nThought: I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.\nCode:\n```py\nanswer = document_qa(document=document, question=\"Who is the oldest person mentioned?\")\nprint(answer)\n```<end_code>\nObservation: \"The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland.\"\n\nThought: I will now generate an image showcasing the oldest person.\n\nCode:\n```py\nimage = image_generator(\"A portrait of John Doe, a 55-year-old man living in Canada.\")\nfinal_answer(image)\n```<end_code>\n\n---\nTask: \"What is the result of the following operation: 5 + 3 + 1294.678?\"\n\nThought: I will use python code to compute the result of the operation and then return the final answer using the `final_answer` tool\n\nCode:\n```py\nresult = 5 + 3 + 1294.678\nfinal_answer(result)\n```<end_code>\n\n---\nTask: \"Which city has the highest population , Guangzhou or Shanghai?\"\n\nThought: I need to get the populations for both cities and compare them: I will use the tool `search` to get the population of both cities.\nCode:\n```py\npopulation_guangzhou = search(\"Guangzhou population\")\nprint(\"Population Guangzhou:\", population_guangzhou)\npopulation_shanghai = search(\"Shanghai population\")\nprint(\"Population Shanghai:\", population_shanghai)\n```<end_code>\nObservation:\nPopulation Guangzhou: ['Guangzhou has a population of 15 million inhabitants as of 2021.']\nPopulation Shanghai: '26 million (2019)'\n\nThought: Now I know that Shanghai has the highest population.\nCode:\n```py\nfinal_answer(\"Shanghai\")\n```<end_code>\n\n---\nTask: \"What is the current age of the pope, raised to the power 0.36?\"\n\nThought: I will use the tool `search` to get the age of the pope, then raise it to the power 0.36.\nCode:\n```py\npope_age = search(query=\"current pope age\")\nprint(\"Pope age:\", pope_age)\n```<end_code>\nObservation:\nPope age: \"The pope Francis is currently 85 years old.\"\n\nThought: I know that the pope is 85 years old. Let's compute the result using python code.\nCode:\n```py\npope_current_age = 85 ** 0.36\nfinal_answer(pope_current_age)\n```<end_code>\n\n\nAbove example were using notional tools that might not exist for you. You only have acces to those tools:\n'image_generator', 'final_answer'\nYou also can perform computations in the python code you generate.\n\nAlways provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.\n\nRemember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.\nPrint results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.\n\nRemember to make sure that variables you use are all defined.\nDO NOT pass the arguments as a dict as in 'answer = ask_search_agent({'query': \"What is the place where James Bond lives?\"})', but use the arguments directly as in 'answer = ask_search_agent(query=\"What is the place where James Bond lives?\")'.\n\nNow Begin!\n",
+    "task": "Generate me a photo of the car that James bond drove in the latest movie.",
+    "agent_memory": [
+      {
+        "role": "system",
+        "content": "You will be given a task to solve as best you can.\nYou have access to the following tools:\n\n- image_generator: This is a tool that creates an image according to a prompt, which is a text description.\n    Takes inputs: {'prompt': {'type': 'text', 'description': \"The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc.\"}}\n\n- final_answer: Provides a final answer to the given problem\n    Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n\nTo solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.\n\nAt each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.\nThen in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.\nDuring each intermediate step, you can use 'print()' to save whatever important information you will then need.\nThese print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.\n\nIn the end you have to return a final answer using the `final_answer` tool.\n\nHere are a few examples using notional tools:\n---\nTask: \"Generate an image of the oldest person in this document.\"\n\nThought: I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.\nCode:\n```py\nanswer = document_qa(document=document, question=\"Who is the oldest person mentioned?\")\nprint(answer)\n```<end_code>\nObservation: \"The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland.\"\n\nThought: I will now generate an image showcasing the oldest person.\n\nCode:\n```py\nimage = image_generator(\"A portrait of John Doe, a 55-year-old man living in Canada.\")\nfinal_answer(image)\n```<end_code>\n\n---\nTask: \"What is the result of the following operation: 5 + 3 + 1294.678?\"\n\nThought: I will use python code to compute the result of the operation and then return the final answer using the `final_answer` tool\n\nCode:\n```py\nresult = 5 + 3 + 1294.678\nfinal_answer(result)\n```<end_code>\n\n---\nTask: \"Which city has the highest population , Guangzhou or Shanghai?\"\n\nThought: I need to get the populations for both cities and compare them: I will use the tool `search` to get the population of both cities.\nCode:\n```py\npopulation_guangzhou = search(\"Guangzhou population\")\nprint(\"Population Guangzhou:\", population_guangzhou)\npopulation_shanghai = search(\"Shanghai population\")\nprint(\"Population Shanghai:\", population_shanghai)\n```<end_code>\nObservation:\nPopulation Guangzhou: ['Guangzhou has a population of 15 million inhabitants as of 2021.']\nPopulation Shanghai: '26 million (2019)'\n\nThought: Now I know that Shanghai has the highest population.\nCode:\n```py\nfinal_answer(\"Shanghai\")\n```<end_code>\n\n---\nTask: \"What is the current age of the pope, raised to the power 0.36?\"\n\nThought: I will use the tool `search` to get the age of the pope, then raise it to the power 0.36.\nCode:\n```py\npope_age = search(query=\"current pope age\")\nprint(\"Pope age:\", pope_age)\n```<end_code>\nObservation:\nPope age: \"The pope Francis is currently 85 years old.\"\n\nThought: I know that the pope is 85 years old. Let's compute the result using python code.\nCode:\n```py\npope_current_age = 85 ** 0.36\nfinal_answer(pope_current_age)\n```<end_code>\n\n\nAbove example were using notional tools that might not exist for you. You only have acces to those tools:\n'image_generator', 'final_answer'\nYou also can perform computations in the python code you generate.\n\nAlways provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.\n\nRemember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.\nPrint results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.\n\nRemember to make sure that variables you use are all defined.\nDO NOT pass the arguments as a dict as in 'answer = ask_search_agent({'query': \"What is the place where James Bond lives?\"})', but use the arguments directly as in 'answer = ask_search_agent(query=\"What is the place where James Bond lives?\")'.\n\nNow Begin!\n"
+      },
+      {
+        "role": "user",
+        "content": "Task: Generate me a photo of the car that James bond drove in the latest movie."
+      }
+    ]
+  },
+  {
+    "llm_output": "Thought: To solve this task, I will first use the `image_generator` tool to generate an image of the car driven by James Bond in the latest movie. However, I need to know what car James Bond drove in the latest movie. I don't have any information about the latest James Bond movie, so I will assume that the latest movie is \"No Time to Die\" and James Bond drove an Aston Martin DB5.\n\nCode:\n```py\nprint(\"Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\")\n```<end_code>\n\n",
+    "rationale": "Thought: To solve this task, I will first use the `image_generator` tool to generate an image of the car driven by James Bond in the latest movie. However, I need to know what car James Bond drove in the latest movie. I don't have any information about the latest James Bond movie, so I will assume that the latest movie is \"No Time to Die\" and James Bond drove an Aston Martin DB5.\n\n",
+    "tool_call": {
+      "tool_name": "code interpreter",
+      "tool_arguments": "print(\"Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\")"
+    },
+    "observation": "Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\n",
+    "agent_memory": [
+      {
+        "role": "system",
+        "content": "You will be given a task to solve as best you can.\nYou have access to the following tools:\n\n- image_generator: This is a tool that creates an image according to a prompt, which is a text description.\n    Takes inputs: {'prompt': {'type': 'text', 'description': \"The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc.\"}}\n\n- final_answer: Provides a final answer to the given problem\n    Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n\nTo solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.\n\nAt each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.\nThen in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.\nDuring each intermediate step, you can use 'print()' to save whatever important information you will then need.\nThese print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.\n\nIn the end you have to return a final answer using the `final_answer` tool.\n\nHere are a few examples using notional tools:\n---\nTask: \"Generate an image of the oldest person in this document.\"\n\nThought: I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.\nCode:\n```py\nanswer = document_qa(document=document, question=\"Who is the oldest person mentioned?\")\nprint(answer)\n```<end_code>\nObservation: \"The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland.\"\n\nThought: I will now generate an image showcasing the oldest person.\n\nCode:\n```py\nimage = image_generator(\"A portrait of John Doe, a 55-year-old man living in Canada.\")\nfinal_answer(image)\n```<end_code>\n\n---\nTask: \"What is the result of the following operation: 5 + 3 + 1294.678?\"\n\nThought: I will use python code to compute the result of the operation and then return the final answer using the `final_answer` tool\n\nCode:\n```py\nresult = 5 + 3 + 1294.678\nfinal_answer(result)\n```<end_code>\n\n---\nTask: \"Which city has the highest population , Guangzhou or Shanghai?\"\n\nThought: I need to get the populations for both cities and compare them: I will use the tool `search` to get the population of both cities.\nCode:\n```py\npopulation_guangzhou = search(\"Guangzhou population\")\nprint(\"Population Guangzhou:\", population_guangzhou)\npopulation_shanghai = search(\"Shanghai population\")\nprint(\"Population Shanghai:\", population_shanghai)\n```<end_code>\nObservation:\nPopulation Guangzhou: ['Guangzhou has a population of 15 million inhabitants as of 2021.']\nPopulation Shanghai: '26 million (2019)'\n\nThought: Now I know that Shanghai has the highest population.\nCode:\n```py\nfinal_answer(\"Shanghai\")\n```<end_code>\n\n---\nTask: \"What is the current age of the pope, raised to the power 0.36?\"\n\nThought: I will use the tool `search` to get the age of the pope, then raise it to the power 0.36.\nCode:\n```py\npope_age = search(query=\"current pope age\")\nprint(\"Pope age:\", pope_age)\n```<end_code>\nObservation:\nPope age: \"The pope Francis is currently 85 years old.\"\n\nThought: I know that the pope is 85 years old. Let's compute the result using python code.\nCode:\n```py\npope_current_age = 85 ** 0.36\nfinal_answer(pope_current_age)\n```<end_code>\n\n\nAbove example were using notional tools that might not exist for you. You only have acces to those tools:\n'image_generator', 'final_answer'\nYou also can perform computations in the python code you generate.\n\nAlways provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.\n\nRemember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.\nPrint results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.\n\nRemember to make sure that variables you use are all defined.\nDO NOT pass the arguments as a dict as in 'answer = ask_search_agent({'query': \"What is the place where James Bond lives?\"})', but use the arguments directly as in 'answer = ask_search_agent(query=\"What is the place where James Bond lives?\")'.\n\nNow Begin!\n"
+      },
+      {
+        "role": "user",
+        "content": "Task: Generate me a photo of the car that James bond drove in the latest movie."
+      },
+      {
+        "role": "assistant",
+        "content": "Thought: To solve this task, I will first use the `image_generator` tool to generate an image of the car driven by James Bond in the latest movie. However, I need to know what car James Bond drove in the latest movie. I don't have any information about the latest James Bond movie, so I will assume that the latest movie is \"No Time to Die\" and James Bond drove an Aston Martin DB5.\n\nCode:\n```py\nprint(\"Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\")\n```<end_code>\n\n\n"
+      },
+      {
+        "role": "tool-response",
+        "content": "Observation: Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\n"
+      }
+    ]
+  },
+  {
+    "llm_output": "Thought: Now that I have the information about the car, I can use the `image_generator` tool to generate an image of the Aston Martin DB5.\n\nCode:\n```py\nimage = image_generator(prompt=\"A high-res, photorealistic image of the Aston Martin DB5 driven by James Bond in No Time to Die.\")\nprint(\"Image generated.\")\n```",
+    "rationale": "Thought: Now that I have the information about the car, I can use the `image_generator` tool to generate an image of the Aston Martin DB5.\n\n",
+    "tool_call": {
+      "tool_name": "code interpreter",
+      "tool_arguments": "image = image_generator(prompt=\"A high-res, photorealistic image of the Aston Martin DB5 driven by James Bond in No Time to Die.\")\nprint(\"Image generated.\")"
+    },
+    "observation": "Image generated.\n",
+    "agent_memory": [
+      {
+        "role": "system",
+        "content": "You will be given a task to solve as best you can.\nYou have access to the following tools:\n\n- image_generator: This is a tool that creates an image according to a prompt, which is a text description.\n    Takes inputs: {'prompt': {'type': 'text', 'description': \"The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc.\"}}\n\n- final_answer: Provides a final answer to the given problem\n    Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n\nTo solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.\n\nAt each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.\nThen in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.\nDuring each intermediate step, you can use 'print()' to save whatever important information you will then need.\nThese print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.\n\nIn the end you have to return a final answer using the `final_answer` tool.\n\nHere are a few examples using notional tools:\n---\nTask: \"Generate an image of the oldest person in this document.\"\n\nThought: I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.\nCode:\n```py\nanswer = document_qa(document=document, question=\"Who is the oldest person mentioned?\")\nprint(answer)\n```<end_code>\nObservation: \"The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland.\"\n\nThought: I will now generate an image showcasing the oldest person.\n\nCode:\n```py\nimage = image_generator(\"A portrait of John Doe, a 55-year-old man living in Canada.\")\nfinal_answer(image)\n```<end_code>\n\n---\nTask: \"What is the result of the following operation: 5 + 3 + 1294.678?\"\n\nThought: I will use python code to compute the result of the operation and then return the final answer using the `final_answer` tool\n\nCode:\n```py\nresult = 5 + 3 + 1294.678\nfinal_answer(result)\n```<end_code>\n\n---\nTask: \"Which city has the highest population , Guangzhou or Shanghai?\"\n\nThought: I need to get the populations for both cities and compare them: I will use the tool `search` to get the population of both cities.\nCode:\n```py\npopulation_guangzhou = search(\"Guangzhou population\")\nprint(\"Population Guangzhou:\", population_guangzhou)\npopulation_shanghai = search(\"Shanghai population\")\nprint(\"Population Shanghai:\", population_shanghai)\n```<end_code>\nObservation:\nPopulation Guangzhou: ['Guangzhou has a population of 15 million inhabitants as of 2021.']\nPopulation Shanghai: '26 million (2019)'\n\nThought: Now I know that Shanghai has the highest population.\nCode:\n```py\nfinal_answer(\"Shanghai\")\n```<end_code>\n\n---\nTask: \"What is the current age of the pope, raised to the power 0.36?\"\n\nThought: I will use the tool `search` to get the age of the pope, then raise it to the power 0.36.\nCode:\n```py\npope_age = search(query=\"current pope age\")\nprint(\"Pope age:\", pope_age)\n```<end_code>\nObservation:\nPope age: \"The pope Francis is currently 85 years old.\"\n\nThought: I know that the pope is 85 years old. Let's compute the result using python code.\nCode:\n```py\npope_current_age = 85 ** 0.36\nfinal_answer(pope_current_age)\n```<end_code>\n\n\nAbove example were using notional tools that might not exist for you. You only have acces to those tools:\n'image_generator', 'final_answer'\nYou also can perform computations in the python code you generate.\n\nAlways provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.\n\nRemember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.\nPrint results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.\n\nRemember to make sure that variables you use are all defined.\nDO NOT pass the arguments as a dict as in 'answer = ask_search_agent({'query': \"What is the place where James Bond lives?\"})', but use the arguments directly as in 'answer = ask_search_agent(query=\"What is the place where James Bond lives?\")'.\n\nNow Begin!\n"
+      },
+      {
+        "role": "user",
+        "content": "Task: Generate me a photo of the car that James bond drove in the latest movie."
+      },
+      {
+        "role": "assistant",
+        "content": "Thought: To solve this task, I will first use the `image_generator` tool to generate an image of the car driven by James Bond in the latest movie. However, I need to know what car James Bond drove in the latest movie. I don't have any information about the latest James Bond movie, so I will assume that the latest movie is \"No Time to Die\" and James Bond drove an Aston Martin DB5.\n\nCode:\n```py\nprint(\"Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\")\n```<end_code>\n\n\n"
+      },
+      {
+        "role": "tool-response",
+        "content": "Observation: Assuming the latest James Bond movie is 'No Time to Die' and James Bond drove an Aston Martin DB5.\n"
+      },
+      {
+        "role": "assistant",
+        "content": "Thought: Now that I have the information about the car, I can use the `image_generator` tool to generate an image of the Aston Martin DB5.\n\nCode:\n```py\nimage = image_generator(prompt=\"A high-res, photorealistic image of the Aston Martin DB5 driven by James Bond in No Time to Die.\")\nprint(\"Image generated.\")\n```\n"
+      },
+      {
+        "role": "tool-response",
+        "content": "Observation: Image generated.\n"
+      },
+      {
+        "role": "user",
+        "content": "Reminder: you are working towards solving the following task: Generate me a photo of the car that James bond drove in the latest movie.\nHere is a summary of your past tool calls and their results:\nStep 1\nStep 2\nTool call:{'tool_name': 'code interpreter', 'tool_arguments': 'print(\"Assuming the latest James Bond movie is \\'No Time to Die\\' and James Bond drove an Aston Martin DB5.\")'}"
+      }
+    ]
+  },
+  {
+    "llm_output": "Thought: I have generated the image of the Aston Martin DB5 using the `image_generator` tool. Now, I need to return this image as the final answer to the task.\n\nCode:\n```py\nfinal_answer(image)\n```",
+    "rationale": "Thought: I have generated the image of the Aston Martin DB5 using the `image_generator` tool. Now, I need to return this image as the final answer to the task.\n\n",
+    "tool_call": {
+      "tool_name": "code interpreter",
+      "tool_arguments": "final_answer(image)"
+    },
+    "observation": ""
+  }
+]