Upload 15 files

agentchat_function_call.ipynb

@@ -0,0 +1,445 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ae1f50ec",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_function_call.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "9a71fa36",
+   "metadata": {},
+   "source": [
+    "# Auto Generated Agent Chat: Task Solving with Provided Tools as Functions\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to make function calls with the new feature of OpenAI models (in model version 0613). A specified prompt and function configs need to be passed to `AssistantAgent` to initialize the agent. The corresponding functions need to be passed to `UserProxyAgent`, which will be responsible for executing any function calls made by `AssistantAgent`. Besides this requirement of matching descriptions with functions, we recommend checking the system message in the `AssistantAgent` to make sure the instructions align with the function call descriptions.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install the [mathchat] option since we will import functions from `MathUserProxyAgent`:\n",
+    "```bash\n",
+    "pip install \"pyautogen[mathchat]\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "2b803c17",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# %pip install \"pyautogen[mathchat]~=0.1.0\""
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "5ebd2397",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_models`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_models) function tries to create a list of configurations using Azure OpenAI endpoints and OpenAI endpoints for the provided list of models. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
+    "\n",
+    "- OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
+    "- Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
+    "- Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
+    "\n",
+    "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base.\n",
+    "If you open this notebook in google colab, you can upload your files by click the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "The following code excludes Azure OpenAI endpoints from the config list because some endpoints don't support functions yet. Remove the `exclude` argument if they do."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "dca301a4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_models(model_list=[\"gpt-4\", \"gpt-3.5-turbo\", \"gpt-3.5-turbo-16k\"], exclude=\"aoai\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "92fde41f",
+   "metadata": {},
+   "source": [
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-3.5-turbo\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo-16k',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-3.5-turbo-16k\n",
+    "]\n",
+    "```\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "2b9526e7",
+   "metadata": {},
+   "source": [
+    "## Making Function Calls\n",
+    "\n",
+    "In this example, we demonstrate function call execution with `AssistantAgent` and `UserProxyAgent`. With the default system prompt of `AssistantAgent`, we allow the LLM assistant to perform tasks with code, and the `UserProxyAgent` would extract code blocks from the LLM response and execute them. With the new \"function_call\" feature, we define functions and specify the description of the function in the OpenAI config for the `AssistantAgent`. Then we register the functions in `UserProxyAgent`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "9fb85afb",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "Draw two agents chatting with each other with an example dialog. Don't add plt.show().\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "\u001b[32m***** Suggested function Call: python *****\u001b[0m\n",
+      "Arguments: \n",
+      "{\n",
+      "  \"cell\": \"\n",
+      "import matplotlib.pyplot as plt\n",
+      "import matplotlib.patches as mpatches\n",
+      "\n",
+      "# Define basic parameters\n",
+      "face_color = '#FFDDC1'\n",
+      "plt.figure(figsize=(10, 2))\n",
+      "\n",
+      "# Agent 1\n",
+      "agent1 = mpatches.FancyBboxPatch((0.02, 0.4), 0.2, 0.6, boxstyle=mpatches.BoxStyle(\\\"Round\\\", pad=0.02))\n",
+      "plt.gca().add_artist(agent1)\n",
+      "plt.gca().text(0.12, 0.7, 'Agent 1', ha='center', va='center', fontsize=12, color='blue')\n",
+      "\n",
+      "# Agent 2\n",
+      "agent2 = mpatches.FancyBboxPatch((0.45, 0.4), 0.2, 0.6, boxstyle=mpatches.BoxStyle(\\\"Round\\\", pad=0.02))\n",
+      "plt.gca().add_artist(agent2)\n",
+      "plt.gca().text(0.55, 0.7, 'Agent 2', ha='center', va='center', fontsize=12, color='red')\n",
+      "\n",
+      "# Dialog\n",
+      "plt.gca().text(0.12, 0.35, '\\\"Hello, how are you?\\\"', ha='center', va='center', fontsize=10)\n",
+      "plt.gca().text(0.55, 0.15, '\\\"I\\'m fine, thank you!\\\"', ha='center', va='center', fontsize=10)\n",
+      "\n",
+      "# Descriptions\n",
+      "plt.gca().text(0.12, 0.15, 'Greeting', ha='center', va='center', fontsize=10)\n",
+      "plt.gca().text(0.55, 0.35, 'Response', ha='center', va='center', fontsize=10)\n",
+      "\n",
+      "plt.axis('off')\n",
+      "\"\n",
+      "}\n",
+      "\u001b[32m*******************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"python\" *****\u001b[0m\n",
+      "Error: Invalid \\escape: line 1 column 785 (char 784)\n",
+      " You argument should follow json format.\n",
+      "\u001b[32m***************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "\u001b[32m***** Suggested function Call: python *****\u001b[0m\n",
+      "Arguments: \n",
+      "{\n",
+      "  \"cell\": \"import matplotlib.pyplot as plt\\nimport matplotlib.patches as mpatches\\n\\n# Define basic parameters\\nface_color = '#FFDDC1'\\nplt.figure(figsize=(10, 2))\\n\\n# Agent 1\\nagent1 = mpatches.FancyBboxPatch((0.02, 0.4), 0.2, 0.6, boxstyle=mpatches.BoxStyle('Round', pad=0.02))\\nplt.gca().add_artist(agent1)\\nplt.gca().text(0.12, 0.7, 'Agent 1', ha='center', va='center', fontsize=12, color='blue')\\n\\n# Agent 2\\nagent2 = mpatches.FancyBboxPatch((0.45, 0.4), 0.2, 0.6, boxstyle=mpatches.BoxStyle('Round', pad=0.02))\\nplt.gca().add_artist(agent2)\\nplt.gca().text(0.55, 0.7, 'Agent 2', ha='center', va='center', fontsize=12, color='red')\\n\\n# Dialog\\nplt.gca().text(0.12, 0.35, '\\\"Hello, how are you?\\\"', ha='center', va='center', fontsize=10)\\nplt.gca().text(0.55, 0.15, '\\\"I\\\\'m fine, thank you!\\\"', ha='center', va='center', fontsize=10)\\n\\n# Descriptions\\nplt.gca().text(0.12, 0.15, 'Greeting', ha='center', va='center', fontsize=10)\\nplt.gca().text(0.55, 0.35, 'Response', ha='center', va='center', fontsize=10)\\n\\nplt.axis('off')\"\n",
+      "}\n",
+      "\u001b[32m*******************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[35m\n",
+      ">>>>>>>> EXECUTING FUNCTION python...\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "(0.0, 1.0, 0.0, 1.0)"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxoAAACuCAYAAACx83usAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAsEElEQVR4nO3deXxM5/4H8M/MZJ3JTppEEomIRGyxRIu0xBLUmlZV0Sa20lJFUbR+DaVq6Y3lVl3u1UiprS6lTdTSK2goGklEVksESQiRbbJn5vn9kZoasQyOxvJ5v155vZwzz3nO9wyek8+c85yRCSEEiIiIiIiIJCSv6wKIiIiIiOjZw6BBRERERESSY9AgIiIiIiLJMWgQEREREZHkGDSIiIiIiEhyDBpERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCRnZGhDpeeLeOGN0MdZC1Gdyt02Fz2aOWLXrl11XQo95cLCwjDjk/+D8+StdV0K0WNTdHw7Ko5vRYm6uK5LIaInlOFXNGSyx1gG0ROA/8ZJSvznRM88/iMnonvjrVNERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCTHoEFERERERJJj0CAiIiIiIskxaBARERERkeQYNIiIiIiISHIMGkREREREJDkGDSIiIiIikhyDBhERERERSY5Bg4iIiIiIJMegQUREREREkmPQICIiIiIiyTFoEBERERGR5Bg0iIiIiIhIcgwaREREREQkOQYNIiIiIiKSHIMGERERERFJjkGDiIiIiIgkx6BBRERERESSY9AgIiIiIiLJMWgQEREREZHkGDSIiIiIiEhyDBpERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCTHoEFERERERJJj0CAiIiIiIskxaBARERERkeQYNIiIiIiISHIMGkREREREJDkGDSIiIiIikhyDBhERERERSY5Bg4iIiIiIJGdU1wU8TYpPuuHGvhYwccqHU/CRui6nluKTbpAZa2DR8rJB7UtSnFB21gEVOTaozlfB1DUPjsN+f8xVEtHz4u2TkZi/bxXinbwQFBxW1+XU8vbJSJQbm2Jbyx73bWtTVoQ3T+1D97PH4Zl3CcZaDc7ZuWBt+4H42afz31AtEdHTh1c0HkBJcgMorEtRmWOLqnxlXZdTS3GcG9SJLg/UvvSsAxSWZZCbVT7GyojoeRSUHI1L1g5onZMOt/zsui6nlnfiIvFG4n6D2rbNSsW0Q+tRaG6JrzsNwZLO76DM2BRf71qMKYe/f8yVEhE9nRg0DFRVYI6KLDvYdUuGXFmBkiTnui7pkdXvFw/XyXvgOPQYFBYVdV0OET1DXAquwC8rBfO7jcZ1pTWCkqLruqRHkl6/IbqOXYOxr89GuN9ArG/bD8Pe+gIxbq3w3rFtMK8sr+sSiYieOLx1ykAlyc6Qm1XCvHEulN5XUJLcADYvn6nVTlNmjPxfm6H0jAMgA5RNrsKq/XnkhHdGvT4Jerc1VeWpUHDIG+UX60FbpYCJfTGsO52Bskmuro060QV5Ub5wGH4EpWmOKElyhqhWwMz9Our1ToRCWXMl4vKqrtAU1VxlyVzUFwDueyuUkRVPjET0eAQlR6PAzAL/a9weu739MTA5GstfHlarnU1ZET779d8IPPM7hEyOfU1ewn/aB2F3+IeY1mey3m1NjfMuYeqh9eh08RTMqyqQZu+GFZ2GYn+Tl3Rt3kjcj6+ilmHQ8MV4NS0GryUdgHl1BQ67t8Gs3hNxQ2kNAPht1Si4FNWMtRcW9QMA/O7aAm8NW3jH47ls41h7pUyGvU06wj/zFBoWXkGavfvDvl1ERM8kXtEwUEmSM5ReVyBTCKh8slCdb4GKHGu9NkIA17a1R0lyA1i0yIJt5zRo1Ka4Hulbq7/KaxbIWe+PqjwLWHU4B7tuKZAZa3Btux9K0x1qtc/f3xxV16xg7X8Glq0vouzsC7ixr7nudbvuyVBYlsHITo16/eJRr188rDuelf6NICIyQFBSNH7x6oQqhTF2+XSGR342WuWk67WRCS3WbvscA5IP4r8tumNJ53dgr87HPyKX1uqvybVM7Fg/DZ55l7GqwxuY3200yozNsGb7fPRKrz1nbu7+1fC5loHl/kOxoXUfdD97HJ/v+5fu9c+7v4tsy/o4a+eCyf2mYnK/qfi645AHPk77knwAwA1zqwfelojoWccrGgaouGKF6hsWUAaeBgCYuuRDYVmGkiRnmDoV6tqVpTuiItsWtt2TYOV3AQBg0SYTuVteQtVtfeb/2hxGVmVwCo6BzEira3v1+47Ij24KpddVvfZys0q8MOQ4ZLKaZSGA4lh3aCuMIDethtLrKgoOe0NuXgmL5lmP5X0gIjJEiytn4XnjMkID3wMAnHBpjmzL+ghKisYpJy9du57pv6Nddirmdn8X4X4DAQAb2vTBhi2za/UZ+usaZFnZY2DwUlQaGQMA1rfpi23ff4wZ0euwx6uTXvt8M0u8M2Qebg6acqHFiNifYFlRgmJTFfZ6dcTUw+uRb26FH5t3fajjtC4rxpCEvTjm0hzXLOweqg8iomcZr2gYoCTJGXJVOcwa5gGoOW+pmuagJNUJQvtXu7IMe0CuhYXvRd06mQywbJOp15+mzBjlmfWgbJoDbaUCmlJjaEqNoS0zhlmja6jOt0B1saneNhatL+pCBgCYud4AhBzVhebSHzAR0SMISjqAayobHG3YsmaFTIafm76C/qmHINdqdO26ZMSiUm6ETb69dOuETI7v2vTT68+6rBidMk8hsunLUFWWwra0sOanrAiHGrWFR342HIqv622zqXVv3DponnBtDiOhhXNhLqQgE1os/+krWFWoMSdwnCR9EhE9a3hF4z6EFihNbQCzhnmoLvzrSVMmDfKhPeGB8sz6MG9Uc4KrLjSHwqIccmOtXh9GtiV6y9X5SgAyFB72RuFh7zvuV1tqClj+NUH79vkUcrOaayTacuOHPjYiIqnJtRr0Tz2Mow1bwbXwryuz8Q28MfbEDvhnJuBwo7YAAJfCXORa2KLc2Eyvj0xbJ71l9/xsyCEw7fAGTDu84Y77rV9aiKuW9XXL2Vb2eq8XmlkAAKzL1Q9/cLeYu281AjJiMaXvR0h5wUOSPomInjUMGvdRnlkfGrUZSlOcUZpS+0lTJcnOuqBhMFHzKZvVi+dg1ujaHZsY2eiHE8jEg+2DiKgOdMo8BQf1DQxIOYQBKYdqvR6UHK0LGoaSi5rxb/WLr+PQXba9YKMfTjSyO1+wl+HRx9JJv21EcFwkFnYZgR0tuj1yf0REzyoGjfsoSW4AubICdn/Oz7hVabojStMdoO0ph9xYCyPrsj+fICXXu6pRna/S287IprTmD3IBc/c8CatlGCGiuhWUHI1rSht89uf8jFv1Tj+KnulHYdqzAhXGprhs/QI6XEyEWVW53lUNt/wcve0u/vnEp2q5AjHurSWrVUB2/0a3eefkz5gSsxFr/QbiXx3ekKwWIqJnEedo3IO2So7SdEeYN86FqumVWj+WbTMhKo1RdrbmKVFmja4BWjnUCQ11fQhR88V4t1KoKmHaMA/q+IaoVuvPxQAATanJQ9UrM9ZAW8HsSER1w7SqAr3Sj9Q80rbpy7V+Itr2g2VlGQLPHgMAHGrUFibaagxN2KPrQya0CI77Wa/fvD/newyL/wX26hu19mtXWlhrnSHKjM1gVVFy/4Z/6pdyCHP2r8GOZgGY123MQ+2TiOh5wt9K76HsrANEpTGUTa7e8XVT5/yaL+9LdobKJwfKJldg4pSP/P/5oDpfBeN6apSecYCm7OY8ir+uONgFnsbV7zsi59vOsPC9CCPrUmhKTVGZZYvqYjM0GHX4ges1cSyEOs4NBUc8YWxTArmqEuZud79iUn7JDuWXap6Uoik1gbZKgYIjngBqJpubudY+oRMR3U3g2WOwrCzT+16LW8U5e+O60hoDk6Pxs09n7G3SAfFOXvj0f2vhlp+Dc/VcEHjmGGzKauZR3HrF4f8C38e27z/Gnm8/wGbfnrho7Yj6pQVom5UKp+LreHXU1w9cb6KjJ96Oi8IHRzYj08YJ11U2OOpW+3HkAOCbnYZ/RIYh39wSR9x8EZQcrfd6rLMPLt3puzaIiJ5jDBr3UJLsDJmRBmbud55HIZMB5h65KEl2hqbMGArzKrzwxgnc+LU51Ked//zCviuw9j+Dq9930j3GFgBM6qvhGPIbCmO8UJLoAk2ZCRTKCpg4FMHav/YXARrCxv8MNEXmKDrmAVFpDFPXvHsHjcx6KIzx0lunm5zun86gQUQPZGByNMqNTHD4Lrc3CZkcBzzaY2ByNGzKilBgboWRb4Qi9Nc1GHT6VwiZHHuadMQy/2HY/v10VBj99bCLs/Ubon/IMkyO2Yg3En+FTVkx8pTWSHLwwHL/oQ9V7wr/t+BclItxx/4Ly8oy/O7a4q5Bo0neJZhqqmFaWoglu5fXen1an8kMGkREt5EJIQy6sV/Z5CW8MOizx13PM6k03QHXdvjBYfgRmLnk13U5dBe5//0cPXwcsGvXrrouhZ5yYWFhmPHp/8F50ta6LuWp1DP9KNbs+AKDhi9GrEuzui6H7qLo+A5UHN+CEnVxXZdCRE8oztGQmLZK/y0VWqD4pDtkJlUwcXi4+4iJiJ5VplUVestyrQYhJ39CkYkSpx0a11FVREQkBd46JbH8/c2hrVbAtEE+oKmZTF6RZQebzqm1vl+DiOh5N3f/aphVV+Bkg6Yw0VShd/pR+GWlYHHnYFQY135YBhERPT0YNCRm5paHouMeKDv7AoRGDmObUtj2OA2rdpn335iI6DlzxK0V3j2+A93OnoCpphKZNg3wWY9x+K5d/7oujYiIHhGDhsRUzbKhapZd12UQET0VdjULwK5mAXVdBhERPQaco0FERERERJJj0CAiIiIiIskxaBARERERkeQYNIiIiIiISHIMGkREREREJDkGDSIiIiIikhyDBhERERERSY5Bg4iIiIiIJMegQUREREREkmPQICIiIiIiyTFoEBERERGR5Bg0iIiIiIhIcgwaREREREQkOQYNIiIiIiKSHIMGERERERFJjkGDiIiIiIgkx6BBRERERESSY9AgIiIiIiLJMWgQEREREZHkGDSIiIiIiEhyDBpERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCTHoEFERERERJJj0CAiIiIiIskxaBARERERkeQYNIiIiIiISHIMGkREREREJDkGDSIiIiIikhyDBhERERERSY5Bg4iIiIiIJMegQUREREREkmPQICIiIiIiyTFoEBERERGR5Bg0iIiIiIhIcgYHDVF8DdqK0sdZC1Gd0VaUQhRfg1KprOtS6BmgVCqhqapAVcGVui6F6LEQQouqvEsw55hJRPdgcNDQFuTg6rfjURwXhcprFyC0msdZ1yNTJ+7HxWVDdMsFv32P7PCJj3Uf9HQRWg0qr11AcVwUrn47HtqCHAQHB9d1WfQM6Nu3LxxecMDVbycg/9B6VGSlQFtVXtdlET0SIQSqC6+iJPU3XNswDepTezHh/ffquiwieoIZGdowLTUFk6dMwc4fvwEAKExMYfqCB2Q2DSA3s6j1IzMygUyuAGRyyGRyQCbT9VUQswkKCztY+vbClfVTUf+1T2FkYYdq9Q1c3/EF6vX9CMZ2znr7z9v7DYxtG8CqfZBB9VYV5ABaDSqy0wAA1cV5EFUVumUp3L4PenRCUwV1wl6UX0yAprQIRlb1Ydm2P0wbeAMAihP2QKO+ARv/ocjdPh/Wnd6CqaPnLR0ICKEFhBZCq4GoroS2XF3rRxRkoyL3PDSVFQCAgUFBWLZ0Kdzd3evgqOlZ4+rqijPpaZg/fz6WLl2GoqNbIJPJYfZCQ8jsGkJhblV7zDQ2qxkz5TfHzKfvztaCmE0oP/9HzYJMDoXSBmZurWDRujdkCuO6LY7uTAgIIQChAbRaCE0VtBUltcfN4muozj2PypJCAEDLVr5YufkQXnnllTo+ACJ6ksmEEOJBNigqKkJcXBxiY2MRGxuLM+fO48aNGygoKEBRQQGqqiofV61ED83Y2ARWNjawsbGBnZ0dmjT2QLt27dCuXTu0adMGVlZWdV0iPaMqKipw+vRp3ZiZlJyC63l5KCgoRFFhPspKeUsqPXnkCgUsraxhY2MLO1tbuDg76cbMdu3awcnJqa5LJKKnwAMHjXsRQqC8vBz5+fkoLy9HdXU1NBoNqqur9drNnj0bDRo0wPjx49GqVSvs3r0bzs7OyMrKwquvvoqtW7eiadOmetuMGjUK3t7emDFjBgCgsrISK1aswC+//IKioiJ4enpiypQpaN++PQBg586dWLx4MWJiYgAA33zzDQ4cOIAffvgBAKDVarFmzRps27YN+fn58PDwwKRJk/Dyyy8bfLw393Hz58qVK2jTpg3mzZsHe3t7g/bz0UcfoX79+vjkk08AAIsWLcL333+PnTt3olGjRqiqqoK/vz9WrFiBDh061KqhoKAACxYswMmTJ1FUVAQXFxeMGTMGffr00XvvPD09oVAoEBkZiSZNmmDt2rU4c+YMwsLCcPLkSZibm6NTp06YPn06bG1ta+2ntLQU3bt3x9y5c9GzZ0/d+v/973+YOXMmDhw4AJVKhfT0dCxatAinTp2CmZkZevTogenTp+vmPtz+9wgAkyZNgqWlJebPn19rv8nJyXjrrbewd+9eODo64ptvvkF2djbmz5+P3r17Y968ebq/85uMjIygUChgZGQEMzMz2Nrawtzc3OC/V6K/U2VlJQoKCqBWq6HRaHRjpoRD899m9uzZKC4uxvLly3XrpkyZgqysLGzduhVarRbffvsttm3bhry8PLi5uWHs2LG6MaWoqAgLFizA0aNHUVpaCgcHB4wZMwZBQUG688OiRYuwceNGpKSkwNXVFZ9++in8/Px0+/vjjz8QFhaGtLQ0WFtbY8CAAfjggw9gZFRzAX/UqFHw8vKCiYkJtm/fDmNjYwwePBjjx48HUHMeW7VqFX788Ufk5eXBxsYGgYGBmDlzJoD7n3ueRgqFQjdmGhsbw8bGBpaWlpDdcicCEdFDEXUgJCREhIaGij9DjsjIyBBCCJGRkSEAiLi4uFrbdOnSRUyaNEm3PGbMGNGpUydx6NAhcfbsWbFkyRJhamoq0tPThRBChIeHC2tra1370NBQ4evrq1sOCwsTVlZWYtOmTSI1NVV8/PHHwtjYWLe9IcLDw4WxsbHo0aOHOHHihIiNjRU+Pj5i2LBhBu9nxYoVonnz5rr2rVu3FvXr1xerVq0SQgjx22+/CWNjY1FSUnLHGi5fviyWLFki4uLixLlz58SKFSuEQqEQx44d03vvLCwsxPTp00VqaqpITU0V+fn5wt7eXsyaNUukpKSIkydPisDAQNG1a9e7Hu+7774r+vTpo7duwIABIjg4WAghhFqtFk5OTuL1118XiYmJ4tdffxWNGjUSISEherXc+vcohBADBw7Ua3NTeXm56NmzpwgMDNStCw0N1bV1c3MTBw4cuGu9RPT3CgkJEQMHDtQtJyYmCkdHR/HSSy8JIYSYP3++aNq0qfjll1/EuXPnRHh4uDA1NRXR0dFCCCEmTJggWrduLU6cOCEyMjLEvn37xK5du4QQf50fXFxcxLZt20RycrIYM2aMsLS0FNevXxdC1IyHSqVSjB8/XqSkpIgdO3aI+vXr6843QtSMQVZWVmLOnDkiPT1dRERECJlMJvbu3SuEEOKHH34QVlZWIioqSmRmZopjx46JNWvW6La/37mHiIj+UidB425unkjMzc2FSqXS+5HL5bpfUDMzM4VCoRBZWVl623fv3l3MmjVLCHH/oNGgQQPxxRdf6G3fvn17MX78eIPrDQ8PFwDE2bNndetWrlwpHBwcDN7PqVOnhEwmE7m5ueLGjRvCxMREzJs3TwwZMkQIUXNi7tSpk8E1CSFE3759xdSpU3XLXbp0EW3atNFrM2/ePNGzZ0+9dZcuXRIARFpa2h37PXbsmFAoFCI7O1sIIcTVq1eFkZGR7peENWvWCFtbW6FWq3XbREZGCrlcLq5cuaKrxZCgUVVVJXr27Ck6deokCgsLDT94IqozISEhQqFQCJVKJUxNTQUAIZfLxbZt20R5eblQKpXiyJEjetuMHj1aDB06VAghRP/+/cXIkSPv2PfN88PChQt166qqqoSLi4tYtGiREEKITz75RHh7ewutVqtrs3LlSmFhYSE0Go0QomYMevnll/X6bt++vZgxY4YQQoh//OMfwsvLS1RWVtaqwZBzDxER/cXgyeB/py1btsDHx0dv3fDhw3V/TkxMhEajgZeXl16biooK1KtX7779FxUVITs7G/7+/nrr/f39kZCQ8EC1KpVKNG7cWLfs5OSE3Nxcg/fTokUL2NnZ4eDBgzAxMUGbNm3Qr18/rFy5EgBw8OBBBAQE3HX/Go0GCxYswNatW5GVlYXKykpUVFTUekxru3bt9JYTEhJw4MABWFhY1Orz3Llztd5bAHjxxRfRvHlzREREYObMmdiwYQPc3NzQuXNnAEBKSgp8fX2hUqn0jlWr1SItLQ0ODg53PY7b7dixA7/99hsuX77M+RNET5GuXbti1apVKCkpwdKlS2FkZIRBgwYhKSkJpaWlCAwM1GtfWVmJNm3aAADef/99DBo0CCdPnkTPnj0RFBSETp066bXv2LGj7s9GRkbw8/NDSkoKgJoxqGPHjnq3/Pj7+0OtVuPy5cto2LAhAKBVq1Z6fd46bg8ePBjLli2Dh4cHevfujT59+qB///4wMjJ65HMPEdHz5okMGq6urvD09NRbd+s99mq1GgqFArGxsVAoFHrt7vSL8+NkbKz/JBWZTPZA91bLZDJ07twZ0dHRMDU1RUBAAFq1aqWbQHrkyBFMmzbtrtsvWbIEy5cvx7Jly9CyZUuoVCpMnjwZlZX6k/Jv/eUfqHkP+/fvj0WLFtXq816T/MaMGYOVK1di5syZCA8Px8iRIx/oPl65XF7r/amqqqrVLjs7G/b29necL0JETy6VSqUbv7/99lv4+vpi7dq1aNGiBQAgMjISzs76TxU0NTUFALz66qvIzMxEVFQU9u3bh+7du2PChAn46quvJK3xTuO2VqsFUHP+SUtLw/79+7Fv3z6MHz8eS5YswcGDB5+ocw8R0dPg6Xt+IoA2bdpAo9EgNzcXnp6eej+Ojo733d7KygoNGjTQTRS/KSYmBs2aNZOsTkP306VLF0RHRyM6OhoBAQGQy+Xo3LkzlixZgoqKilpXRG7va+DAgXj77bfh6+sLDw8PpKen37e2tm3bIikpCe7u7rXew9tDya3efvttZGZmYsWKFUhOTkZISIjuNR8fHyQkJKCkpESvPrlcDm/vmsfT2tvbIycnR/e6RqPB6dOna+1n6NCh+Omnn+57HET05JLL5fjkk08we/ZsNGvWDKamprh48WKtMcfV1VW3jb29PUJCQrBhwwYsW7YMa9as0evz999/1/25uroasbGxuivgPj4+OHr0qN6HGTExMbC0tISLi4vBdZubm6N///5YsWIFoqOjcfToUSQmJj7yuYeI6HnzVAYNLy8vDB8+HMHBwdi+fTsyMjJw/PhxfPnll4iMjDSoj+nTp2PRokXYsmUL0tLSMHPmTMTHx2PSpEmS1mrIfgICApCcnIykpCTd06gCAgLw/fffw8/P756/+Ddp0gT79u3DkSNHkJKSgnHjxuHq1av3rWvChAm4ceMGhg4dihMnTuDcuXPYs2cPRo4cCY3m7l/GaGtri9dffx3Tp09Hz5499U7ew4cPh5mZGUJCQnD69GkcOHAAEydOxDvvvKO7bapbt26IjIxEZGQkUlNT8f7776OgoKDWfrZu3YrJkyff9ziI6Mk2ePBgKBQKrF69GtOmTcOUKVMQERGBc+fO4eTJk/jnP/+JiIgIAMBnn32GnTt34uzZs0hKSsLPP/9c6zbalStXYseOHUhNTcWECROQn5+PUaNGAQDGjx+PS5cuYeLEiUhNTcXOnTsRGhqKjz76CHK5Yae7devWYe3atTh9+jTOnz+PDRs2wNzcHG5ubpKce4iInidP5K1ThggPD8f8+fMxdepUZGVloX79+ujQoQP69etn0PYffvghCgsLMXXqVOTm5qJZs2bYtWsXmjRpomsTEBAAd3d3rFu37qHrNGQ/LVu2hI2NDby8vHSX3wMCAqDRaO45PwOoeZzk+fPn0atXLyiVSowdOxZBQUEoLCy853Y3r7TMmDEDPXv2REVFBdzc3NC7d+/7npBHjx6NjRs36k7uNymVSuzZsweTJk1C+/btoVQqMWjQIISFhenajBo1CgkJCQgODoaRkRGmTJmCrl271trH9evXce7cuXvWQURPPiMjI3zwwQdYvHgxMjIyYG9vjy+//BLnz5+HjY0N2rZtq3u8t4mJCWbNmoULFy7A3Nwcr7zyCjZv3qzX38KFC7Fw4ULEx8fD09MTu3btQv369QEAzs7OiIqKwvTp0+Hr6ws7OzuMHj0as2fPNrheGxsbLFy4EB999BE0Gg1atmyJn376STcH41HPPUREzxNJv0fjWePm5oa5c+dixIgRdV3KE2X9+vWYMmUKsrOzYWJiUtflENFz4MKFC2jUqBHi4uLQunXrui6HiIgM8NRe0XjckpKSYG1tjeDg4Lou5YlRWlqKnJwcLFy4EOPGjWPIICIiIqK7eirnaPwdmjdvjlOnThl8X+/zYPHixWjatCkcHR0xa9asui6HiIiIiJ5gvHWKiIiIiIgkx4/riYiIiIhIcgwaREREREQkOQYNIiIiIiKSHIMGERERERFJjkGDiIiIiIgkx6BBRERERESSY9AgIiIiIiLJMWgQEREREZHkGDSIiIiIiEhyDBpERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCTHoEFERERERJJj0CAiIiIiIskxaBARERERkeQYNJ4BMpkMP/74Y12XQUTPuDVr1sDV1RVyuRzLli3DnDlz0Lp167ouCwBw4cIFyGQyxMfHP/Z9ccwlIjIMg8YDunLlCiZNmgRPT0+YmZnBwcEB/v7+WLVqFUpLSx/rvu92Us/JycGrr776WPdNRHVrxIgRmDNnDoCaX3QvXLgAoPYv2DeXpVZUVIQPPvgAM2bMQFZWFsaOHYtp06bh119/lXxf9zNixAgEBQX97ft9GtwegqKjo+Hu7g5A/98QEdHfwaiuC3ianD9/Hv7+/rCxscGCBQvQsmVLmJqaIjExEWvWrIGzszMGDBhQa7uqqioYGxs/trocHR0fW99ERABw8eJFVFVVoW/fvnByctKtt7CwqMOqiIjoScYrGg9g/PjxMDIywh9//IE333wTPj4+8PDwwMCBAxEZGYn+/fsDqPlEadWqVRgwYABUKhW++OILAMDOnTvRtm1bmJmZwcPDA3PnzkV1dbWu/4KCAowZMwb29vawsrJCt27dkJCQAABYt24d5s6di4SEBMhkMshkMqxbt063v5ufYN38NHP79u3o2rUrlEolfH19cfToUb1j+fe//w1XV1colUq89tprCAsLg42NzeN9A4nob7du3TrY2Njg559/hre3N5RKJd544w2UlpYiIiIC7u7usLW1xYcffgiNRnPXPlq2bAkA8PDw0F1Ruf0q680rDV999RWcnJxQr149TJgwAVVVVbo2FRUVmDZtGpydnaFSqfDSSy8hOjra4OOZM2cOIiIisHPnTt1YeOv258+fv+vYl5eXh6FDh8LZ2RlKpRItW7bEpk2b9PoPCAjAhx9+iI8//hh2dnZwdHS871WA0NBQODk54dSpU7Veu3DhAuRyOf744w+99cuWLYObmxu0Wi0A4ODBg3jxxRdhamoKJycnzJw5U+/84O7ujmXLlun10bp1a16hIKInGoOGgfLy8rB3715MmDABKpXqjm1uvV1hzpw5eO2115CYmIhRo0bh8OHDCA4OxqRJk5CcnIzVq1dj3bp1uhACAIMHD0Zubi52796N2NhYtG3bFt27d8eNGzcwZMgQTJ06Fc2bN0dOTg5ycnIwZMiQu9b76aefYtq0aYiPj4eXlxeGDh2qO2nFxMTgvffew6RJkxAfH4/AwEC9Oojo2VJaWooVK1Zg8+bN+OWXXxAdHY3XXnsNUVFRiIqKwvr167F69Wps27btjtsPGTIE+/fvBwAcP34cOTk5cHV1vWPbAwcO4Ny5czhw4AAiIiKwbt063YciAPDBBx/g6NGj2Lx5M06dOoXBgwejd+/eOHPmjEHHMm3aNLz55pvo3bu3bizs1KmT7vV7jX3l5eVo164dIiMjcfr0aYwdOxbvvPMOjh8/rrePiIgIqFQqHDt2DIsXL8bnn3+Offv21apFCIGJEyfiu+++w+HDh9GqVatabdzd3dGjRw+Eh4frrQ8PD8eIESMgl8uRlZWFPn36oH379khISMCqVauwdu1azJ8/36D3hIjoiSXIIL///rsAILZv3663vl69ekKlUgmVSiU+/vhjIYQQAMTkyZP12nXv3l0sWLBAb9369euFk5OTEEKIw4cPCysrK1FeXq7XpnHjxmL16tVCCCFCQ0OFr69vrdoAiB07dgghhMjIyBAAxH/+8x/d60lJSQKASElJEUIIMWTIENG3b1+9PoYPHy6sra0NeCeI6Ely8/98XFzcHV8PDw8XAMTZs2d168aNGyeUSqUoLi7WrevVq5cYN27cXfcTFxcnAIiMjAzdutvHpJCQEOHm5iaqq6t16wYPHiyGDBkihBAiMzNTKBQKkZWVpdd39+7dxaxZsww5XN1+Bg4cqLfOkLHvTvr27SumTp2qW+7SpYt4+eWX9dq0b99ezJgxQ7cMQPzwww9i2LBhwsfHR1y+fPme9W7ZskXY2trqxvfY2Fghk8l07+Unn3wivL29hVar1W2zcuVKYWFhITQajRBCCDc3N7F06VK9fn19fUVoaKjeulvPB0REdY1XNB7R8ePHER8fj+bNm6OiokK33s/PT69dQkICPv/8c1hYWOh+3n33XeTk5KC0tBQJCQlQq9WoV6+eXpuMjAycO3fugeu69ZO1m/dT5+bmAgDS0tLw4osv6rW/fZmInh1KpRKNGzfWLTs4OMDd3V1vfoWDg4NujHgUzZs3h0Kh0C07OTnp+k1MTIRGo4GXl5feOHfw4MGHGufu5F5jn0ajwbx589CyZUvY2dnBwsICe/bswcWLF+/ax+3HcNOUKVNw7NgxHDp0CM7OzvesKSgoCAqFAjt27ABQcyta165ddZO0U1JS0LFjR72r4v7+/lCr1bh8+fIDHD0R0ZOFk8EN5OnpCZlMhrS0NL31Hh4eAABzc3O99bffXqVWqzF37ly8/vrrtfo2MzODWq2Gk5PTHe9Vfpi5E7dOPr958rp5LzARPV9ufxiFTCa74zopxoh79atWq6FQKBAbG6sXRgDpJpXfa+xbsmQJli9fjmXLlqFly5ZQqVSYPHkyKisrDT6GmwIDA7Fp0ybs2bMHw4cPv2dNJiYmCA4ORnh4OF5//XVs3LgRy5cvf6DjksvlEELorbt17gsR0ZOIQcNA9erVQ2BgIL7++mtMnDjxrvM07qZt27ZIS0uDp6fnXV+/cuUKjIyMdJ9y3c7ExOSukzUfhLe3N06cOKG37vZlIiKptWnTBhqNBrm5uXjllVceup+HHQtjYmIwcOBAvP322wBqAkh6ejqaNWv2wH0NGDAA/fv3x7Bhw6BQKPDWW2/ds/2YMWPQokULfPPNN6iurtb70MnHxwf//e9/IYTQhaOYmBhYWlrCxcUFAGBvb4+cnBzdNkVFRcjIyKi1n9vDCBFRXeKtUw/g5gnCz88PW7ZsQUpKCtLS0rBhwwakpqbW+oTuVp999hm+++47zJ07F0lJSUhJScHmzZsxe/ZsAECPHj3QsWNHBAUFYe/evbhw4QKOHDmCTz/9VPe0End3d2RkZCA+Ph7Xr1/Xu1XrQUycOBFRUVEICwvDmTNnsHr1auzevfuxPHufiOgmLy8vDB8+HMHBwdi+fTsyMjJw/PhxfPnll4iMjDS4H3d3d5w6dQppaWm4fv26wZ/sN2nSBPv27cORI0eQkpKCcePG4erVqw97OHjttdewfv16jBw58q4T6W/y8fFBhw4dMGPGDAwdOlTvKvj48eNx6dIlTJw4Eampqdi5cydCQ0Px0UcfQS6vOU1369YN69evx+HDh5GYmIiQkJBa55ysrCw0bdq0Tr7bhIjoThg0HkDjxo0RFxeHHj16YNasWfD19YWfnx/++c9/Ytq0aZg3b95dt+3Vqxd+/vln7N27F+3bt0eHDh2wdOlSuLm5Aai5NB8VFYXOnTtj5MiR8PLywltvvYXMzEw4ODgAAAYNGoTevXuja9eusLe3r/VYRkP5+/vjX//6F8LCwuDr64tffvkFU6ZMgZmZ2UP1R0RkqPDwcAQHB2Pq1Knw9vZGUFAQTpw4gYYNG+ra3Pr47jt599134e3tDT8/P9jb2yMmJsagfc+ePRtt27ZFr169EBAQAEdHx0f+4r833ngDEREReOedd7B9+/Z7th09ejQqKysxatQovfXOzs6IiorC8ePH4evri/feew+jR4/WfRAFALNmzUKXLl3Qr18/9O3bF0FBQXrzboCaW6nS0tJQXFz8SMdERCQVmeB1VkLNiTs1NRWHDx+u61KI6DmWkZEBLy8vJCcno0mTJnVdjqTmzZuHH3744Y7ft0FE9CziHI3n1FdffYXAwECoVCrs3r0bERER+Oabb+q6LCJ6zkVFRWHs2LHPVMhQq9W4cOECvv76a343BhE9V3hF4zn15ptvIjo6GsXFxfDw8MDEiRPx3nvv1XVZRETPnBEjRmDTpk0ICgrCxo0b7zmfj4joWcKgQUREREREkuNkcCIiIiIikhyDBhERERERSY5Bg4iIiIiIJMegQUREREREkmPQICIiIiIiyTFoEBERERGR5Bg0iIiIiIhIcgwaREREREQkOQYNIiIiIiKSHIMGERERERFJjkGDiIiIiIgkx6BBRERERESSY9AgIiIiIiLJMWgQEREREZHkGDSIiIiIiEhyDBpERERERCQ5Bg0iIiIiIpIcgwYREREREUmOQYOIiIiIiCTHoEFERERERJJj0CAiIiIiIskxaBARERERkeQYNIiIiIiISHIMGkREREREJDkGDSIiIiIiktz/A0/x9tng3v+UAAAAAElFTkSuQmCC",
+      "text/plain": [
+       "<Figure size 1000x200 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"python\" *****\u001b[0m\n",
+      "(0.0, 1.0, 0.0, 1.0)\n",
+      "\u001b[32m***************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "llm_config = {\n",
+    "    \"functions\": [\n",
+    "        {\n",
+    "            \"name\": \"python\",\n",
+    "            \"description\": \"run cell in ipython and return the execution result.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"cell\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"Valid Python cell to execute.\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"cell\"],\n",
+    "            },\n",
+    "        },\n",
+    "        {\n",
+    "            \"name\": \"sh\",\n",
+    "            \"description\": \"run a shell script and return the execution result.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"script\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"Valid shell script to execute.\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"script\"],\n",
+    "            },\n",
+    "        },\n",
+    "    ],\n",
+    "    \"config_list\": config_list,\n",
+    "    \"request_timeout\": 120,\n",
+    "}\n",
+    "chatbot = autogen.AssistantAgent(\n",
+    "    name=\"chatbot\",\n",
+    "    system_message=\"For coding tasks, only use the functions you have been provided with. Reply TERMINATE when the task is done.\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "\n",
+    "# create a UserProxyAgent instance named \"user_proxy\"\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"user_proxy\",\n",
+    "    is_termination_msg=lambda x: x.get(\"content\", \"\") and x.get(\"content\", \"\").rstrip().endswith(\"TERMINATE\"),\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    max_consecutive_auto_reply=10,\n",
+    "    code_execution_config={\"work_dir\": \"coding\"},\n",
+    ")\n",
+    "\n",
+    "# define functions according to the function desription\n",
+    "from IPython import get_ipython\n",
+    "\n",
+    "def exec_python(cell):\n",
+    "    ipython = get_ipython()\n",
+    "    result = ipython.run_cell(cell)\n",
+    "    log = str(result.result)\n",
+    "    if result.error_before_exec is not None:\n",
+    "        log += f\"\\n{result.error_before_exec}\"\n",
+    "    if result.error_in_exec is not None:\n",
+    "        log += f\"\\n{result.error_in_exec}\"\n",
+    "    return log\n",
+    "\n",
+    "def exec_sh(script):\n",
+    "    return user_proxy.execute_code_blocks([(\"sh\", script)])\n",
+    "\n",
+    "# register the functions\n",
+    "user_proxy.register_function(\n",
+    "    function_map={\n",
+    "        \"python\": exec_python,\n",
+    "        \"sh\": exec_sh,\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# start the conversation\n",
+    "user_proxy.initiate_chat(\n",
+    "    chatbot,\n",
+    "    message=\"Draw two agents chatting with each other with an example dialog. Don't add plt.show().\",\n",
+    ")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "e9531d55",
+   "metadata": {},
+   "source": [
+    "## Another example with Wolfram Alpha API\n",
+    "\n",
+    "We give another example of querying Wolfram Alpha API to solve math problem. We use the predefined function `MathUserProxyAgent().execute_one_wolfram_query` as the function to be called."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "4a917492",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "Problem: Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "\u001b[32m***** Suggested function Call: query_wolfram *****\u001b[0m\n",
+      "Arguments: \n",
+      "{\n",
+      "  \"query\": \"solve (2x+10)(x+3)<(3x+9)(x+8) for x\"\n",
+      "}\n",
+      "\u001b[32m**************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[35m\n",
+      ">>>>>>>> EXECUTING FUNCTION query_wolfram...\u001b[0m\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"query_wolfram\" *****\u001b[0m\n",
+      "('Assumption: solve (2 x + 10) (x + 3)<(3 x + 9) (x + 8) for x \\nAnswer: ans 0: x<-14\\nans 1: x>-3\\n', True)\n",
+      "\u001b[32m**********************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "The solution to the inequality $(2x+10)(x+3)<(3x+9)(x+8)$ is $x \\in (-\\infty, -14) \\cup (-3, +\\infty)$. TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser_proxy\u001b[0m (to chatbot):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mchatbot\u001b[0m (to user_proxy):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "from autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
+    "\n",
+    "# you need to provide a wolfram alpha appid to run this example\n",
+    "if not os.environ.get(\"WOLFRAM_ALPHA_APPID\"):\n",
+    "    os.environ[\"WOLFRAM_ALPHA_APPID\"] = open(\"wolfram.txt\").read().strip()\n",
+    "\n",
+    "llm_config = {\n",
+    "    \"model\": \"gpt-4-0613\",\n",
+    "    \"functions\": [\n",
+    "        {\n",
+    "            \"name\": \"query_wolfram\",\n",
+    "            \"description\": \"Return the API query result from the Wolfram Alpha. the ruturn is a tuple of (result, is_success).\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"query\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The Wolfram Alpha code to be executed.\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"query\"],\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"config_list\": config_list,\n",
+    "}\n",
+    "chatbot = autogen.AssistantAgent(\n",
+    "    name=\"chatbot\",\n",
+    "    system_message=\"Only use the functions you have been provided with. Do not ask user to perform other actions than executing the functions. Reply TERMINATE when the task is done.\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "\n",
+    "# the key in `function_map` should match the function name in \"functions\" above\n",
+    "# we register a class instance method directly\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    \"user_proxy\",\n",
+    "    max_consecutive_auto_reply=2,\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    function_map={\"query_wolfram\": MathUserProxyAgent().execute_one_wolfram_query},\n",
+    ")\n",
+    "\n",
+    "# start the conversation\n",
+    "user_proxy.initiate_chat(\n",
+    "    chatbot,\n",
+    "    message=\"Problem: Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.\",\n",
+    ")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "flaml_dev",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

agentchat_groupchat.ipynb

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_groupchat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Auto Generated Agent Chat: Group Chat\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "This notebook is modified based on https://github.com/microsoft/FLAML/blob/4ea686af5c3e8ff24d9076a7a626c8b28ab5b1d7/notebook/autogen_multiagent_roleplay_chat.ipynb\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
+    "```bash\n",
+    "pip install pyautogen\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 105,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "# %pip install pyautogen~=0.1.0"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 106,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list_gpt4 = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")\n",
+    "# config_list_gpt35 = autogen.config_list_from_json(\n",
+    "#     \"OAI_CONFIG_LIST\",\n",
+    "#     filter_dict={\n",
+    "#         \"model\": {\n",
+    "#             \"gpt-3.5-turbo\",\n",
+    "#             \"gpt-3.5-turbo-16k\",\n",
+    "#             \"gpt-3.5-turbo-0301\",\n",
+    "#             \"chatgpt-35-turbo-0301\",\n",
+    "#             \"gpt-35-turbo-v0301\",\n",
+    "#         },\n",
+    "#     },\n",
+    "# )"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). Only the gpt-4 models are kept in the list based on the filter condition.\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 107,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_config = {\"config_list\": config_list_gpt4, \"seed\": 42}\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "   name=\"User_proxy\",\n",
+    "   system_message=\"A human admin.\",\n",
+    "   code_execution_config={\"last_n_messages\": 2, \"work_dir\": \"groupchat\"},\n",
+    "   human_input_mode=\"TERMINATE\"\n",
+    ")\n",
+    "coder = autogen.AssistantAgent(\n",
+    "    name=\"Coder\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "pm = autogen.AssistantAgent(\n",
+    "    name=\"Product_manager\",\n",
+    "    system_message=\"Creative in software product ideas.\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)\n",
+    "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start Chat"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 108,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCoder\u001b[0m (to chat_manager):\n",
+      "\n",
+      "To find the latest paper about GPT-4 on arxiv, I'll provide you with a Python code that fetches the most recent papers from the arxiv API and filters the results to get the most relevant paper related to GPT-4. After fetching the paper, I'll extract the information for potential applications in software. Please execute the following Python code:\n",
+      "\n",
+      "```python\n",
+      "import requests\n",
+      "from bs4 import BeautifulSoup\n",
+      "import re\n",
+      "\n",
+      "def fetch_arxiv_papers(query):\n",
+      "    base_url = \"http://export.arxiv.org/api/query?\"\n",
+      "    search_query = \"all:\" + query\n",
+      "    response = requests.get(base_url, params={\"search_query\": search_query, \"sortBy\": \"submittedDate\", \"sortOrder\": \"descending\"})\n",
+      "    return BeautifulSoup(response.content, \"xml\")\n",
+      "\n",
+      "def find_gpt4_paper():\n",
+      "    papers = fetch_arxiv_papers(\"gpt-4\")\n",
+      "    for entry in papers.find_all(\"entry\"):\n",
+      "        title = entry.title.text.strip()\n",
+      "        summary = entry.summary.text.strip()\n",
+      "        if \"gpt-4\" in title.lower() or \"gpt-4\" in summary.lower():\n",
+      "            return {\"title\": title, \"summary\": summary}\n",
+      "\n",
+      "gpt4_paper = find_gpt4_paper()\n",
+      "if gpt4_paper:\n",
+      "    print(\"Title:\", gpt4_paper[\"title\"])\n",
+      "    print(\"Summary:\", gpt4_paper[\"summary\"])\n",
+      "else:\n",
+      "    print(\"No recent GPT-4 papers found.\")\n",
+      "```\n",
+      "\n",
+      "Once we have the paper details, I'll analyze the summary to identify potential applications in software development.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "Title: FIMO: A Challenge Formal Dataset for Automated Theorem Proving\n",
+      "Summary: We present FIMO, an innovative dataset comprising formal mathematical problem\n",
+      "statements sourced from the International Mathematical Olympiad (IMO)\n",
+      "Shortlisted Problems. Designed to facilitate advanced automated theorem proving\n",
+      "at the IMO level, FIMO is currently tailored for the Lean formal language. It\n",
+      "comprises 149 formal problem statements, accompanied by both informal problem\n",
+      "descriptions and their corresponding LaTeX-based informal proofs. Through\n",
+      "initial experiments involving GPT-4, our findings underscore the existing\n",
+      "limitations in current methodologies, indicating a substantial journey ahead\n",
+      "before achieving satisfactory IMO-level automated theorem proving outcomes.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mProduct_manager\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Based on the paper titled \"FIMO: A Challenge Formal Dataset for Automated Theorem Proving\" and its summary, the potential applications of GPT-4 in software development can be related to the field of automated theorem proving.\n",
+      "\n",
+      "1. **Automated theorem proving**: GPT-4 can be utilized in the development of automated theorem proving software that attempts to prove complex mathematical problems taken from International Mathematical Olympiad (IMO) or other challenging sources. By fine-tuning GPT-4 with a dataset like FIMO consisting of formal mathematical problems, the model can potentially better understand the problem statements and generate appropriate proofs.\n",
+      "\n",
+      "2. **Mathematical problem-solving assistants**: Software tools can be developed using GPT-4 to guide users in solving complex mathematical problems. The AI model can be integrated into educational platforms, online math tutoring services, or even standalone tools to help make solving problems easier and faster for students and professionals alike.\n",
+      "\n",
+      "3. **Formal language translation**: GPT-4 can potentially be integrated into software for translating between formal languages, assisting in the understanding and comparison of various formal systems. This would be especially useful in research communities employing different formal languages and wanting to share ideas and results.\n",
+      "\n",
+      "4. **Mathematical proof checking**: GPT-4 can be employed in proof-checking software to identify and correct inconsistencies. By improving the correctness of proofs, this application would ultimately help users save time and contribute to the overall quality of mathematical research.\n",
+      "\n",
+      "Please note that this paper highlights the current limitations of GPT-4 in the context of IMO-level theorem proving. Nevertheless, these potential applications suggest directions for further research and software development as the model and related techniques continue to improve.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCoder\u001b[0m (to chat_manager):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "user_proxy.initiate_chat(manager, message=\"Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\")\n",
+    "# type exit to terminate the chat"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "flaml",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_groupchat_research.ipynb

@@ -0,0 +1,566 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/groupchat_research.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Auto Generated Agent Chat: Performs Research with Multi-Agent Group Chat\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
+    "```bash\n",
+    "pip install pyautogen\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "# %pip install pyautogen~=0.1.0"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list_gpt4 = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well).\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k-0314',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gpt4_config = {\n",
+    "    \"seed\": 42,  # change the seed for different trials\n",
+    "    \"temperature\": 0,\n",
+    "    \"config_list\": config_list_gpt4,\n",
+    "    \"request_timeout\": 120,\n",
+    "}\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "   name=\"Admin\",\n",
+    "   system_message=\"A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.\",\n",
+    "   code_execution_config=False,\n",
+    ")\n",
+    "engineer = autogen.AssistantAgent(\n",
+    "    name=\"Engineer\",\n",
+    "    llm_config=gpt4_config,\n",
+    "    system_message='''Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.\n",
+    "Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.\n",
+    "If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n",
+    "''',\n",
+    ")\n",
+    "scientist = autogen.AssistantAgent(\n",
+    "    name=\"Scientist\",\n",
+    "    llm_config=gpt4_config,\n",
+    "    system_message=\"\"\"Scientist. You follow an approved plan. You are able to categorize papers after seeing their abstracts printed. You don't write code.\"\"\"\n",
+    ")\n",
+    "planner = autogen.AssistantAgent(\n",
+    "    name=\"Planner\",\n",
+    "    system_message='''Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.\n",
+    "The plan may involve an engineer who can write code and a scientist who doesn't write code.\n",
+    "Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.\n",
+    "''',\n",
+    "    llm_config=gpt4_config,\n",
+    ")\n",
+    "executor = autogen.UserProxyAgent(\n",
+    "    name=\"Executor\",\n",
+    "    system_message=\"Executor. Execute the code written by the engineer and report the result.\",\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    code_execution_config={\"last_n_messages\": 3, \"work_dir\": \"paper\"},\n",
+    ")\n",
+    "critic = autogen.AssistantAgent(\n",
+    "    name=\"Critic\",\n",
+    "    system_message=\"Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.\",\n",
+    "    llm_config=gpt4_config,\n",
+    ")\n",
+    "groupchat = autogen.GroupChat(agents=[user_proxy, engineer, scientist, planner, executor, critic], messages=[], max_round=50)\n",
+    "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=gpt4_config)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start Chat"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mPlanner\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Plan:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link.\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCritic\u001b[0m (to chat_manager):\n",
+      "\n",
+      "The plan seems solid and well-structured. However, it lacks the inclusion of verifiable information such as source URLs. Here's a revised version:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link (source URL).\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate. Check the source URLs to verify the information.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table, ensuring all entries have a valid source URL.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mEngineer\u001b[0m (to chat_manager):\n",
+      "\n",
+      "I'm sorry for the confusion, but as an AI model developed by OpenAI, I don't have the ability to directly scrape websites or access real-time data from external databases or websites, including arXiv. However, I can help you write a Python script that uses the arXiv API to fetch the data you need.\n",
+      "\n",
+      "Here's a Python script that uses the `arxiv` package to search for papers related to \"LLM applications\" from the last week. This script will print out the title, authors, summary, and URL of each paper.\n",
+      "\n",
+      "```python\n",
+      "import arxiv\n",
+      "import datetime\n",
+      "\n",
+      "# Calculate the date one week ago\n",
+      "one_week_ago = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime('%Y%m%d%H%M%S')\n",
+      "\n",
+      "# Search for papers on LLM applications\n",
+      "search = arxiv.Search(\n",
+      "  query=\"LLM applications\",\n",
+      "  max_results=100,\n",
+      "  sort_by=arxiv.SortCriterion.SubmittedDate,\n",
+      "  sort_order=arxiv.SortOrder.Descending\n",
+      ")\n",
+      "\n",
+      "for result in search.get():\n",
+      "  # Check if the paper was submitted in the last week\n",
+      "  if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"Title: {result.title}\")\n",
+      "    print(f\"Authors: {', '.join(author.name for author in result.authors)}\")\n",
+      "    print(f\"Summary: {result.summary}\")\n",
+      "    print(f\"URL: {result.entry_id}\")\n",
+      "    print(\"\\n\")\n",
+      "```\n",
+      "\n",
+      "You can install the `arxiv` package using pip:\n",
+      "\n",
+      "```shell\n",
+      "pip install arxiv\n",
+      "```\n",
+      "\n",
+      "Please note that you'll need to replace `\"LLM applications\"` with the actual search term you're interested in. The arXiv API doesn't support searching by the submission date, so the script fetches the 100 most recent results and filters them manually. You may need to increase `max_results` if there are more than 100 papers submitted per week on your topic.\n",
+      "\n",
+      "To create a markdown table, you'll need to format the output like this:\n",
+      "\n",
+      "```markdown\n",
+      "| Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|\n",
+      "| ...   | ...     | ...     | ... |\n",
+      "```\n",
+      "\n",
+      "You can do this in Python using f-strings:\n",
+      "\n",
+      "```python\n",
+      "print(\"| Title | Authors | Summary | URL |\")\n",
+      "print(\"|-------|---------|---------|-----|\")\n",
+      "for result in search.get():\n",
+      "  # Check if the paper was submitted in the last week\n",
+      "  if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"| {result.title} | {', '.join(author.name for author in result.authors)} | {result.summary} | {result.entry_id} |\")\n",
+      "```\n",
+      "\n",
+      "This will print a markdown table to the console. If you want to write the table to a file, you can open a file and use the `write` method instead of `print`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mExecutor\u001b[0m (to chat_manager):\n",
+      "\n",
+      "exitcode: 1 (execution failed)\n",
+      "Code output: \n",
+      ":15: DeprecationWarning: The 'get' method is deprecated, use 'results' instead\n",
+      "  for result in search.get():\n",
+      "Traceback (most recent call last):\n",
+      "  File \"\", line 17, in <module>\n",
+      "    if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "AttributeError: 'Result' object has no attribute 'submitted'\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mEngineer\u001b[0m (to chat_manager):\n",
+      "\n",
+      "I apologize for the confusion. It seems there was a mistake in the code. The `arxiv` package's `Result` object doesn't have a `submitted` attribute. Instead, we should use the `published` attribute to get the date when the paper was published.\n",
+      "\n",
+      "Here's the corrected code:\n",
+      "\n",
+      "```python\n",
+      "import arxiv\n",
+      "import datetime\n",
+      "\n",
+      "# Calculate the date one week ago\n",
+      "one_week_ago = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime('%Y%m%d%H%M%S')\n",
+      "\n",
+      "# Search for papers on LLM applications\n",
+      "search = arxiv.Search(\n",
+      "  query=\"LLM applications\",\n",
+      "  max_results=100,\n",
+      "  sort_by=arxiv.SortCriterion.Relevance,\n",
+      "  sort_order=arxiv.SortOrder.Descending\n",
+      ")\n",
+      "\n",
+      "print(\"| Title | Authors | Summary | URL |\")\n",
+      "print(\"|-------|---------|---------|-----|\")\n",
+      "for result in search.results():\n",
+      "  # Check if the paper was published in the last week\n",
+      "  if result.published.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"| {result.title} | {', '.join(author.name for author in result.authors)} | {result.summary} | {result.entry_id} |\")\n",
+      "```\n",
+      "\n",
+      "This script will print a markdown table of papers related to \"LLM applications\" that were published in the last week. The table includes the title, authors, summary, and URL of each paper.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mExecutor\u001b[0m (to chat_manager):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "| Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|\n",
+      "| Large Language Models as Data Preprocessors | Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada | Large Language Models (LLMs), typified by OpenAI's GPT series and Meta's\n",
+      "LLaMA variants, have marked a significant advancement in artificial\n",
+      "intelligence. Trained on vast amounts of text data, LLMs are capable of\n",
+      "understanding and generating human-like text across a diverse range of topics.\n",
+      "This study expands on the applications of LLMs, exploring their potential in\n",
+      "data preprocessing, a critical stage in data mining and analytics applications.\n",
+      "We delve into the applicability of state-of-the-art LLMs such as GPT-3.5,\n",
+      "GPT-4, and Vicuna-13B for error detection, data imputation, schema matching,\n",
+      "and entity matching tasks. Alongside showcasing the inherent capabilities of\n",
+      "LLMs, we highlight their limitations, particularly in terms of computational\n",
+      "expense and inefficiency. We propose an LLM-based framework for data\n",
+      "preprocessing, which integrates cutting-edge prompt engineering techniques,\n",
+      "coupled with traditional methods like contextualization and feature selection,\n",
+      "to improve the performance and efficiency of these models. The effectiveness of\n",
+      "LLMs in data preprocessing is evaluated through an experimental study spanning\n",
+      "12 datasets. GPT-4 emerged as a standout, achieving 100\\% accuracy or F1 score\n",
+      "on 4 datasets, suggesting LLMs' immense potential in these tasks. Despite\n",
+      "certain limitations, our study underscores the promise of LLMs in this domain\n",
+      "and anticipates future developments to overcome current hurdles. | http://arxiv.org/abs/2308.16361v1 |\n",
+      "| Large language models in medicine: the potentials and pitfalls | Jesutofunmi A. Omiye, Haiwen Gui, Shawheen J. Rezaei, James Zou, Roxana Daneshjou | Large language models (LLMs) have been applied to tasks in healthcare,\n",
+      "ranging from medical exam questions to responding to patient questions. With\n",
+      "increasing institutional partnerships between companies producing LLMs and\n",
+      "healthcare systems, real world clinical application is coming closer to\n",
+      "reality. As these models gain traction, it is essential for healthcare\n",
+      "practitioners to understand what LLMs are, their development, their current and\n",
+      "potential applications, and the associated pitfalls when utilized in medicine.\n",
+      "This review and accompanying tutorial aim to give an overview of these topics\n",
+      "to aid healthcare practitioners in understanding the rapidly changing landscape\n",
+      "of LLMs as applied to medicine. | http://arxiv.org/abs/2309.00087v1 |\n",
+      "| Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following | Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, Pheng-Ann Heng | We introduce Point-Bind, a 3D multi-modality model aligning point clouds with\n",
+      "2D image, language, audio, and video. Guided by ImageBind, we construct a joint\n",
+      "embedding space between 3D and multi-modalities, enabling many promising\n",
+      "applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D\n",
+      "open-world understanding. On top of this, we further present Point-LLM, the\n",
+      "first 3D large language model (LLM) following 3D multi-modal instructions. By\n",
+      "parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of\n",
+      "Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction\n",
+      "data, but exhibits superior 3D and multi-modal question-answering capacity. We\n",
+      "hope our work may cast a light on the community for extending 3D point clouds\n",
+      "to multi-modality applications. Code is available at\n",
+      "https://github.com/ZiyuGuo99/Point-Bind_Point-LLM. | http://arxiv.org/abs/2309.00615v1 |\n",
+      "| Where Would I Go Next? Large Language Models as Human Mobility Predictors | Xinglei Wang, Meng Fang, Zichao Zeng, Tao Cheng | Accurate human mobility prediction underpins many important applications\n",
+      "across a variety of domains, including epidemic modelling, transport planning,\n",
+      "and emergency responses. Due to the sparsity of mobility data and the\n",
+      "stochastic nature of people's daily activities, achieving precise predictions\n",
+      "of people's locations remains a challenge. While recently developed large\n",
+      "language models (LLMs) have demonstrated superior performance across numerous\n",
+      "language-related tasks, their applicability to human mobility studies remains\n",
+      "unexplored. Addressing this gap, this article delves into the potential of LLMs\n",
+      "for human mobility prediction tasks. We introduce a novel method, LLM-Mob,\n",
+      "which leverages the language understanding and reasoning capabilities of LLMs\n",
+      "for analysing human mobility data. We present concepts of historical stays and\n",
+      "context stays to capture both long-term and short-term dependencies in human\n",
+      "movement and enable time-aware prediction by using time information of the\n",
+      "prediction target. Additionally, we design context-inclusive prompts that\n",
+      "enable LLMs to generate more accurate predictions. Comprehensive evaluations of\n",
+      "our method reveal that LLM-Mob excels in providing accurate and interpretable\n",
+      "predictions, highlighting the untapped potential of LLMs in advancing human\n",
+      "mobility prediction techniques. We posit that our research marks a significant\n",
+      "paradigm shift in human mobility modelling, transitioning from building complex\n",
+      "domain-specific models to harnessing general-purpose LLMs that yield accurate\n",
+      "predictions through language instructions. The code for this work is available\n",
+      "at https://github.com/xlwang233/LLM-Mob. | http://arxiv.org/abs/2308.15197v1 |\n",
+      "| Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model | Kazuki Hori, Kanata Suzuki, Tetsuya Ogata | The application of the Large Language Model (LLM) to robot action planning\n",
+      "has been actively studied. The instructions given to the LLM by natural\n",
+      "language may include ambiguity and lack of information depending on the task\n",
+      "context. It is possible to adjust the output of LLM by making the instruction\n",
+      "input more detailed; however, the design cost is high. In this paper, we\n",
+      "propose the interactive robot action planning method that allows the LLM to\n",
+      "analyze and gather missing information by asking questions to humans. The\n",
+      "method can minimize the design cost of generating precise robot instructions.\n",
+      "We demonstrated the effectiveness of our method through concrete examples in\n",
+      "cooking tasks. However, our experiments also revealed challenges in robot\n",
+      "action planning with LLM, such as asking unimportant questions and assuming\n",
+      "crucial information without asking. Shedding light on these issues provides\n",
+      "valuable insights for future research on utilizing LLM for robotics. | http://arxiv.org/abs/2308.15684v1 |\n",
+      "| AskIt: Unified Programming Interface for Programming with Large Language Models | Katsumi Okuda, Saman Amarasinghe | In the evolving landscape of software development, Large Language Models\n",
+      "(LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating\n",
+      "adeptness across numerous tasks, from text summarization to code generation.\n",
+      "While these abilities open up novel avenues in software design and crafting,\n",
+      "their incorporation presents substantial challenges. Developers grapple with\n",
+      "decisions surrounding the direct embedding of LLMs within applications versus\n",
+      "employing them for code generation. Moreover, effective prompt design becomes a\n",
+      "critical concern, given the necessity of data extraction from natural language\n",
+      "outputs. To address these intricacies, this paper introduces AskIt, a\n",
+      "domain-specific language (DSL) specifically designed for LLMs. AskIt simplifies\n",
+      "LLM integration, offering type-guided output control, template-based function\n",
+      "definitions, and a unified interface that diminishes the distinction between\n",
+      "LLM-based code generation and application integration. Furthermore, through\n",
+      "Programming by Example (PBE), AskIt harnesses the power of few-shot learning at\n",
+      "the programming language level. Our evaluations underscore AskIt's potency.\n",
+      "Across 50 tasks, AskIt generated concise prompts for the given tasks, achieving\n",
+      "a 16.14% reduction in prompt length relative to benchmarks. Additionally, by\n",
+      "enabling the transition from direct LLM application usage to function\n",
+      "generation, AskIt achieved significant speedups, as observed in our GSM8K\n",
+      "benchmark experiments. Through these advancements, AskIt streamlines the\n",
+      "integration of LLMs in software development, offering a more efficient,\n",
+      "versatile approach for leveraging emergent abilities. The implementations of\n",
+      "AskIt in TypeScript and Python are available at\n",
+      "https://github.com/katsumiok/ts-askit and https://github.com/katsumiok/pyaskit,\n",
+      "respectively. | http://arxiv.org/abs/2308.15645v1 |\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mScientist\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Here are the papers related to LLM applications published in the last week:\n",
+      "\n",
+      "| Domain | Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|-----|\n",
+      "| Data Preprocessing | [Large Language Models as Data Preprocessors](http://arxiv.org/abs/2308.16361v1) | Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada | This study expands on the applications of LLMs, exploring their potential in data preprocessing, a critical stage in data mining and analytics applications. | [Link](http://arxiv.org/abs/2308.16361v1) |\n",
+      "| Medicine | [Large language models in medicine: the potentials and pitfalls](http://arxiv.org/abs/2309.00087v1) | Jesutofunmi A. Omiye, Haiwen Gui, Shawheen J. Rezaei, James Zou, Roxana Daneshjou | This review and accompanying tutorial aim to give an overview of these topics to aid healthcare practitioners in understanding the rapidly changing landscape of LLMs as applied to medicine. | [Link](http://arxiv.org/abs/2309.00087v1) |\n",
+      "| 3D Understanding, Generation, and Instruction Following | [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](http://arxiv.org/abs/2309.00615v1) | Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, Pheng-Ann Heng | We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. | [Link](http://arxiv.org/abs/2309.00615v1) |\n",
+      "| Human Mobility Prediction | [Where Would I Go Next? Large Language Models as Human Mobility Predictors](http://arxiv.org/abs/2308.15197v1) | Xinglei Wang, Meng Fang, Zichao Zeng, Tao Cheng | This article delves into the potential of LLMs for human mobility prediction tasks. | [Link](http://arxiv.org/abs/2308.15197v1) |\n",
+      "| Robotics | [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](http://arxiv.org/abs/2308.15684v1) | Kazuki Hori, Kanata Suzuki, Tetsuya Ogata | In this paper, we propose the interactive robot action planning method that allows the LLM to analyze and gather missing information by asking questions to humans. | [Link](http://arxiv.org/abs/2308.15684v1) |\n",
+      "| Software Development | [AskIt: Unified Programming Interface for Programming with Large Language Models](http://arxiv.org/abs/2308.15645v1) | Katsumi Okuda, Saman Amarasinghe | This paper introduces AskIt, a domain-specific language (DSL) specifically designed for LLMs. | [Link](http://arxiv.org/abs/2308.15645v1) |\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCritic\u001b[0m (to chat_manager):\n",
+      "\n",
+      "The scientist has done a good job categorizing the papers into different domains and providing a summary for each. The markdown table is correctly formatted and includes the source URL for each paper, which allows for verification of the information. The domains identified are diverse, indicating a broad range of applications for Large Language Models (LLMs). This review and categorization should provide a useful resource for anyone interested in the recent applications of LLMs.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "user_proxy.initiate_chat(\n",
+    "    manager,\n",
+    "    message=\"\"\"\n",
+    "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Group Chat without Critic for Comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mPlanner\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Plan:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link.\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "groupchat_nocritic = autogen.GroupChat(agents=[user_proxy, engineer, scientist, planner, executor], messages=[], max_round=50)\n",
+    "for agent in groupchat.agents:\n",
+    "    agent.reset()\n",
+    "manager_nocritic = autogen.GroupChatManager(groupchat=groupchat_nocritic, llm_config=gpt4_config)\n",
+    "user_proxy.initiate_chat(\n",
+    "    manager_nocritic,\n",
+    "    message=\"\"\"\n",
+    "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "flaml",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_human_feedback.ipynb

@@ -0,0 +1,801 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_human_feedback.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to solve a challenging math problem with human feedback. Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. By setting `human_input_mode` properly, the `UserProxyAgent` can also prompt the user for feedback to `AssistantAgent`. For example, when `human_input_mode` is set to \"ALWAYS\", the `UserProxyAgent` will always prompt the user for feedback. When user feedback is provided, the `UserProxyAgent` will directly pass the feedback to `AssistantAgent`. When no user feedback is provided, the `UserProxyAgent` will execute the code written by `AssistantAgent` and return the execution results (success or failure and corresponding outputs) to `AssistantAgent`.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
+    "```bash\n",
+    "pip install pyautogen\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install pyautogen~=0.1.0"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\".\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your first Azure OpenAI API key here>',\n",
+    "        'api_base': '<your first Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your second Azure OpenAI API key here>',\n",
+    "        'api_base': '<your second Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # another Azure OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-3.5-turbo\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your first Azure OpenAI API key here>',\n",
+    "        'api_base': '<your first Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-3.5-turbo\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your second Azure OpenAI API key here>',\n",
+    "        'api_base': '<your second Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # another Azure OpenAI API endpoint for gpt-3.5-turbo\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents\n",
+    "\n",
+    "We construct the assistant agent and the user proxy agent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create an AssistantAgent instance named \"assistant\"\n",
+    "assistant = autogen.AssistantAgent(\n",
+    "    name=\"assistant\",\n",
+    "    llm_config={\n",
+    "        \"seed\": 41,\n",
+    "        \"config_list\": config_list,\n",
+    "    }\n",
+    ")\n",
+    "# create a UserProxyAgent instance named \"user_proxy\"\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"user_proxy\",\n",
+    "    human_input_mode=\"ALWAYS\",\n",
+    "    is_termination_msg=lambda x: x.get(\"content\", \"\").rstrip().endswith(\"TERMINATE\"),\n",
+    ")\n",
+    "\n",
+    "# the purpose of the following line is to log the conversation history\n",
+    "autogen.ChatCompletion.start_logging()\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Perform a task\n",
+    "\n",
+    "We invoke the `initiate_chat()` method of the user proxy agent to start the conversation. When you run the cell below, you will be prompted to provide feedback after receving a message from the assistant agent. If you don't provide any feedback (by pressing Enter directly), the user proxy agent will try to execute the code suggested by the assistant agent on behalf of you, or terminate if the assistant agent sends a \"TERMINATE\" signal in the end of the message."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "\n",
+      "Find $a + b + c$, given that $x+y \\neq -1$ and \n",
+      "\\begin{align}\n",
+      "\tax + by + c & = x + 7,\\\n",
+      "\ta + bx + cy & = 2x + 6y,\\\n",
+      "\tay + b + cx & = 4x + y.\n",
+      "\\end{align}.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "To solve the system of linear equations, I will use NumPy library in Python to calculate the coefficients and constants and then use the linalg.solve() function to find the values of a, b, and c. Finally, I will add them up to find $a + b + c$.\n",
+      "\n",
+      "Step 1: Define the coefficients and constants\n",
+      "\n",
+      "Step 2: Solve the system of equations using NumPy\n",
+      "\n",
+      "Step 3: Calculate the sum of a, b, and c\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "use sympy\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Sure, I will use Sympy to solve the system of linear equations.\n",
+      "\n",
+      "Step 1: Import Sympy library and define the symbols\n",
+      "``` python\n",
+      "from sympy import symbols, Eq, solve\n",
+      "\n",
+      "a, b, c, x, y = symbols('a b c x y')\n",
+      "```\n",
+      "\n",
+      "Step 2: Define the equations\n",
+      "``` python\n",
+      "eq1 = Eq(a * x + b * y + c, x + 7)\n",
+      "eq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\n",
+      "eq3 = Eq(a * y + b + c * x, 4 * x + y)\n",
+      "```\n",
+      "\n",
+      "Step 3: Solve the system of equations\n",
+      "``` python\n",
+      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "```\n",
+      "\n",
+      "Step 4: Calculate the sum of a, b, and c\n",
+      "``` python\n",
+      "a_value, b_value, c_value = solutions[a], solutions[b], solutions[c]\n",
+      "sum_abc = a_value + b_value + c_value\n",
+      "```\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 1 (execution failed)\n",
+      "Code output: \n",
+      "  File \"\", line 2\n",
+      "    Step 2: Define the equations\n",
+      "         ^\n",
+      "SyntaxError: invalid syntax\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Apologies for the confusion. I should have provided the code in a single block without the step-by-step comments. Here is the code to solve the system of linear equations using Sympy:\n",
+      "\n",
+      "```python\n",
+      "from sympy import symbols, Eq, solve\n",
+      "\n",
+      "a, b, c, x, y = symbols('a b c x y')\n",
+      "\n",
+      "eq1 = Eq(a * x + b * y + c, x + 7)\n",
+      "eq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\n",
+      "eq3 = Eq(a * y + b + c * x, 4 * x + y)\n",
+      "\n",
+      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "\n",
+      "a_value, b_value, c_value = solutions[a], solutions[b], solutions[c]\n",
+      "sum_abc = a_value + b_value + c_value\n",
+      "print(sum_abc)\n",
+      "```\n",
+      "\n",
+      "Please run the provided code to find the sum of a, b, and c.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "(2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "I apologize for the confusion. The code provided the sum of a, b, and c in a symbolic representation. To get an answer, we need to put some constraints on x and y for which x + y is not equal to -1.\n",
+      "\n",
+      "We can choose any values for x and y that don't satisfy x + y = -1, and then substitute them into the equation to find the corresponding sum of a, b, and c. Let's use x = 1 and y = 0, which satisfy the given condition.\n",
+      "\n",
+      "Here is the updated code with the chosen values for x and y:\n",
+      "\n",
+      "```python\n",
+      "from sympy import symbols, Eq, solve\n",
+      "\n",
+      "a, b, c, x, y = symbols('a b c x y')\n",
+      "\n",
+      "eq1 = Eq(a * x + b * y + c, x + 7)\n",
+      "eq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\n",
+      "eq3 = Eq(a * y + b + c * x, 4 * x + y)\n",
+      "\n",
+      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "\n",
+      "chosen_x = 1\n",
+      "chosen_y = 0\n",
+      "\n",
+      "a_value, b_value, c_value = solutions[a].subs(x, chosen_x).subs(y, chosen_y), solutions[b].subs(x, chosen_x).subs(y, chosen_y), solutions[c].subs(x, chosen_x).subs(y, chosen_y)\n",
+      "sum_abc = a_value + b_value + c_value\n",
+      "print(sum_abc)\n",
+      "```\n",
+      "\n",
+      "Please run the provided code again to find the sum of a, b, and c.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "7\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Great! Now we have the sum of a, b, and c. The answer is:\n",
+      "\n",
+      "$$a + b + c = 7$$\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "math_problem_to_solve = \"\"\"\n",
+    "Find $a + b + c$, given that $x+y \\\\neq -1$ and \n",
+    "\\\\begin{align}\n",
+    "\tax + by + c & = x + 7,\\\\\n",
+    "\ta + bx + cy & = 2x + 6y,\\\\\n",
+    "\tay + b + cx & = 4x + y.\n",
+    "\\\\end{align}.\n",
+    "\"\"\"\n",
+    "\n",
+    "# the assistant receives a message from the user, which contains the task description\n",
+    "user_proxy.initiate_chat(assistant, message=math_problem_to_solve)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analyze the conversation\n",
+    "\n",
+    "The human user can provide feedback at each step. When the human user didn't provide feedback, the code was executed. The executed results and error messages are returned to the assistant, and the assistant was able to modify the code based on the feedback. In the end, the task is complete and a \"TERMINATE\" signal is sent from the assistant. The user skipped feedback in the end and the conversation is finished.\n",
+    "\n",
+    "After the conversation is finished, we can save the log of the conversation between the two agents. The log can be accessed from `autogen.ChatCompletion.logged_history`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'[{\"content\": \"You are a helpful AI assistant.\\\\nSolve tasks using your coding and language skills.\\\\nIn the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.\\\\n    1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.\\\\n    2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.\\\\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\\\\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can\\'t modify your code. So do not suggest incomplete code which requires users to modify. Don\\'t use a code block if it\\'s not intended to be executed by the user.\\\\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don\\'t include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use \\'print\\' function for the output when relevant. Check the execution result returned by the user.\\\\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can\\'t be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\\\\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\\\\nReply \\\\\"TERMINATE\\\\\" in the end when everything is done.\\\\n    \", \"role\": \"system\"}, {\"content\": \"\\\\nFind $a + b + c$, given that $x+y \\\\\\\\neq -1$ and \\\\n\\\\\\\\begin{align}\\\\n\\\\tax + by + c & = x + 7,\\\\\\\\\\\\n\\\\ta + bx + cy & = 2x + 6y,\\\\\\\\\\\\n\\\\tay + b + cx & = 4x + y.\\\\n\\\\\\\\end{align}.\\\\n\", \"role\": \"user\"}, {\"content\": \"To solve the system of linear equations, I will use NumPy library in Python to calculate the coefficients and constants and then use the linalg.solve() function to find the values of a, b, and c. Finally, I will add them up to find $a + b + c$.\\\\n\\\\nStep 1: Define the coefficients and constants\\\\n\\\\nStep 2: Solve the system of equations using NumPy\\\\n\\\\nStep 3: Calculate the sum of a, b, and c\", \"role\": \"assistant\"}, {\"content\": \"use sympy\", \"role\": \"user\"}, {\"content\": \"Sure, I will use Sympy to solve the system of linear equations.\\\\n\\\\nStep 1: Import Sympy library and define the symbols\\\\n``` python\\\\nfrom sympy import symbols, Eq, solve\\\\n\\\\na, b, c, x, y = symbols(\\'a b c x y\\')\\\\n```\\\\n\\\\nStep 2: Define the equations\\\\n``` python\\\\neq1 = Eq(a * x + b * y + c, x + 7)\\\\neq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\\\\neq3 = Eq(a * y + b + c * x, 4 * x + y)\\\\n```\\\\n\\\\nStep 3: Solve the system of equations\\\\n``` python\\\\nsolutions = solve((eq1, eq2, eq3), (a, b, c))\\\\n```\\\\n\\\\nStep 4: Calculate the sum of a, b, and c\\\\n``` python\\\\na_value, b_value, c_value = solutions[a], solutions[b], solutions[c]\\\\nsum_abc = a_value + b_value + c_value\\\\n```\\\\n\\\\n\", \"role\": \"assistant\"}, {\"content\": \"exitcode: 1 (execution failed)\\\\nCode output: \\\\n  File \\\\\"\\\\\", line 2\\\\n    Step 2: Define the equations\\\\n         ^\\\\nSyntaxError: invalid syntax\\\\n\", \"role\": \"user\"}, {\"content\": \"Apologies for the confusion. I should have provided the code in a single block without the step-by-step comments. Here is the code to solve the system of linear equations using Sympy:\\\\n\\\\n```python\\\\nfrom sympy import symbols, Eq, solve\\\\n\\\\na, b, c, x, y = symbols(\\'a b c x y\\')\\\\n\\\\neq1 = Eq(a * x + b * y + c, x + 7)\\\\neq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\\\\neq3 = Eq(a * y + b + c * x, 4 * x + y)\\\\n\\\\nsolutions = solve((eq1, eq2, eq3), (a, b, c))\\\\n\\\\na_value, b_value, c_value = solutions[a], solutions[b], solutions[c]\\\\nsum_abc = a_value + b_value + c_value\\\\nprint(sum_abc)\\\\n```\\\\n\\\\nPlease run the provided code to find the sum of a, b, and c.\", \"role\": \"assistant\"}, {\"content\": \"exitcode: 0 (execution succeeded)\\\\nCode output: \\\\n(2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\\\\n\", \"role\": \"user\"}, {\"content\": \"I apologize for the confusion. The code provided the sum of a, b, and c in a symbolic representation. To get an answer, we need to put some constraints on x and y for which x + y is not equal to -1.\\\\n\\\\nWe can choose any values for x and y that don\\'t satisfy x + y = -1, and then substitute them into the equation to find the corresponding sum of a, b, and c. Let\\'s use x = 1 and y = 0, which satisfy the given condition.\\\\n\\\\nHere is the updated code with the chosen values for x and y:\\\\n\\\\n```python\\\\nfrom sympy import symbols, Eq, solve\\\\n\\\\na, b, c, x, y = symbols(\\'a b c x y\\')\\\\n\\\\neq1 = Eq(a * x + b * y + c, x + 7)\\\\neq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\\\\neq3 = Eq(a * y + b + c * x, 4 * x + y)\\\\n\\\\nsolutions = solve((eq1, eq2, eq3), (a, b, c))\\\\n\\\\nchosen_x = 1\\\\nchosen_y = 0\\\\n\\\\na_value, b_value, c_value = solutions[a].subs(x, chosen_x).subs(y, chosen_y), solutions[b].subs(x, chosen_x).subs(y, chosen_y), solutions[c].subs(x, chosen_x).subs(y, chosen_y)\\\\nsum_abc = a_value + b_value + c_value\\\\nprint(sum_abc)\\\\n```\\\\n\\\\nPlease run the provided code again to find the sum of a, b, and c.\", \"role\": \"assistant\"}, {\"content\": \"exitcode: 0 (execution succeeded)\\\\nCode output: \\\\n7\\\\n\", \"role\": \"user\"}, {\"content\": \"Great! Now we have the sum of a, b, and c. The answer is:\\\\n\\\\n$$a + b + c = 7$$\\\\n\\\\nTERMINATE\", \"role\": \"assistant\"}]': {'created_at': [0, 1, 2, 3, 4], 'cost': [0.022019999999999998, 0.03305999999999999, 0.04019999999999999, 0.058589999999999996, 0.050969999999999994]}}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(autogen.ChatCompletion.logged_history)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "json.dump(autogen.ChatCompletion.logged_history, open(\"conversations.json\", \"w\"), indent=2)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_planning.ipynb

@@ -0,0 +1,818 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_planning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate how to use multiple agents to work together and accomplish a task which requires finding info from the web and coding. `AssistantAgent` is an LLM-based agent that can write and debug Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We further create a planning agent for the assistant agent to consult. The planning agent is a variation of the LLM-based `AssistantAgent` with a different system message.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n",
+    "```bash\n",
+    "pip install pyautogen docker\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install pyautogen~=0.1.0 docker"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "* The [`config_list_openai_aoai`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_openai_aoai) function tries to create a list of configurations using Azure OpenAI endpoints and OpenAI endpoints. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
+    "  - OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
+    "  - Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
+    "  - Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
+    "* The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file. It first looks for an environment variable with a specified name. The value of the environment variable needs to be a valid json string. If that variable is not found, it then looks for a json file with the same name. It filters the configs by filter_dict.\n",
+    "\n",
+    "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base. If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-07-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-07-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-4-32k\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file.\n",
+    "\n",
+    "## Construct Agents\n",
+    "\n",
+    "We construct the planning agent named \"planner\" and a user proxy agent for the planner named \"planner_user\". We specify `human_input_mode` as \"NEVER\" in the user proxy agent, which will never ask for human feedback. We define `ask_planner` function to send a message to planner and return the suggestion from the planner."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "planner = autogen.AssistantAgent(\n",
+    "    name=\"planner\",\n",
+    "    llm_config={\"config_list\": config_list},\n",
+    "    # the default system message of the AssistantAgent is overwritten here\n",
+    "    system_message=\"You are a helpful AI assistant. You suggest coding and reasoning steps for another AI assistant to accomplish a task. Do not suggest concrete code. For any action beyond writing code or reasoning, convert it to a step which can be implemented by writing code. For example, the action of browsing the web can be implemented by writing code which reads and prints the content of a web page. Finally, inspect the execution result. If the plan is not good, suggest a better plan. If the execution is wrong, analyze the error and suggest a fix.\"\n",
+    ")\n",
+    "planner_user = autogen.UserProxyAgent(\n",
+    "    name=\"planner_user\",\n",
+    "    max_consecutive_auto_reply=0,  # terminate without auto-reply\n",
+    "    human_input_mode=\"NEVER\",\n",
+    ")\n",
+    "\n",
+    "def ask_planner(message):\n",
+    "    planner_user.initiate_chat(planner, message=message)\n",
+    "    # return the last message received from the planner\n",
+    "    return planner_user.last_message()[\"content\"]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We construct the assistant agent and the user proxy agent. We specify `human_input_mode` as \"TERMINATE\" in the user proxy agent, which will ask for feedback when it receives a \"TERMINATE\" signal from the assistant agent. We set the `functions` in `AssistantAgent` and `function_map` in `UserProxyAgent` to use the created `ask_planner` function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create an AssistantAgent instance named \"assistant\"\n",
+    "assistant = autogen.AssistantAgent(\n",
+    "    name=\"assistant\",\n",
+    "    llm_config={\n",
+    "        \"temperature\": 0,\n",
+    "        \"request_timeout\": 600,\n",
+    "        \"seed\": 42,\n",
+    "        \"model\": \"gpt-4-0613\",\n",
+    "        \"config_list\": autogen.config_list_openai_aoai(exclude=\"aoai\"),\n",
+    "        \"functions\": [\n",
+    "            {\n",
+    "                \"name\": \"ask_planner\",\n",
+    "                \"description\": \"ask planner to: 1. get a plan for finishing a task, 2. verify the execution result of the plan and potentially suggest new plan.\",\n",
+    "                \"parameters\": {\n",
+    "                    \"type\": \"object\",\n",
+    "                    \"properties\": {\n",
+    "                        \"message\": {\n",
+    "                            \"type\": \"string\",\n",
+    "                            \"description\": \"question to ask planner. Make sure the question include enough context, such as the code and the execution result. The planner does not know the conversation between you and the user, unless you share the conversation with the planner.\",\n",
+    "                        },\n",
+    "                    },\n",
+    "                    \"required\": [\"message\"],\n",
+    "                },\n",
+    "            },\n",
+    "        ],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# create a UserProxyAgent instance named \"user_proxy\"\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"user_proxy\",\n",
+    "    human_input_mode=\"TERMINATE\",\n",
+    "    max_consecutive_auto_reply=10,\n",
+    "    # is_termination_msg=lambda x: \"content\" in x and x[\"content\"] is not None and x[\"content\"].rstrip().endswith(\"TERMINATE\"),\n",
+    "    code_execution_config={\"work_dir\": \"planning\"},\n",
+    "    function_map={\"ask_planner\": ask_planner},\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Perform a task\n",
+    "\n",
+    "We invoke the `initiate_chat()` method of the user proxy agent to start the conversation. When you run the cell below, you will be prompted to provide feedback after the assistant agent sends a \"TERMINATE\" signal in the end of the message. If you don't provide any feedback (by pressing Enter directly), the conversation will finish. Before the \"TERMINATE\" signal, the user proxy agent will try to execute the code suggested by the assistant agent on behalf of the user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "Suggest a fix to an open good first issue of flaml\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "To suggest a fix to an open good first issue of FLAML, we first need to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository. We can do this using the GitHub API.\n",
+      "\n",
+      "Here is a Python script that uses the requests library to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository.\n",
+      "\n",
+      "```python\n",
+      "# filename: fetch_issues.py\n",
+      "\n",
+      "import requests\n",
+      "import json\n",
+      "\n",
+      "def fetch_issues():\n",
+      "    url = \"https://api.github.com/repos/microsoft/FLAML/issues\"\n",
+      "    params = {\n",
+      "        \"state\": \"open\",\n",
+      "        \"labels\": \"good first issue\"\n",
+      "    }\n",
+      "    response = requests.get(url, params=params)\n",
+      "    issues = response.json()\n",
+      "    for issue in issues:\n",
+      "        print(f\"Issue ID: {issue['id']}, Title: {issue['title']}, URL: {issue['html_url']}\")\n",
+      "\n",
+      "fetch_issues()\n",
+      "```\n",
+      "\n",
+      "Please run this script to fetch the list of open issues. After that, we can select one issue and suggest a fix for it.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "Issue ID: 1809297895, Title: Moving function execution out of UserProxyAgent to be an openai util, URL: https://github.com/microsoft/FLAML/issues/1135\n",
+      "Issue ID: 1799114476, Title: use_label_encoder warning with xgboost, URL: https://github.com/microsoft/FLAML/issues/1120\n",
+      "Issue ID: 1705274482, Title: Use appropriate wait time for retry based on the error message. , URL: https://github.com/microsoft/FLAML/issues/1034\n",
+      "Issue ID: 1702580697, Title: Issues with Adding Custom APIs in Auto Generation, URL: https://github.com/microsoft/FLAML/issues/1029\n",
+      "Issue ID: 1658981020, Title: Running flaml[tune] using \"-O\" flag for python interpreter (optimization - disables assertions) crashes, URL: https://github.com/microsoft/FLAML/issues/981\n",
+      "Issue ID: 1560969891, Title: Conditional parameter flow2 crash, URL: https://github.com/microsoft/FLAML/issues/903\n",
+      "Issue ID: 1538549388, Title: indentation space, URL: https://github.com/microsoft/FLAML/issues/884\n",
+      "Issue ID: 1531028010, Title: Check if openml version is required, URL: https://github.com/microsoft/FLAML/issues/882\n",
+      "Issue ID: 1470354491, Title: Adjust the indent, URL: https://github.com/microsoft/FLAML/issues/834\n",
+      "Issue ID: 1456950742, Title: pip install flaml FAIL, URL: https://github.com/microsoft/FLAML/issues/821\n",
+      "Issue ID: 1441047067, Title: Isolate the ensemble part and expose it to users, URL: https://github.com/microsoft/FLAML/issues/807\n",
+      "Issue ID: 1440171793, Title: how to pass categorical features names or indices to learner, URL: https://github.com/microsoft/FLAML/issues/805\n",
+      "Issue ID: 1429945686, Title: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?, URL: https://github.com/microsoft/FLAML/issues/785\n",
+      "Issue ID: 1408240042, Title: Add an announcement of the discord channel, URL: https://github.com/microsoft/FLAML/issues/764\n",
+      "Issue ID: 1396515109, Title: Documentation about small budget, URL: https://github.com/microsoft/FLAML/issues/748\n",
+      "Issue ID: 1378268096, Title: Make zero-shot automl more discoverable, URL: https://github.com/microsoft/FLAML/issues/737\n",
+      "Issue ID: 1189515901, Title: New HCrystalBall release, URL: https://github.com/microsoft/FLAML/issues/509\n",
+      "Issue ID: 1114253143, Title: samples about conversion to ONNX, URL: https://github.com/microsoft/FLAML/issues/429\n",
+      "Issue ID: 1107488969, Title: support anomaly detection, URL: https://github.com/microsoft/FLAML/issues/413\n",
+      "Issue ID: 1061332179, Title: CatBoost Fails with Keyword 'groups', URL: https://github.com/microsoft/FLAML/issues/304\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "\u001b[32m***** Suggested function Call: ask_planner *****\u001b[0m\n",
+      "Arguments: \n",
+      "{\n",
+      "\"message\": \"We have fetched a list of open issues labeled as 'good first issue' from the FLAML GitHub repository. Now, we need to select one issue and suggest a fix for it. Could you please provide a plan for this?\"\n",
+      "}\n",
+      "\u001b[32m************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[35m\n",
+      ">>>>>>>> EXECUTING FUNCTION ask_planner...\u001b[0m\n",
+      "\u001b[33mplanner_user\u001b[0m (to planner):\n",
+      "\n",
+      "We have fetched a list of open issues labeled as 'good first issue' from the FLAML GitHub repository. Now, we need to select one issue and suggest a fix for it. Could you please provide a plan for this?\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mplanner\u001b[0m (to planner_user):\n",
+      "\n",
+      "Sure, here's a plan for selecting one issue from the list and suggesting a fix for it:\n",
+      "\n",
+      "1. Import the fetched list of open issues labeled as 'good first issue' from the FLAML GitHub repository into your AI assistant. \n",
+      "2. Examine the list for common issue attributes like 'title', 'description', 'labels', 'issue number', 'created at', and 'updated at'. \n",
+      "3. To select a suitable issue for fixing, apply a selection criteria based on your preferences, such as prioritizing by the 'created at' attribute in descending order to pick the most recent issue, or filtering by a specific label in addition to 'good first issue'. Write code to filter and sort the issues accordingly.\n",
+      "4. Inspect the execution result. If the selection criteria are not applied correctly, modify the code to fix any errors.\n",
+      "5. Once the issue is selected, read the issue's title, description, and any linked resources or documents to understand the problem to be solved.\n",
+      "6. Break down the issue into smaller tasks that can be addressed by writing code, and create a step-by-step plan.\n",
+      "\n",
+      "For instance, the following could be smaller tasks to address the selected issue:\n",
+      "  a. Understand the issue's background and requirements.\n",
+      "  b. Write clear and concise instructions to reproduce the issue.\n",
+      "  c. Analyze existing code or tests related to the issue.\n",
+      "  d. Devise a solution to fix the issue.\n",
+      "  e. Implement the solution in separate code pieces.\n",
+      "  f. Verify that the solution addresses the issue.\n",
+      "  g. Write unit tests to ensure the solution is robust and handles edge cases.\n",
+      "\n",
+      "7. Inspect the execution result. If the issue is misunderstood or the tasks' breakdown is incorrect, revise the understanding of the issue and modify the tasks accordingly.\n",
+      "8. With the defined tasks and step-by-step plan, work on each task, and test the implemented code to ensure the issue is solved.\n",
+      "9. If any issues arise during the task execution, analyze the errors and adjust the plan or code accordingly.\n",
+      "10. Once the issue is fixed, prepare a pull request on GitHub, mentioning the issue number and giving a brief description of the solution in the merge request.\n",
+      "\n",
+      "Remember that this is meant to be a general plan, and the specific tasks may vary depending on the selected issue. Adjust the plan as needed, based on the selected issue's requirements and your problem-solving approach.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"ask_planner\" *****\u001b[0m\n",
+      "Sure, here's a plan for selecting one issue from the list and suggesting a fix for it:\n",
+      "\n",
+      "1. Import the fetched list of open issues labeled as 'good first issue' from the FLAML GitHub repository into your AI assistant. \n",
+      "2. Examine the list for common issue attributes like 'title', 'description', 'labels', 'issue number', 'created at', and 'updated at'. \n",
+      "3. To select a suitable issue for fixing, apply a selection criteria based on your preferences, such as prioritizing by the 'created at' attribute in descending order to pick the most recent issue, or filtering by a specific label in addition to 'good first issue'. Write code to filter and sort the issues accordingly.\n",
+      "4. Inspect the execution result. If the selection criteria are not applied correctly, modify the code to fix any errors.\n",
+      "5. Once the issue is selected, read the issue's title, description, and any linked resources or documents to understand the problem to be solved.\n",
+      "6. Break down the issue into smaller tasks that can be addressed by writing code, and create a step-by-step plan.\n",
+      "\n",
+      "For instance, the following could be smaller tasks to address the selected issue:\n",
+      "  a. Understand the issue's background and requirements.\n",
+      "  b. Write clear and concise instructions to reproduce the issue.\n",
+      "  c. Analyze existing code or tests related to the issue.\n",
+      "  d. Devise a solution to fix the issue.\n",
+      "  e. Implement the solution in separate code pieces.\n",
+      "  f. Verify that the solution addresses the issue.\n",
+      "  g. Write unit tests to ensure the solution is robust and handles edge cases.\n",
+      "\n",
+      "7. Inspect the execution result. If the issue is misunderstood or the tasks' breakdown is incorrect, revise the understanding of the issue and modify the tasks accordingly.\n",
+      "8. With the defined tasks and step-by-step plan, work on each task, and test the implemented code to ensure the issue is solved.\n",
+      "9. If any issues arise during the task execution, analyze the errors and adjust the plan or code accordingly.\n",
+      "10. Once the issue is fixed, prepare a pull request on GitHub, mentioning the issue number and giving a brief description of the solution in the merge request.\n",
+      "\n",
+      "Remember that this is meant to be a general plan, and the specific tasks may vary depending on the selected issue. Adjust the plan as needed, based on the selected issue's requirements and your problem-solving approach.\n",
+      "\u001b[32m********************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Based on the plan, let's select the most recent issue from the list. The most recent issue is \"Moving function execution out of UserProxyAgent to be an openai util\" with the URL: https://github.com/microsoft/FLAML/issues/1135\n",
+      "\n",
+      "To suggest a fix for this issue, we need to understand the problem first. Please visit the issue URL and read the issue's description and any comments to understand the problem.\n",
+      "\n",
+      "Once you understand the problem, here are the steps you can follow to suggest a fix:\n",
+      "\n",
+      "1. Understand the issue's background and requirements: Read the issue description and any linked resources or documents to understand what needs to be done.\n",
+      "\n",
+      "2. Write clear and concise instructions to reproduce the issue: Based on the issue description, write down the steps to reproduce the issue. This will help you understand the problem better and will be useful when you test your fix.\n",
+      "\n",
+      "3. Analyze existing code or tests related to the issue: Look at the code in the FLAML repository that is related to the issue. Try to understand how it works and how it might be causing the issue.\n",
+      "\n",
+      "4. Devise a solution to fix the issue: Based on your understanding of the problem and the code, come up with a solution to fix the issue. This might involve writing new code or modifying existing code.\n",
+      "\n",
+      "5. Implement the solution in separate code pieces: Write the code for your solution. Make sure to break it down into small, manageable pieces. This will make it easier to understand and test.\n",
+      "\n",
+      "6. Verify that the solution addresses the issue: Test your code to make sure it solves the issue. You can do this by following the instructions you wrote to reproduce the issue.\n",
+      "\n",
+      "7. Write unit tests to ensure the solution is robust and handles edge cases: In addition to testing your solution manually, write unit tests to automatically test your code. This will help ensure that your solution works in all cases and that future changes to the code won't break your solution.\n",
+      "\n",
+      "8. Prepare a pull request on GitHub: Once you're confident that your solution works, prepare a pull request on GitHub. In the pull request description, mention the issue number and give a brief description of your solution.\n",
+      "\n",
+      "Please note that this is a general plan and the specific steps may vary depending on the issue. Adjust the plan as needed based on the issue's requirements and your problem-solving approach. \n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# the assistant receives a message from the user, which contains the task description\n",
+    "user_proxy.initiate_chat(\n",
+    "    assistant,\n",
+    "    message=\"\"\"Suggest a fix to an open good first issue of flaml\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When the assistant needs to consult the planner, it suggests a function call to `ask_planner`. When this happens, a line like the following will be displayed:\n",
+    "\n",
+    "***** Suggested function Call: ask_planner *****\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_stream.ipynb

@@ -0,0 +1,781 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_stream.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Interactive LLM Agent Dealing with Data Stream\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate how to use customized agents to continuously acquires news from the web and ask for investment suggestions.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
+    "```bash\n",
+    "pip install pyautogen\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install pyautogen~=0.1.0"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). Only the models with matching names are kept in the list based on the filter condition.\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your first Azure OpenAI API key here>',\n",
+    "        'api_base': '<your first Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-4',\n",
+    "        'api_key': '<your second Azure OpenAI API key here>',\n",
+    "        'api_base': '<your second Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # another Azure OpenAI API endpoint for gpt-4\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },  # OpenAI API endpoint for gpt-3.5-turbo\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your first Azure OpenAI API key here>',\n",
+    "        'api_base': '<your first Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # Azure OpenAI API endpoint for gpt-3.5-turbo\n",
+    "    {\n",
+    "        'model': 'gpt-3.5-turbo',\n",
+    "        'api_key': '<your second Azure OpenAI API key here>',\n",
+    "        'api_base': '<your second Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # another Azure OpenAI API endpoint for gpt-3.5-turbo\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example Task: Investment suggestion with realtime data\n",
+    "\n",
+    "We consider a scenario where news data are streamed from a source, and we use an assistant agent to continually provide investment suggestions based on the data.\n",
+    "\n",
+    "First, we use the following code to simulate the data stream process."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "\n",
+    "def get_market_news(ind, ind_upper):\n",
+    "    import requests\n",
+    "    import json\n",
+    "    # replace the \"demo\" apikey below with your own key from https://www.alphavantage.co/support/#api-key\n",
+    "    # url = 'https://www.alphavantage.co/query?function=NEWS_SENTIMENT&tickers=AAPL&sort=LATEST&limit=5&apikey=demo'\n",
+    "    # r = requests.get(url)\n",
+    "    # data = r.json()\n",
+    "    # with open('market_news_local.json', 'r') as file:\n",
+    "    #     # Load JSON data from file\n",
+    "    #     data = json.load(file)\n",
+    "    data = {\n",
+    "        \"feed\": [\n",
+    "            {\n",
+    "                \"title\": \"Palantir CEO Says Our Generation's Atomic Bomb Could Be AI Weapon - And Arrive Sooner Than You Think - Palantir Technologies  ( NYSE:PLTR ) \",\n",
+    "                \"summary\": \"Christopher Nolan's blockbuster movie \\\"Oppenheimer\\\" has reignited the public discourse surrounding the United States' use of an atomic bomb on Japan at the end of World War II.\",\n",
+    "                \"overall_sentiment_score\": 0.009687,\n",
+    "            },\n",
+    "            {\n",
+    "                \"title\": '3 \"Hedge Fund Hotels\" Pulling into Support',\n",
+    "                \"summary\": \"Institutional quality stocks have several benefits including high-liquidity, low beta, and a long runway. Strategist Andrew Rocco breaks down what investors should look for and pitches 3 ideas.\",\n",
+    "                \"banner_image\": \"https://staticx-tuner.zacks.com/images/articles/main/92/87.jpg\",\n",
+    "                \"overall_sentiment_score\": 0.219747,\n",
+    "            },\n",
+    "            {\n",
+    "                \"title\": \"PDFgear, Bringing a Completely-Free PDF Text Editing Feature\",\n",
+    "                \"summary\": \"LOS ANGELES, July 26, 2023 /PRNewswire/ -- PDFgear, a leading provider of PDF solutions, announced a piece of exciting news for everyone who works extensively with PDF documents.\",\n",
+    "                \"overall_sentiment_score\": 0.360071,\n",
+    "            },\n",
+    "            {\n",
+    "                \"title\": \"Researchers Pitch 'Immunizing' Images Against Deepfake Manipulation\",\n",
+    "                \"summary\": \"A team at MIT says injecting tiny disruptive bits of code can cause distorted deepfake images.\",\n",
+    "                \"overall_sentiment_score\": -0.026894,\n",
+    "            },\n",
+    "            {\n",
+    "                \"title\": \"Nvidia wins again - plus two more takeaways from this week's mega-cap earnings\",\n",
+    "                \"summary\": \"We made some key conclusions combing through quarterly results for Microsoft and Alphabet and listening to their conference calls with investors.\",\n",
+    "                \"overall_sentiment_score\": 0.235177,\n",
+    "            },\n",
+    "        ]\n",
+    "    }\n",
+    "    feeds = data[\"feed\"][ind:ind_upper]\n",
+    "    feeds_summary = \"\\n\".join(\n",
+    "        [\n",
+    "            f\"News summary: {f['title']}. {f['summary']} overall_sentiment_score: {f['overall_sentiment_score']}\"\n",
+    "            for f in feeds\n",
+    "        ]\n",
+    "    )\n",
+    "    return feeds_summary\n",
+    "\n",
+    "data = asyncio.Future()\n",
+    "\n",
+    "async def add_stock_price_data():\n",
+    "    # simulating the data stream\n",
+    "    for i in range(0, 5, 1):\n",
+    "        latest_news = get_market_news(i, i + 1)\n",
+    "        if data.done():\n",
+    "            data.result().append(latest_news)\n",
+    "        else:\n",
+    "            data.set_result([latest_news])\n",
+    "        # print(data.result())\n",
+    "        await asyncio.sleep(5)\n",
+    "\n",
+    "data_task = asyncio.create_task(add_stock_price_data())\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then, we construct agents. An assistant agent is created to answer the question using LLM. A UserProxyAgent is created to ask questions, and add the new data in the conversation when they are available."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "# create an AssistantAgent instance named \"assistant\"\n",
+    "assistant = autogen.AssistantAgent(\n",
+    "    name=\"assistant\",\n",
+    "    llm_config={\n",
+    "        \"request_timeout\": 600,\n",
+    "        \"seed\": 41,\n",
+    "        \"config_list\": config_list,\n",
+    "        \"temperature\": 0,\n",
+    "    },\n",
+    "    system_message=\"You are a financial expert.\",\n",
+    ")\n",
+    "# create a UserProxyAgent instance named \"user\"\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"user\",\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    max_consecutive_auto_reply=5,\n",
+    "    code_execution_config=False,\n",
+    "    default_auto_reply=None,\n",
+    ")\n",
+    "\n",
+    "async def add_data_reply(recipient, messages, sender, config):\n",
+    "    await asyncio.sleep(0.1)\n",
+    "    data = config[\"news_stream\"]\n",
+    "    if data.done():\n",
+    "        result = data.result()\n",
+    "        if result:\n",
+    "            news_str = \"\\n\".join(result)\n",
+    "            result.clear()\n",
+    "            return (\n",
+    "                True,\n",
+    "                f\"Just got some latest market news. Merge your new suggestion with previous ones.\\n{news_str}\",\n",
+    "            )\n",
+    "        return False, None\n",
+    "\n",
+    "user_proxy.register_reply(autogen.AssistantAgent, add_data_reply, 1, config={\"news_stream\": data})"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We invoke the `a_initiate_chat()` method of the user proxy agent to start the conversation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser\u001b[0m (to assistant):\n",
+      "\n",
+      "Give me investment suggestion in 3 bullet points.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user):\n",
+      "\n",
+      "1. Diversify Your Portfolio: Don't put all your eggs in one basket. Spread your investments across a variety of asset classes such as stocks, bonds, real estate, and commodities. This can help to mitigate risk and potentially increase returns.\n",
+      "\n",
+      "2. Invest for the Long Term: Investing is not about making a quick buck, but about growing your wealth over time. Stick to a long-term investment strategy and avoid the temptation to engage in frequent buying and selling.\n",
+      "\n",
+      "3. Regularly Review Your Investments: The market is dynamic and constantly changing. Regularly review your investment portfolio to ensure it aligns with your financial goals and risk tolerance. Adjust your investments as necessary based on changes in your personal circumstances and market conditions.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser\u001b[0m (to assistant):\n",
+      "\n",
+      "Just got some latest market news. Merge your new suggestion with previous ones.\n",
+      "News summary: Palantir CEO Says Our Generation's Atomic Bomb Could Be AI Weapon - And Arrive Sooner Than You Think - Palantir Technologies  ( NYSE:PLTR ) . Christopher Nolan's blockbuster movie \"Oppenheimer\" has reignited the public discourse surrounding the United States' use of an atomic bomb on Japan at the end of World War II. overall_sentiment_score: 0.009687\n",
+      "News summary: 3 \"Hedge Fund Hotels\" Pulling into Support. Institutional quality stocks have several benefits including high-liquidity, low beta, and a long runway. Strategist Andrew Rocco breaks down what investors should look for and pitches 3 ideas. overall_sentiment_score: 0.219747\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user):\n",
+      "\n",
+      "1. Diversify Your Portfolio: Given the recent news about AI technology and its potential impact, consider investing in tech companies like Palantir Technologies that are at the forefront of AI development. However, remember to maintain a diversified portfolio across various sectors and asset classes to mitigate risk.\n",
+      "\n",
+      "2. Long-Term Investment Strategy: Despite the potential for rapid advancements in AI, it's important to maintain a long-term investment perspective. While these developments may bring short-term volatility, they could also present long-term growth opportunities.\n",
+      "\n",
+      "3. Regularly Review and Adjust Your Investments: With the news about \"Hedge Fund Hotels\" and their potential benefits, consider reviewing your portfolio to see if these high-liquidity, low beta stocks fit into your investment strategy. However, always adjust your investments based on your personal circumstances, risk tolerance, and market conditions. \n",
+      "\n",
+      "4. Invest in AI and Tech Stocks: Given the potential of AI as highlighted by Palantir's CEO, consider investing in companies that are leading in AI and other technological advancements. This could provide significant growth opportunities.\n",
+      "\n",
+      "5. Consider \"Hedge Fund Hotels\": The news suggests that these stocks have several benefits including high-liquidity, low beta, and a long runway. If these align with your investment goals, they could be a good addition to your portfolio.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser\u001b[0m (to assistant):\n",
+      "\n",
+      "Just got some latest market news. Merge your new suggestion with previous ones.\n",
+      "News summary: PDFgear, Bringing a Completely-Free PDF Text Editing Feature. LOS ANGELES, July 26, 2023 /PRNewswire/ -- PDFgear, a leading provider of PDF solutions, announced a piece of exciting news for everyone who works extensively with PDF documents. overall_sentiment_score: 0.360071\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user):\n",
+      "\n",
+      "1. Diversify Your Portfolio: With the latest news about PDFgear, consider investing in companies that provide digital solutions, as they are becoming increasingly important in our digital age. However, continue to maintain a diversified portfolio across various sectors and asset classes to mitigate risk.\n",
+      "\n",
+      "2. Long-Term Investment Strategy: The announcement from PDFgear could potentially lead to short-term gains, but remember to maintain a long-term investment perspective. The digital solutions sector is expected to grow in the long run, providing potential growth opportunities.\n",
+      "\n",
+      "3. Regularly Review and Adjust Your Investments: Given the news about PDFgear and its potential impact on the digital solutions sector, it's important to review your portfolio to see if it aligns with these new market trends. Adjust your investments based on your personal circumstances, risk tolerance, and market conditions.\n",
+      "\n",
+      "4. Invest in Digital Solutions: The news about PDFgear highlights the potential growth in the digital solutions sector. Consider investing in companies that are leading in this field, as they could provide significant growth opportunities.\n",
+      "\n",
+      "5. Consider Tech Stocks: With the continuous advancements in technology, tech stocks, including those in AI, digital solutions, and other tech sub-sectors, could be a good addition to your portfolio, given their potential for high growth.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser\u001b[0m (to assistant):\n",
+      "\n",
+      "Just got some latest market news. Merge your new suggestion with previous ones.\n",
+      "News summary: Researchers Pitch 'Immunizing' Images Against Deepfake Manipulation. A team at MIT says injecting tiny disruptive bits of code can cause distorted deepfake images. overall_sentiment_score: -0.026894\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user):\n",
+      "\n",
+      "1. Diversify Your Portfolio: The latest news about deepfake manipulation suggests potential growth in cybersecurity and AI sectors. Consider investing in companies that are working on these technologies, but continue to maintain a diversified portfolio across various sectors and asset classes to mitigate risk.\n",
+      "\n",
+      "2. Long-Term Investment Strategy: While the news about deepfake manipulation might cause short-term volatility in the tech sector, it's important to maintain a long-term investment perspective. The cybersecurity and AI sectors are expected to grow in the long run, providing potential growth opportunities.\n",
+      "\n",
+      "3. Regularly Review and Adjust Your Investments: Given the news about deepfake manipulation and its potential impact on the tech sector, it's important to review your portfolio to see if it aligns with these new market trends. Adjust your investments based on your personal circumstances, risk tolerance, and market conditions.\n",
+      "\n",
+      "4. Invest in Cybersecurity and AI: The news about deepfake manipulation highlights the potential growth in the cybersecurity and AI sectors. Consider investing in companies that are leading in these fields, as they could provide significant growth opportunities.\n",
+      "\n",
+      "5. Consider Tech Stocks: With the continuous advancements in technology, tech stocks, including those in AI, digital solutions, cybersecurity, and other tech sub-sectors, could be a good addition to your portfolio, given their potential for high growth.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser\u001b[0m (to assistant):\n",
+      "\n",
+      "Just got some latest market news. Merge your new suggestion with previous ones.\n",
+      "News summary: Nvidia wins again - plus two more takeaways from this week's mega-cap earnings. We made some key conclusions combing through quarterly results for Microsoft and Alphabet and listening to their conference calls with investors. overall_sentiment_score: 0.235177\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user):\n",
+      "\n",
+      "1. Diversify Your Portfolio: The latest news about Nvidia, Microsoft, and Alphabet suggests potential growth in the tech sector. Consider investing in these companies, but continue to maintain a diversified portfolio across various sectors and asset classes to mitigate risk.\n",
+      "\n",
+      "2. Long-Term Investment Strategy: While the news about these tech giants might cause short-term volatility, it's important to maintain a long-term investment perspective. The tech sector, particularly companies like Nvidia, Microsoft, and Alphabet, are expected to grow in the long run, providing potential growth opportunities.\n",
+      "\n",
+      "3. Regularly Review and Adjust Your Investments: Given the news about Nvidia, Microsoft, and Alphabet and their potential impact on the tech sector, it's important to review your portfolio to see if it aligns with these new market trends. Adjust your investments based on your personal circumstances, risk tolerance, and market conditions.\n",
+      "\n",
+      "4. Invest in Tech Giants: The news about Nvidia, Microsoft, and Alphabet highlights the potential growth in the tech sector. Consider investing in these tech giants, as they could provide significant growth opportunities.\n",
+      "\n",
+      "5. Consider Tech Stocks: With the continuous advancements in technology, tech stocks, including those in AI, digital solutions, cybersecurity, and other tech sub-sectors, could be a good addition to your portfolio, given their potential for high growth.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "await user_proxy.a_initiate_chat(\n",
+    "    assistant,\n",
+    "    message=\"\"\"Give me investment suggestion in 3 bullet points.\"\"\",\n",
+    ")\n",
+    "while not data_task.done() and not data_task.cancelled():\n",
+    "    reply = await user_proxy.a_generate_reply(sender=assistant)\n",
+    "    if reply is not None:\n",
+    "        await user_proxy.a_send(reply, assistant)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_two_users.ipynb

@@ -0,0 +1,886 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_two_users.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Auto Generated Agent Chat: Collaborative Task Solving with Multiple Agents and Human Users\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate an application involving multiple agents and human users to work together and accomplish a task. `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We create multiple `UserProxyAgent` instances which can represent different human users.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
+    "```bash\n",
+    "pip install pyautogen\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install pyautogen~=0.1.1"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n",
+    "\n",
+    "It first looks for an environment variable of a specified name (\"OAI_CONFIG_LIST\" in this example) which needs to be a valid json string. If that variable is not found, it then looks for a json file with the same name. It filters the configs by models (you can filter by other keys as well).\n",
+    "\n",
+    "The json looks like the following:\n",
+    "```json\n",
+    "[\n",
+    "    {\n",
+    "        \"model\": \"gpt-4\",\n",
+    "        \"api_key\": \"<your OpenAI API key here>\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"model\": \"gpt-4\",\n",
+    "        \"api_key\": \"<your Azure OpenAI API key here>\",\n",
+    "        \"api_base\": \"<your Azure OpenAI API base here>\",\n",
+    "        \"api_type\": \"azure\",\n",
+    "        \"api_version\": \"2023-07-01-preview\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"model\": \"gpt-4-32k\",\n",
+    "        \"api_key\": \"<your Azure OpenAI API key here>\",\n",
+    "        \"api_base\": \"<your Azure OpenAI API base here>\",\n",
+    "        \"api_type\": \"azure\",\n",
+    "        \"api_version\": \"2023-07-01-preview\"\n",
+    "    }\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt-4\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents\n",
+    "\n",
+    "We define `ask_expert` function to start a conversation between two agents and return a summary of the result. We construct an assistant agent named \"assistant_for_expert\" and a user proxy agent named \"expert\". We specify `human_input_mode` as \"ALWAYS\" in the user proxy agent, which will always ask for feedback from the expert user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def ask_expert(message):\n",
+    "    assistant_for_expert = autogen.AssistantAgent(\n",
+    "        name=\"assistant_for_expert\",\n",
+    "        llm_config={\n",
+    "            \"temperature\": 0,\n",
+    "            \"config_list\": config_list,\n",
+    "        },\n",
+    "    )\n",
+    "    expert = autogen.UserProxyAgent(\n",
+    "        name=\"expert\",\n",
+    "        human_input_mode=\"ALWAYS\",\n",
+    "        code_execution_config={\"work_dir\": \"expert\"},\n",
+    "    )\n",
+    "\n",
+    "    expert.initiate_chat(assistant_for_expert, message=message)\n",
+    "    expert.stop_reply_at_receive(assistant_for_expert)\n",
+    "    # expert.human_input_mode, expert.max_consecutive_auto_reply = \"NEVER\", 0\n",
+    "    # final message sent from the expert\n",
+    "    expert.send(\"summarize the solution and explain the answer in an easy-to-understand way\", assistant_for_expert)\n",
+    "    # return the last message the expert received\n",
+    "    return expert.last_message()[\"content\"]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We construct another assistant agent named \"assistant_for_student\" and a user proxy agent named \"student\". We specify `human_input_mode` as \"TERMINATE\" in the user proxy agent, which will ask for feedback when it receives a \"TERMINATE\" signal from the assistant agent. We set the `functions` in `AssistantAgent` and `function_map` in `UserProxyAgent` to use the created `ask_expert` function.\n",
+    "\n",
+    "For simplicity, the `ask_expert` function is defined to run locally. For real applications, the function should run remotely to interact with an expert user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assistant_for_student = autogen.AssistantAgent(\n",
+    "    name=\"assistant_for_student\",\n",
+    "    system_message=\"You are a helpful assistant. Reply TERMINATE when the task is done.\",\n",
+    "    llm_config={\n",
+    "        \"request_timeout\": 600,\n",
+    "        \"seed\": 42,\n",
+    "        # Excluding azure openai endpoints from the config list.\n",
+    "        # Change to `exclude=\"openai\"` to exclude openai endpoints, or remove the `exclude` argument to include both.\n",
+    "        \"config_list\": autogen.config_list_openai_aoai(exclude=\"aoai\"),\n",
+    "        \"model\": \"gpt-4-0613\",  # make sure the endpoint you use supports the model\n",
+    "        \"temperature\": 0,\n",
+    "        \"functions\": [\n",
+    "            {\n",
+    "                \"name\": \"ask_expert\",\n",
+    "                \"description\": \"ask expert when you can't solve the problem satisfactorily.\",\n",
+    "                \"parameters\": {\n",
+    "                    \"type\": \"object\",\n",
+    "                    \"properties\": {\n",
+    "                        \"message\": {\n",
+    "                            \"type\": \"string\",\n",
+    "                            \"description\": \"question to ask expert. Make sure the question include enough context, such as the code and the execution result. The expert does not know the conversation between you and the user, unless you share the conversation with the expert.\",\n",
+    "                        },\n",
+    "                    },\n",
+    "                    \"required\": [\"message\"],\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "student = autogen.UserProxyAgent(\n",
+    "    name=\"student\",\n",
+    "    human_input_mode=\"TERMINATE\",\n",
+    "    max_consecutive_auto_reply=10,\n",
+    "    code_execution_config={\"work_dir\": \"student\"},\n",
+    "    function_map={\"ask_expert\": ask_expert},\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Perform a task\n",
+    "\n",
+    "We invoke the `initiate_chat()` method of the student proxy agent to start the conversation. When you run the cell below, you will be prompted to provide feedback after the assistant agent sends a \"TERMINATE\" signal in the end of the message. If you don't provide any feedback (by pressing Enter directly), the conversation will finish. Before the \"TERMINATE\" signal, the student proxy agent will try to execute the code suggested by the assistant agent on behalf of the user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "Find $a + b + c$, given that $x+y \\neq -1$ and \n",
+      "\\begin{align}\n",
+      "\tax + by + c & = x + 7,\\\n",
+      "\ta + bx + cy & = 2x + 6y,\\\n",
+      "\tay + b + cx & = 4x + y.\n",
+      "\\end{align}.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "To find the values of $a$, $b$, and $c$, we need to solve the system of equations. However, the system of equations you provided seems to be incomplete or incorrect. The equations are not properly formatted, and it's unclear how $x$ and $y$ are related to $a$, $b$, and $c$. \n",
+      "\n",
+      "Could you please provide the correct system of equations?\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "\u001b[32m***** Suggested function Call: ask_expert *****\u001b[0m\n",
+      "Arguments: \n",
+      "{\n",
+      "  \"message\": \"The user provided a system of equations and asked to find $a + b + c$, given that $x+y \\\\neq -1$ and \\n\\\\begin{align}\\n\\tax + by + c & = x + 7,\\\\\\n\\ta + bx + cy & = 2x + 6y,\\\\\\n\\tay + b + cx & = 4x + y.\\n\\\\end{align}. However, the system of equations seems to be incomplete or incorrectly formatted. How should I proceed?\"\n",
+      "}\n",
+      "\u001b[32m***********************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[35m\n",
+      ">>>>>>>> EXECUTING FUNCTION ask_expert...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "The user provided a system of equations and asked to find $a + b + c$, given that $x+y \\neq -1$ and \n",
+      "\\begin{align}\n",
+      "\tax + by + c & = x + 7,\\\n",
+      "\ta + bx + cy & = 2x + 6y,\\\n",
+      "\tay + b + cx & = 4x + y.\n",
+      "\\end{align}. However, the system of equations seems to be incomplete or incorrectly formatted. How should I proceed?\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "To proceed, we can first try to solve the given system of equations and see if we can find a unique solution for $a$, $b$, and $c$. If we can't find a unique solution, we can then determine if there's any additional information needed or if the system is indeed incomplete or incorrectly formatted.\n",
+      "\n",
+      "Let's solve the given system of equations using Python's SymPy library.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "make sure you simplify the answer\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "Sure, I will simplify the answer. Let's solve the given system of equations using Python's SymPy library and simplify the result.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "First, let's install the SymPy library if you haven't already. Execute the following command to install it:\n",
+      "\n",
+      "```sh\n",
+      "pip install sympy\n",
+      "```\n",
+      "\n",
+      "Now, let's solve the given system of equations using Python's SymPy library and simplify the result.\n",
+      "\n",
+      "```python\n",
+      "from sympy import symbols, Eq, solve, simplify\n",
+      "\n",
+      "a, b, c, x, y = symbols('a b c x y')\n",
+      "\n",
+      "eq1 = Eq(a * x + b * y + c, x + 7)\n",
+      "eq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\n",
+      "eq3 = Eq(a * y + b + c * x, 4 * x + y)\n",
+      "\n",
+      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "\n",
+      "simplified_solutions = {key: simplify(value) for key, value in solutions.items()}\n",
+      "a_val, b_val, c_val = simplified_solutions[a], simplified_solutions[b], simplified_solutions[c]\n",
+      "sum_abc = simplify(a_val + b_val + c_val)\n",
+      "\n",
+      "print(f\"a: {a_val}\")\n",
+      "print(f\"b: {b_val}\")\n",
+      "print(f\"c: {c_val}\")\n",
+      "print(f\"a + b + c: {sum_abc}\")\n",
+      "```\n",
+      "\n",
+      "Please execute the above Python code to solve the system of equations and simplify the result.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 1 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "Defaulting to user installation because normal site-packages is not writeable\n",
+      "Requirement already satisfied: sympy in /home/vscode/.local/lib/python3.9/site-packages (1.12)\n",
+      "Requirement already satisfied: mpmath>=0.19 in /home/vscode/.local/lib/python3.9/site-packages (from sympy) (1.3.0)\n",
+      "\n",
+      "a: (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "b: (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "c: (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "a + b + c: 7\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "The given system of equations has been solved and simplified. The result is:\n",
+      "\n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "The sum $a + b + c$ is equal to $7$. Since we were able to find a unique solution for $a$, $b$, and $c$, the system of equations is complete and correctly formatted.\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "summarize the solution and explain the answer in an easy-to-understand way\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "Sure, let me summarize the solution and explain the answer in an easy-to-understand way.\n",
+      "\n",
+      "We were given a system of three equations with three unknowns, $a$, $b$, and $c$. The equations were:\n",
+      "\n",
+      "1. $ax + by + c = x + 7$\n",
+      "2. $a + bx + cy = 2x + 6y$\n",
+      "3. $ay + b + cx = 4x + y$\n",
+      "\n",
+      "We were asked to find the sum $a + b + c$ given that $x + y \\neq -1$.\n",
+      "\n",
+      "We used Python's SymPy library to solve the system of equations and found the following solutions for $a$, $b$, and $c$:\n",
+      "\n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we were able to find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "In conclusion, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"ask_expert\" *****\u001b[0m\n",
+      "Sure, let me summarize the solution and explain the answer in an easy-to-understand way.\n",
+      "\n",
+      "We were given a system of three equations with three unknowns, $a$, $b$, and $c$. The equations were:\n",
+      "\n",
+      "1. $ax + by + c = x + 7$\n",
+      "2. $a + bx + cy = 2x + 6y$\n",
+      "3. $ay + b + cx = 4x + y$\n",
+      "\n",
+      "We were asked to find the sum $a + b + c$ given that $x + y \\neq -1$.\n",
+      "\n",
+      "We used Python's SymPy library to solve the system of equations and found the following solutions for $a$, $b$, and $c$:\n",
+      "\n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we were able to find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "In conclusion, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\u001b[32m*******************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "The solution to the system of equations you provided is:\n",
+      "\n",
+      "1. $a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "2. $b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "3. $c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we can find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "So, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# the assistant receives a message from the student, which contains the task description\n",
+    "student.initiate_chat(\n",
+    "    assistant_for_student,\n",
+    "    message=\"\"\"Find $a + b + c$, given that $x+y \\\\neq -1$ and \n",
+    "\\\\begin{align}\n",
+    "\tax + by + c & = x + 7,\\\\\n",
+    "\ta + bx + cy & = 2x + 6y,\\\\\n",
+    "\tay + b + cx & = 4x + y.\n",
+    "\\\\end{align}.\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When the assistant needs to consult the expert, it suggests a function call to `ask_expert`. When this happens, a line like the following will be displayed:\n",
+    "\n",
+    "***** Suggested function Call: ask_expert *****\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

agentchat_web_info.ipynb

@@ -0,0 +1,987 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_web_info.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Auto Generated Agent Chat: Solving Tasks Requiring Web Info\n",
+    "\n",
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to perform tasks which require acquiring info from the web:\n",
+    "* discuss a paper based on its URL.\n",
+    "* discuss about stock market.\n",
+    "\n",
+    "Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. By setting `human_input_mode` properly, the `UserProxyAgent` can also prompt the user for feedback to `AssistantAgent`. For example, when `human_input_mode` is set to \"TERMINATE\", the `UserProxyAgent` will execute the code written by `AssistantAgent` directly and return the execution results (success or failure and corresponding outputs) to `AssistantAgent`, and prompt the user for feedback when the task is finished. When user feedback is provided, the `UserProxyAgent` will directly pass the feedback to `AssistantAgent`.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n",
+    "```bash\n",
+    "pip install pyautogen docker\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install pyautogen~=0.1.0 docker"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "llm_config={\n",
+    "    \"request_timeout\": 600,\n",
+    "    \"seed\": 42,\n",
+    "    \"config_list\": config_list,\n",
+    "    \"temperature\": 0,\n",
+    "}"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). Only the models with matching names are kept in the list based on the filter condition.\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt4',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k-0314',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents\n",
+    "\n",
+    "We construct the assistant agent and the user proxy agent. We specify `human_input_mode` as \"TERMINATE\" in the user proxy agent, which will ask for human feedback when it receives a \"TERMINATE\" signal from the assistant agent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create an AssistantAgent instance named \"assistant\"\n",
+    "assistant = autogen.AssistantAgent(\n",
+    "    name=\"assistant\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "# create a UserProxyAgent instance named \"user_proxy\"\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"user_proxy\",\n",
+    "    human_input_mode=\"TERMINATE\",\n",
+    "    max_consecutive_auto_reply=10,\n",
+    "    is_termination_msg=lambda x: x.get(\"content\", \"\").rstrip().endswith(\"TERMINATE\"),\n",
+    "    code_execution_config={\"work_dir\": \"web\"},\n",
+    "    llm_config=llm_config,\n",
+    "    system_message=\"\"\"Reply TERMINATE if the task has been solved at full satisfaction.\n",
+    "Otherwise, reply CONTINUE, or the reason why the task is not solved yet.\"\"\"\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example Task: Paper Talk from URL\n",
+    "\n",
+    "We invoke the `initiate_chat()` method of the user proxy agent to start the conversation. When you run the cell below, you will be prompted to provide feedback after the assistant agent sends a \"TERMINATE\" signal in the end of the message. If you don't provide any feedback (by pressing Enter directly), the conversation will finish. Before the \"TERMINATE\" signal, the user proxy agent will try to execute the code suggested by the assistant agent on behalf of the user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "\n",
+      "Who should read this paper: https://arxiv.org/abs/2306.01337\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "To determine who should read the paper, we need to first understand the content and context of the paper. We can do this by fetching the abstract of the paper from the provided URL and analyzing it. \n",
+      "\n",
+      "Here is a Python script that uses the BeautifulSoup library to scrape the abstract of the paper from the webpage. \n",
+      "\n",
+      "```python\n",
+      "# Python script to scrape the abstract of the paper\n",
+      "\n",
+      "import requests\n",
+      "from bs4 import BeautifulSoup\n",
+      "\n",
+      "def get_abstract(url):\n",
+      "    response = requests.get(url)\n",
+      "    soup = BeautifulSoup(response.text, 'html.parser')\n",
+      "    abstract = soup.find('blockquote', attrs={'class': 'abstract mathjax'}).text.strip()\n",
+      "    return abstract\n",
+      "\n",
+      "url = \"https://arxiv.org/abs/2306.01337\"\n",
+      "abstract = get_abstract(url)\n",
+      "print(abstract)\n",
+      "```\n",
+      "\n",
+      "Please run this script and provide the output. Based on the abstract, I can suggest who might be interested in reading this paper.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "Abstract:  Employing Large Language Models (LLMs) to address mathematical problems is an\n",
+      "intriguing research endeavor, considering the abundance of math problems\n",
+      "expressed in natural language across numerous science and engineering fields.\n",
+      "While several prior works have investigated solving elementary mathematics\n",
+      "using LLMs, this work explores the frontier of using GPT-4 for solving more\n",
+      "complex and challenging math problems. We evaluate various ways of using GPT-4.\n",
+      "Some of them are adapted from existing work, and one is MathChat, a\n",
+      "conversational problem-solving framework newly proposed in this work. We\n",
+      "perform the evaluation on difficult high school competition problems from the\n",
+      "MATH dataset, which shows the advantage of the proposed conversational\n",
+      "approach.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Based on the abstract, the paper is about using Large Language Models (LLMs), specifically GPT-4, to solve complex mathematical problems. The paper introduces a new conversational problem-solving framework called MathChat and evaluates its performance on difficult high school competition problems from the MATH dataset.\n",
+      "\n",
+      "Given this, the paper would be of interest to the following groups:\n",
+      "\n",
+      "1. **Researchers in Artificial Intelligence and Natural Language Processing**: The paper discusses the use of a large language model (GPT-4) for problem-solving, which is a key research area in AI and NLP.\n",
+      "\n",
+      "2. **Mathematicians and Math Educators**: The paper focuses on solving complex mathematical problems, so those with a background in mathematics might find the techniques and results interesting.\n",
+      "\n",
+      "3. **Data Scientists and Machine Learning Engineers**: These professionals often use models like GPT-4 in their work and might be interested in new applications and techniques.\n",
+      "\n",
+      "4. **Students studying AI, NLP, or Mathematics**: The paper could provide valuable insights for these students into how AI can be used in problem-solving.\n",
+      "\n",
+      "5. **Developers working on AI-based chatbots or conversational agents**: The paper introduces a new conversational problem-solving framework, which could be of interest to these developers.\n",
+      "\n",
+      "Please note that while the paper is likely to be of interest to these groups, the specific relevance will depend on the individual's specific interests and research needs.\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# the assistant receives a message from the user, which contains the task description\n",
+    "user_proxy.initiate_chat(\n",
+    "    assistant,\n",
+    "    message=\"\"\"\n",
+    "Who should read this paper: https://arxiv.org/abs/2308.08155\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example Task: Chat about Stock Market"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "Show me the YTD gain of 10 largest technology companies as of today.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "To get the YTD gain of the 10 largest technology companies, we need to do the following:\n",
+      "\n",
+      "1. Identify the 10 largest technology companies. We can use the list of the largest technology companies by market capitalization. This list can change frequently, so we need to get the latest data. We can use web scraping to get this data from a reliable source.\n",
+      "\n",
+      "2. Get the YTD gain for each of these companies. We can use a financial data API to get this data. Yahoo Finance is a popular source for this kind of data.\n",
+      "\n",
+      "Here is a Python script that uses the BeautifulSoup library for web scraping and the yfinance library to get data from Yahoo Finance. This script will print the 10 largest technology companies and their YTD gains.\n",
+      "\n",
+      "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
+      "import requests\n",
+      "from bs4 import BeautifulSoup\n",
+      "import yfinance as yf\n",
+      "import pandas as pd\n",
+      "\n",
+      "# Get the list of largest tech companies by market cap\n",
+      "url = \"https://companiesmarketcap.com/technology/largest-technology-companies-by-market-cap/\"\n",
+      "response = requests.get(url)\n",
+      "soup = BeautifulSoup(response.text, 'html.parser')\n",
+      "\n",
+      "# Find the table with the data\n",
+      "table = soup.find('table')\n",
+      "\n",
+      "# Get the ticker symbols of the 10 largest companies\n",
+      "tickers = []\n",
+      "for row in table.find_all('tr')[1:11]:  # Skip the header row\n",
+      "    ticker = row.find_all('td')[1].text\n",
+      "    tickers.append(ticker)\n",
+      "\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    ticker_data = yf.Ticker(ticker)\n",
+      "    hist = ticker_data.history(period=\"ytd\")\n",
+      "    start_price = hist.iloc[0]['Close']\n",
+      "    end_price = hist.iloc[-1]['Close']\n",
+      "    ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "    ytd_gains[ticker] = ytd_gain\n",
+      "\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
+      "```\n",
+      "\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the BeautifulSoup, requests, yfinance, and pandas libraries. If you don't have these libraries installed, you can install them with the command `pip install beautifulsoup4 requests yfinance pandas`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 1 (execution failed)\n",
+      "Code output: \n",
+      "Traceback (most recent call last):\n",
+      "  File \"ytd_gain.py\", line 18, in <module>\n",
+      "    for row in table.find_all('tr')[1:11]:  # Skip the header row\n",
+      "AttributeError: 'NoneType' object has no attribute 'find_all'\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "I apologize for the error. It seems that the script failed to find the table in the webpage. This could be due to changes in the webpage structure or because the page uses JavaScript to load the table, which BeautifulSoup can't handle.\n",
+      "\n",
+      "Let's try a different approach. We'll use the yfinance library to get the data for the 10 largest technology companies. These are currently: Apple (AAPL), Microsoft (MSFT), Alphabet (GOOGL), Amazon (AMZN), Facebook (FB), Tesla (TSLA), Nvidia (NVDA), Paypal (PYPL), Adobe (ADBE), and Netflix (NFLX).\n",
+      "\n",
+      "Here's the updated Python script:\n",
+      "\n",
+      "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
+      "import yfinance as yf\n",
+      "\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'FB', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
+      "\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    ticker_data = yf.Ticker(ticker)\n",
+      "    hist = ticker_data.history(period=\"ytd\")\n",
+      "    start_price = hist.iloc[0]['Close']\n",
+      "    end_price = hist.iloc[-1]['Close']\n",
+      "    ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "    ytd_gains[ticker] = ytd_gain\n",
+      "\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
+      "```\n",
+      "\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 1 (execution failed)\n",
+      "Code output: \n",
+      "FB: No data found, symbol may be delisted\n",
+      "Traceback (most recent call last):\n",
+      "  File \"ytd_gain.py\", line 13, in <module>\n",
+      "    start_price = hist.iloc[0]['Close']\n",
+      "  File \"/home/vscode/.local/lib/python3.9/site-packages/pandas/core/indexing.py\", line 931, in __getitem__\n",
+      "    return self._getitem_axis(maybe_callable, axis=axis)\n",
+      "  File \"/home/vscode/.local/lib/python3.9/site-packages/pandas/core/indexing.py\", line 1566, in _getitem_axis\n",
+      "    self._validate_integer(key, axis)\n",
+      "  File \"/home/vscode/.local/lib/python3.9/site-packages/pandas/core/indexing.py\", line 1500, in _validate_integer\n",
+      "    raise IndexError(\"single positional indexer is out-of-bounds\")\n",
+      "IndexError: single positional indexer is out-of-bounds\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "I apologize for the error. It seems that the script failed to get the data for Facebook (FB). This could be due to a temporary issue with the Yahoo Finance API.\n",
+      "\n",
+      "Let's modify the script to handle this situation. We'll add a try-except block to skip any companies for which we can't get data.\n",
+      "\n",
+      "Here's the updated Python script:\n",
+      "\n",
+      "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
+      "import yfinance as yf\n",
+      "\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'FB', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
+      "\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    try:\n",
+      "        ticker_data = yf.Ticker(ticker)\n",
+      "        hist = ticker_data.history(period=\"ytd\")\n",
+      "        start_price = hist.iloc[0]['Close']\n",
+      "        end_price = hist.iloc[-1]['Close']\n",
+      "        ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "        ytd_gains[ticker] = ytd_gain\n",
+      "    except IndexError:\n",
+      "        print(f\"Could not get data for {ticker}\")\n",
+      "\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
+      "```\n",
+      "\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "Could not get data for FB\n",
+      "AAPL: 46.41%\n",
+      "MSFT: 37.63%\n",
+      "GOOGL: 51.13%\n",
+      "AMZN: 56.22%\n",
+      "TSLA: 131.46%\n",
+      "NVDA: 237.68%\n",
+      "PYPL: -16.49%\n",
+      "ADBE: 59.79%\n",
+      "NFLX: 43.41%\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "The script has successfully fetched the Year-to-Date (YTD) gain for the largest technology companies, except for Facebook (FB). The YTD gains are as follows:\n",
+      "\n",
+      "- Apple (AAPL): 46.41%\n",
+      "- Microsoft (MSFT): 37.63%\n",
+      "- Alphabet (GOOGL): 51.13%\n",
+      "- Amazon (AMZN): 56.22%\n",
+      "- Tesla (TSLA): 131.46%\n",
+      "- Nvidia (NVDA): 237.68%\n",
+      "- Paypal (PYPL): -16.49%\n",
+      "- Adobe (ADBE): 59.79%\n",
+      "- Netflix (NFLX): 43.41%\n",
+      "\n",
+      "Unfortunately, the script could not fetch the data for Facebook (FB). This could be due to a temporary issue with the Yahoo Finance API or the ticker symbol might have changed.\n",
+      "\n",
+      "Please note that these values are subject to change as the stock market fluctuates. \n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "FB -> META\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "Thank you for the update. Facebook Inc. has indeed rebranded to Meta Platforms Inc. and its ticker symbol has changed from FB to META on the NASDAQ stock exchange.\n",
+      "\n",
+      "Let's update the script to reflect this change:\n",
+      "\n",
+      "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
+      "import yfinance as yf\n",
+      "\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "# Updated Facebook (FB) to Meta Platforms (META)\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'META', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
+      "\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    try:\n",
+      "        ticker_data = yf.Ticker(ticker)\n",
+      "        hist = ticker_data.history(period=\"ytd\")\n",
+      "        start_price = hist.iloc[0]['Close']\n",
+      "        end_price = hist.iloc[-1]['Close']\n",
+      "        ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "        ytd_gains[ticker] = ytd_gain\n",
+      "    except IndexError:\n",
+      "        print(f\"Could not get data for {ticker}\")\n",
+      "\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
+      "```\n",
+      "\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "AAPL: 46.34%\n",
+      "MSFT: 37.55%\n",
+      "GOOGL: 51.08%\n",
+      "AMZN: 56.17%\n",
+      "META: 133.67%\n",
+      "TSLA: 131.02%\n",
+      "NVDA: 237.38%\n",
+      "PYPL: -16.55%\n",
+      "ADBE: 59.71%\n",
+      "NFLX: 43.44%\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "The script has successfully fetched the Year-to-Date (YTD) gain for the largest technology companies, including Meta Platforms (formerly Facebook). The YTD gains are as follows:\n",
+      "\n",
+      "- Apple (AAPL): 46.34%\n",
+      "- Microsoft (MSFT): 37.55%\n",
+      "- Alphabet (GOOGL): 51.08%\n",
+      "- Amazon (AMZN): 56.17%\n",
+      "- Meta Platforms (META): 133.67%\n",
+      "- Tesla (TSLA): 131.02%\n",
+      "- Nvidia (NVDA): 237.38%\n",
+      "- Paypal (PYPL): -16.55%\n",
+      "- Adobe (ADBE): 59.71%\n",
+      "- Netflix (NFLX): 43.44%\n",
+      "\n",
+      "Please note that these values are subject to change as the stock market fluctuates. \n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "user_proxy.initiate_chat(\n",
+    "    assistant,\n",
+    "    message=\"\"\"Show me the YTD gain of 10 largest technology companies as of today.\"\"\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

oai_chatgpt_gpt4.ipynb

@@ -0,0 +1,2199 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/oai_chatgpt_gpt4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved. \n",
+    "\n",
+    "Licensed under the MIT License.\n",
+    "\n",
+    "# Use AutoGen to Tune ChatGPT\n",
+    "\n",
+    "AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
+    "Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n",
+    "\n",
+    "In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning.\n",
+    "\n",
+    "Related link: [Blogpost](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n",
+    "```bash\n",
+    "pip install \"pyautogen[blendsearch]\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install \"pyautogen[blendsearch]\" datasets"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "AutoGen has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:54.634335Z",
+     "iopub.status.busy": "2023-02-13T23:40:54.633929Z",
+     "iopub.status.idle": "2023-02-13T23:40:56.105700Z",
+     "shell.execute_reply": "2023-02-13T23:40:56.105085Z"
+    },
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import autogen"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_openai_aoai`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_openai_aoai) function tries to create a list of  Azure OpenAI endpoints and OpenAI endpoints. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
+    "\n",
+    "- OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
+    "- Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
+    "- Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
+    "\n",
+    "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.324240Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.323783Z",
+     "iopub.status.idle": "2023-02-13T23:40:52.330570Z",
+     "shell.execute_reply": "2023-02-13T23:40:52.329750Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "config_list = autogen.config_list_openai_aoai()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {'api_key': '<your OpenAI API key here>'},  # only if OpenAI API key is found\n",
+    "    {\n",
+    "        'api_key': '<your first Azure OpenAI API key here>',\n",
+    "        'api_base': '<your first Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # only if the at least one Azure OpenAI API key is found\n",
+    "    {\n",
+    "        'api_key': '<your second Azure OpenAI API key here>',\n",
+    "        'api_base': '<your second Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },  # only if the second Azure OpenAI API key is found\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "You can directly override it if the above function returns an empty list, i.e., it doesn't find the keys in the specified locations."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Load dataset\n",
+    "\n",
+    "We load the competition_math dataset. The dataset contains 201 \"Level 2\" Algebra examples. We use a random sample of 20 examples for tuning the generation hyperparameters and the remaining for evaluation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:52.339977Z",
+     "iopub.status.busy": "2023-02-13T23:40:52.339556Z",
+     "iopub.status.idle": "2023-02-13T23:40:54.603349Z",
+     "shell.execute_reply": "2023-02-13T23:40:54.602630Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "20 201\n"
+     ]
+    }
+   ],
+   "source": [
+    "import datasets\n",
+    "\n",
+    "seed = 41\n",
+    "data = datasets.load_dataset(\"competition_math\")\n",
+    "train_data = data[\"train\"].shuffle(seed=seed)\n",
+    "test_data = data[\"test\"].shuffle(seed=seed)\n",
+    "n_tune_data = 20\n",
+    "tune_data = [\n",
+    "    {\n",
+    "        \"problem\": train_data[x][\"problem\"],\n",
+    "        \"solution\": train_data[x][\"solution\"],\n",
+    "    }\n",
+    "    for x in range(len(train_data)) if train_data[x][\"level\"] == \"Level 2\" and train_data[x][\"type\"] == \"Algebra\"\n",
+    "][:n_tune_data]\n",
+    "test_data = [\n",
+    "    {\n",
+    "        \"problem\": test_data[x][\"problem\"],\n",
+    "        \"solution\": test_data[x][\"solution\"],\n",
+    "    }\n",
+    "    for x in range(len(test_data)) if test_data[x][\"level\"] == \"Level 2\" and test_data[x][\"type\"] == \"Algebra\"\n",
+    "]\n",
+    "print(len(tune_data), len(test_data))\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "Check a tuning example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:54.607152Z",
+     "iopub.status.busy": "2023-02-13T23:40:54.606441Z",
+     "iopub.status.idle": "2023-02-13T23:40:54.610504Z",
+     "shell.execute_reply": "2023-02-13T23:40:54.609759Z"
+    },
+    "slideshow": {
+     "slide_type": "subslide"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "If $3+a=4-b$ and $4+b=7+a$, what is $3-a$?\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(tune_data[1][\"problem\"])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here is one example of the canonical solution:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:54.613590Z",
+     "iopub.status.busy": "2023-02-13T23:40:54.613168Z",
+     "iopub.status.idle": "2023-02-13T23:40:54.616873Z",
+     "shell.execute_reply": "2023-02-13T23:40:54.616193Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "First we begin by solving the system of equations \\begin{align*}\n",
+      "3+a&=4-b, \\\\\n",
+      "4+b&=7+a.\n",
+      "\\end{align*}Adding the two equations, we get $3+a+4+b=4-b+7+a$, which simplifies to $7+a+b=11+a-b$. Cancelling $a$ from both sides, we get $7+b=11-b$. Solving for $b$, we find that $b=2$. Plugging this into the first equation above, we obtain $3+a=4-2$. Hence $a=-1$ and $3-a=\\boxed{4}$.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(tune_data[1][\"solution\"])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define Success Metric\n",
+    "\n",
+    "Before we start tuning, we need to define the success metric we want to optimize. For each math task, we use voting to select a response with the most common answers out of all the generated responses. If it has an equivalent answer to the canonical solution, we consider the task as successfully solved. Then we can optimize the mean success rate of a collection of tasks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:54.626998Z",
+     "iopub.status.busy": "2023-02-13T23:40:54.626593Z",
+     "iopub.status.idle": "2023-02-13T23:40:54.631383Z",
+     "shell.execute_reply": "2023-02-13T23:40:54.630770Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from autogen.math_utils import eval_math_responses"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Use the tuning data to find a good configuration\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:56.109177Z",
+     "iopub.status.busy": "2023-02-13T23:40:56.108624Z",
+     "iopub.status.idle": "2023-02-13T23:40:56.112651Z",
+     "shell.execute_reply": "2023-02-13T23:40:56.112076Z"
+    },
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "autogen.ChatCompletion.set_cache(seed)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This will create a disk cache in \".cache/{seed}\". You can change `cache_path_root` from \".cache\" to a different path in `set_cache()`. The cache for different seeds are stored separately.\n",
+    "\n",
+    "### Perform tuning\n",
+    "\n",
+    "The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
+    "\n",
+    "* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.004 means the target inference budget is 0.004 dollars, which translates to 2000 tokens (input + output combined) if the gpt-3.5-turbo model is used.\n",
+    "* `optimization_budget` is the total budget allowed to perform the tuning. For example, 1 means 1 dollars are allowed in total, which translates to 500K tokens for the gpt-3.5-turbo model.\n",
+    "* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
+    "\n",
+    "Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
+    "\n",
+    "```python\n",
+    "default_search_space = {\n",
+    "    \"model\": tune.choice([\n",
+    "        \"gpt-3.5-turbo\",\n",
+    "        \"gpt-4\",\n",
+    "    ]),\n",
+    "    \"temperature_or_top_p\": tune.choice(\n",
+    "        [\n",
+    "            {\"temperature\": tune.uniform(0, 2)},\n",
+    "            {\"top_p\": tune.uniform(0, 1)},\n",
+    "        ]\n",
+    "    ),\n",
+    "    \"max_tokens\": tune.lograndint(50, 1000),\n",
+    "    \"n\": tune.randint(1, 100),\n",
+    "    \"prompt\": \"{prompt}\",\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "The default search space can be overridden by users' input.\n",
+    "For example, the following code specifies a fixed prompt template. For hyperparameters which don't appear in users' input, the default search space will be used."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:40:56.115383Z",
+     "iopub.status.busy": "2023-02-13T23:40:56.114975Z",
+     "iopub.status.idle": "2023-02-13T23:41:55.045654Z",
+     "shell.execute_reply": "2023-02-13T23:41:55.044973Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "\n",
+    "prompts = [\"{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.\"]\n",
+    "config, analysis = autogen.ChatCompletion.tune(\n",
+    "    data=tune_data,  # the data for tuning\n",
+    "    metric=\"success_vote\",  # the metric to optimize\n",
+    "    mode=\"max\",  # the optimization mode\n",
+    "    eval_func=eval_math_responses,  # the evaluation function to return the success metrics\n",
+    "    # log_file_name=\"logs/math.log\",  # the log file name\n",
+    "    inference_budget=0.02,  # the inference budget (dollar per instance)\n",
+    "    optimization_budget=1,  # the optimization budget (dollar in total)\n",
+    "    # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
+    "    # -1 means decided by the optimization budget only\n",
+    "    num_samples=20,\n",
+    "    model=\"gpt-3.5-turbo\",  # comment to tune both gpt-3.5-turbo and gpt-4\n",
+    "    prompt=prompts,  # the prompt templates to choose from\n",
+    "    # stop=\"###\",  # the stop sequence\n",
+    "    config_list=config_list,  # the endpoint list\n",
+    "    allow_format_str_template=True,  # whether to allow format string template\n",
+    "    # logging_level=logging.INFO,  # the logging level\n",
+    ")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Output tuning results\n",
+    "\n",
+    "After the tuning, we can print out the config and the result found by AutoGen, which uses flaml for tuning."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:41:55.049204Z",
+     "iopub.status.busy": "2023-02-13T23:41:55.048871Z",
+     "iopub.status.idle": "2023-02-13T23:41:55.053284Z",
+     "shell.execute_reply": "2023-02-13T23:41:55.052574Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "optimized config {'max_tokens': 375, 'n': 44, 'prompt': '{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.', 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True, 'temperature': 0.7466815201029384}\n",
+      "best result on tuning data {'expected_success': 0.9818164607828072, 'success': 1.0, 'success_vote': 0.95, 'voted_answer': 'To find the number of integers in the sequence, we need to find when each term becomes less than 1. \\n\\nStarting with 6075, we divide by 3 to get $\\\\frac{6075}{3} = 2025$. Since 2025 is an integer, it is included in the sequence.\\n\\nDividing 2025 by 3, we get $\\\\frac{2025}{3} = 675$. Again, 675 is an integer, so it is included in the sequence.\\n\\nIf we divide 675 by 3, we get $\\\\frac{675}{3} = 225$. 225 is an integer, so it is included in the sequence.\\n\\nDividing 225 by 3, we get $\\\\frac{225}{3} = 75$. 75 is an integer, so it is included in the sequence.\\n\\nDividing 75 by 3, we get $\\\\frac{75}{3} = 25$. 25 is an integer, so it is included in the sequence.\\n\\nIf we divide 25 by 3, we get $\\\\frac{25}{3} \\\\approx 8.3333$, which is not an integer. Thus, 25 is the last integer in the sequence.\\n\\nThere are a total of $\\\\boxed{6}$ integers in the sequence.', 'votes': 34.85, 'total_cost': 0.463802, 'cost': 0.27552199999999993, 'inference_cost': 0.01310685, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 0.7466815201029384}, 'max_tokens': 375, 'n': 44, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 0.7466815201029384}, 'config/max_tokens': 375, 'config/n': 44, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 201.2768588066101}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"optimized config\", config)\n",
+    "print(\"best result on tuning data\", analysis.best_result)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "### Make a request with the tuned config\n",
+    "\n",
+    "We can apply the tuned config on the request for an example task:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:41:55.056205Z",
+     "iopub.status.busy": "2023-02-13T23:41:55.055631Z",
+     "iopub.status.idle": "2023-02-13T23:41:56.039259Z",
+     "shell.execute_reply": "2023-02-13T23:41:56.038427Z"
+    },
+    "slideshow": {
+     "slide_type": "subslide"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "response on an example data instance: {\n",
+      "  \"id\": \"chatcmpl-7isNR6uGRH8VfNvrTX9YHj7cKdp49\",\n",
+      "  \"object\": \"chat.completion\",\n",
+      "  \"created\": 1690929813,\n",
+      "  \"model\": \"gpt-35-turbo\",\n",
+      "  \"prompt_annotations\": [\n",
+      "    {\n",
+      "      \"prompt_index\": 0,\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    }\n",
+      "  ],\n",
+      "  \"choices\": [\n",
+      "    {\n",
+      "      \"index\": 0,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We start by solving the first equation for $a$: $$3+a=4-b.$$Adding $-3$ to both sides gives $a=1-b$. Substituting this expression for $a$ into the second equation gives $$4+b=7+(1-b).$$Simplifying this expression, we find that $b=2$. Substituting $b=2$ into the first equation to solve for $a$, we find that $a=1-2=-1$. Finally, we have $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 1,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding $a$ to both sides of the first equation gives $3+a+a=4-b+a$, which simplifies to $3+2a=4+a-b$. Adding $b$ to both sides of the second equation gives $4+b+b=7+a+b$, which simplifies to $4+2b=7+a+b$. Rearranging the equations gives $2a-b=1$ and $2b-a=3$. Adding these two equations gives $3a=4$, so $a=\\\\frac{4}{3}$. Substituting this into the first equation gives $2\\\\left(\\\\frac{4}{3}\\\\right)-b=1$, so $b=\\\\frac{5}{3}$. Thus, $3-a=3-\\\\frac{4}{3}=\\\\frac{9}{3}-\\\\frac{4}{3}=\\\\boxed{\\\\frac{5}{3}}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 2,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can first rewrite the given equations as $a+b=1$ and $a-b=3$. Adding these equations, we get $(a+b)+(a-b)=1+3$, which simplifies to $2a=4$. Dividing both sides by 2, we find $a=2$. Substituting this value of $a$ into the first equation, we have $2+b=1$, so $b=-1$. Finally, we can compute $3-a=3-2=\\\\boxed{1}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 3,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $a=4-b-3$. Substituting this into the second equation, we get $4+b=7+4-b-3$. Simplifying the right side gives us $4+b=8-b$. Adding $b$ to both sides gives $4+2b=8$. Subtracting 4 from both sides gives $2b=4$ and dividing both sides by 2 gives $b=2$. Substituting this into the first equation gives $3+a=4-2$, so $a=-1$. Finally, $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 4,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two equations, we have $(3+a)+(4+b)=(4-b)+(7+a)$. Expanding both sides gives $7+a+b=11+a-b$. Subtracting $a$ and $b$ from both sides gives $7=11$, which is impossible. Thus, there is no solution to the given equations.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 5,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We are given the equations $3+a=4-b$ and $4+b=7+a$. We can solve for $a$ and $b$ by using elimination.\\n\\nWe can eliminate $a$ by adding the two equations together. This gives us $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides gives us $7+a+b=11+a-b$. We can then eliminate $a$ by subtracting $a$ from both sides, which gives us $7+b=11-b$. Adding $b$ to both sides gives us $7+2b=11$. Subtracting $7$ from both sides gives us $2b=4$. Dividing both sides by $2$, we get $b=2$.\\n\\nNow that we know the value of $b$, we can substitute it back into one of the original equations to solve for $a$. Let's use the equation $4+b=7+a$. Since we found that $b=2$, we have $4+2=7+a$. Simplifying gives us $6=7+a$. Subtracting $7$ from both sides gives us $-1=a$.\\n\\nFinally, we can find $3-a$ by substituting the value of $a$ that we found back into the expression. We have $3-a=3-(-1)$. Simplifying gives us $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 6,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by solving the given equations separately. From $3+a=4-b$, we can add $b$ to both sides to find $3+a+b=4$. Similarly, from $4+b=7+a$, we can subtract $a$ from both sides to find $4+b-a=7$. Adding these two equations together, we have $3+a+b+4+b-a=4+7$, which simplifies to $7+b=11$. Subtracting $7$ from both sides gives us $b=4$. Substituting this value of $b$ into the equation $4+b-a=7$, we find $4+4-a=7$. Simplifying this equation further, we have $8-a=7$. Subtracting $8$ from both sides gives us $-a=-1$, or $a=1$. Finally, we can substitute the values of $a$ and $b$ into the expression $3-a$, which gives us $3-1=\\\\boxed{2}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 7,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Starting with the first equation, we have $3+a=4-b$. Rearranging, we get $a=-1-b$. Substituting this into the second equation, we have $4+b=7+(-1-b)$. Simplifying, we get $b=-2$. Substituting this value of $b$ back into the first equation, we have $3+a=4-(-2)$. Simplifying further, we have $3+a=6$. Subtracting $3$ from both sides, we get $a=3$. Finally, we can find $3-a=3-3=\\\\boxed{0}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 8,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two equations gives $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides gives $7+a+b=11+a-b$. Subtracting $a$ from both sides gives $7+b=11-b$. Adding $b$ to both sides gives $b+b=11-7$, so $2b=4$ and $b=2$. Substituting this value back into either equation gives $4+2=7+a$, so $a=4$. Therefore, $3-a=3-4=\\\\boxed{-1}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 9,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Rearranging the first equation, we have $a=4-b-3$ and rearranging the second equation, we have $b=7+a-4$. Substituting these expressions for $a$ and $b$ into the equation $3-a$, we have $3-(4-b-3)$. Simplifying the expression inside the parentheses, we have $3-(4-b-3)=3-4+b+3=9+b$. Substituting the expression for $b$ into $9+b$, we have $9+(7+a-4)=9+7+a-4=12+a$. Finally, substituting the expression for $a$ into $12+a$, we have $12+(4-b-3)=12+4-b-3=\\\\boxed{10-b}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 10,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We have the system of equations \\\\begin{align*}\\n3+a&=4-b\\\\\\\\\\n4+b&=7+a\\n\\\\end{align*} Rearranging the first equation, we have $a+b=1$. Substituting this into the second equation, we get $4+1=7+a$, so $a=-4$. Thus, $3-a=\\\\boxed{7}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 11,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Simplifying the first equation, we have $a=1-b$. Substituting this into the second equation, we have $4+b=7+(1-b)$. Expanding the right side gives $4+b=7+1-b$. Combining like terms gives $2b=4$, so $b=2$. Substituting this back into $a=1-b$, we find that $a=-1$. Thus, $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 12,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $a=4-b-3$. Substituting this into the second equation, we have $4+b=7+(4-b-3)$. Simplifying the right side of the equation gives $4+b=8-b$. Adding $b$ to both sides gives $4+2b=8$. Subtracting 4 from both sides gives $2b=4$. Dividing both sides by 2 gives $b=2$. Substituting this value back into the first equation gives $3+a=4-2$. Simplifying the right side gives $3+a=2$. Subtracting 3 from both sides gives $a=-1$. Finally, we have $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 13,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, subtracting $a$ and adding $4$ to both sides gives $7=b-a$. Substituting this into the second equation gives $4+(b-a)=7+a$, so $4+7=b+a$.  Combining these equations gives $3+b+a=11+a$, so $b=\\\\boxed{8}$.  Substituting into the first equation gives $3+a=4-8$ which gives $a=\\\\boxed{-9}$.   Finally, $3-a=3-(-9)=3+9=\\\\boxed{12}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 14,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by solving the first equation for $a$ in terms of $b$. Subtracting $3$ from both sides of the equation $3+a=4-b$ gives $a=1-b$. We can substitute this expression for $a$ in the second equation to solve for $b$: \\\\begin{align*}\\n4+b&=7+a\\\\\\\\\\n4+b&=7+(1-b)\\\\\\\\\\n4+b&=8-b\\\\\\\\\\n2b&=4\\\\\\\\\\nb&=2.\\n\\\\end{align*}Substituting this value of $b$ back into the first equation to solve for $a$, we have $3+a=4-2$, so $a=-1$. Finally, we can find $3-a=3-(-1)=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 15,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $a=1-b$. Substituting this into the second equation gives $4+b=7+1-b$. Simplifying gives $2b=4$, so $b=2$. Substituting this back into the first equation gives $3+a=4-2$, so $a=-1$. Therefore, $3-a=3-(-1)=4$. Simplifying gives $\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 16,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides gives $7+a+b=11+a-b$. We can subtract $a$ from both sides to get $7+b=11-b$. Adding $b$ to both sides gives $7+2b=11$. Subtracting 7 from both sides gives $2b=4$. Dividing by 2 gives $b=2$.\\n\\nWe can substitute $b=2$ into the first equation $3+a=4-b$ to solve for $a$. We have $3+a=4-2$ which simplifies to $3+a=2$. Subtracting 3 from both sides gives $a=-1$.\\n\\nFinally, we can substitute $a=-1$ into $3-a$ to find $3-a=3-(-1)$. Simplifying gives $3-a=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 17,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides gives $7+a+b=11+a-b$. Subtracting $a$ from both sides gives $7+b=11-b$. Adding $b$ to both sides gives $7+2b=11$. Subtracting $7$ from both sides gives $2b=4$. Finally, dividing both sides by $2$ gives $b=2$. Substituting this value for $b$ into the second given equation, we have $4+2=7+a$. Simplifying gives $a=-1$. Therefore, $3-a=3-(-1)=4$. Thus, the final answer is $\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 18,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Let's start by simplifying the given equations. We have $3+a=4-b$, which we can rearrange to get $a=-b+1$. Similarly, we have $4+b=7+a$, which rearranges to $b=a+3$. \\n\\nWe can substitute the value of $b$ from the second equation into the first equation to get $a=(-a-3)+1$. Simplifying this equation gives $2a=-2$, so $a=-1$. \\n\\nSubstituting this value of $a$ into the second equation gives $b=(-1)+3$, so $b=2$. \\n\\nFinally, we can find $3-a$ by substituting $a=-1$ into $3-a$. This gives $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 19,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We have the equations $3+a=4-b$ and $4+b=7+a$. We can solve these equations using substitution or elimination. Let's solve it using elimination.\\n\\nTo eliminate $a$, we can add the first equation to the second equation. This gives us $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides, we have $7 + a + b = 11 + a - b$.\\n\\nNow, let's isolate $b$ by subtracting $a$ from both sides: $7 + b = 11 - b$.\\n\\nTo isolate $b$ on one side, we can add $b$ to both sides: $7 + 2b = 11$.\\n\\nSubtracting $7$ from both sides gives $2b= 4$. Dividing both sides by $2$, we find $b=2$.\\n\\nNow, we can substitute $b=2$ into the second equation $4+b=7+a$. This gives $4+2=7+a$, or $6=7+a$. Subtracting $7$ from both sides gives $-1=a$.\\n\\nFinally, we can find $3-a$ by substituting $a=-1$ into $3-a$. This gives $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 20,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by combining like terms in both equations. From the first equation, we have $a+3=b-4$, and from the second equation, we have $b+4=a-7$. \\n\\nNow, we can substitute $b-4$ for $a+3$ in the second equation, since they are equal. This gives us $b+4=(b-4)-7$. \\n\\nSimplifying, we have $b+4=b-11$. \\n\\nSubtracting $b$ from both sides, we get $4=-11$. \\n\\nThis is a contradiction, since $4$ does not equal $-11$. \\n\\nTherefore, the solution to this system of equations does not exist, and we cannot find the value of $3-a$. Thus, the answer is $\\\\boxed{\\\\text{DNE}}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 21,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by solving the first equation, $3+a=4-b$, for $a$ in terms of $b$ by subtracting $3$ from both sides and then adding $b$ to both sides.  This gives us $a = 1-b$.\\n\\nWe can substitute this expression for $a$ into the second equation, $4+b=7+a$, to solve for $b$ in terms of $a$.  After simplifying, we have $b=4-a$.\\n\\nTo find $3-a$, we substitute $b=4-a$ into the first equation $3+a=4-b$.  This gives us $3+a=4-(4-a)$.\\n\\nSimplifying this equation gives $3+a=4-4+a$, so $3+a=a$.\\n\\nTherefore, $3-a = \\\\boxed{3}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 22,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"To solve this problem, we can start by solving the first equation $3+a=4-b$ for $b$. Subtracting $3$ from both sides gives $a=1-b$. We can substitute this into the second equation $4+b=7+a$ to get $4+b=7+(1-b)$. Expanding the right side gives $4+b=7+1-b$. Combining like terms gives $b+b=7+1-4$. Simplifying the right side gives $2b=4$. Dividing both sides by $2$ gives $b=2$. Now we can substitute this back into the first equation to solve for $a$. We have $3+a=4-2$, so $3+a=2$. Subtracting $3$ from both sides gives $a=-1$. Finally, we can find $3-a$ by subtracting $a$ from $3$. We have $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 23,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Starting with the first equation, we have $3+a=4-b$. Rearranging this equation, we get $a=-b+1$. \\n\\nSubstituting this expression for $a$ into the second equation, we have $4+b=7+(-b+1)$. Simplifying this equation gives $b=-2$. \\n\\nSubstituting this value of $b$ into the equation $a=-b+1$, we find $a=3$. \\n\\nFinally, we can evaluate $3-a$ using the value we found for $a$. We have $3-a=3-3=\\\\boxed{0}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 24,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $(3+a)+(4+b)=(4-b)+(7+a)$. Simplifying both sides gives $7+a+b=11+a-b$. Subtracting $a$ from both sides gives $7+b=11-b$. Adding $b$ to both sides gives $7+2b=11$. Subtracting $7$ from both sides gives $2b=4$. Dividing both sides by $2$ gives $b=2$. Substituting this value of $b$ into the first given equation, we have $3+a=4-2$, or $a=-1$. Finally, substituting these values into $3-a$, we have $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 25,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, $3+a=4-b$, we can subtract $3$ from both sides to find that $a=1-b$. Substituting this into the second equation, we have $4+b=7+(1-b)$. Expanding the right side gives $4+b=7+1-b$. We can then simplify this to $b+4=8-b$ by combining like terms. Adding $b$ to both sides gives $2b+4=8$. Subtracting $4$ from both sides gives $2b=4$. Finally, dividing both sides by $2$ gives $b=2$. \\n\\nNow that we have found the value of $b$, we can substitute it back into the first equation to find $3+a=4-2$. Simplifying gives $3+a=2$. Subtracting $3$ from both sides gives $a=-1$. \\n\\nFinally, we can find $3-a$ by subtracting $-1$ from $3$. This gives $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 26,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by solving the first equation, $3+a=4-b$, for $a$. Subtracting 3 from both sides gives $a=1-b$. \\n\\nWe can substitute this value for $a$ into the second equation $4+b=7+a$. Substituting gives $4+b=7+(1-b)$. Expanding the right side gives $4+b=7+1-b$. Combining like terms gives $4+b=8-b$. \\n\\nAdding $b$ to both sides gives $4=8-2b$. Subtracting 8 from both sides gives $-4=-2b$. Dividing both sides by $-2$ gives $2=b$.\\n\\nNow that we know $b=2$, we can substitute this value back into the first equation to find $a$. Substituting gives $3+a=4-2$. Simplifying gives $3+a=2$. Subtracting 3 from both sides gives $a=-1$.\\n\\nFinally, we can find $3-a$ by substituting the value of $a$ we found into $3-a$. Substituting gives $3-(-1)$. Simplifying gives $3+1=4$.\\n\\nThus, $3-a=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 27,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Starting with the first equation, we can subtract $a$ from both sides to isolate $3$:\\n\\n\\\\[3+a-a=4-b-a\\\\qquad\\\\Rightarrow\\\\qquad 3=4-b-a.\\\\]\\n\\nRearranging the terms, we have $b+a=4-3=1.$  Similarly, starting with the second equation, we can subtract $b$ from both sides to obtain $a+b=7-4=3.$  Adding these two equations, we have $2a+2b=4,$ so $a+b=2.$  Subtracting this equation from $b+a=1,$ we get $b-a=\\\\boxed{-1}.$\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 28,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $a=4-b-3=-b+1$. Substituting this into the second equation, we get $4+b=7+(-b+1)$, which simplifies to $4+b=8-b$. Solving for $b$, we find $2b=4$, so $b=2$. Substituting this back into $a=-b+1$, we find $a=-2+1=-1$. Finally, $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 29,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We start by solving the first equation for $a$ by subtracting 3 from both sides to find $a=1-b$. Substituting this into the second equation gives $4+b=7+(1-b)$. Expanding the brackets gives $4+b=7+1-b$. Simplifying gives $2b=4$, so $b=2$. Substituting this into $a=1-b$ gives $a=1-2=-1$. Finally, substituting this into $3-a$ gives $3-(-1)=4$, so our final answer is $\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 30,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $(3+a)+(4+b)=(4-b)+(7+a)$. Applying the commutative property of addition, we can rearrange the terms to get $(3+4)+(a+b)=(4+7)+(-b+a)$. Simplifying both sides gives $7+(a+b)=11+(a-b)$. We can rewrite this equation as $a+b+7=a-b+11$. Subtracting $a+b+4$ from both sides yields $7-4=a-b+11-(a+b+4)$. Simplifying gives $3=-4-b$. Adding $b$ to both sides gives $3+b=-4$. Subtracting 4 from both sides yields $b-1=-4$. Then, adding 1 to both sides gives $b=-3$. Substituting this into the first equation $3+a=4-b$, we can substitute $-3$ for $b$ to get $3+a=4-(-3)$. Simplifying gives $3+a=4+3$. Subtracting 3 from both sides yields $a=4$. Finally, substituting this into $3-a$, we can substitute $4$ for $a$ to get $3-4=\\\\boxed{-1}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 31,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $a=4-b-3=1-b$. Substituting this into the second equation gives $4+b=7+(1-b)$. Expanding the right side gives $4+b=7+1-b$. Simplifying the right side gives $4+b=8-b$. Adding $b$ to both sides gives $4+2b=8$. Subtracting 4 from both sides gives $2b=4$. Dividing both sides by 2 gives $b=2$. Substituting this into the first equation gives $3+a=4-2$. Simplifying the right side gives $3+a=2$. Subtracting 3 from both sides gives $a=-1$. Finally, we have $3-a=3-(-1)=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 32,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We start by solving the first equation for $a$: \\\\begin{align*}\\n3+a&=4-b \\\\\\\\\\na&=1-b.\\n\\\\end{align*}We substitute this expression for $a$ into the second equation: \\\\begin{align*}\\n4+b&=7+a \\\\\\\\\\n4+b&=7+(1-b) \\\\\\\\\\n4+b&=8-b.\\n\\\\end{align*}Adding $b$ to both sides gives $4+2b=8$, so $2b=4$ and $b=2$. Substituting this value into $a=1-b$ gives $a=1-2=-1$. Therefore, $3-a=3-(-1)=3+1=4$. Thus, $3-a=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 33,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We start by solving the first equation, $3+a=4-b$, for $a$. Subtracting $3$ from both sides gives $a=1-b$.\\n\\nWe can substitute this expression for $a$ into the second equation, $4+b=7+a$. Substituting $1-b$ for $a$ gives $4+b=7+(1-b)$.\\n\\nExpanding the parentheses gives $4+b=7+1-b$. Simplifying the right side gives $4+b=8-b$.\\n\\nAdding $b$ to both sides gives $4+2b=8$. Subtracting $4$ from both sides gives $2b=4$.\\n\\nFinally, dividing both sides by $2$ gives $b=2$. We can substitute this value back into the equation $a=1-b$ to find $a=1-2=-1$.\\n\\nNow, we can find $3-a$ by subtracting $(-1)$ from $3$. This gives $3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 34,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Combining the two equations, we have $3+a+4+b=4-b+7+a$. Simplifying both sides, we have $7+a+b=11+a-b$. Subtracting $a$ from both sides, we have $7+b=11-b$. Adding $b$ to both sides, we have $7+2b=11$. Subtracting 7 from both sides, we have $2b=4$. Dividing both sides by 2, we have $b=2$. Substituting this value back into the first equation, we have $3+a=4-2$, so $3+a=2$. Subtracting 3 from both sides, we have $a=-1$. Therefore, $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 35,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by subtracting $a$ from both sides of the first equation to find that $3 = 4 - b - a$.  Similarly, we can subtract $a$ from both sides of the second equation to find that $4 = 7 + a - b$.  Adding these two equations gives $3 + 4 = 4 - b - a + 7 + a - b$, which simplifies to $7 = 11 - 2b$.  Solving for $b$ gives $b = 2$.  Substituting this value into the first equation gives $3 + a = 4 - 2$, so $a = 3$.  Then $3 - a = 3 - 3 = \\\\boxed{0}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 36,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $$(3+a)+(4+b)=(4-b)+(7+a).$$Expanding both sides gives $$7+a+b=11+a-b.$$We can then cancel out the $a$ term on both sides to get $$7+b=11-b.$$Adding $b$ to both sides gives $$7+2b=11.$$Subtracting $7$ from both sides gives $$2b=4.$$Dividing both sides by $2$ gives $$b=2.$$Plugging this value of $b$ into either of the original equations, we can solve for $a$. Using the first equation, we have $$3+a=4-2 \\\\Rightarrow a=-1.$$Finally, we can find $3-a$ as $$3-a=3-(-1)=3+1=\\\\boxed{4}.$$\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 37,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by adding $a$ to both sides of the first equation and subtracting $b$ from both sides of the second equation to obtain \\\\begin{align*}\\na+b&=1, \\\\\\\\\\na-b&=-3.\\n\\\\end{align*} We can then add these equations to eliminate $b$: $$2a=1+(-3)=-2.$$Dividing both sides by $2$ gives $a=-1$. Substituting into the second equation gives $-1-b=-3$, so $b=2$. Finally, we find that $3-a=3-(-1)=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 38,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We can start by subtracting $a$ from both sides of the first equation and subtracting $b$ from both sides of the second equation to obtain \\\\begin{align*}\\n3&=4-b-a,\\\\\\\\\\n4&=7+a-b.\\n\\\\end{align*}We can rearrange the first equation to get $b+a=4-3=1$. Similarly, we can rearrange the second equation to get $a-b=4-7=-3$. Adding these equations, we find that $(b+a)+(a-b)=1+(-3)$, which implies $2a= -2$. Hence,  $a=-1$. We can substitute this value of $a$ into $a-b=-3$ to find that $-1-b=-3$, so $b=-1-(-3)=2$. Finally, we have \\\\begin{align*}\\n3-a&=3-(-1)=3+1=\\\\boxed{4}.\\n\\\\end{align*}\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 39,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Adding the two given equations, we have $$(3+a)+(4+b)=(4-b)+(7+a).$$Simplifying both sides gives $7+a+b=11+a-b$. Subtracting $a$ and $b$ from both sides gives $7=11$, which is a contradiction. Therefore, there are no solutions to the given equations, and the value of $3-a$ is undefined. So we have $3-a=\\\\boxed{ \\\\text{undefined}}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 40,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"To solve this problem, we can start by isolating $a$ in both equations. \\n\\nFrom the first equation, $3+a=4-b$, we can subtract 3 from both sides to get $a=1-b$. \\n\\nFrom the second equation, $4+b=7+a$, we can subtract 4 from both sides to get $b=3+a$. \\n\\nNow, we can substitute $1-b$ for $a$ in the second equation to get $b=3+1-b$. Simplifying this equation gives $2b=4$, so $b=2$. \\n\\nSubstituting $b=2$ into the equation $a=1-b$, we find that $a=1-2=-1$. \\n\\nFinally, we can find $3-a$ by subtracting $(-1)$ from 3, which gives us $3-(-1)=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 41,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"We start by solving the first equation for $a$ in terms of $b$. Subtracting $3$ from both sides, we have $a=1-b$. Substituting this into the second equation, we get $4+b=7+(1-b)$. Simplifying, we have $4+b=8-b$. Adding $b$ to both sides, we have $4+2b=8$. Subtracting $4$ from both sides, we have $2b=4$. Dividing both sides by $2$, we have $b=2$. Substituting this into the equation $a=1-b$, we have $a=1-2=-1$. Thus, $3-a=3-(-1)=3+1=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 42,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"From the first equation, we have $3+a=4-b$. Rearranging this equation, we get $a=-1-b$. Substituting this value of $a$ into the second equation, we have $4+b=7+(-1-b)$. Simplifying this equation, we get $b+1=-b+6$. Adding $b$ to both sides and subtracting $1$ from both sides, we have $2b=5$. Therefore, $b=\\\\frac{5}{2}$. Substituting this value of $b$ into the first equation, we have $3+a=4-\\\\frac{5}{2}$. Simplifying this equation, we get $a=\\\\frac{3}{2}$. Finally, we have $3-a=3-\\\\frac{3}{2}=\\\\boxed{\\\\frac{3}{2}}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    },\n",
+      "    {\n",
+      "      \"index\": 43,\n",
+      "      \"finish_reason\": \"stop\",\n",
+      "      \"message\": {\n",
+      "        \"role\": \"assistant\",\n",
+      "        \"content\": \"Manipulating the given equations, we have $a=1-b$ and $b=3+a$. Substituting $a=1-b$ into the second equation, we get $b=3+1-b$, which implies $2b=4$, or $b=2$. Substituting $b=2$ into the first equation, we get $a=1-2=-1$. Finally, evaluating $3-a$, we find that $3-a=3-(-1)=\\\\boxed{4}$.\"\n",
+      "      },\n",
+      "      \"content_filter_results\": {\n",
+      "        \"hate\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"self_harm\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"sexual\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        },\n",
+      "        \"violence\": {\n",
+      "          \"filtered\": false,\n",
+      "          \"severity\": \"safe\"\n",
+      "        }\n",
+      "      }\n",
+      "    }\n",
+      "  ],\n",
+      "  \"usage\": {\n",
+      "    \"completion_tokens\": 7178,\n",
+      "    \"prompt_tokens\": 52,\n",
+      "    \"total_tokens\": 7230\n",
+      "  },\n",
+      "  \"cost\": 0.01446,\n",
+      "  \"config_id\": 0,\n",
+      "  \"pass_filter\": true\n",
+      "}\n",
+      "metric_results on the example data instance: {'expected_success': 1.0, 'success': True, 'success_vote': 1.0, 'voted_answer': 'We start by solving the first equation for $a$: $$3+a=4-b.$$Adding $-3$ to both sides gives $a=1-b$. Substituting this expression for $a$ into the second equation gives $$4+b=7+(1-b).$$Simplifying this expression, we find that $b=2$. Substituting $b=2$ into the first equation to solve for $a$, we find that $a=1-2=-1$. Finally, we have $3-a=3-(-1)=3+1=\\\\boxed{4}$.', 'votes': 27}\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = autogen.ChatCompletion.create(context=tune_data[1], config_list=config_list, **config)\n",
+    "metric_results = eval_math_responses(autogen.ChatCompletion.extract_text(response), **tune_data[1])\n",
+    "print(\"response on an example data instance:\", response)\n",
+    "print(\"metric_results on the example data instance:\", metric_results)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Evaluate the success rate on the test data\n",
+    "\n",
+    "You can use `autogen.ChatCompletion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while (30 mins to 1 hour) to evaluate all the test data instances if uncommented and run. It will cost roughly $3. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-13T23:41:56.042764Z",
+     "iopub.status.busy": "2023-02-13T23:41:56.042086Z",
+     "iopub.status.idle": "2023-02-13T23:53:05.597643Z",
+     "shell.execute_reply": "2023-02-13T23:53:05.596603Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# result = autogen.ChatCompletion.test(test_data, logging_level=logging.INFO, config_list=config_list, **config)\n",
+    "# print(\"performance on test data with the tuned config:\", result)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "What about the default, untuned gpt-4 config (with the same prompt as the tuned config)? We can evaluate it and compare:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "performance on test data from gpt-4 with a default config: {'expected_success': 0.6965174129353234, 'success': 0.6965174129353234, 'success_vote': 0.6965174129353234, 'votes': 1.0, 'cost': 1.9264799999999993, 'inference_cost': 0.009584477611940295}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# the following code will cost roughly $2 if uncommented and run.\n",
+    "\n",
+    "# default_config = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"allow_format_str_template\": True}\n",
+    "# default_result = autogen.ChatCompletion.test(test_data, config_list=config_list, **default_config)\n",
+    "# print(\"performance on test data from gpt-4 with a default config:\", default_result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "tuned config succeeds in 90.5% test cases\n",
+      "untuned config succeeds in 69.7% test cases\n"
+     ]
+    }
+   ],
+   "source": [
+    "# print(\"tuned config succeeds in {:.1f}% test cases\".format(result[\"success_vote\"] * 100))\n",
+    "# print(\"untuned config succeeds in {:.1f}% test cases\".format(default_result[\"success_vote\"] * 100))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The default use of GPT-4 has a much lower accuracy. Note that the default config has a lower inference cost. What if we heuristically increase the number of responses n?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# The following evaluation costs $3 and longer than one hour if you uncomment it and run it.\n",
+    "\n",
+    "# config_n2 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 2, \"allow_format_str_template\": True}\n",
+    "# result_n2 = autogen.ChatCompletion.test(test_data, config_list=config_list, **config_n2)\n",
+    "# print(\"performance on test data from gpt-4 with a default config and n=2:\", result_n2)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The inference cost is doubled and matches the tuned config. But the success rate doesn't improve much. What if we further increase the number of responses n to 5?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# The following evaluation costs $8 and longer than one hour if you uncomment it and run it.\n",
+    "\n",
+    "# config_n5 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 5, \"allow_format_str_template\": True}\n",
+    "# result_n5 = autogen.ChatCompletion.test(test_data, config_list=config_list, **config_n5)\n",
+    "# print(\"performance on test data from gpt-4 with a default config and n=5:\", result_n5)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n",
+    "\n",
+    "A developer could use AutoGen to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "454146d0f7224f038689031002906e6f": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
+        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
+        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
+       ],
+       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "577e1e3cc4db4942b0883577b3b52755": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "6086462a12d54bafa59d3c4566f06cb2": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "74a6ba0c3cbc4051be0a83e152fe1e62": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "7d3f3d9e15894d05a4d188ff4f466554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
+      }
+     },
+     "ca245376fd9f4354af6b2befe4af4466": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "dc83c7bff2f241309537a8119dfc7555": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
+       "placeholder": "​",
+       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "f1355871cc6f4dd4b50d9df5af20e5c8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

oai_completion.ipynb

@@ -0,0 +1,1189 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/oai_completion.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved. \n",
+    "\n",
+    "Licensed under the MIT License.\n",
+    "\n",
+    "# Use AutoGen to Tune OpenAI Models\n",
+    "\n",
+    "AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
+    "Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n",
+    "\n",
+    "In this notebook, we tune OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n",
+    "```bash\n",
+    "pip install pyautogen[blendsearch]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:36.910966Z",
+     "iopub.status.busy": "2023-02-24T23:25:36.910473Z",
+     "iopub.status.idle": "2023-02-24T23:25:36.914554Z",
+     "shell.execute_reply": "2023-02-24T23:25:36.914030Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install \"pyautogen[blendsearch]~=0.1.0\" datasets"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "* The [`config_list_openai_aoai`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_openai_aoai) function tries to create a list of configurations using Azure OpenAI endpoints and OpenAI endpoints. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
+    "  - OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
+    "  - Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
+    "  - Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
+    "* The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file. It first looks for environment variable `env_or_file` which needs to be a valid json string. If that variable is not found, it then looks for a json file with the same name. It filters the configs by filter_dict.\n",
+    "\n",
+    "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base. If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:36.917301Z",
+     "iopub.status.busy": "2023-02-24T23:25:36.917011Z",
+     "iopub.status.idle": "2023-02-24T23:25:36.923156Z",
+     "shell.execute_reply": "2023-02-24T23:25:36.922619Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "endpoint_list = autogen.config_list_openai_aoai()\n",
+    "# the endpoint_list looks like this:\n",
+    "# endpoint_list = [\n",
+    "#     {\n",
+    "#         'api_key': '<your OpenAI API key here>',\n",
+    "#     },  # OpenAI API endpoint for gpt-4\n",
+    "#     {\n",
+    "#         'api_key': '<your first Azure OpenAI API key here>',\n",
+    "#         'api_base': '<your first Azure OpenAI API base here>',\n",
+    "#         'api_type': 'azure',\n",
+    "#         'api_version': '2023-03-15-preview',\n",
+    "#     },  # Azure OpenAI API endpoint for gpt-4\n",
+    "#     {\n",
+    "#         'api_key': '<your second Azure OpenAI API key here>',\n",
+    "#         'api_base': '<your second Azure OpenAI API base here>',\n",
+    "#         'api_type': 'azure',\n",
+    "#         'api_version': '2023-03-15-preview',\n",
+    "#     },  # another Azure OpenAI API endpoint for gpt-4\n",
+    "# ]\n",
+    "\n",
+    "config_list = autogen.config_list_from_json(\n",
+    "    env_or_file=\"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": {\n",
+    "            \"gpt-3.5-turbo\",\n",
+    "            \"gpt-3.5-turbo-16k\",\n",
+    "            \"gpt-3.5-turbo-0301\",\n",
+    "            \"chatgpt-35-turbo-0301\",\n",
+    "            \"gpt-35-turbo-v0301\",\n",
+    "            \"gpt\",\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "# the config_list looks like this:\n",
+    "# config_list = [\n",
+    "#     {\n",
+    "#         'model': 'gpt-3.5-turbo',\n",
+    "#         'api_key': '<your OpenAI API key here>',\n",
+    "#     },  # OpenAI API endpoint for gpt-3.5-turbo\n",
+    "#     {\n",
+    "#         'model': 'gpt-3.5-turbo',\n",
+    "#         'api_key': '<your first Azure OpenAI API key here>',\n",
+    "#         'api_base': '<your first Azure OpenAI API base here>',\n",
+    "#         'api_type': 'azure',\n",
+    "#         'api_version': '2023-06-01-preview',\n",
+    "#     },  # Azure OpenAI API endpoint for gpt-3.5-turbo\n",
+    "#     {\n",
+    "#         'model': 'gpt-35-turbo-v0301',\n",
+    "#         'api_key': '<your second Azure OpenAI API key here>',\n",
+    "#         'api_base': '<your second Azure OpenAI API base here>',\n",
+    "#         'api_type': 'azure',\n",
+    "#         'api_version': '2023-06-01-preview',\n",
+    "#     },  # another Azure OpenAI API endpoint for gpt-3.5-turbo with deployment name gpt-35-turbo-v0301\n",
+    "# ]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you don't use the two provided utility functions above, you can define the lists in other ways you prefer.\n",
+    "\n",
+    "## Load dataset\n",
+    "\n",
+    "First, we load the humaneval dataset. The dataset contains 164 examples. We use the first 20 for tuning the generation hyperparameters and the remaining for evaluation. In each example, the \"prompt\" is the prompt string for eliciting the code generation (renamed into \"definition\"), \"test\" is the Python code for unit test for the example, and \"entry_point\" is the function name to be tested."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:36.931255Z",
+     "iopub.status.busy": "2023-02-24T23:25:36.930838Z",
+     "iopub.status.idle": "2023-02-24T23:25:39.148799Z",
+     "shell.execute_reply": "2023-02-24T23:25:39.148113Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset openai_humaneval (/home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8e08cc907707418a86a3da668e45326b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75/cache-1e8448101c1b32e8.arrow\n"
+     ]
+    }
+   ],
+   "source": [
+    "import datasets\n",
+    "\n",
+    "seed = 41\n",
+    "data = datasets.load_dataset(\"openai_humaneval\")[\"test\"].shuffle(seed=seed)\n",
+    "n_tune_data = 20\n",
+    "tune_data = [\n",
+    "    {\n",
+    "        \"definition\": data[x][\"prompt\"],\n",
+    "        \"test\": data[x][\"test\"],\n",
+    "        \"entry_point\": data[x][\"entry_point\"],\n",
+    "    }\n",
+    "    for x in range(n_tune_data)\n",
+    "]\n",
+    "test_data = [\n",
+    "    {\n",
+    "        \"definition\": data[x][\"prompt\"],\n",
+    "        \"test\": data[x][\"test\"],\n",
+    "        \"entry_point\": data[x][\"entry_point\"],\n",
+    "    }\n",
+    "    for x in range(n_tune_data, len(data))\n",
+    "]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "Check a tuning example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:39.152156Z",
+     "iopub.status.busy": "2023-02-24T23:25:39.151531Z",
+     "iopub.status.idle": "2023-02-24T23:25:39.155313Z",
+     "shell.execute_reply": "2023-02-24T23:25:39.154731Z"
+    },
+    "slideshow": {
+     "slide_type": "subslide"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "def compare(game,guess):\n",
+      "    \"\"\"I think we all remember that feeling when the result of some long-awaited\n",
+      "    event is finally known. The feelings and thoughts you have at that moment are\n",
+      "    definitely worth noting down and comparing.\n",
+      "    Your task is to determine if a person correctly guessed the results of a number of matches.\n",
+      "    You are given two arrays of scores and guesses of equal length, where each index shows a match. \n",
+      "    Return an array of the same length denoting how far off each guess was. If they have guessed correctly,\n",
+      "    the value is 0, and if not, the value is the absolute difference between the guess and the score.\n",
+      "    \n",
+      "    \n",
+      "    example:\n",
+      "\n",
+      "    compare([1,2,3,4,5,1],[1,2,3,4,2,-2]) -> [0,0,0,0,3,3]\n",
+      "    compare([0,5,0,0,0,4],[4,1,1,0,0,-2]) -> [4,4,1,0,0,6]\n",
+      "    \"\"\"\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(tune_data[1][\"definition\"])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here is one example of the unit test code for verifying the correctness of the generated code:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:39.158398Z",
+     "iopub.status.busy": "2023-02-24T23:25:39.157766Z",
+     "iopub.status.idle": "2023-02-24T23:25:39.161396Z",
+     "shell.execute_reply": "2023-02-24T23:25:39.160797Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "def check(candidate):\n",
+      "\n",
+      "    # Check some simple cases\n",
+      "    assert candidate([1,2,3,4,5,1],[1,2,3,4,2,-2])==[0,0,0,0,3,3], \"This prints if this assert fails 1 (good for debugging!)\"\n",
+      "    assert candidate([0,0,0,0,0,0],[0,0,0,0,0,0])==[0,0,0,0,0,0], \"This prints if this assert fails 1 (good for debugging!)\"\n",
+      "    assert candidate([1,2,3],[-1,-2,-3])==[2,4,6], \"This prints if this assert fails 1 (good for debugging!)\"\n",
+      "    assert candidate([1,2,3,5],[-1,2,3,4])==[2,0,0,1], \"This prints if this assert fails 1 (good for debugging!)\"\n",
+      "\n",
+      "    # Check some edge cases that are easy to work out by hand.\n",
+      "    assert True, \"This prints if this assert fails 2 (also good for debugging!)\"\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(tune_data[1][\"test\"])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define Success Metric\n",
+    "\n",
+    "Before we start tuning, we need to define the success metric we want to optimize. For each code generation task, we can use the model to generate multiple candidates, and then select one from them. If the final selected response can pass a unit test, we consider the task as successfully solved. Then we can define the mean success rate of a collection of tasks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:39.164187Z",
+     "iopub.status.busy": "2023-02-24T23:25:39.163867Z",
+     "iopub.status.idle": "2023-02-24T23:25:39.169009Z",
+     "shell.execute_reply": "2023-02-24T23:25:39.168427Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from functools import partial\n",
+    "\n",
+    "eval_with_generated_assertions = partial(\n",
+    "    autogen.code_utils.eval_function_completions,\n",
+    "    assertions=partial(autogen.code_utils.generate_assertions, config_list=config_list),\n",
+    "    use_docker=False,\n",
+    "    # Please set use_docker=True if you have docker available to run the generated code.\n",
+    "    # Using docker is safer than running the generated code directly.\n",
+    ")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "This function will first generate assertion statements for each problem. Then, it uses the assertions to select the generated responses.\n",
+    "\n",
+    "## Use the tuning data to find a good configuration\n",
+    "\n",
+    "AutoGen has provided an API for hyperparameter optimization of OpenAI models: `autogen.Completion.tune` and to make a request with the tuned config: `autogen.Completion.create`.\n",
+    "\n",
+    "For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:40.587815Z",
+     "iopub.status.busy": "2023-02-24T23:25:40.587283Z",
+     "iopub.status.idle": "2023-02-24T23:25:40.590826Z",
+     "shell.execute_reply": "2023-02-24T23:25:40.590158Z"
+    },
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "autogen.Completion.set_cache(seed)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This will create a disk cache in \".cache/{seed}\". You can change `cache_path_root` from \".cache\" to a different path in `set_cache()`. The cache for different seeds are stored separately.\n",
+    "\n",
+    "### Perform tuning\n",
+    "\n",
+    "The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
+    "\n",
+    "* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.02 means the target inference budget is 0.02 dollars, which translates to 1000 tokens (input + output combined) if the text Davinci model is used.\n",
+    "* `optimization_budget` is the total budget allowed to perform the tuning. For example, 5 means 5 dollars are allowed in total, which translates to 250K tokens for the text Davinci model.\n",
+    "* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
+    "\n",
+    "Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
+    "\n",
+    "```python\n",
+    "default_search_space = {\n",
+    "    \"model\": tune.choice([\n",
+    "        \"text-ada-001\",\n",
+    "        \"text-babbage-001\",\n",
+    "        \"text-davinci-003\",\n",
+    "        \"gpt-3.5-turbo\",\n",
+    "        \"gpt-4\",\n",
+    "    ]),\n",
+    "    \"temperature_or_top_p\": tune.choice(\n",
+    "        [\n",
+    "            {\"temperature\": tune.uniform(0, 1)},\n",
+    "            {\"top_p\": tune.uniform(0, 1)},\n",
+    "        ]\n",
+    "    ),\n",
+    "    \"max_tokens\": tune.lograndint(50, 1000),\n",
+    "    \"n\": tune.randint(1, 100),\n",
+    "    \"prompt\": \"{prompt}\",\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "The default search space can be overridden by users' input.\n",
+    "For example, the following code specifies three choices for the prompt and two choices of stop sequences. For hyperparameters which don't appear in users' input, the default search space will be used. If you don't have access to gpt-4 or would like to modify the choice of models, you can provide a different search space for model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:25:40.593603Z",
+     "iopub.status.busy": "2023-02-24T23:25:40.593269Z",
+     "iopub.status.idle": "2023-02-24T23:26:38.349191Z",
+     "shell.execute_reply": "2023-02-24T23:26:38.348392Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\u001b[32m[I 2023-07-30 04:19:08,150]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
+      "\u001b[32m[I 2023-07-30 04:19:08,153]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[flaml.tune.tune: 07-30 04:19:08] {805} INFO - trial 1 config: {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-ada-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}\n",
+      "[flaml.tune.tune: 07-30 04:22:35] {197} INFO - result: {'index_selected': 26.0, 'succeed_assertions': 0.0, 'success': 0.0, 'gen_cost': 0.000460625, 'assertions': 'assert vowels_count(\"abcde\") == 2\\nassert vowels_count(\"ACEDY\") == 3', 'total_cost': 0.010514800000000003, 'cost': 0.010514800000000003, 'inference_cost': 0.00023534000000000003, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-ada-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'text-ada-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 207.29033374786377}\n",
+      "[flaml.tune.tune: 07-30 04:22:35] {805} INFO - trial 2 config: {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-babbage-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}\n",
+      "[flaml.tune.tune: 07-30 04:23:18] {197} INFO - result: {'index_selected': 26.0, 'succeed_assertions': 0.0, 'success': 0.0, 'gen_cost': 0.000460625, 'assertions': 'assert vowels_count(\"abcde\") == 2\\nassert vowels_count(\"ACEDY\") == 3', 'total_cost': 0.0300243, 'cost': 0.019509500000000003, 'inference_cost': 0.0009754750000000001, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-babbage-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'text-babbage-001', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 42.417603969573975}\n",
+      "[flaml.tune.tune: 07-30 04:23:18] {805} INFO - trial 3 config: {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-davinci-003', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}\n",
+      "[flaml.tune.tune: 07-30 04:24:20] {197} INFO - result: {'index_selected': 2.35, 'succeed_assertions': 0.95, 'success': 0.65, 'gen_cost': 0.000460625, 'assertions': 'assert vowels_count(\"abcde\") == 2\\nassert vowels_count(\"ACEDY\") == 3', 'total_cost': 0.8658043000000002, 'cost': 0.8357800000000002, 'inference_cost': 0.04093000000000001, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-davinci-003', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'text-davinci-003', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 62.81497287750244}\n",
+      "[flaml.tune.tune: 07-30 04:24:20] {805} INFO - trial 4 config: {'prompt': 1, 'stop': 0, 'subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}\n",
+      "[flaml.tune.tune: 07-30 04:25:39] {197} INFO - result: {'index_selected': 13.95, 'succeed_assertions': 0.55, 'success': 0.5, 'gen_cost': 0.000460625, 'assertions': 'assert vowels_count(\"abcde\") == 2\\nassert vowels_count(\"ACEDY\") == 3', 'total_cost': 0.9462703000000001, 'cost': 0.08046600000000001, 'inference_cost': 0.00399515, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 79.03474521636963}\n",
+      "[flaml.tune.tune: 07-30 04:25:39] {805} INFO - trial 5 config: {'prompt': 1, 'stop': 0, 'subspace': {'model': 'gpt-4', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}\n",
+      "[flaml.tune.tune: 07-30 04:25:50] {197} INFO - result: {'success': 0, 'total_cost': 1.0053703, 'cost': 0.0591, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'gpt-4', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'gpt-4', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 10.245523691177368}\n",
+      "[flaml.tune.tune: 07-30 04:25:50] {828} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
+     ]
+    }
+   ],
+   "source": [
+    "config, analysis = autogen.Completion.tune(\n",
+    "    data=tune_data,  # the data for tuning\n",
+    "    metric=\"success\",  # the metric to optimize\n",
+    "    mode=\"max\",  # the optimization mode\n",
+    "    eval_func=eval_with_generated_assertions,  # the evaluation function to return the success metrics\n",
+    "    # log_file_name=\"logs/humaneval.log\",  # the log file name\n",
+    "    inference_budget=0.05,  # the inference budget (dollar per instance)\n",
+    "    optimization_budget=1,  # the optimization budget (dollar in total)\n",
+    "    # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
+    "    # -1 means decided by the optimization budget only\n",
+    "    num_samples=-1,\n",
+    "    prompt=[\n",
+    "        \"{definition}\",\n",
+    "        \"# Python 3{definition}\",\n",
+    "        \"Complete the following Python function:{definition}\",\n",
+    "    ],  # the prompt templates to choose from\n",
+    "    stop=[[\"\\nclass\", \"\\ndef\", \"\\nif\", \"\\nprint\"], None],  # the stop sequences\n",
+    "    config_list=endpoint_list,  # optional: a list of endpoints to use\n",
+    "    allow_format_str_template=True,  # whether to allow format string template\n",
+    ")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Output tuning results\n",
+    "\n",
+    "After the tuning, we can print out the config and the result found by autogen:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:26:38.352710Z",
+     "iopub.status.busy": "2023-02-24T23:26:38.352378Z",
+     "iopub.status.idle": "2023-02-24T23:26:38.356939Z",
+     "shell.execute_reply": "2023-02-24T23:26:38.356217Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "optimized config {'prompt': '# Python 3{definition}', 'stop': ['\\nclass', '\\ndef', '\\nif', '\\nprint'], 'model': 'text-davinci-003', 'max_tokens': 148, 'n': 27, 'top_p': 0.755486898036596}\n",
+      "best result on tuning data {'index_selected': 2.35, 'succeed_assertions': 0.95, 'success': 0.65, 'gen_cost': 0.000460625, 'assertions': 'assert vowels_count(\"abcde\") == 2\\nassert vowels_count(\"ACEDY\") == 3', 'total_cost': 0.8658043000000002, 'cost': 0.8357800000000002, 'inference_cost': 0.04093000000000001, 'training_iteration': 0, 'config': {'prompt': 1, 'stop': 0, 'subspace': {'model': 'text-davinci-003', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}}, 'config/prompt': 1, 'config/stop': 0, 'config/subspace': {'model': 'text-davinci-003', 'max_tokens': 148, 'temperature_or_top_p': {'top_p': 0.755486898036596}, 'n': 27}, 'experiment_tag': 'exp', 'time_total_s': 62.81497287750244}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"optimized config\", config)\n",
+    "print(\"best result on tuning data\", analysis.best_result)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "### Make a request with the tuned config\n",
+    "\n",
+    "We can apply the tuned config on the request for an example task:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:26:38.359902Z",
+     "iopub.status.busy": "2023-02-24T23:26:38.359506Z",
+     "iopub.status.idle": "2023-02-24T23:26:39.343921Z",
+     "shell.execute_reply": "2023-02-24T23:26:39.343051Z"
+    },
+    "slideshow": {
+     "slide_type": "subslide"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "  \"id\": \"cmpl-7hsFhPX6faeWYaT4y3C7IkQAgNbZR\",\n",
+      "  \"warning\": \"This model version is deprecated. Migrate before January 4, 2024 to avoid disruption of service. Learn more https://platform.openai.com/docs/deprecations\",\n",
+      "  \"object\": \"text_completion\",\n",
+      "  \"created\": 1690691005,\n",
+      "  \"model\": \"text-davinci-003\",\n",
+      "  \"choices\": [\n",
+      "    {\n",
+      "      \"text\": \"    results = []\\n    for i in range(len(game)):\\n        if game[i] == guess[i]:\\n            results.append(0)\\n        else:\\n            results.append(abs(game[i]-guess[i]))\\n    return results\",\n",
+      "      \"index\": 0,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 1,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i]-guess[i]))\\n    return result\",\n",
+      "      \"index\": 2,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    res = []\\n    for i in range(len(game)):\\n        res.append(abs(game[i]-guess[i]))\\n    return res\",\n",
+      "      \"index\": 3,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 4,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 5,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 6,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    results = []\\n    for i in range(len(game)):\\n        results.append(abs(game[i] - guess[i]))\\n    return results\",\n",
+      "      \"index\": 7,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    res = []\\n    for i in range(len(game)):\\n        res.append(abs(game[i]-guess[i]))\\n    return res\",\n",
+      "      \"index\": 8,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i]-guess[i]))\\n    return result\",\n",
+      "      \"index\": 9,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 10,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 11,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        if game[i] == guess[i]:\\n            result.append(0)\\n        else:\\n            result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 12,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    # set up empty list to store differences\\n    diff = []\\n    # iterate through the game list and guess list\\n    for i in range(len(game)):\\n        # check if the guess is equal to the game\\n        if game[i] == guess[i]:\\n            # if so, append 0 to the diff list\\n            diff.append(0)\\n        # otherwise, calculate the difference between the guess and the game\\n        else:\\n            diff.append(abs(game[i]-guess[i]))\\n    # return the diff list\\n    return diff\",\n",
+      "      \"index\": 13,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i]-guess[i]))\\n    return result\",\n",
+      "      \"index\": 14,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        if game[i] == guess[i]:\\n            result.append(0)\\n        else:\\n            result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 15,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 16,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 17,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 18,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 19,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 20,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 21,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 22,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    # your code here\\n    result = []\\n    for i in range(len(game)):\\n        if game[i] == guess[i]:\\n            result.append(0)\\n        else:\\n            result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 23,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    results = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        results.append(diff)\\n    return results\",\n",
+      "      \"index\": 24,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        diff = abs(game[i] - guess[i])\\n        result.append(diff)\\n    return result\",\n",
+      "      \"index\": 25,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    },\n",
+      "    {\n",
+      "      \"text\": \"    result = []\\n    for i in range(len(game)):\\n        result.append(abs(game[i] - guess[i]))\\n    return result\",\n",
+      "      \"index\": 26,\n",
+      "      \"logprobs\": null,\n",
+      "      \"finish_reason\": \"stop\"\n",
+      "    }\n",
+      "  ],\n",
+      "  \"usage\": {\n",
+      "    \"prompt_tokens\": 243,\n",
+      "    \"completion_tokens\": 1264,\n",
+      "    \"total_tokens\": 1507\n",
+      "  },\n",
+      "  \"cost\": 0.03014,\n",
+      "  \"config_id\": 0,\n",
+      "  \"pass_filter\": true\n",
+      "}\n",
+      "{'index_selected': 0, 'succeed_assertions': True, 'success': True, 'gen_cost': 0.000702, 'assertions': 'assert compare([1,2,3,4,5,1],[1,2,3,4,2,-2]) == [0,0,0,0,3,3]\\nassert compare([0,5,0,0,0,4],[4,1,1,0,0,-2]) == [4,4,1,0,0,6]'}\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = autogen.Completion.create(context=tune_data[1], config_list=endpoint_list, **config)\n",
+    "print(response)\n",
+    "print(eval_with_generated_assertions(autogen.Completion.extract_text(response), **tune_data[1]))\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Evaluate the success rate on the test data\n",
+    "\n",
+    "You can use `autogen.Completion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while to evaluate all the 144 test data instances. The cost is about $6 if you uncomment it and run it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2023-02-24T23:26:39.347295Z",
+     "iopub.status.busy": "2023-02-24T23:26:39.346994Z",
+     "iopub.status.idle": "2023-02-24T23:29:27.160335Z",
+     "shell.execute_reply": "2023-02-24T23:29:27.159519Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "performance on test data with the tuned config: {'index_selected': 5.222222222222222, 'succeed_assertions': 0.8402777777777778, 'success': 0.7569444444444444, 'gen_cost': 0.00044632638888888885, 'cost': 5.704979999999999, 'inference_cost': 0.03961791666666666}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# result = autogen.Completion.test(test_data, config_list=endpoint_list, **config)\n",
+    "# print(\"performance on test data with the tuned config:\", result)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The result will vary with the inference budget and optimization budget.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "24dd93300e0442788ee6cc1310e5bf14": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "35cd066a31b242bb87b2c106ee72e5f2": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_8e7ee7687a99410d88a98a74ecfcea99",
+        "IPY_MODEL_421e02a11a974b40b3ddb75382b3b640",
+        "IPY_MODEL_77db9797e78b49438d21c5c8da34b4cb"
+       ],
+       "layout": "IPY_MODEL_47d3046236a54b0e8f9ae455a82c7e0b",
+       "tabbable": null,
+       "tooltip": null
+      }
+     },
+     "3d5d106a38954af2bb3bde5777702f4e": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "background": null,
+       "description_width": "",
+       "font_size": null,
+       "text_color": null
+      }
+     },
+     "3e1ebb31412443b0bca86a301cbdac11": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "421e02a11a974b40b3ddb75382b3b640": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_e6398d4027c9459a97965b9d91ae484f",
+       "max": 1,
+       "min": 0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_3e1ebb31412443b0bca86a301cbdac11",
+       "tabbable": null,
+       "tooltip": null,
+       "value": 1
+      }
+     },
+     "47d3046236a54b0e8f9ae455a82c7e0b": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "754800f7feb04acea977696e4787d1ff": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "77db9797e78b49438d21c5c8da34b4cb": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_7b6c4e1c11e249409a1edcd63be450d8",
+       "placeholder": "​",
+       "style": "IPY_MODEL_3d5d106a38954af2bb3bde5777702f4e",
+       "tabbable": null,
+       "tooltip": null,
+       "value": " 1/1 [00:00&lt;00:00, 44.40it/s]"
+      }
+     },
+     "7b6c4e1c11e249409a1edcd63be450d8": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "8e7ee7687a99410d88a98a74ecfcea99": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "2.0.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "2.0.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "2.0.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_allow_html": false,
+       "layout": "IPY_MODEL_754800f7feb04acea977696e4787d1ff",
+       "placeholder": "​",
+       "style": "IPY_MODEL_24dd93300e0442788ee6cc1310e5bf14",
+       "tabbable": null,
+       "tooltip": null,
+       "value": "100%"
+      }
+     },
+     "e6398d4027c9459a97965b9d91ae484f": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "2.0.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "2.0.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "2.0.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border_bottom": null,
+       "border_left": null,
+       "border_right": null,
+       "border_top": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}