update 1

.gitattributes

@@ -0,0 +1,2 @@
+*.lance filter=lfs diff=lfs merge=lfs -text
+*.idx filter=lfs diff=lfs merge=lfs -text

Copy_of_rag_homework_fin.ipynb

@@ -0,0 +1,1315 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "844fe3af-9cf1-4c66-aa78-b88a3429acc6",
+      "metadata": {
+        "id": "844fe3af-9cf1-4c66-aa78-b88a3429acc6"
+      },
+      "source": [
+        "### 0. Setup\n",
+        "1) Clone https://github.com/plaggy/rag-gradio-sample-project and set up an environment with gradio_app/requirements.txt.\n",
+        "\n",
+        "There you'll find the following files:\n",
+        "- [prep_scripts/markdown_to_text.py](https://github.com/plaggy/rag-gradio-sample-project/blob/main/prep_scripts/markdown_to_text.py) processes markdown into text; you won't need to change it.\n",
+        "- [prep_scripts/lancedb_setup.py](https://github.com/plaggy/rag-gradio-sample-project/blob/main/prep_scripts/lancedb_setup.py) is the file where the database is created and, in particular, an embedding model is defined.\n",
+        "- [gradio_app/backend/query_llm.py](https://github.com/plaggy/rag-gradio-sample-project/blob/main/gradio_app/backend/query_llm.py) defines what LLM is used.\n",
+        "- [gradio_app/app.py](https://github.com/plaggy/rag-gradio-sample-project/blob/main/gradio_app/app.py) creates the gradio app.\n",
+        "\n",
+        "In this task you'll try not only OpenAI models, but also open-source models from Hugging Face Hub through InferenceClient interface (see [gradio_app/backend/query_llm.py](https://github.com/plaggy/rag-gradio-sample-project/blob/main/gradio_app/backend/query_llm.py)). Please don't forget to obtain a Hugging Face token for that (see here https://huggingface.co/settings/tokens).\n",
+        "\n",
+        "\n",
+        "A convenient way to work through the project is to test locally and keep committing the changes to the [HF Spaces](https://huggingface.co/spaces) repo. A space gets automatically rebuilt after each commit and you get a new version of your application up and running.\n",
+        "\n",
+        "2) Create a new space with Gradio SDK. You'll get an almost empty repo, the only thing you'll need from it is README.md which has a config letting a space builder know that it's a Gradio app. Reset a remote upstream of your local rag-gradio-sample-project clone to be your freshly created Spaces repository.\n",
+        "\n",
+        "The easiest way to set your space up is to set up the gradio_app folder as a git repo, set remote origin to your space repo and checkout the remote README:\n",
+        "\n",
+        "```\n",
+        "cd gradio_app\n",
+        "git init\n",
+        "git remote add origin <your spaces repo url>\n",
+        "git fetch\n",
+        "git checkout origin/main README.md\n",
+        "```\n",
+        "\n",
+        "The space is not working yet. You'll get the first working version after the Step 3.\n",
+        "\n",
+        "- Clone https://github.com/huggingface/transformers to a local machine and run prep_scripts/markdown_to_text.py script to extract raw text from transformers/docs/source/en/. This will be your knowledge base, you don't need it to be a part of your repository\n",
+        "\n",
+        "Run the command as follows (pass arguments that work for you)\n",
+        "```\n",
+        "python prep_scripts/markdown_to_text.py --input-dir transformers/docs/source/en/ --output-dir docs\n",
+        "```\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "762e9fde-c1f4-464c-b12b-dca602fac5ba",
+      "metadata": {
+        "id": "762e9fde-c1f4-464c-b12b-dca602fac5ba"
+      },
+      "source": [
+        "**By design, you'll be running your experiments in a [Gradio space](https://huggingface.co/docs/hub/en/spaces-sdks-gradio). Apart from deliverables for each step you'll need to provide a link to a functioning RAG space in it final state!**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!git clone https://github.com/plaggy/rag-gradio-sample-project"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "BUHKUeqR7unC",
+        "outputId": "92617e28-da69-45e3-b34e-2b88876ae3dd"
+      },
+      "id": "BUHKUeqR7unC",
+      "execution_count": 1,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Cloning into 'rag-gradio-sample-project'...\n",
+            "remote: Enumerating objects: 73, done.\u001b[K\n",
+            "remote: Counting objects: 100% (73/73), done.\u001b[K\n",
+            "remote: Compressing objects: 100% (59/59), done.\u001b[K\n",
+            "remote: Total 73 (delta 23), reused 57 (delta 14), pack-reused 0\u001b[K\n",
+            "Receiving objects: 100% (73/73), 31.10 KiB | 10.37 MiB/s, done.\n",
+            "Resolving deltas: 100% (23/23), done.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install -r /content/rag-gradio-sample-project/gradio_app/requirements.txt"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "FFIvgBYDcVMt",
+        "outputId": "3c53faf0-f87e-4d19-bbac-90401cc70b71"
+      },
+      "id": "FFIvgBYDcVMt",
+      "execution_count": 2,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Collecting lancedb==0.5.3 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading lancedb-0.5.3-py3-none-any.whl (106 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m107.0/107.0 kB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting openai==1.11.1 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2))\n",
+            "  Downloading openai-1.11.1-py3-none-any.whl (226 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.1/226.1 kB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting sentence-transformers==2.3.1 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3))\n",
+            "  Downloading sentence_transformers-2.3.1-py3-none-any.whl (132 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.8/132.8 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting tqdm==4.66.1 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 4))\n",
+            "  Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting torch==2.1.1 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading torch-2.1.1-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m670.2/670.2 MB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting transformers==4.37.2 (from -r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 6))\n",
+            "  Downloading transformers-4.37.2-py3-none-any.whl (8.4 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m89.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting deprecation (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB)\n",
+            "Collecting pylance==0.9.12 (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading pylance-0.9.12-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21.4 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.4/21.4 MB\u001b[0m \u001b[31m68.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting ratelimiter~=1.0 (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading ratelimiter-1.2.0.post0-py3-none-any.whl (6.6 kB)\n",
+            "Collecting retry>=0.9.2 (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading retry-0.9.2-py2.py3-none-any.whl (8.0 kB)\n",
+            "Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (2.6.1)\n",
+            "Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (23.2.0)\n",
+            "Collecting semver>=3.0 (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading semver-3.0.2-py3-none-any.whl (17 kB)\n",
+            "Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (5.3.2)\n",
+            "Requirement already satisfied: pyyaml>=6.0 in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (6.0.1)\n",
+            "Requirement already satisfied: click>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (8.1.7)\n",
+            "Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (2.31.0)\n",
+            "Collecting overrides>=0.7 (from lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading overrides-7.7.0-py3-none-any.whl (17 kB)\n",
+            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (3.7.1)\n",
+            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (1.7.0)\n",
+            "Collecting httpx<1,>=0.23.0 (from openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2))\n",
+            "  Downloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (1.3.0)\n",
+            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (4.9.0)\n",
+            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (1.25.2)\n",
+            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (1.2.2)\n",
+            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (1.11.4)\n",
+            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (3.8.1)\n",
+            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (0.1.99)\n",
+            "Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (0.20.3)\n",
+            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (9.4.0)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (3.13.1)\n",
+            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (1.12)\n",
+            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (3.2.1)\n",
+            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (3.1.3)\n",
+            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (2023.6.0)\n",
+            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m59.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m43.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m80.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m764.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-curand-cu12==10.3.2.106 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m209.8/209.8 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting nvidia-nvtx-cu12==12.1.105 (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (2.1.0)\n",
+            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.2->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 6)) (23.2)\n",
+            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.2->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 6)) (2023.12.25)\n",
+            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.2->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 6)) (0.15.2)\n",
+            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.2->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 6)) (0.4.2)\n",
+            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5))\n",
+            "  Downloading nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (20.5 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m72.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting pyarrow>=12 (from pylance==0.9.12->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (38.3 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.3/38.3 MB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (3.6)\n",
+            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (1.2.0)\n",
+            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2)) (2024.2.2)\n",
+            "Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2))\n",
+            "  Downloading httpcore-1.0.3-py3-none-any.whl (77 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.0/77.0 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai==1.11.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 2))\n",
+            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (0.6.0)\n",
+            "Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (2.16.2)\n",
+            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (3.3.2)\n",
+            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (2.0.7)\n",
+            "Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1)) (4.4.2)\n",
+            "Collecting py<2.0.0,>=1.4.26 (from retry>=0.9.2->lancedb==0.5.3->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 1))\n",
+            "  Downloading py-1.11.0-py2.py3-none-any.whl (98 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (2.1.5)\n",
+            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (1.3.2)\n",
+            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers==2.3.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 3)) (3.2.0)\n",
+            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.1.1->-r /content/rag-gradio-sample-project/gradio_app/requirements.txt (line 5)) (1.3.0)\n",
+            "Installing collected packages: ratelimiter, tqdm, semver, pyarrow, py, overrides, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, h11, deprecation, retry, pylance, nvidia-cusparse-cu12, nvidia-cudnn-cu12, httpcore, nvidia-cusolver-cu12, lancedb, httpx, transformers, torch, openai, sentence-transformers\n",
+            "  Attempting uninstall: tqdm\n",
+            "    Found existing installation: tqdm 4.66.2\n",
+            "    Uninstalling tqdm-4.66.2:\n",
+            "      Successfully uninstalled tqdm-4.66.2\n",
+            "  Attempting uninstall: pyarrow\n",
+            "    Found existing installation: pyarrow 10.0.1\n",
+            "    Uninstalling pyarrow-10.0.1:\n",
+            "      Successfully uninstalled pyarrow-10.0.1\n",
+            "  Attempting uninstall: transformers\n",
+            "    Found existing installation: transformers 4.35.2\n",
+            "    Uninstalling transformers-4.35.2:\n",
+            "      Successfully uninstalled transformers-4.35.2\n",
+            "  Attempting uninstall: torch\n",
+            "    Found existing installation: torch 2.1.0+cu121\n",
+            "    Uninstalling torch-2.1.0+cu121:\n",
+            "      Successfully uninstalled torch-2.1.0+cu121\n",
+            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+            "llmx 0.0.15a0 requires cohere, which is not installed.\n",
+            "llmx 0.0.15a0 requires tiktoken, which is not installed.\n",
+            "ibis-framework 7.1.0 requires pyarrow<15,>=2, but you have pyarrow 15.0.0 which is incompatible.\n",
+            "torchaudio 2.1.0+cu121 requires torch==2.1.0, but you have torch 2.1.1 which is incompatible.\n",
+            "torchdata 0.7.0 requires torch==2.1.0, but you have torch 2.1.1 which is incompatible.\n",
+            "torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.1.1 which is incompatible.\n",
+            "torchvision 0.16.0+cu121 requires torch==2.1.0, but you have torch 2.1.1 which is incompatible.\u001b[0m\u001b[31m\n",
+            "\u001b[0mSuccessfully installed deprecation-2.1.0 h11-0.14.0 httpcore-1.0.3 httpx-0.26.0 lancedb-0.5.3 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 openai-1.11.1 overrides-7.7.0 py-1.11.0 pyarrow-15.0.0 pylance-0.9.12 ratelimiter-1.2.0.post0 retry-0.9.2 semver-3.0.2 sentence-transformers-2.3.1 torch-2.1.1 tqdm-4.66.1 transformers-4.37.2\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install huggingface_hub"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "uVHHnyoedIPy",
+        "outputId": "527c17ae-a7db-45db-cf12-46cb07f90342"
+      },
+      "id": "uVHHnyoedIPy",
+      "execution_count": 3,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.10/dist-packages (0.20.3)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (3.13.1)\n",
+            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (2023.6.0)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (2.31.0)\n",
+            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (4.66.1)\n",
+            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (6.0.1)\n",
+            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (4.9.0)\n",
+            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub) (23.2)\n",
+            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub) (3.3.2)\n",
+            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub) (3.6)\n",
+            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub) (2.0.7)\n",
+            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub) (2024.2.2)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!huggingface-cli login"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "8-M0jyfGdKYe",
+        "outputId": "c7f7d369-c51b-43e6-8aa4-182af93a7f4a"
+      },
+      "id": "8-M0jyfGdKYe",
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "\n",
+            "    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|\n",
+            "    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|\n",
+            "    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|\n",
+            "    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|\n",
+            "    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|\n",
+            "\n",
+            "    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .\n",
+            "Token: \n",
+            "Add token as git credential? (Y/n) \n",
+            "Token is valid (permission: read).\n",
+            "\u001b[1m\u001b[31mCannot authenticate through git-credential as no helper is defined on your machine.\n",
+            "You might have to re-authenticate when pushing to the Hugging Face Hub.\n",
+            "Run the following command in your terminal in case you want to set the 'store' credential helper as default.\n",
+            "\n",
+            "git config --global credential.helper store\n",
+            "\n",
+            "Read https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage for more details.\u001b[0m\n",
+            "Token has not been saved to git credential helper.\n",
+            "Your token has been saved to /root/.cache/huggingface/token\n",
+            "Login successful\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "%cd rag-gradio-sample-project/gradio_app/\n",
+        "%ls"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "HjXOD1nH1fx5",
+        "outputId": "874e8e62-8730-47e2-b185-c7c1e9cc6cfe"
+      },
+      "id": "HjXOD1nH1fx5",
+      "execution_count": 5,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "/content/rag-gradio-sample-project/gradio_app\n",
+            "app.py  \u001b[0m\u001b[01;34mbackend\u001b[0m/  requirements.txt  \u001b[01;34mtemplates\u001b[0m/\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "%pwd"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 36
+        },
+        "id": "55IyOwgr1kNR",
+        "outputId": "6235574d-e278-40eb-f389-bdae96090556"
+      },
+      "id": "55IyOwgr1kNR",
+      "execution_count": 6,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'/content/rag-gradio-sample-project/gradio_app'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 6
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!git init\n",
+        "!git remote add origin https://huggingface.co/spaces/Ahmadzei/RAG\n",
+        "!git config --global init.defaultBranch main\n",
+        "!git fetch\n",
+        "!git checkout origin/main README.md"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "1wY0VaL-9c14",
+        "outputId": "27d6b540-2d9a-4dee-d397-25601878c187"
+      },
+      "id": "1wY0VaL-9c14",
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "\u001b[33mhint: Using 'master' as the name for the initial branch. This default branch name\u001b[m\n",
+            "\u001b[33mhint: is subject to change. To configure the initial branch name to use in all\u001b[m\n",
+            "\u001b[33mhint: of your new repositories, which will suppress this warning, call:\u001b[m\n",
+            "\u001b[33mhint: \u001b[m\n",
+            "\u001b[33mhint: \tgit config --global init.defaultBranch <name>\u001b[m\n",
+            "\u001b[33mhint: \u001b[m\n",
+            "\u001b[33mhint: Names commonly chosen instead of 'master' are 'main', 'trunk' and\u001b[m\n",
+            "\u001b[33mhint: 'development'. The just-created branch can be renamed via this command:\u001b[m\n",
+            "\u001b[33mhint: \u001b[m\n",
+            "\u001b[33mhint: \tgit branch -m <name>\u001b[m\n",
+            "Initialized empty Git repository in /content/rag-gradio-sample-project/gradio_app/.git/\n",
+            "remote: Enumerating objects: 4, done.\u001b[K\n",
+            "remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 4\u001b[K\n",
+            "Unpacking objects: 100% (4/4), 1.27 KiB | 1.27 MiB/s, done.\n",
+            "From https://huggingface.co/spaces/Ahmadzei/RAG\n",
+            " * [new branch]      main       -> origin/main\n",
+            "Updated 1 path from b4805fb\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!git clone https://github.com/huggingface/transformers"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "cgX7Aqujk37U",
+        "outputId": "6294a191-642f-41f2-bb07-0ce528fae8c2"
+      },
+      "id": "cgX7Aqujk37U",
+      "execution_count": 7,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Cloning into 'transformers'...\n",
+            "remote: Enumerating objects: 185037, done.\u001b[K\n",
+            "remote: Counting objects: 100% (1681/1681), done.\u001b[K\n",
+            "remote: Compressing objects: 100% (1231/1231), done.\u001b[K\n",
+            "remote: Total 185037 (delta 824), reused 742 (delta 374), pack-reused 183356\u001b[K\n",
+            "Receiving objects: 100% (185037/185037), 205.20 MiB | 19.65 MiB/s, done.\n",
+            "Resolving deltas: 100% (130045/130045), done.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# !python transformers/prep_scripts/markdown_to_text.py --input_dir transformers/docs/source/en/ --output_dir /content/knowledge_base/\n",
+        "!python /content/rag-gradio-sample-project/prep_scripts/markdown_to_text.py --input-dir /content/rag-gradio-sample-project/gradio_app/transformers/docs/source/en/ --output-dir /content/docs/"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "2NYMq3KIlMAz",
+        "outputId": "d24cd17b-2f77-4f3a-b8c0-449acd9b0f80"
+      },
+      "id": "2NYMq3KIlMAz",
+      "execution_count": 8,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "\r0it [00:00, ?it/s]/content/rag-gradio-sample-project/prep_scripts/markdown_to_text.py:22: DeprecationWarning: The 'text' argument to find()-type methods is deprecated. Use 'string' instead.\n",
+            "  text = ''.join(soup.findAll(text=True))\n",
+            "385it [00:06, 60.38it/s]\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "6c813d03-33a7-4ce1-836f-11afc541f291",
+      "metadata": {
+        "id": "6c813d03-33a7-4ce1-836f-11afc541f291"
+      },
+      "outputs": [],
+      "source": [
+        "# Add the link to the space you've just created here:\n",
+        "# https://huggingface.co/spaces/Ahmadzei/RAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "c970d0a4-fee8-48ac-9377-4a6def7712b2",
+      "metadata": {
+        "id": "c970d0a4-fee8-48ac-9377-4a6def7712b2"
+      },
+      "source": [
+        "### Step 1: Chunk Your Data\n",
+        "\n",
+        "To efficiently pull up documents relevant to a query from a knowledge base documents are embedded and stored as vectors. Documents in your knowledge base are not expected to fit into the context length of an embedding model (most have 512 token limit). Hence chunking your documents into smaller pieces is required. Take a deeper dive into why chunking is important and what are the options [here](https://www.pinecone.io/learn/chunking-strategies/).\n",
+        "\n",
+        "Your task is to implement and compare two chunking strategies: fixed-sized chunking and content-aware chunking. For content-aware you could split by sentences, paragraphs or in some other way that makes sense.\n",
+        "\n",
+        "The deliverables are:\n",
+        "- The code for chunk splitting"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f7bad8c8",
+      "metadata": {
+        "id": "f7bad8c8"
+      },
+      "outputs": [],
+      "source": [
+        "# Chunk splitting deliverables"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def fixed_size_chunking(text, chunk_size=512):\n",
+        "    \"\"\"\n",
+        "    Splits the text into fixed-sized chunks.\n",
+        "\n",
+        "    :param text: The input text to be chunked.\n",
+        "    :param chunk_size: The size of each chunk in number of characters.\n",
+        "    :return: A list of chunks.\n",
+        "    \"\"\"\n",
+        "    return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]\n"
+      ],
+      "metadata": {
+        "id": "n9qEj8jfvlPj"
+      },
+      "id": "n9qEj8jfvlPj",
+      "execution_count": 9,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def content_aware_chunking(text, max_chunk_size=512):\n",
+        "    \"\"\"\n",
+        "    Splits the text into content-aware chunks by sentences.\n",
+        "\n",
+        "    :param text: The input text to be chunked.\n",
+        "    :param max_chunk_size: The maximum size of each chunk in number of characters.\n",
+        "    :return: A list of chunks.\n",
+        "    \"\"\"\n",
+        "    sentences = text.split('. ')  # Simple sentence splitting, can be improved with NLP libraries\n",
+        "    chunks = []\n",
+        "    current_chunk = \"\"\n",
+        "\n",
+        "    for sentence in sentences:\n",
+        "        if len(current_chunk) + len(sentence) < max_chunk_size:\n",
+        "            current_chunk += sentence + \". \"\n",
+        "        else:\n",
+        "            chunks.append(current_chunk.strip())\n",
+        "            current_chunk = sentence + \". \"\n",
+        "    if current_chunk:\n",
+        "        chunks.append(current_chunk.strip())\n",
+        "\n",
+        "    return chunks"
+      ],
+      "metadata": {
+        "id": "DB5IlJAdL6Bq"
+      },
+      "id": "DB5IlJAdL6Bq",
+      "execution_count": 10,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import nltk\n",
+        "nltk.download('punkt')\n",
+        "from nltk.tokenize import sent_tokenize\n",
+        "\n",
+        "def nltk_chunking(text):\n",
+        "    \"\"\"\n",
+        "    Divide text into chunks based on sentences.\n",
+        "\n",
+        "    Args:\n",
+        "    text (str): The text to be chunked.\n",
+        "\n",
+        "    Returns:\n",
+        "    list of str: A list containing the text chunks (sentences).\n",
+        "    \"\"\"\n",
+        "    return sent_tokenize(text)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "8eYOiabGvl00",
+        "outputId": "abf76bf5-09cb-43f6-b40e-0fffcbf37b3a"
+      },
+      "id": "8eYOiabGvl00",
+      "execution_count": 11,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
+            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def paragraph_chunking(text):\n",
+        "    \"\"\"\n",
+        "    Divide text into chunks based on paragraphs.\n",
+        "\n",
+        "    Args:\n",
+        "    text (str): The text to be chunked.\n",
+        "\n",
+        "    Returns:\n",
+        "    list of str: A list containing the text chunks (paragraphs).\n",
+        "    \"\"\"\n",
+        "    return text.split('\\n\\n')"
+      ],
+      "metadata": {
+        "id": "Sk2M6tYmvosj"
+      },
+      "id": "Sk2M6tYmvosj",
+      "execution_count": 12,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import os\n",
+        "import glob\n",
+        "\n",
+        "def chunk_and_write_docs(input_dir, output_dir_fixed, output_dir_content_aware):\n",
+        "    # Ensure output directories exist\n",
+        "    os.makedirs(output_dir_fixed, exist_ok=True)\n",
+        "    os.makedirs(output_dir_content_aware, exist_ok=True)\n",
+        "\n",
+        "    # List all text files in the input directory\n",
+        "    file_paths = glob.glob(os.path.join(input_dir, '*.txt'))\n",
+        "\n",
+        "    for file_path in file_paths:\n",
+        "        # Read the content of the file\n",
+        "        with open(file_path, 'r', encoding='utf-8') as file:\n",
+        "            text_content = file.read()\n",
+        "\n",
+        "        # Generate chunks using both methods\n",
+        "        fixed_chunks = fixed_size_chunking(text_content)\n",
+        "        content_aware_chunks = content_aware_chunking(text_content)\n",
+        "\n",
+        "        # Extract base name without extension for use in chunk file names\n",
+        "        base_name = os.path.splitext(os.path.basename(file_path))[0]\n",
+        "\n",
+        "        # Fixed-size chunking\n",
+        "        fixed_chunk_dir = os.path.join(output_dir_fixed, base_name.replace('.txt', ''))\n",
+        "        os.makedirs(fixed_chunk_dir, exist_ok=True)\n",
+        "        for i, chunk in enumerate(fixed_chunks):\n",
+        "            with open(os.path.join(fixed_chunk_dir, f'chunk_{i}.txt'), 'w', encoding='utf-8') as chunk_file:\n",
+        "                chunk_file.write(chunk)\n",
+        "\n",
+        "        # Content-aware chunking\n",
+        "        content_aware_chunk_dir = os.path.join(output_dir_content_aware, base_name.replace('.txt', ''))\n",
+        "        os.makedirs(content_aware_chunk_dir, exist_ok=True)\n",
+        "        for i, chunk in enumerate(content_aware_chunks):\n",
+        "            with open(os.path.join(content_aware_chunk_dir, f'chunk_{i}.txt'), 'w', encoding='utf-8') as chunk_file:\n",
+        "                chunk_file.write(chunk)\n",
+        "\n",
+        "# Define input and output directories\n",
+        "input_dir = '/content/docs'\n",
+        "output_dir_fixed = '/content/chunked/fixed_size_chunking'\n",
+        "output_dir_content_aware = '/content/chunked/content_aware_chunking'\n",
+        "\n",
+        "# Process the documents\n",
+        "chunk_and_write_docs(input_dir, output_dir_fixed, output_dir_content_aware)\n",
+        "\n",
+        "# To indicate completion and the count of processed files\n",
+        "processed_files_count = len(glob.glob(os.path.join(input_dir, '*.txt')))\n",
+        "processed_files_count\n"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "FGDf40tqSK2C",
+        "outputId": "39033395-444e-4579-a387-1128ec73bc41"
+      },
+      "id": "FGDf40tqSK2C",
+      "execution_count": 13,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "381"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 13
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5e5ebaad-8d42-430c-b00b-18198cdb9ce8",
+      "metadata": {
+        "id": "5e5ebaad-8d42-430c-b00b-18198cdb9ce8"
+      },
+      "source": [
+        "### Step 2: Ingest chunks into a database and create an index\n",
+        "\n",
+        "Chunks need to be vectorized and made accessible to an LLM to enable semantic search with embedding models. A current industry standard is to use a vector database to store and retrieve texts both conveniently and efficiently. There are many products out there, we'll be using [LanceDB](https://lancedb.github.io/lancedb/). LanceDB is a young product, one way it stands out is that it's embedded - it's designed not to be a standalone service but rather a part of an application, more on this [here](https://lancedb.github.io/lancedb/basic/).\n",
+        "\n",
+        "Find more details on how different databases compare in [this](https://thedataquarry.com/tags/vector-db/) series of posts.\n",
+        "\n",
+        "Your task is to vectorize and ingest chunked documents into the database.\n",
+        "**For each chunking strategy from the previous step create a separate table with one of the embedding models. Compare the chunking strategies and choose one. Perform vectorization+ingestion with the second model only with one chunking strategy of your choice**.\n",
+        "Use prep_scrips/lancedb_setup.py to vectorize chunks and store vector representations along with raw text in a Lancedb instance. The script also creates an index for fast ANN retrieval (not really needed for this exercise but necessary at scale). Try different embedding models and see how results differ. The options are:\n",
+        "\n",
+        "- `sentence-transformers/all-MiniLM-L6-v2`: a light model, produces vectors of length 384\n",
+        "- `BAAI/bge-large-en-v1.5`: a much heavier model, embedding vector length is 1024\n",
+        "\n",
+        "Feel free to explore other embedding models and justify your choice.\n",
+        "For different embedding models and different chunking strategies create different tables in the database so you can easily switch between them and compare.\n",
+        "\n",
+        "Run the embedding+ingestion script as follows, make sure to look into the script and go over the arguments. Note that the number of sub-vectors for indexing must be a divisor of the model embedding size.\n",
+        "\n",
+        "```\n",
+        "python prep_scrips/lancedb_setup.py --emb-model <model name> --table <db table name> --input-dir <folder with chunked docs> --num-sub-vectors <a number which is a divisor of the embedding dim>\n",
+        "```\n",
+        "\n",
+        "Before committing to your space set up environment variables on the settings tab of your space, use `.env` as a ference list of all the things you can customize. Make sure to add HF_TOKEN and OPENAI_API_KEY as secrets.\n",
+        "Not all the parameters are required to set via environment variables, most have default values.\n",
+        "\n",
+        "*The database is expected to be in the `gradio_app` folder under `.lancedb`, make sure to move it there if was initialized elsewhere.* It can be parametrized but it's unnecessary here.\n",
+        "\n",
+        "To commit large files to Github use `git lfs`:\n",
+        "```\n",
+        "git lfs install\n",
+        "git lfs track \"*.lance\"\n",
+        "git lfs track \"*.idx\"\n",
+        "git add .gitattributes\n",
+        "```\n",
+        "Then proceed as usual.\n",
+        "\n",
+        "For experimenting you can easily switch between embedding models/tables by changing the values of the corresponding env variables in your space (`EMB_MODEL`, `TABLE_NAME`). Overall, every time you change the value of an environment variable a space gets automatically rebuilt.\n",
+        "\n",
+        "The deliverables are:\n",
+        "1. The illustration of how retrieved documents differ depending on the embedding model and the chunking strategy. You should create at least 3 tables: model_1 + chunking_strategy_1, model_1 + chunking_strategy_2, model_2 + chunking_strategy_<1 or 2>\n",
+        "2. The analysis of pros and cons of chunking strategies\n",
+        "3. The analysis of how retrieved document differ between embedding models (is one better than the other?)\n",
+        "4. The analysis of how the embedding time differs between models"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f7db282e-e03c-41de-9c03-54abf455481f",
+      "metadata": {
+        "id": "f7db282e-e03c-41de-9c03-54abf455481f"
+      },
+      "outputs": [],
+      "source": [
+        "# Embed documents with different chunking strategies and ingest into the database"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install lancedb openai pyarrow pandas numpy sentence-transformers"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "vrrCjCs3-lNy",
+        "outputId": "c1a20049-d733-4390-ef65-cd9df1c0109f"
+      },
+      "id": "vrrCjCs3-lNy",
+      "execution_count": 14,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Requirement already satisfied: lancedb in /usr/local/lib/python3.10/dist-packages (0.5.3)\n",
+            "Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (1.11.1)\n",
+            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (15.0.0)\n",
+            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)\n",
+            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.25.2)\n",
+            "Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.10/dist-packages (2.3.1)\n",
+            "Requirement already satisfied: deprecation in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.1.0)\n",
+            "Requirement already satisfied: pylance==0.9.12 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.9.12)\n",
+            "Requirement already satisfied: ratelimiter~=1.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (1.2.0.post0)\n",
+            "Requirement already satisfied: retry>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.9.2)\n",
+            "Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (4.66.1)\n",
+            "Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.6.1)\n",
+            "Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (23.2.0)\n",
+            "Requirement already satisfied: semver>=3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (3.0.2)\n",
+            "Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb) (5.3.2)\n",
+            "Requirement already satisfied: pyyaml>=6.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (6.0.1)\n",
+            "Requirement already satisfied: click>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from lancedb) (8.1.7)\n",
+            "Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.31.0)\n",
+            "Requirement already satisfied: overrides>=0.7 in /usr/local/lib/python3.10/dist-packages (from lancedb) (7.7.0)\n",
+            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
+            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n",
+            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.26.0)\n",
+            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.0)\n",
+            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai) (4.9.0)\n",
+            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
+            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4)\n",
+            "Requirement already satisfied: transformers<5.0.0,>=4.32.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.37.2)\n",
+            "Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (2.1.1)\n",
+            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.2.2)\n",
+            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.11.4)\n",
+            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (3.8.1)\n",
+            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.1.99)\n",
+            "Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.20.3)\n",
+            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (9.4.0)\n",
+            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (3.6)\n",
+            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.0)\n",
+            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2024.2.2)\n",
+            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.3)\n",
+            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (3.13.1)\n",
+            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (2023.6.0)\n",
+            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (23.2)\n",
+            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (0.6.0)\n",
+            "Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (2.16.2)\n",
+            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n",
+            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (3.3.2)\n",
+            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (2.0.7)\n",
+            "Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (4.4.2)\n",
+            "Requirement already satisfied: py<2.0.0,>=1.4.26 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (1.11.0)\n",
+            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (1.12)\n",
+            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.2.1)\n",
+            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.1.3)\n",
+            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
+            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
+            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
+            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (8.9.2.26)\n",
+            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.3.1)\n",
+            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.0.2.54)\n",
+            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (10.3.2.106)\n",
+            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.4.5.107)\n",
+            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.0.106)\n",
+            "Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.18.1)\n",
+            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
+            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.1.0)\n",
+            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence-transformers) (12.3.101)\n",
+            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence-transformers) (2023.12.25)\n",
+            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence-transformers) (0.15.2)\n",
+            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence-transformers) (0.4.2)\n",
+            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->sentence-transformers) (1.3.2)\n",
+            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (3.2.0)\n",
+            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence-transformers) (2.1.5)\n",
+            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11.0->sentence-transformers) (1.3.0)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Setting environment variables\n",
+        "os.environ['EMB_MODEL'] = 'sentence-transformers/all-MiniLM-L6-v2' #sentence-transformers/all-MiniLM-L6-v2: a light model, produces vectors of length 384 / BAAI/bge-large-en-v1.5: a much heavier model, embedding vector length is 1024\n",
+        "os.environ['TABLE_NAME'] = 'fixed_size_chunking' # fixed_size_chunking / content_aware_chunking\n",
+        "os.environ['INPUT_DIR'] = '/content/chunked/docs/fixed_size_chunking/'  # fixed_size_chunking / content_aware_chunking\n",
+        "os.environ['NUM_SUB_VECTORS'] = '12'"
+      ],
+      "metadata": {
+        "id": "o3TCdDIEYwk6"
+      },
+      "id": "o3TCdDIEYwk6",
+      "execution_count": 15,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "EMB_MODEL = os.getenv('EMB_MODEL')\n",
+        "TABLE_NAME = os.getenv('TABLE_NAME')\n",
+        "INPUT_DIR = os.getenv('INPUT_DIR')\n",
+        "NUM_SUB_VECTORS = os.getenv('NUM_SUB_VECTORS')"
+      ],
+      "metadata": {
+        "id": "1tVGE7JYZc3i"
+      },
+      "id": "1tVGE7JYZc3i",
+      "execution_count": 16,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "print(INPUT_DIR)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "uL8Gzk6TgLtK",
+        "outputId": "68c608cf-e685-45c6-fc5f-e51ba204c074"
+      },
+      "id": "uL8Gzk6TgLtK",
+      "execution_count": 17,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "/content/chunked/docs/fixed_size_chunking/\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!python /content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py --emb-model {EMB_MODEL} --table {TABLE_NAME} --input-dir {INPUT_DIR} --num-sub-vectors {NUM_SUB_VECTORS}"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "Xy1cyu7_zFgO",
+        "outputId": "89ade558-d3bf-4aab-9b29-35f72950a07d"
+      },
+      "id": "Xy1cyu7_zFgO",
+      "execution_count": 19,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
+            "/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
+            "  return self.fget.__get__(instance, owner)()\n",
+            "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: cpu\n",
+            "INFO:__main__:using cpu device\n",
+            "0it [00:00, ?it/s]\n",
+            "Traceback (most recent call last):\n",
+            "  File \"/content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py\", line 96, in <module>\n",
+            "    main()\n",
+            "  File \"/content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py\", line 88, in main\n",
+            "    tbl.create_index(\n",
+            "  File \"/usr/local/lib/python3.10/dist-packages/lancedb/table.py\", line 858, in create_index\n",
+            "    self._dataset.create_index(\n",
+            "  File \"/usr/local/lib/python3.10/dist-packages/lance/dataset.py\", line 1269, in create_index\n",
+            "    self._ds.create_index(column, index_type, name, replace, kwargs)\n",
+            "OSError: LanceError(Index): KMeans: can not train 256 centroids with 0 vectors, choose a smaller K (< 0) instead, /home/runner/work/lance/lance/rust/lance-index/src/vector/kmeans.rs:45:21\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Setting environment variables\n",
+        "os.environ['EMB_MODEL'] = 'sentence-transformers/all-MiniLM-L6-v2' #sentence-transformers/all-MiniLM-L6-v2: a light model, produces vectors of length 384 / BAAI/bge-large-en-v1.5: a much heavier model, embedding vector length is 1024\n",
+        "os.environ['TABLE_NAME'] = 'content_aware_chunking' # fixed_size_chunking / content_aware_chunking\n",
+        "os.environ['INPUT_DIR'] = '/content/chunked/docs/content_aware_chunking/'  # fixed_size_chunking / content_aware_chunking\n",
+        "os.environ['NUM_SUB_VECTORS'] = '12'"
+      ],
+      "metadata": {
+        "id": "t7aqMOI3bh2s"
+      },
+      "id": "t7aqMOI3bh2s",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "EMB_MODEL2 = os.getenv('EMB_MODEL')\n",
+        "TABLE_NAME2 = os.getenv('TABLE_NAME')\n",
+        "INPUT_DIR2 = os.getenv('INPUT_DIR')\n",
+        "NUM_SUB_VECTORS2 = os.getenv('NUM_SUB_VECTORS')"
+      ],
+      "metadata": {
+        "id": "Gk9ynF4Bbslu"
+      },
+      "id": "Gk9ynF4Bbslu",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!python /content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py --emb-model {EMB_MODEL2} --table {TABLE_NAME2} --input-dir {INPUT_DIR2} --num-sub-vectors {NUM_SUB_VECTORS2}"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "rc0n7a9zbwh2",
+        "outputId": "50251872-bad0-473b-9ac3-36ed6d7a2e5f"
+      },
+      "id": "rc0n7a9zbwh2",
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
+            "/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
+            "  return self.fget.__get__(instance, owner)()\n",
+            "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: cpu\n",
+            "INFO:__main__:using cpu device\n",
+            "0it [00:00, ?it/s]\n",
+            "Traceback (most recent call last):\n",
+            "  File \"/content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py\", line 100, in <module>\n",
+            "    main()\n",
+            "  File \"/content/rag-gradio-sample-project/prep_scripts/lancedb_setup.py\", line 92, in main\n",
+            "    tbl.create_index(\n",
+            "  File \"/usr/local/lib/python3.10/dist-packages/lancedb/table.py\", line 858, in create_index\n",
+            "    self._dataset.create_index(\n",
+            "  File \"/usr/local/lib/python3.10/dist-packages/lance/dataset.py\", line 1269, in create_index\n",
+            "    self._ds.create_index(column, index_type, name, replace, kwargs)\n",
+            "OSError: LanceError(Index): KMeans: can not train 256 centroids with 0 vectors, choose a smaller K (< 0) instead, /home/runner/work/lance/lance/rust/lance-index/src/vector/kmeans.rs:45:21\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!git lfs install\n",
+        "!git lfs track \"*.lance\"\n",
+        "!git lfs track \"*.idx\"\n",
+        "!git add .gitattributes\n",
+        "# Then commit and push as usual\n"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "3Mlmy4j7x9Ln",
+        "outputId": "c4940d06-37a5-4861-a101-d6cbf753b5d2"
+      },
+      "id": "3Mlmy4j7x9Ln",
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Updated git hooks.\n",
+            "Git LFS initialized.\n",
+            "Tracking \"*.lance\"\n",
+            "Tracking \"*.idx\"\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "7d818b4f-ba5a-4c81-b6d7-f3474c398d9c",
+      "metadata": {
+        "id": "7d818b4f-ba5a-4c81-b6d7-f3474c398d9c"
+      },
+      "source": [
+        "### Step 3: Add a reranker\n",
+        "\n",
+        "A reranker is a second-level model which produces similarity scores for pairs of (input query + retrieved document). Cross-encoders are conventionally used for reranking, their architecture is slightly different from retrieval models (more on it [here] and [here](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)). Cross-encoders are much more costly to run, therefore a retrieval model is used to get a few (dozens) highest-scoring items, and a reranker picks the best among these. The overall pipeline is similar to the recommender system industry standard: a light model retrieves top-n, a precise and heavy model reranks n to get top k, n is orders of magnitude larger than k.\n",
+        "\n",
+        "Cross-encoders are optional because of the overhead their usage implies. Your task is to implement a reranker using a cross-encoder and assess pros and cons of having it. Do not forget that the process of pulling the most relevant documents becomes two-staged: retrieve a larger number of items first, than rerank and keep the best top-k for context.\n",
+        "\n",
+        "The models fit for the task:\n",
+        "1. BAAI/bge-reranker-large\n",
+        "2. cross-encoder/ms-marco-MiniLM-L-6-v2\n",
+        "\n",
+        "As usual, feel free to pick another model and provide some description to it.\n",
+        "\n",
+        "The deliverables are:\n",
+        "\n",
+        "1. The code that enables a reranker.\n",
+        "3. A comparison of how the prompt and the model output change after adding a reranker\n",
+        "4. The analysis of pros and cons. The evaluation aspects should include the relevance of the top-k documents, the response time.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ee1b0160-0ba0-4b5f-81c4-ef3ea76850e5",
+      "metadata": {
+        "id": "ee1b0160-0ba0-4b5f-81c4-ef3ea76850e5"
+      },
+      "outputs": [],
+      "source": [
+        "# Implement code for selecting the final documents using a cross-encoder and compare with and without"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from sentence_transformers import SentenceTransformer\n",
+        "\n",
+        "# Load the model\n",
+        "model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2') # BAAI/bge-reranker-large\n",
+        "\n",
+        "# Vectorize the query\n",
+        "query = \"Your search query here\"\n",
+        "query_vector = model.encode(query)"
+      ],
+      "metadata": {
+        "id": "peSWSL0lXOK5"
+      },
+      "id": "peSWSL0lXOK5",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import lancedb\n",
+        "import numpy as np\n",
+        "\n",
+        "# Connect to LanceDB and open your table\n",
+        "db = lancedb.connect(\"/content/rag-gradio-sample-project/gradio_app/.lancedb/\")\n",
+        "tbl = db.open_table({TABLE_NAME2})\n",
+        "\n",
+        "# Perform a vector search for the top-N documents\n",
+        "df = tbl.search(query_vector) \\\n",
+        "    .metric(\"cosine\") \\\n",
+        "    .limit(10) \\\n",
+        "    .to_list()  # Or use .to_pandas(), .to_arrow(), etc., based on your preference\n",
+        "\n",
+        "# `df` now contains the top-N documents and their similarity scores"
+      ],
+      "metadata": {
+        "id": "xd10rndiUCIW"
+      },
+      "id": "xd10rndiUCIW",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Assuming `df` contains document IDs or keys to fetch the actual documents\n",
+        "documents = [db.fetch_document(table_name, doc_id) for doc_id in df]"
+      ],
+      "metadata": {
+        "id": "8KWuDzhxTLTX"
+      },
+      "id": "8KWuDzhxTLTX",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
+        "from torch.utils.data import DataLoader\n",
+        "import torch\n",
+        "\n",
+        "# Initialize the tokenizer and model\n",
+        "tokenizer = AutoTokenizer.from_pretrained(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n",
+        "model = AutoModelForSequenceClassification.from_pretrained(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n",
+        "\n",
+        "def rerank(query, documents):\n",
+        "    # Assuming `documents` is a list of texts\n",
+        "    pairs = [[query, doc['text']] for doc in documents]  # Adjust based on your `results` structure\n",
+        "    inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors=\"pt\")\n",
+        "    with torch.no_grad():\n",
+        "        scores = rerank_model(**inputs).logits[:,1]  # Scores for each pair\n",
+        "    # Sort documents by scores in descending order and return\n",
+        "    documents = [doc for _, doc in sorted(zip(scores, documents), key=lambda x: x[0], reverse=True)]\n",
+        "    return documents"
+      ],
+      "metadata": {
+        "id": "O6xMyqFjRp_m"
+      },
+      "id": "O6xMyqFjRp_m",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "top_k_documents = rerank(query, documents)[:K]  # Keep top K after reranking"
+      ],
+      "metadata": {
+        "id": "dZtiwhPBRtnS"
+      },
+      "id": "dZtiwhPBRtnS",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "id": "f5816c54-a290-4cb0-b7db-3b8901998cb0",
+      "metadata": {
+        "id": "f5816c54-a290-4cb0-b7db-3b8901998cb0"
+      },
+      "source": [
+        "### Step 4: Try a different LLM\n",
+        "\n",
+        "The suggested `Mistral-7b-instruct` is a great but small model for an LLM. A larger model can be applied to a wider range of problems and do more complex reasoning. Within the scope of this project a larger model may not be beneficial but for more complex cases the difference would become apparent. Another dimension to explore is a base model which was not instruction fine-tuned - it won't respond to your queries the way you'd expect. It may be a great exercise to see the value of fine-tuning.\n",
+        "\n",
+        "The task here is to try out an alternative LLM to explore the differences.\n",
+        "\n",
+        "The options are:\n",
+        "1. mistralai/Mistral-7B-v0.1\n",
+        "2. mistralai/Mixtral-8x7B-Instruct-v0.1\n",
+        "\n",
+        "Of course, feel free to choose another one and give some details on how different it is from the initial model.\n",
+        "\n",
+        "The deliverables are:\n",
+        "\n",
+        "1. The comparison between outputs of the Mistral-7b-instuct and a different model of your choice.\n",
+        "2. The difference in response times if a larger model was chosen. Make sure to make multiple queries to make the comparison meaningful.\n",
+        "3. Analyse the differences between outputs and share the conclusions.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "942f39d4-eb27-4f2d-ae47-a5d65f102faa",
+      "metadata": {
+        "id": "942f39d4-eb27-4f2d-ae47-a5d65f102faa"
+      },
+      "outputs": [],
+      "source": [
+        "# Analysis of the difference between LLMs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "70c16440",
+      "metadata": {
+        "id": "70c16440"
+      },
+      "source": [
+        "### Step 5 (Bonus): Use an LLM to quantitatively compare outputs of different variants of the system (LLM as a Judge)\n",
+        "\n",
+        "Use a powerful LLM (e.g. GPT-4) to quantitatively evaluate outputs of two alternative setups (different embedding models, different LLMs, both etc.). For inspiration and for prompts refer to [1](https://arxiv.org/pdf/2306.05685.pdf), [2](https://arxiv.org/pdf/2401.10020.pdf), [3](https://www.airtrain.ai/blog/the-comprehensive-guide-to-llm-evaluation#high-level-approach)\n",
+        "\n",
+        "The deliverables:\n",
+        "\n",
+        "1. The code you put together\n",
+        "2. The high-level description of the setup\n",
+        "3. The results of the qualitative comparison\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "39c18ba0-e54a-478f-9e60-0ea65c29238a",
+      "metadata": {
+        "id": "39c18ba0-e54a-478f-9e60-0ea65c29238a"
+      },
+      "outputs": [],
+      "source": [
+        "# The code implementing LLM-as-a-Judge and the evaluation results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "2ce78700-2578-4719-8b6b-d59fc669d1c1",
+      "metadata": {
+        "id": "2ce78700-2578-4719-8b6b-d59fc669d1c1"
+      },
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.11"
+    },
+    "colab": {
+      "provenance": []
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

README.md

@@ -1,10 +1,11 @@
 ---
 title: RAG
 emoji: ⚡
-colorFrom: pink
-colorTo: gray
+colorFrom: yellow
+colorTo: indigo
 sdk: gradio
-sdk_version: 4.19.0
-app_file: gradio_app\app.py
+sdk_version: 4.4.1
+app_file: app.py
 pinned: false
+license: apache-2.0
 ---

chunked/content_aware_chunking/__config/chunk_0.txt

@@ -0,0 +1,13 @@
+docstyle-ignore
+INSTALL_CONTENT = """
+Transformers installation
+! pip install transformers datasets
+To install from source instead of the last release, comment the command above and uncomment the following one.
+! pip install git+https://github.com/huggingface/transformers.git
+"""
+notebook_first_cells = [{"type": "code", "content": INSTALL_CONTENT}]
+black_avoid_patterns = {
+    "{processor_class}": "FakeProcessorClass",
+    "{model_class}": "FakeModelClass",
+    "{object_class}": "FakeObjectClass",
+}.

chunked/content_aware_chunking/__redirects/chunk_0.txt

@@ -0,0 +1,2 @@
+Optimizing inference
+perf_infer_gpu_many: perf_infer_gpu_one.

chunked/content_aware_chunking/__toctree/chunk_1.txt

@@ -0,0 +1,836 @@
+sections:
+local: index
+    title: 🤗 Transformers
+local: quicktour
+    title: Quick tour
+local: installation
+    title: Installation
+  title: Get started
+sections:
+local: pipeline_tutorial
+    title: Run inference with pipelines
+local: autoclass_tutorial
+    title: Write portable code with AutoClass
+local: preprocessing
+    title: Preprocess data
+local: training
+    title: Fine-tune a pretrained model
+local: run_scripts
+    title: Train with a script
+local: accelerate
+    title: Set up distributed training with 🤗 Accelerate
+local: peft
+    title: Load and train adapters with 🤗 PEFT
+local: model_sharing
+    title: Share your model
+local: transformers_agents
+    title: Agents
+local: llm_tutorial
+    title: Generation with LLMs
+  title: Tutorials
+sections:
+isExpanded: false
+    sections:
+local: tasks/sequence_classification
+  title: Text classification
+local: tasks/token_classification
+  title: Token classification
+local: tasks/question_answering
+  title: Question answering
+local: tasks/language_modeling
+  title: Causal language modeling
+local: tasks/masked_language_modeling
+  title: Masked language modeling
+local: tasks/translation
+  title: Translation
+local: tasks/summarization
+  title: Summarization
+local: tasks/multiple_choice
+  title: Multiple choice
+title: Natural Language Processing
+
+isExpanded: false
+    sections:
+local: tasks/audio_classification
+  title: Audio classification
+local: tasks/asr
+  title: Automatic speech recognition
+title: Audio
+
+isExpanded: false
+    sections:
+local: tasks/image_classification
+  title: Image classification
+local: tasks/semantic_segmentation
+  title: Image segmentation
+local: tasks/video_classification
+  title: Video classification
+local: tasks/object_detection
+  title: Object detection
+local: tasks/zero_shot_object_detection
+  title: Zero-shot object detection
+local: tasks/zero_shot_image_classification
+  title: Zero-shot image classification
+local: tasks/monocular_depth_estimation
+  title: Depth estimation
+local: tasks/image_to_image
+  title: Image-to-Image
+local: tasks/mask_generation
+  title: Mask Generation
+local: tasks/knowledge_distillation_for_image_classification
+  title: Knowledge Distillation for Computer Vision
+title: Computer Vision
+
+isExpanded: false
+    sections:
+local: tasks/image_captioning
+  title: Image captioning
+local: tasks/document_question_answering
+  title: Document Question Answering
+local: tasks/visual_question_answering
+  title: Visual Question Answering
+local: tasks/text-to-speech
+  title: Text to speech
+title: Multimodal
+
+isExpanded: false
+    sections:
+local: generation_strategies
+  title: Customize the generation strategy
+title: Generation
+
+isExpanded: false
+    sections:
+local: tasks/idefics
+  title: Image tasks with IDEFICS
+local: tasks/prompting
+  title: LLM prompting guide
+title: Prompting
+  title: Task Guides
+
+sections:
+local: fast_tokenizers
+    title: Use fast tokenizers from 🤗 Tokenizers
+local: multilingual
+    title: Run inference with multilingual models
+local: create_a_model
+    title: Use model-specific APIs
+local: custom_models
+    title: Share a custom model
+local: chat_templating
+    title: Templates for chat models
+local: trainer
+    title: Trainer
+local: sagemaker
+    title: Run training on Amazon SageMaker
+local: serialization
+    title: Export to ONNX
+local: tflite
+    title: Export to TFLite
+local: torchscript
+    title: Export to TorchScript
+local: benchmarks
+    title: Benchmarks
+local: notebooks
+    title: Notebooks with examples
+local: community
+    title: Community resources
+local: custom_tools
+    title: Custom Tools and Prompts
+local: troubleshooting
+    title: Troubleshoot  
+local: hf_quantizer
+    title: Contribute new quantization method
+  title: Developer guides
+sections:
+local: performance
+    title: Overview
+local: quantization
+    title: Quantization
+sections:
+local: perf_train_gpu_one
+  title: Methods and tools for efficient training on a single GPU
+local: perf_train_gpu_many
+  title: Multiple GPUs and parallelism
+local: fsdp
+  title: Fully Sharded Data Parallel
+local: deepspeed
+  title: DeepSpeed
+local: perf_train_cpu
+  title: Efficient training on CPU
+local: perf_train_cpu_many
+  title: Distributed CPU training
+local: perf_train_tpu_tf
+  title: Training on TPU with TensorFlow
+local: perf_train_special
+  title: PyTorch training on Apple silicon
+local: perf_hardware
+  title: Custom hardware for training
+local: hpo_train
+  title: Hyperparameter Search using Trainer API
+title: Efficient training techniques
+
+sections:
+local: perf_infer_cpu
+  title: CPU inference
+local: perf_infer_gpu_one
+  title: GPU inference
+title: Optimizing inference
+
+local: big_models
+    title: Instantiating a big model
+local: debugging
+    title: Debugging
+local: tf_xla
+    title: XLA Integration for TensorFlow Models
+local: perf_torch_compile
+    title: Optimize inference using torch.compile()
+  title: Performance and scalability
+sections:
+local: contributing
+    title: How to contribute to 🤗 Transformers?
+local: add_new_model
+    title: How to add a model to 🤗 Transformers?
+local: add_tensorflow_model
+    title: How to convert a 🤗 Transformers model to TensorFlow?
+local: add_new_pipeline
+    title: How to add a pipeline to 🤗 Transformers?
+local: testing
+    title: Testing
+local: pr_checks
+    title: Checks on a Pull Request
+  title: Contribute
+sections:
+local: philosophy
+    title: Philosophy
+local: glossary
+    title: Glossary
+local: task_summary
+    title: What 🤗 Transformers can do
+local: tasks_explained
+    title: How 🤗 Transformers solve tasks
+local: model_summary
+    title: The Transformer model family
+local: tokenizer_summary
+    title: Summary of the tokenizers
+local: attention
+    title: Attention mechanisms
+local: pad_truncation
+    title: Padding and truncation
+local: bertology
+    title: BERTology
+local: perplexity
+    title: Perplexity of fixed-length models
+local: pipeline_webserver
+    title: Pipelines for webserver inference
+local: model_memory_anatomy
+    title: Model training anatomy
+local: llm_tutorial_optimization
+    title: Getting the most out of LLMs
+  title: Conceptual guides
+sections:
+sections:
+local: main_classes/agent
+  title: Agents and Tools
+local: model_doc/auto
+  title: Auto Classes
+local: main_classes/backbones
+  title: Backbones
+local: main_classes/callback
+  title: Callbacks
+local: main_classes/configuration
+  title: Configuration
+local: main_classes/data_collator
+  title: Data Collator
+local: main_classes/keras_callbacks
+  title: Keras callbacks
+local: main_classes/logging
+  title: Logging
+local: main_classes/model
+  title: Models
+local: main_classes/text_generation
+  title: Text Generation
+local: main_classes/onnx
+  title: ONNX
+local: main_classes/optimizer_schedules
+  title: Optimization
+local: main_classes/output
+  title: Model outputs
+local: main_classes/pipelines
+  title: Pipelines
+local: main_classes/processors
+  title: Processors
+local: main_classes/quantization
+  title: Quantization
+local: main_classes/tokenizer
+  title: Tokenizer
+local: main_classes/trainer
+  title: Trainer
+local: main_classes/deepspeed
+  title: DeepSpeed
+local: main_classes/feature_extractor
+  title: Feature Extractor
+local: main_classes/image_processor
+  title: Image Processor
+title: Main Classes
+
+sections:
+isExpanded: false
+  sections:
+local: model_doc/albert
+    title: ALBERT
+local: model_doc/bart
+    title: BART
+local: model_doc/barthez
+    title: BARThez
+local: model_doc/bartpho
+    title: BARTpho
+local: model_doc/bert
+    title: BERT
+local: model_doc/bert-generation
+    title: BertGeneration
+local: model_doc/bert-japanese
+    title: BertJapanese
+local: model_doc/bertweet
+    title: Bertweet
+local: model_doc/big_bird
+    title: BigBird
+local: model_doc/bigbird_pegasus
+    title: BigBirdPegasus
+local: model_doc/biogpt
+    title: BioGpt
+local: model_doc/blenderbot
+    title: Blenderbot
+local: model_doc/blenderbot-small
+    title: Blenderbot Small
+local: model_doc/bloom
+    title: BLOOM
+local: model_doc/bort
+    title: BORT
+local: model_doc/byt5
+    title: ByT5
+local: model_doc/camembert
+    title: CamemBERT
+local: model_doc/canine
+    title: CANINE
+local: model_doc/codegen
+    title: CodeGen
+local: model_doc/code_llama
+    title: CodeLlama
+local: model_doc/convbert
+    title: ConvBERT
+local: model_doc/cpm
+    title: CPM
+local: model_doc/cpmant
+    title: CPMANT
+local: model_doc/ctrl
+    title: CTRL
+local: model_doc/deberta
+    title: DeBERTa
+local: model_doc/deberta-v2
+    title: DeBERTa-v2
+local: model_doc/dialogpt
+    title: DialoGPT
+local: model_doc/distilbert
+    title: DistilBERT
+local: model_doc/dpr
+    title: DPR
+local: model_doc/electra
+    title: ELECTRA
+local: model_doc/encoder-decoder
+    title: Encoder Decoder Models
+local: model_doc/ernie
+    title: ERNIE
+local: model_doc/ernie_m
+    title: ErnieM
+local: model_doc/esm
+    title: ESM
+local: model_doc/falcon
+    title: Falcon
+local: model_doc/fastspeech2_conformer
+    title: FastSpeech2Conformer
+local: model_doc/flan-t5
+    title: FLAN-T5
+local: model_doc/flan-ul2
+    title: FLAN-UL2
+local: model_doc/flaubert
+    title: FlauBERT
+local: model_doc/fnet
+    title: FNet
+local: model_doc/fsmt
+    title: FSMT
+local: model_doc/funnel
+    title: Funnel Transformer
+local: model_doc/fuyu
+    title: Fuyu
+local: model_doc/openai-gpt
+    title: GPT
+local: model_doc/gpt_neo
+    title: GPT Neo
+local: model_doc/gpt_neox
+    title: GPT NeoX
+local: model_doc/gpt_neox_japanese
+    title: GPT NeoX Japanese
+local: model_doc/gptj
+    title: GPT-J
+local: model_doc/gpt2
+    title: GPT2
+local: model_doc/gpt_bigcode
+    title: GPTBigCode
+local: model_doc/gptsan-japanese
+    title: GPTSAN Japanese
+local: model_doc/gpt-sw3
+    title: GPTSw3
+local: model_doc/herbert
+    title: HerBERT
+local: model_doc/ibert
+    title: I-BERT
+local: model_doc/jukebox
+    title: Jukebox
+local: model_doc/led
+    title: LED
+local: model_doc/llama
+    title: LLaMA
+local: model_doc/llama2
+    title: Llama2
+local: model_doc/longformer
+    title: Longformer
+local: model_doc/longt5
+    title: LongT5
+local: model_doc/luke
+    title: LUKE
+local: model_doc/m2m_100
+    title: M2M100
+local: model_doc/madlad-400
+    title: MADLAD-400
+local: model_doc/marian
+    title: MarianMT
+local: model_doc/markuplm
+    title: MarkupLM
+local: model_doc/mbart
+    title: MBart and MBart-50
+local: model_doc/mega
+    title: MEGA
+local: model_doc/megatron-bert
+    title: MegatronBERT
+local: model_doc/megatron_gpt2
+    title: MegatronGPT2
+local: model_doc/mistral
+    title: Mistral
+local: model_doc/mixtral
+    title: Mixtral
+local: model_doc/mluke
+    title: mLUKE
+local: model_doc/mobilebert
+    title: MobileBERT
+local: model_doc/mpnet
+    title: MPNet
+local: model_doc/mpt
+    title: MPT
+local: model_doc/mra
+    title: MRA
+local: model_doc/mt5
+    title: MT5
+local: model_doc/mvp
+    title: MVP
+local: model_doc/nezha
+    title: NEZHA
+local: model_doc/nllb
+    title: NLLB
+local: model_doc/nllb-moe
+    title: NLLB-MoE
+local: model_doc/nystromformer
+    title: Nyströmformer
+local: model_doc/open-llama
+    title: Open-Llama
+local: model_doc/opt
+    title: OPT
+local: model_doc/pegasus
+    title: Pegasus
+local: model_doc/pegasus_x
+    title: PEGASUS-X
+local: model_doc/persimmon
+    title: Persimmon
+local: model_doc/phi
+    title: Phi
+local: model_doc/phobert
+    title: PhoBERT
+local: model_doc/plbart
+    title: PLBart
+local: model_doc/prophetnet
+    title: ProphetNet
+local: model_doc/qdqbert
+    title: QDQBert
+local: model_doc/qwen2
+    title: Qwen2
+local: model_doc/rag
+    title: RAG
+local: model_doc/realm
+    title: REALM
+local: model_doc/reformer
+    title: Reformer
+local: model_doc/rembert
+    title: RemBERT
+local: model_doc/retribert
+    title: RetriBERT
+local: model_doc/roberta
+    title: RoBERTa
+local: model_doc/roberta-prelayernorm
+    title: RoBERTa-PreLayerNorm
+local: model_doc/roc_bert
+    title: RoCBert
+local: model_doc/roformer
+    title: RoFormer
+local: model_doc/rwkv
+    title: RWKV
+local: model_doc/splinter
+    title: Splinter
+local: model_doc/squeezebert
+    title: SqueezeBERT
+local: model_doc/stablelm
+    title: StableLm
+local: model_doc/switch_transformers
+    title: SwitchTransformers
+local: model_doc/t5
+    title: T5
+local: model_doc/t5v1.1
+    title: T5v1.1
+local: model_doc/tapex
+    title: TAPEX
+local: model_doc/transfo-xl
+    title: Transformer XL
+local: model_doc/ul2
+    title: UL2
+local: model_doc/umt5
+    title: UMT5
+local: model_doc/xmod
+    title: X-MOD
+local: model_doc/xglm
+    title: XGLM
+local: model_doc/xlm
+    title: XLM
+local: model_doc/xlm-prophetnet
+    title: XLM-ProphetNet
+local: model_doc/xlm-roberta
+    title: XLM-RoBERTa
+local: model_doc/xlm-roberta-xl
+    title: XLM-RoBERTa-XL
+local: model_doc/xlm-v
+    title: XLM-V
+local: model_doc/xlnet
+    title: XLNet
+local: model_doc/yoso
+    title: YOSO
+  title: Text models
+isExpanded: false
+  sections:
+local: model_doc/beit
+    title: BEiT
+local: model_doc/bit
+    title: BiT
+local: model_doc/conditional_detr
+    title: Conditional DETR
+local: model_doc/convnext
+    title: ConvNeXT
+local: model_doc/convnextv2
+    title: ConvNeXTV2
+local: model_doc/cvt
+    title: CvT
+local: model_doc/deformable_detr
+    title: Deformable DETR
+local: model_doc/deit
+    title: DeiT
+local: model_doc/depth_anything
+    title: Depth Anything
+local: model_doc/deta
+    title: DETA
+local: model_doc/detr
+    title: DETR
+local: model_doc/dinat
+    title: DiNAT
+local: model_doc/dinov2
+    title: DINOV2
+local: model_doc/dit
+    title: DiT
+local: model_doc/dpt
+    title: DPT
+local: model_doc/efficientformer
+    title: EfficientFormer
+local: model_doc/efficientnet
+    title: EfficientNet
+local: model_doc/focalnet
+    title: FocalNet
+local: model_doc/glpn
+    title: GLPN
+local: model_doc/imagegpt
+    title: ImageGPT
+local: model_doc/levit
+    title: LeViT
+local: model_doc/mask2former
+    title: Mask2Former
+local: model_doc/maskformer
+    title: MaskFormer
+local: model_doc/mobilenet_v1
+    title: MobileNetV1
+local: model_doc/mobilenet_v2
+    title: MobileNetV2
+local: model_doc/mobilevit
+    title: MobileViT
+local: model_doc/mobilevitv2
+    title: MobileViTV2
+local: model_doc/nat
+    title: NAT
+local: model_doc/poolformer
+    title: PoolFormer
+local: model_doc/pvt
+    title: Pyramid Vision Transformer (PVT)
+local: model_doc/regnet
+    title: RegNet
+local: model_doc/resnet
+    title: ResNet
+local: model_doc/segformer
+    title: SegFormer
+local: model_doc/swiftformer
+    title: SwiftFormer
+local: model_doc/swin
+    title: Swin Transformer
+local: model_doc/swinv2
+    title: Swin Transformer V2
+local: model_doc/swin2sr
+    title: Swin2SR
+local: model_doc/table-transformer
+    title: Table Transformer
+local: model_doc/upernet
+    title: UperNet
+local: model_doc/van
+    title: VAN
+local: model_doc/vit
+    title: Vision Transformer (ViT)
+local: model_doc/vit_hybrid
+    title: ViT Hybrid
+local: model_doc/vitdet
+    title: ViTDet
+local: model_doc/vit_mae
+    title: ViTMAE
+local: model_doc/vitmatte
+    title: ViTMatte
+local: model_doc/vit_msn
+    title: ViTMSN
+local: model_doc/yolos
+    title: YOLOS
+  title: Vision models
+isExpanded: false
+  sections:
+local: model_doc/audio-spectrogram-transformer
+    title: Audio Spectrogram Transformer
+local: model_doc/bark
+    title: Bark
+local: model_doc/clap
+    title: CLAP
+local: model_doc/encodec
+    title: EnCodec
+local: model_doc/hubert
+    title: Hubert
+local: model_doc/mctct
+    title: MCTCT
+local: model_doc/mms
+    title: MMS
+local: model_doc/musicgen
+    title: MusicGen
+local: model_doc/pop2piano
+    title: Pop2Piano
+local: model_doc/seamless_m4t
+    title: Seamless-M4T
+local: model_doc/seamless_m4t_v2
+    title: SeamlessM4T-v2
+local: model_doc/sew
+    title: SEW
+local: model_doc/sew-d
+    title: SEW-D
+local: model_doc/speech_to_text
+    title: Speech2Text
+local: model_doc/speech_to_text_2
+    title: Speech2Text2
+local: model_doc/speecht5
+    title: SpeechT5
+local: model_doc/unispeech
+    title: UniSpeech
+local: model_doc/unispeech-sat
+    title: UniSpeech-SAT
+local: model_doc/univnet
+    title: UnivNet
+local: model_doc/vits
+    title: VITS
+local: model_doc/wav2vec2
+    title: Wav2Vec2
+local: model_doc/wav2vec2-bert
+    title: Wav2Vec2-BERT
+local: model_doc/wav2vec2-conformer
+    title: Wav2Vec2-Conformer
+local: model_doc/wav2vec2_phoneme
+    title: Wav2Vec2Phoneme
+local: model_doc/wavlm
+    title: WavLM
+local: model_doc/whisper
+    title: Whisper
+local: model_doc/xls_r
+    title: XLS-R
+local: model_doc/xlsr_wav2vec2
+    title: XLSR-Wav2Vec2
+  title: Audio models
+isExpanded: false
+  sections:
+local: model_doc/timesformer
+    title: TimeSformer
+local: model_doc/videomae
+    title: VideoMAE
+local: model_doc/vivit
+    title: ViViT
+  title: Video models 
+isExpanded: false
+  sections:
+local: model_doc/align
+    title: ALIGN
+local: model_doc/altclip
+    title: AltCLIP
+local: model_doc/blip
+    title: BLIP
+local: model_doc/blip-2
+    title: BLIP-2
+local: model_doc/bridgetower
+    title: BridgeTower
+local: model_doc/bros
+    title: BROS
+local: model_doc/chinese_clip
+    title: Chinese-CLIP
+local: model_doc/clip
+    title: CLIP
+local: model_doc/clipseg
+    title: CLIPSeg
+local: model_doc/clvp
+    title: CLVP
+local: model_doc/data2vec
+    title: Data2Vec
+local: model_doc/deplot
+    title: DePlot
+local: model_doc/donut
+    title: Donut
+local: model_doc/flava
+    title: FLAVA
+local: model_doc/git
+    title: GIT
+local: model_doc/groupvit
+    title: GroupViT
+local: model_doc/idefics
+    title: IDEFICS
+local: model_doc/instructblip
+    title: InstructBLIP
+local: model_doc/kosmos-2
+    title: KOSMOS-2
+local: model_doc/layoutlm
+    title: LayoutLM
+local: model_doc/layoutlmv2
+    title: LayoutLMV2
+local: model_doc/layoutlmv3
+    title: LayoutLMV3
+local: model_doc/layoutxlm
+    title: LayoutXLM
+local: model_doc/lilt
+    title: LiLT
+local: model_doc/llava
+    title: Llava
+local: model_doc/lxmert
+    title: LXMERT
+local: model_doc/matcha
+    title: MatCha
+local: model_doc/mgp-str
+    title: MGP-STR
+local: model_doc/nougat
+    title: Nougat
+local: model_doc/oneformer
+    title: OneFormer
+local: model_doc/owlvit
+    title: OWL-ViT
+local: model_doc/owlv2
+    title: OWLv2
+local: model_doc/perceiver
+    title: Perceiver
+local: model_doc/pix2struct
+    title: Pix2Struct
+local: model_doc/sam
+    title: Segment Anything
+local: model_doc/siglip
+    title: SigLIP
+local: model_doc/speech-encoder-decoder
+    title: Speech Encoder Decoder Models
+local: model_doc/tapas
+    title: TAPAS
+local: model_doc/trocr
+    title: TrOCR
+local: model_doc/tvlt
+    title: TVLT
+local: model_doc/tvp
+    title: TVP
+local: model_doc/vilt
+    title: ViLT
+local: model_doc/vipllava
+    title: VipLlava
+local: model_doc/vision-encoder-decoder
+    title: Vision Encoder Decoder Models
+local: model_doc/vision-text-dual-encoder
+    title: Vision Text Dual Encoder
+local: model_doc/visual_bert
+    title: VisualBERT
+local: model_doc/xclip
+    title: X-CLIP
+  title: Multimodal models
+isExpanded: false
+  sections:
+local: model_doc/decision_transformer
+    title: Decision Transformer
+local: model_doc/trajectory_transformer
+    title: Trajectory Transformer
+  title: Reinforcement learning models
+isExpanded: false
+  sections:
+local: model_doc/autoformer
+    title: Autoformer
+local: model_doc/informer
+    title: Informer
+local: model_doc/patchtsmixer
+    title: PatchTSMixer
+local: model_doc/patchtst
+    title: PatchTST
+local: model_doc/time_series_transformer
+    title: Time Series Transformer
+  title: Time series models
+isExpanded: false
+  sections:
+local: model_doc/graphormer
+    title: Graphormer
+  title: Graph models
+title: Models
+
+sections:
+local: internal/modeling_utils
+  title: Custom Layers and Utilities
+local: internal/pipelines_utils
+  title: Utilities for pipelines
+local: internal/tokenization_utils
+  title: Utilities for Tokenizers
+local: internal/trainer_utils
+  title: Utilities for Trainer
+local: internal/generation_utils
+  title: Utilities for Generation
+local: internal/image_processing_utils
+  title: Utilities for Image Processors
+local: internal/audio_utils
+  title: Utilities for Audio processing
+local: internal/file_utils
+  title: General Utilities
+local: internal/time_series_utils
+  title: Utilities for Time Series
+title: Internal Helpers
+  title: API
+
+.

chunked/content_aware_chunking/_accelerate/chunk_0.txt

@@ -0,0 +1,2 @@
+Distributed training with 🤗 Accelerate
+As models get bigger, parallelism has emerged as a strategy for training larger models on limited hardware and accelerating training speed by several orders of magnitude. At Hugging Face, we created the 🤗 Accelerate library to help users easily train a 🤗 Transformers model on any type of distributed setup, whether it is multiple GPU's on one machine or multiple GPU's across several machines.

chunked/content_aware_chunking/_accelerate/chunk_1.txt

@@ -0,0 +1,6 @@
+In this tutorial, learn how to customize your native PyTorch training loop to enable training in a distributed environment.
+Setup
+Get started by installing 🤗 Accelerate:
+
+pip install accelerate
+Then import and create an [~accelerate.Accelerator] object. The [~accelerate.Accelerator] will automatically detect your type of distributed setup and initialize all the necessary components for training.

chunked/content_aware_chunking/_accelerate/chunk_2.txt

@@ -0,0 +1,7 @@
+You don't need to explicitly place your model on a device.
+
+from accelerate import Accelerator
+accelerator = Accelerator()
+
+Prepare to accelerate
+The next step is to pass all the relevant training objects to the [~accelerate.Accelerator.prepare] method.

chunked/content_aware_chunking/_accelerate/chunk_3.txt

@@ -0,0 +1,72 @@
+This includes your training and evaluation DataLoaders, a model and an optimizer:
+
+train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )
+
+Backward
+The last addition is to replace the typical loss.backward() in your training loop with 🤗 Accelerate's [~accelerate.Accelerator.backward]method:
+
+for epoch in range(num_epochs):
+     for batch in train_dataloader:
+         outputs = model(**batch)
+         loss = outputs.loss
+         accelerator.backward(loss)
+
+         optimizer.step()
+         lr_scheduler.step()
+         optimizer.zero_grad()
+         progress_bar.update(1)
+
+As you can see in the following code, you only need to add four additional lines of code to your training loop to enable distributed training!
+
++ from accelerate import Accelerator
+  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler
+
+accelerator = Accelerator()
+
+model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
+  optimizer = AdamW(model.parameters(), lr=3e-5)
+
+device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+
+model.to(device)
+
+train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+
+train_dataloader, eval_dataloader, model, optimizer
+)
+
+num_epochs = 3
+  num_training_steps = num_epochs * len(train_dataloader)
+  lr_scheduler = get_scheduler(
+      "linear",
+      optimizer=optimizer,
+      num_warmup_steps=0,
+      num_training_steps=num_training_steps
+  )
+progress_bar = tqdm(range(num_training_steps))
+model.train()
+  for epoch in range(num_epochs):
+      for batch in train_dataloader:
+
+          outputs = model(**batch)
+          loss = outputs.loss
+
++         accelerator.backward(loss)
+      optimizer.step()
+      lr_scheduler.step()
+      optimizer.zero_grad()
+      progress_bar.update(1)
+
+Train
+Once you've added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.
+Train with a script
+If you are running your training from a script, run the following command to create and save a configuration file:
+
+accelerate config
+Then launch your training with:
+
+accelerate launch train.py
+Train with a notebook
+🤗 Accelerate can also run in a notebook if you're planning on using Colaboratory's TPUs.

chunked/content_aware_chunking/_accelerate/chunk_4.txt

@@ -0,0 +1,6 @@
+Wrap all the code responsible for training in a function, and pass it to [~accelerate.notebook_launcher]:
+
+from accelerate import notebook_launcher
+notebook_launcher(training_function)
+
+For more information about 🤗 Accelerate and its rich features, refer to the documentation..

chunked/content_aware_chunking/_add_new_model/chunk_0.txt

@@ -0,0 +1,2 @@
+How to add a model to 🤗 Transformers?
+The 🤗 Transformers library is often able to offer new models thanks to community contributors. But this can be a challenging project and requires an in-depth knowledge of the 🤗 Transformers library and the model to implement.

chunked/content_aware_chunking/_add_new_model/chunk_1.txt

@@ -0,0 +1,12 @@
+At Hugging Face, we're trying to empower more of the community to actively add models and we've put together this guide to walk you through the process of adding a PyTorch model (make sure you have PyTorch installed).
+
+If you're interested in implementing a TensorFlow model, take a look at the How to convert a 🤗 Transformers model to TensorFlow guide!
+
+Along the way, you'll:
+
+get insights into open-source best practices
+understand the design principles behind one of the most popular deep learning libraries
+learn how to efficiently test large models
+learn how to integrate Python utilities like black, ruff, and make fix-copies to ensure clean and readable code
+
+A Hugging Face team member will be available to help you along the way so you'll never be alone.

chunked/content_aware_chunking/_add_new_model/chunk_10.txt

@@ -0,0 +1,2 @@
+Note that the configuration and the model are always serialized into two
+different formats - the model to a pytorch_model.bin file and the configuration to a config.json file.

chunked/content_aware_chunking/_add_new_model/chunk_100.txt

@@ -0,0 +1,5 @@
+Hence, the documentation must be understandable and concise. It is very useful for
+the community to add some Tips to show how the model should be used. Don't hesitate to ping the Hugging Face team
+regarding the docstrings.
+Next, make sure that the docstring added to src/transformers/models/brand_new_bert/modeling_brand_new_bert.py is
+correct and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our docstring format here.

chunked/content_aware_chunking/_add_new_model/chunk_101.txt

@@ -0,0 +1,5 @@
+It is always to good to remind oneself that documentation should
+be treated at least as carefully as the code in 🤗 Transformers since the documentation is usually the first contact
+point of the community with the model.
+Code refactor
+Great, now you have added all the necessary code for brand_new_bert.

chunked/content_aware_chunking/_add_new_model/chunk_102.txt

@@ -0,0 +1,10 @@
+At this point, you should correct some potential
+incorrect code style by running:
+
+make style
+and verify that your coding style passes the quality check:
+
+make quality
+There are a couple of other very strict design tests in 🤗 Transformers that might still be failing, which shows up in
+the tests of your pull request. This is often because of some missing information in the docstring or some incorrect
+naming.

chunked/content_aware_chunking/_add_new_model/chunk_103.txt

@@ -0,0 +1,5 @@
+The Hugging Face team will surely help you if you're stuck here.
+Lastly, it is always a good idea to refactor one's code after having ensured that the code works correctly. With all
+tests passing, now it's a good time to go over the added code again and do some refactoring.
+You have now finished the coding part, congratulation! 🎉 You are Awesome! 😎
+12.

chunked/content_aware_chunking/_add_new_model/chunk_104.txt

@@ -0,0 +1,5 @@
+Upload the models to the model hub
+In this final part, you should convert and upload all checkpoints to the model hub and add a model card for each
+uploaded model checkpoint. You can get familiar with the hub functionalities by reading our Model sharing and uploading Page. You should work alongside the Hugging Face team here to decide on a fitting name for each
+checkpoint and to get the required access rights to be able to upload the model under the author's organization of
+brand_new_bert.

chunked/content_aware_chunking/_add_new_model/chunk_105.txt

@@ -0,0 +1,8 @@
+The push_to_hub method, present in all models in transformers, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:
+thon
+brand_new_bert.push_to_hub("brand_new_bert")
+Uncomment the following line to push to an organization.
+brand_new_bert.push_to_hub("/brand_new_bert")
+
+It is worth spending some time to create fitting model cards for each checkpoint. The model cards should highlight the
+specific characteristics of this particular checkpoint, e.g.

chunked/content_aware_chunking/_add_new_model/chunk_106.txt

@@ -0,0 +1,7 @@
+On which dataset was the checkpoint
+pretrained/fine-tuned on? On what down-stream task should the model be used? And also include some code on how to
+correctly use the model.
+13. (Optional) Add notebook
+It is very helpful to add a notebook that showcases in-detail how brand_new_bert can be used for inference and/or
+fine-tuned on a downstream task. This is not mandatory to merge your PR, but very useful for the community.
+14.

chunked/content_aware_chunking/_add_new_model/chunk_107.txt

@@ -0,0 +1,2 @@
+Submit your finished PR
+You're done programming now and can move to the last step, which is getting your PR merged into main.

chunked/content_aware_chunking/_add_new_model/chunk_108.txt

@@ -0,0 +1,7 @@
+Usually, the
+Hugging Face team should have helped you already at this point, but it is worth taking some time to give your finished
+PR a nice description and eventually add comments to your code, if you want to point out certain design choices to your
+reviewer.
+Share your work!!
+Now, it's time to get some credit from the community for your work! Having completed a model addition is a major
+contribution to Transformers and the whole NLP community.

chunked/content_aware_chunking/_add_new_model/chunk_109.txt

@@ -0,0 +1,4 @@
+Your code and the ported pre-trained models will certainly be
+used by hundreds and possibly even thousands of developers and researchers. You should be proud of your work and share
+your achievements with the community.
+You have made another model that is super easy to access for everyone in the community! 🤯.

chunked/content_aware_chunking/_add_new_model/chunk_11.txt

@@ -0,0 +1,9 @@
+Calling
+[~PreTrainedModel.save_pretrained] will automatically call
+[~PretrainedConfig.save_pretrained], so that both model and configuration are saved.
+Code style
+When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
+own regarding how code should be written :-)
+
+The forward pass of your model should be fully written in the modeling file while being fully independent of other
+   models in the library.

chunked/content_aware_chunking/_add_new_model/chunk_12.txt

@@ -0,0 +1,5 @@
+If you want to reuse a block from another model, copy the code and paste it with a
+   # Copied from comment on top (see here
+   for a good example and there for more documentation on Copied from). 
+The code should be fully understandable, even by a non-native English speaker. This means you should pick
+   descriptive variable names and avoid abbreviations.

chunked/content_aware_chunking/_add_new_model/chunk_13.txt

@@ -0,0 +1,6 @@
+As an example, activation is preferred to act.
+   One-letter variable names are strongly discouraged unless it's an index in a for loop.
+More generally we prefer longer explicit code to short magical one.
+Avoid subclassing nn.Sequential in PyTorch but subclass nn.Module and write the forward pass, so that anyone
+   using your code can quickly debug it by adding print statements or breaking points.
+Your function signature should be type-annotated.

chunked/content_aware_chunking/_add_new_model/chunk_14.txt

@@ -0,0 +1,8 @@
+For the rest, good variable names are way more readable and
+   understandable than type annotations.
+
+Overview of tokenizers
+Not quite ready yet :-( This section will be added soon!
+Step-by-step recipe to add a model to 🤗 Transformers
+Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
+of how other contributors ported models to Hugging Face.

chunked/content_aware_chunking/_add_new_model/chunk_15.txt

@@ -0,0 +1,11 @@
+Here is a list of community blog posts on how to port a model:
+
+Porting GPT2 Model by Thomas
+Porting WMT19 MT Model by Stas
+
+From experience, we can tell you that the most important things to keep in mind when adding a model are:
+
+Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
+  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
+  from. grep and rg are your
+  friends.

chunked/content_aware_chunking/_add_new_model/chunk_16.txt

@@ -0,0 +1,4 @@
+Note that it might very well happen that your model's tokenizer is based on one model implementation, and
+  your model's modeling code on another one. E.g. FSMT's modeling code is based on BART, while FSMT's tokenizer code
+  is based on XLM.
+It's more of an engineering challenge than a scientific challenge.

chunked/content_aware_chunking/_add_new_model/chunk_17.txt

@@ -0,0 +1,4 @@
+You should spend more time creating an
+  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.
+Ask for help, when you're stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
+  than happy to help you at every step to add your model.

chunked/content_aware_chunking/_add_new_model/chunk_18.txt

@@ -0,0 +1,21 @@
+Don't hesitate to ask if you notice you are not making
+  progress.
+
+In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.
+The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
+List:
+☐ (Optional) Understood the model's theoretical aspects
+☐ Prepared 🤗 Transformers dev environment
+☐ Set up debugging environment of the original repository
+☐ Created script that successfully runs the forward() pass using the original repository and checkpoint
+☐ Successfully added the model skeleton to 🤗 Transformers
+☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint
+☐ Successfully ran forward() pass in 🤗 Transformers that gives identical output to original checkpoint
+☐ Finished model tests in 🤗 Transformers
+☐ Successfully added tokenizer in 🤗 Transformers
+☐ Run end-to-end integration tests
+☐ Finished docs
+☐ Uploaded model weights to the Hub
+☐ Submitted the pull request
+☐ (Optional) Added a demo notebook
+To begin with, we usually recommend starting by getting a good theoretical understanding of BrandNewBert.

chunked/content_aware_chunking/_add_new_model/chunk_19.txt

@@ -0,0 +1,6 @@
+However,
+if you prefer to understand the theoretical aspects of the model on-the-job, then it is totally fine to directly dive
+into the BrandNewBert's code-base. This option might suit you better if your engineering skills are better than
+your theoretical skill, if you have trouble understanding BrandNewBert's paper, or if you just enjoy programming
+much more than reading scientific papers.
+1.

chunked/content_aware_chunking/_add_new_model/chunk_2.txt

@@ -0,0 +1,5 @@
+🤗 ❤️
+To get started, open a New model addition issue for the model you want to see in 🤗 Transformers. If you're not especially picky about contributing a specific model, you can filter by the New model label to see if there are any unclaimed model requests and work on it.
+Once you've opened a new model request, the first step is to get familiar with 🤗 Transformers if you aren't already!
+General overview of 🤗 Transformers
+First, you should get a general overview of 🤗 Transformers.

chunked/content_aware_chunking/_add_new_model/chunk_20.txt

@@ -0,0 +1,5 @@
+(Optional) Theoretical aspects of BrandNewBert
+You should take some time to read BrandNewBert's paper, if such descriptive work exists. There might be large
+sections of the paper that are difficult to understand. If this is the case, this is fine - don't worry! The goal is
+not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
+effectively re-implement the model in 🤗 Transformers.

chunked/content_aware_chunking/_add_new_model/chunk_21.txt

@@ -0,0 +1,15 @@
+That being said, you don't have to spend too much time on the
+theoretical aspects, but rather focus on the practical ones, namely:
+
+What type of model is brand_new_bert? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
+  encoder-decoder model? Look at the model_summary if you're not familiar with the differences between those.
+What are the applications of brand_new_bert? Text classification? Text generation? Seq2Seq tasks, e.g.,
+  summarization?
+What is the novel feature of the model that makes it different from BERT/GPT-2/BART?
+Which of the already existing 🤗 Transformers models is most
+  similar to brand_new_bert?
+What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
+  for BERT or BART?
+
+After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
+Hugging Face team with any questions you might have.

chunked/content_aware_chunking/_add_new_model/chunk_22.txt

@@ -0,0 +1,6 @@
+This might include questions regarding the model's architecture,
+its attention layer, etc. We will be more than happy to help you.
+2. Next prepare your environment
+
+Fork the repository by clicking on the ‘Fork' button on the
+   repository's page.

chunked/content_aware_chunking/_add_new_model/chunk_23.txt

@@ -0,0 +1,15 @@
+This creates a copy of the code under your GitHub user account.
+
+Clone your transformers fork to your local disk, and add the base repository as a remote:
+
+git clone https://github.com/[your Github handle]/transformers.git
+cd transformers
+git remote add upstream https://github.com/huggingface/transformers.git
+
+Set up a development environment, for instance by running the following command:
+
+python -m venv .env
+source .env/bin/activate
+pip install -e ".[dev]"
+Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
+failure with this command.

chunked/content_aware_chunking/_add_new_model/chunk_24.txt

@@ -0,0 +1,12 @@
+If that's the case make sure to install the Deep Learning framework you are working with
+(PyTorch, TensorFlow and/or Flax) then do:
+
+pip install -e ".[quality]"
+which should be enough for most use cases. You can then return to the parent directory
+
+cd ..
+
+We recommend adding the PyTorch version of brand_new_bert to Transformers. To install PyTorch, please follow the
+   instructions on https://pytorch.org/get-started/locally/.
+
+Note: You don't need to have CUDA installed.

chunked/content_aware_chunking/_add_new_model/chunk_25.txt

@@ -0,0 +1,10 @@
+Making the new model work on CPU is sufficient.
+
+To port brand_new_bert, you will also need access to its original repository:
+
+git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
+cd brand_new_bert
+pip install -e .
+Now you have set up a development environment to port brand_new_bert to 🤗 Transformers.
+3.-4. Run a pretrained checkpoint using the original repository
+At first, you will work on the original brand_new_bert repository.

chunked/content_aware_chunking/_add_new_model/chunk_26.txt

@@ -0,0 +1,5 @@
+Often, the original implementation is very
+“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
+be exactly your motivation to reimplement brand_new_bert. At Hugging Face, one of our main goals is to make people
+stand on the shoulders of giants which translates here very well into taking a working model and rewriting it to make
+it as accessible, user-friendly, and beautiful as possible.

chunked/content_aware_chunking/_add_new_model/chunk_27.txt

@@ -0,0 +1,5 @@
+This is the number-one motivation to re-implement
+models into 🤗 Transformers - trying to make complex new NLP technology accessible to everybody.
+You should start thereby by diving into the original repository.
+Successfully running the official pretrained model in the original repository is often the most difficult step.
+From our experience, it is very important to spend some time getting familiar with the original code-base.

chunked/content_aware_chunking/_add_new_model/chunk_28.txt

@@ -0,0 +1,10 @@
+You need to
+figure out the following:
+
+Where to find the pretrained weights?
+How to load the pretrained weights into the corresponding model?
+How to run the tokenizer independently from the model?
+Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
+  you only have to reimplement those functions.
+Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,
+  e.g.

chunked/content_aware_chunking/_add_new_model/chunk_29.txt

@@ -0,0 +1,2 @@
+EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
+  e.g.

chunked/content_aware_chunking/_add_new_model/chunk_3.txt

@@ -0,0 +1,5 @@
+🤗 Transformers is a very opinionated library, so there is a
+chance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we
+found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
+Transformers while keeping maintenance costs at a reasonable level.
+A good first starting point to better understand the library is to read the documentation of our philosophy.

chunked/content_aware_chunking/_add_new_model/chunk_30.txt

@@ -0,0 +1,7 @@
+self-attention, cross-attention?
+How can you debug the model in the original environment of the repo? Do you have to add print statements, can you
+  work with an interactive debugger like ipdb, or should you use an efficient IDE to debug the model, like PyCharm?
+
+It is very important that before you start the porting process, you can efficiently debug code in the original
+repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
+even a pull request in the original repository.

chunked/content_aware_chunking/_add_new_model/chunk_31.txt

@@ -0,0 +1,5 @@
+The maintainers of this repository are most likely very happy about
+someone looking into their code!
+At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
+model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
+dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model.

chunked/content_aware_chunking/_add_new_model/chunk_32.txt

@@ -0,0 +1,10 @@
+Only
+at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
+model also works as expected on GPU.
+In general, there are two possible debugging environments for running the original model
+
+Jupyter notebooks / google colab
+Local python scripts.
+
+Jupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split
+logical components from one another and to have faster debugging cycles as intermediate results can be stored.