Welcome to H2O LLM Studio, a framework and no-code GUI designed for
+ fine-tuning state-of-the-art large language models (LLMs).
+
+
+## Jump to
+
+- [With H2O LLM Studio, you can](#with-h2o-llm-studio-you-can)
+- [Quickstart](#quickstart)
+- [What's New](#whats-new)
+- [Setup](#setup)
+ - [Recommended Install](#recommended-install)
+ - [Using requirements.txt](#using-requirementstxt)
+- [Run H2O LLM Studio GUI](#run-h2o-llm-studio-gui)
+- [Run H2O LLM Studio GUI using Docker from a nightly build](#run-h2o-llm-studio-gui-using-docker-from-a-nightly-build)
+- [Run H2O LLM Studio GUI by building your own Docker image](#run-h2o-llm-studio-gui-by-building-your-own-docker-image)
+- [Run H2O LLM Studio with command line interface (CLI)](#run-h2o-llm-studio-with-command-line-interface-cli)
+- [Data format and example data](#data-format-and-example-data)
+- [Training your model](#training-your-model)
+- [Example: Run on OASST data via CLI](#example-run-on-oasst-data-via-cli)
+- [Model checkpoints](#model-checkpoints)
+- [Documentation](#documentation)
+- [Contributing](#contributing)
+- [License](#license)
+
+## With H2O LLM Studio, you can
+
+- easily and effectively fine-tune LLMs **without the need for any coding experience**.
+- use a **graphic user interface (GUI)** specially designed for large language models.
+- finetune any LLM using a large variety of hyperparameters.
+- use recent finetuning techniques such as [Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685) and 8-bit model training with a low memory footprint.
+- use Reinforcement Learning (RL) to finetune your model (experimental)
+- use advanced evaluation metrics to judge generated answers by the model.
+- track and compare your model performance visually. In addition, [Neptune](https://neptune.ai/) integration can be used.
+- chat with your model and get instant feedback on your model performance.
+- easily export your model to the [Hugging Face Hub](https://huggingface.co/) and share it with the community.
+
+## Quickstart
+
+For questions, discussing, or just hanging out, come and join our [Discord](https://discord.gg/WKhYMWcVbq)!
+
+We offer several ways of getting started quickly.
+
+Using CLI for fine-tuning LLMs:
+
+[](https://www.kaggle.com/code/ilu000/h2o-llm-studio-cli/) [](https://colab.research.google.com/drive/1soqfJjwDJwjjH-VzZYO_pUeLx5xY4N1K?usp=sharing)
+
+## What's New
+
+- [PR 592](https://github.com/h2oai/h2o-llmstudio/pull/599) Added `KTOPairLoss` for DPO modeling allowing to train models with simple preference data. Data currently needs to be manually prepared by randomly matching positive and negative examples as pairs.
+- [PR 592](https://github.com/h2oai/h2o-llmstudio/pull/592) Starting to deprecate RLHF in favor of DPO/IPO optimization. Training is disabled, but old experiments are still viewable. RLHF will be fully removed in a future release.
+- [PR 530](https://github.com/h2oai/h2o-llmstudio/pull/530) Introduced a new problem type for DPO/IPO optimization. This optimization technique can be used as an alternative to RLHF.
+- [PR 288](https://github.com/h2oai/h2o-llmstudio/pull/288) Introduced Deepspeed for sharded training allowing to train larger models on machines with multiple GPUs. Requires NVLink. This feature replaces FSDP and offers more flexibility. Deepspeed requires a system installation of cudatoolkit and we recommend using version 11.8. See [Recommended Install](#recommended-install).
+- [PR 449](https://github.com/h2oai/h2o-llmstudio/pull/449) New problem type for Causal Classification Modeling allows to train binary and multiclass models using LLMs.
+- [PR 364](https://github.com/h2oai/h2o-llmstudio/pull/364) User secrets are now handled more securely and flexible. Support for handling secrets using the 'keyring' library was added. User settings are tried to be migrated automatically.
+
+Please note that due to current rapid development we cannot guarantee full backwards compatibility of new functionality. We thus recommend to pin the version of the framework to the one you used for your experiments. For resetting, please delete/backup your `data` and `output` folders.
+
+## Setup
+
+H2O LLM Studio requires a machine with Ubuntu 16.04+ and at least one recent Nvidia GPU with Nvidia drivers version >= 470.57.02. For larger models, we recommend at least 24GB of GPU memory.
+
+For more information about installation prerequisites, see the [Set up H2O LLM Studio](https://docs.h2o.ai/h2o-llmstudio/get-started/set-up-llm-studio#prerequisites) guide in the documentation.
+
+For a performance comparison of different GPUs, see the [H2O LLM Studio performance](https://h2oai.github.io/h2o-llmstudio/get-started/llm-studio-performance) guide in the documentation.
+
+### Recommended Install
+
+The recommended way to install H2O LLM Studio is using pipenv with Python 3.10. To install Python 3.10 on Ubuntu 16.04+, execute the following commands:
+
+#### System installs (Python 3.10)
+
+```bash
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt install python3.10
+sudo apt-get install python3.10-distutils
+curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+```
+
+#### Installing NVIDIA Drivers (if required)
+
+If deploying on a 'bare metal' machine running Ubuntu, one may need to install the required Nvidia drivers and CUDA. The following commands show how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an example. One can update the following based on their OS.
+
+```bash
+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
+sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
+wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2004-11-8-local_11.8.0-520.61.05-1_amd64.deb
+sudo dpkg -i cuda-repo-ubuntu2004-11-8-local_11.8.0-520.61.05-1_amd64.deb
+sudo cp /var/cuda-repo-ubuntu2004-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
+sudo apt-get update
+sudo apt-get -y install cuda
+```
+
+alternatively, one can install cudatoolkits in a cuda environment:
+
+```bash
+conda create -n llmstudio python=3.10
+conda activate llmstudio
+conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
+```
+
+#### Create virtual environment (pipenv)
+
+The following command will create a virtual environment using pipenv and will install the dependencies using pipenv:
+
+```bash
+make setup
+```
+
+If you are having troubles installing the flash_attn package, consider running
+
+```bash
+make setup-no-flash
+```
+
+instead. This will install the dependencies without the flash_attn package. Note that this will disable the use of Flash Attention 2 and model training will be slower and consume more memory.
+
+### Using requirements.txt
+
+If you wish to use conda or another virtual environment, you can also install the dependencies using the requirements.txt file:
+
+```bash
+pip install -r requirements.txt
+pip install flash-attn==2.5.5 --no-build-isolation # optional for Flash Attention 2
+```
+
+## Run H2O LLM Studio GUI
+
+You can start H2O LLM Studio using the following command:
+
+```bash
+make llmstudio
+```
+
+This command will start the [H2O wave](https://github.com/h2oai/wave) server and app.
+Navigate to + The recommended way to install H2O LLM Studio is using pipenv with Python + 3.10. To install Python 3.10 on Ubuntu 16.04+, execute the following + commands. +
++ System installs (Python 3.10) +
+
+
+ sudo add-apt-repository ppa:deadsnakes/ppa
+ sudo apt install python3.10
+ sudo apt-get install python3.10-distutils
+ curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+
+
+
+ Install NVIDIA drivers (if required)
+
+ If you are deploying on a 'bare metal' machine running Ubuntu, you may need
+ to install the required Nvidia drivers and CUDA. The following commands show
+ how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an
+ example. You can update the following based on your respective operating system.
+
+
+ wget
+ https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin{" "}
+
+ sudo mv cuda-ubuntu2004.pin
+ /etc/apt/preferences.d/cuda-repository-pin-600
+ wget
+ https://developer.download.nvidia.com/compute/cuda/11.4.3/local_installers/cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb{" "}
+
+ sudo dpkg -i
+ cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb
+ sudo apt-key add /var/cuda-repo-ubuntu2004-11-4-local/7fa2af80.pub
+ sudo apt-get -y update
+ sudo apt-get -y install cuda
+
+
+
+ Create virtual environment (pipenv)
+
+ The following command creates a virtual environment using pipenv and will install
+ the dependencies using pipenv.
+
+ make setup
+
+
+
+ If you wish to use conda or another virtual environment, you can also
+ install the dependencies using the requirements.txt{" "}
+ file.{" "}
+
+ pip install -r requirements.txt
+
+ + Follow the steps below to install H2O LLM Studio on a Windows machine + using Windows Subsystem for Linux{" "} + WSL2 +
++ 1. Download the{" "} + + latest nvidia driver + {" "} + for Windows.{" "} +
++ 2. Open PowerShell or a Windows Command Prompt window in administrator + mode.{" "} +
++ 3. Run the following command to confirm that the driver is installed + properly and see the driver version. +
+ nvidia-smi
+
+
+ + 4. Run the following command to install WSL2. +
+ wsl --install
+
+
+ 5. Launch the WSL2 Ubuntu installation.
++ 6. Install the{" "} + + WSL2 Nvidia Cuda Drivers + + . +
+
+ wget
+ https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin{" "}
+
+ sudo mv cuda-ubuntu2004.pin
+ /etc/apt/preferences.d/cuda-repository-pin-600
+ wget
+ https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb{" "}
+
+ sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb
+ sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg
+ /usr/share/keyrings/
+ sudo apt-get update
+ sudo apt-get -y install cuda
+
+
+
+ + 7. Set up the required python system installs (Python 3.10). +
+
+ sudo add-apt-repository ppa:deadsnakes/ppa
+ sudo apt install python3.10
+ sudo apt-get install python3.10-distutils
+ curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+
+
+
+ + 8. Create the virtual environment. +
+
+ sudo apt install -y python3.10-venv
+ python3 -m venv llmstudio
+ source llmstudio/bin/activate
+
+
+
+ + 9.Clone the H2O LLM Studio repository locally. +
+
+ git clone https://github.com/h2oai/h2o-llmstudio.git
+ cd h2o-llmstudio
+
+
+
+ + 10. Install H2O LLM Studio using the `requirements.txt`. +
+ pip install -r requirements.txt
+
+
+ + 11. Run the H2O LLM Studio application. +
+
+ H2O_WAVE_MAX_REQUEST_SIZE=25MB \
+ H2O_WAVE_NO_LOG=True \
+ H2O_WAVE_PRIVATE_DIR="/download/@output/download" \
+ wave run app
+
+
+
+ + This will start the H2O Wave server and the H2O LLM Studio app. Navigate + to http://localhost:10101/ (we recommend using Chrome) to access + H2O LLM Studio and start fine-tuning your models. +
+For more information, see AWS credentials and Methods for accessing a bucket in the AWS Documentation.
+python generate.py --base_model=HuggingFaceH4/zephyr-7b-betapython generate.py --base_model=zephyr-7b-beta.Q5_K_M.ggufpython generate.py --base_model=TheBloke/zephyr-7B-beta-AWQ{traceback.format_exc()}")],
+ ),
+ ]
+
+ buttons = [
+ ui.button(
+ name="dataset/import/6", label="Continue", primary=valid_visualization
+ ),
+ ui.button(
+ name="dataset/import/3/edit",
+ label="Back",
+ primary=not valid_visualization,
+ ),
+ ui.button(name="dataset/list", label="Abort"),
+ ]
+
+ q.page["dataset/import"] = ui.form_card(box="content", items=items)
+ q.client.delete_cards.add("dataset/import")
+
+ q.page["dataset/import/footer"] = ui.form_card(
+ box="footer", items=[ui.inline(items=buttons, justify="start")]
+ )
+ q.client.delete_cards.add("dataset/import/footer")
+
+ elif step == 6: # create dataset
+ if q.client["dataset/import/name"] == "":
+ await clean_dashboard(q, mode="full")
+ await dataset_import(q, step=2, error="Please enter all required fields!")
+
+ else:
+ folder_name = q.client["dataset/import/path"].split("/")[-1]
+ new_folder = q.client["dataset/import/name"]
+ act_path = q.client["dataset/import/path"]
+ new_path = new_folder.join(act_path.rsplit(folder_name, 1))
+
+ try:
+ shutil.move(q.client["dataset/import/path"], new_path)
+
+ cfg = q.client["dataset/import/cfg"]
+
+ # remap old path to new path
+ for k in default_cfg.dataset_folder_keys:
+ old_path = getattr(cfg.dataset, k, None)
+ if old_path is not None:
+ setattr(
+ cfg.dataset,
+ k,
+ old_path.replace(q.client["dataset/import/path"], new_path),
+ )
+
+ # change the default validation strategy if validation df set
+ if cfg.dataset.validation_dataframe != "None":
+ cfg.dataset.validation_strategy = "custom"
+ cfg_path = f"{new_path}/{q.client['dataset/import/cfg_file']}.yaml"
+ save_config_yaml(cfg_path, cfg)
+
+ train_rows = None
+ if os.path.exists(cfg.dataset.train_dataframe):
+ train_rows = read_dataframe_drop_missing_labels(
+ cfg.dataset.train_dataframe, cfg
+ ).shape[0]
+ validation_rows = None
+ if os.path.exists(cfg.dataset.validation_dataframe):
+ validation_rows = read_dataframe_drop_missing_labels(
+ cfg.dataset.validation_dataframe, cfg
+ ).shape[0]
+
+ dataset = Dataset(
+ id=q.client["dataset/import/id"],
+ name=q.client["dataset/import/name"],
+ path=new_path,
+ config_file=cfg_path,
+ train_rows=train_rows,
+ validation_rows=validation_rows,
+ )
+ if q.client["dataset/import/id"] is not None:
+ q.client.app_db.delete_dataset(dataset.id)
+ q.client.app_db.add_dataset(dataset)
+ await dataset_list(q)
+
+ except Exception as exception:
+ logger.error("Dataset error:", exc_info=True)
+ q.client.app_db._session.rollback()
+ error = clean_error(str(exception))
+ await clean_dashboard(q, mode="full")
+ await dataset_import(q, step=2, error=str(error))
+
+
+async def dataset_merge(q: Q, step, error=""):
+ if step == 1: # Select which dataset to merge
+ await clean_dashboard(q, mode="full")
+ q.client["nav/active"] = "dataset/merge"
+
+ q.page["dataset/merge"] = ui.form_card(box="content", items=[])
+ q.client.delete_cards.add("dataset/merge")
+
+ datasets_df = q.client.app_db.get_datasets_df()
+ import_choices = [
+ ui.choice(x["path"], x["name"]) for idx, x in datasets_df.iterrows()
+ ]
+
+ items = [
+ ui.text_l("Merge current dataset with an existing dataset"),
+ ui.dropdown(
+ name="dataset/merge/target",
+ label="Dataset",
+ value=datasets_df.iloc[0]["path"],
+ choices=import_choices,
+ trigger=False,
+ tooltip="Source of dataset import",
+ ),
+ ]
+
+ if error:
+ items.append(ui.message_bar(type="error", text=error))
+
+ q.page["dataset/merge"].items = items
+
+ buttons = [
+ ui.button(name="dataset/merge/action", label="Merge", primary=True),
+ ui.button(name="dataset/import/3", label="Back", primary=False),
+ ui.button(name="dataset/list", label="Abort"),
+ ]
+
+ q.page["dataset/import/footer"] = ui.form_card(
+ box="footer", items=[ui.inline(items=buttons, justify="start")]
+ )
+ q.client.delete_cards.add("dataset/import/footer")
+
+ elif step == 2: # copy file to dataset and go to edit dataset
+ current_dir = q.client["dataset/import/path"]
+ target_dir = q.args["dataset/merge/target"]
+
+ if current_dir == target_dir:
+ await dataset_merge(q, step=1, error="Cannot merge dataset with itself")
+ return
+
+ datasets_df = q.client.app_db.get_datasets_df().set_index("path")
+ has_dataset_entry = current_dir in datasets_df.index
+
+ if has_dataset_entry:
+ experiment_df = q.client.app_db.get_experiments_df()
+ source_id = int(datasets_df.loc[current_dir, "id"])
+ has_experiment = any(experiment_df["dataset"].astype(int) == source_id)
+ else:
+ source_id = None
+ has_experiment = False
+
+ current_files = os.listdir(current_dir)
+ current_files = [x for x in current_files if not x.endswith(".yaml")]
+ target_files = os.listdir(target_dir)
+ overlapping_files = list(set(current_files).intersection(set(target_files)))
+ rename_map = {}
+
+ for file in overlapping_files:
+ tmp_str = file.split(".")
+ if len(tmp_str) == 1:
+ file_name, extension = file, ""
+ else:
+ file_name, extension = ".".join(tmp_str[:-1]), f".{tmp_str[-1]}"
+
+ cnt = 1
+ while f"{file_name}_{cnt}{extension}" in target_files:
+ cnt += 1
+
+ rename_map[file] = f"{file_name}_{cnt}{extension}"
+ target_files.append(rename_map[file])
+
+ if len(overlapping_files):
+ warning = (
+ f"Renamed {', '.join(rename_map.keys())} to "
+ f"{', '.join(rename_map.values())} due to duplicated entries."
+ )
+ else:
+ warning = ""
+
+ for file in current_files:
+ new_file = rename_map.get(file, file)
+ src = os.path.join(current_dir, file)
+ dst = os.path.join(target_dir, new_file)
+
+ if has_experiment:
+ if os.path.isdir(src):
+ shutil.copytree(src, dst)
+ else:
+ shutil.copy(src, dst)
+ else:
+ shutil.move(src, dst)
+
+ if not has_experiment:
+ shutil.rmtree(current_dir)
+ if has_dataset_entry:
+ q.client.app_db.delete_dataset(source_id)
+
+ dataset_id = int(datasets_df.loc[target_dir, "id"])
+ await dataset_edit(q, dataset_id, warning=warning, allow_merge=False)
+
+
+async def dataset_list_table(
+ q: Q,
+ show_experiment_datasets: bool = True,
+) -> None:
+ """Pepare dataset list form card
+
+ Args:
+ q: Q
+ show_experiment_datasets: whether to also show datasets linked to experiments
+ """
+
+ q.client["dataset/list/df_datasets"] = get_datasets(
+ q=q,
+ show_experiment_datasets=show_experiment_datasets,
+ )
+
+ df_viz = q.client["dataset/list/df_datasets"].copy()
+
+ columns_to_drop = [
+ "id",
+ "path",
+ "config_file",
+ "validation dataframe",
+ ]
+
+ df_viz = df_viz.drop(columns=columns_to_drop, errors="ignore")
+ if "problem type" in df_viz.columns:
+ df_viz["problem type"] = df_viz["problem type"].str.replace("Text ", "")
+
+ widths = {
+ "name": "200",
+ "problem type": "210",
+ "train dataframe": "190",
+ "train rows": "120",
+ "validation rows": "130",
+ "labels": "120",
+ "actions": "5",
+ }
+
+ actions_dict = {
+ "dataset/newexperiment": "New experiment",
+ "dataset/edit": "Edit dataset",
+ "dataset/delete/dialog/single": "Delete dataset",
+ }
+
+ q.page["dataset/list"] = ui.form_card(
+ box="content",
+ items=[
+ ui_table_from_df(
+ q=q,
+ df=df_viz,
+ name="dataset/list/table",
+ sortables=["train rows", "validation rows"],
+ filterables=["name", "problem type"],
+ searchables=[],
+ min_widths=widths,
+ link_col="name",
+ height="calc(100vh - 245px)",
+ actions=actions_dict,
+ ),
+ ui.message_bar(type="info", text=""),
+ ],
+ )
+ q.client.delete_cards.add("dataset/list")
+
+
+async def dataset_list(q: Q, reset: bool = True) -> None:
+ """Display all datasets."""
+ q.client["nav/active"] = "dataset/list"
+
+ if reset:
+ await clean_dashboard(q, mode="full")
+ await dataset_list_table(q)
+
+ q.page["dataset/display/footer"] = ui.form_card(
+ box="footer",
+ items=[
+ ui.inline(
+ items=[
+ ui.button(
+ name="dataset/import", label="Import dataset", primary=True
+ ),
+ ui.button(
+ name="dataset/list/delete",
+ label="Delete datasets",
+ primary=False,
+ ),
+ ],
+ justify="start",
+ )
+ ],
+ )
+ q.client.delete_cards.add("dataset/display/footer")
+ remove_temp_files(q)
+
+ await q.page.save()
+
+
+async def dataset_newexperiment(q: Q, dataset_id: int):
+ """Start a new experiment from given dataset."""
+
+ dataset = q.client.app_db.get_dataset(dataset_id)
+
+ q.client["experiment/start/cfg_file"] = dataset.config_file.split("/")[-1].replace(
+ ".yaml", ""
+ )
+ q.client["experiment/start/cfg_category"] = q.client[
+ "experiment/start/cfg_file"
+ ].split("_")[0]
+ q.client["experiment/start/dataset"] = str(dataset_id)
+
+ await experiment_start(q)
+
+
+async def dataset_edit(
+ q: Q, dataset_id: int, error: str = "", warning: str = "", allow_merge: bool = True
+):
+ """Edit selected dataset.
+
+ Args:
+ q: Q
+ dataset_id: dataset id to edit
+ error: optional error message
+ warning: optional warning message
+ allow_merge: whether to allow merging dataset when editing
+ """
+
+ dataset = q.client.app_db.get_dataset(dataset_id)
+
+ experiments_df = q.client.app_db.get_experiments_df()
+ experiments_df = experiments_df[experiments_df["dataset"] == str(dataset_id)]
+ statuses, _ = get_experiments_status(experiments_df)
+ num_invalid = len([stat for stat in statuses if stat in ["running", "queued"]])
+
+ if num_invalid:
+ info = "s" if num_invalid > 1 else ""
+ info_str = (
+ f"Dataset {dataset.name} is linked to {num_invalid} "
+ f"running or queued experiment{info}. Wait for them to finish or stop them "
+ "first before editing the dataset."
+ )
+ q.page["dataset/list"].items[1].message_bar.text = info_str
+ return
+
+ q.client["dataset/import/id"] = dataset_id
+
+ q.client["dataset/import/cfg_file"] = dataset.config_file.split("/")[-1].replace(
+ ".yaml", ""
+ )
+ q.client["dataset/import/cfg_category"] = q.client["dataset/import/cfg_file"].split(
+ "_"
+ )[0]
+ q.client["dataset/import/path"] = dataset.path
+ q.client["dataset/import/name"] = dataset.name
+ q.client["dataset/import/original_name"] = dataset.name
+ q.client["dataset/import/cfg"] = load_config_yaml(dataset.config_file)
+
+ if allow_merge and experiments_df.shape[0]:
+ allow_merge = False
+
+ await dataset_import(
+ q=q, step=2, edit=True, error=error, warning=warning, allow_merge=allow_merge
+ )
+
+
+async def dataset_list_delete(q: Q):
+ """Allow to select multiple datasets for deletion."""
+
+ await dataset_list_table(q, show_experiment_datasets=False)
+
+ q.page["dataset/list"].items[0].table.multiple = True
+
+ info_str = "Only datasets not linked to experiments can be deleted."
+
+ q.page["dataset/list"].items[1].message_bar.text = info_str
+
+ q.page["dataset/display/footer"].items = [
+ ui.inline(
+ items=[
+ ui.button(
+ name="dataset/delete/dialog", label="Delete datasets", primary=True
+ ),
+ ui.button(name="dataset/list/delete/abort", label="Abort"),
+ ]
+ )
+ ]
+
+
+async def dataset_delete(q: Q, dataset_ids: List[int]):
+ """Delete selected datasets.
+
+ Args:
+ q: Q
+ dataset_ids: list of dataset ids to delete
+ """
+
+ for dataset_id in dataset_ids:
+ dataset = q.client.app_db.get_dataset(dataset_id)
+ q.client.app_db.delete_dataset(dataset.id)
+
+ try:
+ shutil.rmtree(dataset.path)
+ except OSError:
+ pass
+
+
+async def dataset_delete_single(q: Q, dataset_id: int):
+ dataset = q.client.app_db.get_dataset(dataset_id)
+
+ experiments_df = q.client.app_db.get_experiments_df()
+ num_experiments = sum(experiments_df["dataset"] == str(dataset_id))
+ if num_experiments:
+ info = "s" if num_experiments > 1 else ""
+ info_str = (
+ f"Dataset {dataset.name} is linked to {num_experiments} "
+ f"experiment{info}. Only datasets not linked to experiments can be deleted."
+ )
+ await dataset_list(q)
+ q.page["dataset/list"].items[1].message_bar.text = info_str
+ else:
+ await dataset_delete(q, [dataset_id])
+ await dataset_list(q)
+
+
+async def dataset_display(q: Q) -> None:
+ """Display a selected dataset."""
+
+ dataset_id = q.client["dataset/list/df_datasets"]["id"].iloc[
+ q.client["dataset/display/id"]
+ ]
+ dataset: Dataset = q.client.app_db.get_dataset(dataset_id)
+ config_filename = dataset.config_file
+ cfg = load_config_yaml(config_filename)
+ dataset_filename = cfg.dataset.train_dataframe
+
+ if (
+ q.client["dataset/display/tab"] is None
+ or q.args["dataset/display/data"] is not None
+ ):
+ q.client["dataset/display/tab"] = "dataset/display/data"
+
+ if q.args["dataset/display/visualization"] is not None:
+ q.client["dataset/display/tab"] = "dataset/display/visualization"
+
+ if q.args["dataset/display/statistics"] is not None:
+ q.client["dataset/display/tab"] = "dataset/display/statistics"
+
+ if q.args["dataset/display/summary"] is not None:
+ q.client["dataset/display/tab"] = "dataset/display/summary"
+
+ await clean_dashboard(q, mode=q.client["dataset/display/tab"])
+
+ items: List[Tab] = [
+ ui.tab(name="dataset/display/data", label="Sample Train Data"),
+ ui.tab(
+ name="dataset/display/visualization", label="Sample Train Visualization"
+ ),
+ ui.tab(name="dataset/display/statistics", label="Train Data Statistics"),
+ ui.tab(name="dataset/display/summary", label="Summary"),
+ ]
+
+ q.page["dataset/display/tab"] = ui.tab_card(
+ box="nav2",
+ link=True,
+ items=items,
+ value=q.client["dataset/display/tab"],
+ )
+ q.client.delete_cards.add("dataset/display/tab")
+
+ if q.client["dataset/display/tab"] == "dataset/display/data":
+ await show_data_tab(q=q, cfg=cfg, filename=dataset_filename)
+
+ elif q.client["dataset/display/tab"] == "dataset/display/visualization":
+ await show_visualization_tab(q, cfg)
+
+ elif q.client["dataset/display/tab"] == "dataset/display/statistics":
+ await show_statistics_tab(
+ q, dataset_filename=dataset_filename, config_filename=config_filename
+ )
+
+ elif q.client["dataset/display/tab"] == "dataset/display/summary":
+ await show_summary_tab(q, dataset_id)
+
+ q.page["dataset/display/footer"] = ui.form_card(
+ box="footer",
+ items=[
+ ui.inline(
+ items=[
+ ui.button(
+ name="dataset/newexperiment/from_current",
+ label="Create experiment",
+ primary=False,
+ disabled=False,
+ tooltip=None,
+ ),
+ ui.button(name="dataset/list", label="Back", primary=False),
+ ],
+ justify="start",
+ )
+ ],
+ )
+ q.client.delete_cards.add("dataset/display/footer")
+
+
+async def show_data_tab(q, cfg, filename: str):
+ fill_columns = get_fill_columns(cfg)
+ df = read_dataframe(filename, n_rows=200, fill_columns=fill_columns)
+ q.page["dataset/display/data"] = ui.form_card(
+ box="first",
+ items=[
+ ui_table_from_df(
+ q=q,
+ df=df,
+ name="dataset/display/data/table",
+ sortables=list(df.columns),
+ height="calc(100vh - 265px)",
+ cell_overflow="wrap",
+ )
+ ],
+ )
+ q.client.delete_cards.add("dataset/display/data")
+
+
+async def show_visualization_tab(q, cfg):
+ try:
+ plot = cfg.logging.plots_class.plot_data(cfg)
+ except Exception as error:
+ logger.error(f"Error while plotting data preview: {error}", exc_info=True)
+ plot = PlotData("" in line: + in_pre += 1 + if "" in line: + in_pre -= 1 + if in_pre == 0: + text += "
"
+ + subprocess.check_output(["nvidia-smi"]).decode("utf-8")
+ + ""
+ )
+
+ if torch.cuda.is_available():
+ logging.info(
+ "" + torch.cuda.memory_summary().replace("-", "=") + ""
+ )
+
+ elif isinstance(exception, LLMDataException):
+ logging.error(
+ "Data error occurred during H2O LLM Studio run:", exc_info=True
+ )
+ write_flag(flag_path.format(local_rank), "info", "Data error")
+ elif isinstance(exception, LLMTrainingException):
+ logging.error(
+ "Training error occurred during H2O LLM Studio run:", exc_info=True
+ )
+ write_flag(flag_path.format(local_rank), "info", "Training error")
+ elif isinstance(exception, LLMMetricException):
+ logging.error(
+ "Validation metric failed. Please make sure selected validation "
+ "metric is suitable for your current problem setup.",
+ exc_info=True,
+ )
+ write_flag(flag_path.format(local_rank), "info", "Metric error")
+ elif isinstance(exception, LLMAugmentationsException):
+ logging.error(
+ "Custom augmentations error occurred during " "H2O LLM Studio run:",
+ exc_info=True,
+ )
+ write_flag(flag_path.format(local_rank), "info", "Augmentations error")
+ elif isinstance(exception, LLMModelException):
+ logging.error(
+ "Model error occurred during H2O LLM Studio run:",
+ exc_info=True,
+ )
+ write_flag(flag_path.format(local_rank), "info", "Model error")
+ else:
+ logging.error(
+ "Exception occurred during H2O LLM Studio run:", exc_info=True
+ )
+ write_flag(flag_path.format(local_rank), "info", "See logs")
+ kill_ddp_processes()