|
--- |
|
base_model: |
|
- codellama/CodeLlama-13b-Instruct-hf |
|
language: |
|
- en |
|
license: apache-2.0 |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- unsloth |
|
- llama |
|
- trl |
|
datasets: |
|
- rasa/command-generation-calm-demo-v1 |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# Model Card for Command Generator |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This is a Dialogue Understanding (DU) model developed by Rasa. |
|
It can be used to power assistants built with the [Conversational AI with Language Models (CALM) approach](https://rasa.com/docs/rasa-pro/calm) developed by Rasa. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
This model takes as input a transcript of an ongoing conversation between an AI assistant and a user, |
|
as well as structured information about the assistant's business logic. |
|
As output, it produces a short sequence of [commands](https://rasa.com/docs/rasa-pro/concepts/dialogue-understanding#command-reference) |
|
(typically 1-3) from the following list: |
|
|
|
* `StartFlow(flow_name)` |
|
* `SetSlot(slot_name, slot_value)` |
|
* `CorrectSlot(slot_name, slot_value)` |
|
* `Clarify(flow_name_1, flow_name_2, ...)` |
|
* `ChitChat` |
|
* `KnowledgeAnswer` |
|
* `HumanHandoff` |
|
* `Error` |
|
|
|
Note that this model can only produce commands to be interpreted by Rasa. |
|
It **cannot** be used to generate arbitrary text. |
|
|
|
The Command Generator translates user messages into this internal grammar, allowing CALM to progress the conversation. |
|
|
|
Examples: |
|
|
|
> I want to transfer money |
|
|
|
`StartFlow(transfer_money)` |
|
|
|
> I want to transfer $55 to John |
|
|
|
`StartFlow(transfer_money), SetSlot(recipient, John), SetSlot(amount, 55)` |
|
|
|
- **Developed by:** Rasa Technologies |
|
- **Model type:** Text Generation |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
- **Finetuned from model [optional]:** [CodeLlama 13b Instruct](https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf) |
|
|
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
The Command Generator is used as part of an AI assistant developed with Rasa's CALM paradigm. |
|
Typical use cases include customer-facing chatbots, voice assistants, IVR systems, and internal chatbots in large organizations. |
|
|
|
### Direct Use |
|
|
|
This model can be directly used as part of the command generator component if the flows of your CALM assistant are similar to flows used in the [rasa-calm-demo assistant](https://github.com/RasaHQ/rasa-calm-demo). |
|
|
|
### Downstream Use [optional] |
|
|
|
The model can also be used as a base model to fine-tune further on your own assistant's data using the [fine-tuning recipe feature](https://rasa.com/docs/rasa-pro/building-assistants/fine-tuning-recipe#step-2-prepare-the-fine-tuning-dataset) available in rasa pro |
|
|
|
### Out-of-Scope Use |
|
|
|
Since the model has been explicitly fine-tuned to output the grammar of commands, it shouldn't be used to generate any other free form content. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
The Command Generator interprets conversations and translates user messages into commands. |
|
These commands are processed by Rasa to advance the conversations. |
|
This model does not generate text to be sent to and end user, and is incapable of generating problematic |
|
or harmful text. |
|
|
|
However, as with any pre-trained model, its predictions are susceptible to bias. |
|
For example, the accuracy of the model varies with the language used. The authors have tested the performance on English but haven't tried the model in any other language. |
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
Trained on the `train` split of [rasa/command-generation-calm-demo-v1](https://huggingface.co/datasets/rasa/command-generation-calm-demo-v1). |
|
|
|
### Training Procedure |
|
|
|
Trained using the notebook available [here](https://github.com/RasaHQ/notebooks/blob/main/cmd_gen_finetuning.ipynb). Used a single A100 GPU with 80GB VRAM. |
|
|
|
## Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
Evaluated on the `test` split of [rasa/command-generation-calm-demo-v1](https://huggingface.co/datasets/rasa/command-generation-calm-demo-v1). |
|
|
|
|
|
#### Metrics |
|
|
|
F1 score per command type (StartFlow, SetSlot, etc.) is the main metric chosen to evaluate the model on the test split. |
|
This helps us understand which commands the model has learnt well and the ones that the model needs more training on. |
|
|
|
### Results |
|
|
|
The below table shows the F1 score per command for this model. |
|
|
|
| Model | StartFlow | SetSlot | Cancel | Clarify | Chitchat | SearchAndReply | SkipQuestion | |
|
|--------------------------------------|-----------|---------|--------|---------|----------|----------------|--------------| |
|
| rasa/cmd_gen_codellama_13b_calm_demo | 0.9722 | 0.9239 | 0.6667 | 0.8889 | 1 | 0 | 0.8 | |
|
|
|
|
|
|
|
|
|
|
|
## Model Card Contact |
|
|
|
[More Information Needed] |
