Create README.md

README.md

@@ -0,0 +1,40 @@
+---
+library_name: transformers
+datasets:
+- julep-ai/samantha_finetune_dataset_03
+language:
+- en
+---
+
+## Samantha
+
+### Technical notes
+
+This model is trained on a specialized dataset and uses special sentinel tokens to demarcate conversations.
+
+**Important Note: These sentinels are similar to gpt2-style special tokens but they are <u>NOT</u> added as special tokens in the tokenizer.**
+
+**Concepts**:
+- Each conversation consists of n "sections"
+- Each section can be one of:
+  + `me`: The model
+  + `person`: The speaker
+  + `situation`: relevant background information to set the context of the conversation
+  + `thought`: Thoughts generated by the model for parsing intermediate steps etc
+  + `information`: External information added into the context by the system running the model
+- The model and speaker sections can optionally include a name like `me (Samantha)` or `person (Dmitry)`
+
+**Tokens**:
+- `<|section|>` token marks the start of a "section"
+- `<|endsection|>` token marks the end of a "section". This is also set to be the default `EOS` token in the tokenizer
+- these are both "special" tokens and are not split up by the tokenizer
+
+### Example
+
+```
+<|section|>situation
+I am talking to Diwank. I want to ask him about his food preferences.<|endsection|>
+<|section|>person (Diwank)
+Hey Samantha! What do you want to talk about?<|endsection|>
+<|section|>me (Samantha)
+```