Update README.md

README.md

@@ -1,3 +1,112 @@
 ---
 license: wtfpl
+datasets:
+- migtissera/Synthia-v1.3
+language:
+- en
 ---
+
+
+# MAMBA (2.8B) 🐍 fine-tuned on Synthia-v1.3
+
+<div style="text-align:center;width:250px;height:250px;">
+    <img src="https://huggingface.co/clibrain/mamba-2.8b-instruct-openhermes/resolve/main/mamba_hermes_logo_1.png?download=true" alt="mamba-hermes logo"">
+</div>
+
+Model Card is still WIP!
+
+
+## Base model info
+
+Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers.
+It is based on the line of progress on [structured state space models](https://github.com/state-spaces/s4),
+with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).
+
+## Dataset info
+
+The OpenHermes dataset is composed of 242,000 entries of primarily GPT-4 generated data, from open datasets across the AI landscape, including:
+
+OpenHermes 13B is the first fine tune of the Hermes dataset that has a fully open source dataset!
+
+OpenHermes was trained on 242,000 entries of primarily GPT-4 generated data, from open datasets across the AI landscape, including:
+
+- GPTeacher - General Instruct, Roleplay v1, Roleplay v2, and Code Instruct Datasets, by Teknium
+- WizardLM (v1, evol_instruct 70k), by WizardLM Team/nlpxucan
+- Airoboros GPT-4 (v1.0), by JonDurbin
+- Camel-AI's domain expert datasets, by the Camel-AI Team
+- CodeAlpaca, by Sahil2801
+- GPT4-LLM and Unnatural Instructions, by Microsoft
+Filtering included removal of OpenAI refusals, disclaimers, and "As an AI" type examples and more
+The base dataset mix is identical to the original Nous-Hermes', minus the Nous-Instruct and PDACTL datasets which were private datasets.
+
+## Usage
+
+```sh
+pip install torch==2.1.0 transformers==4.35.0 causal-conv1d==1.0.0 mamba-ssm==1.0.1
+```
+
+```py
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
+
+CHAT_TEMPLATE_ID = "HuggingFaceH4/zephyr-7b-beta"
+
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model_name = "clibrain/mamba-2.8b-ft-synthia-v1.3"
+
+eos_token = "<|endoftext|>"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+tokenizer.eos_token = eos_token
+tokenizer.pad_token = tokenizer.eos_token
+tokenizer.chat_template = AutoTokenizer.from_pretrained(CHAT_TEMPLATE_ID).chat_template
+
+model = MambaLMHeadModel.from_pretrained(
+        model_name, device=device, dtype=torch.float16)
+
+messages = []
+prompt = "Tell me 5 sites to visit in Spain"
+messages.append(dict(role="user", content=prompt))
+
+input_ids = tokenizer.apply_chat_template(
+            messages, return_tensors="pt", add_generation_prompt=True
+).to(device)
+
+out = model.generate(
+    input_ids=input_ids,
+    max_length=2000,
+    temperature=0.9,
+    top_p=0.7,
+    eos_token_id=tokenizer.eos_token_id,
+)
+
+decoded = tokenizer.batch_decode(out)
+assistant_message = (
+    decoded[0].split("<|assistant|>\n")[-1].replace(eos, "")
+)
+
+print(assistant_message)
+```
+
+
+## Gradio Demo
+
+```sh
+git clone https://github.com/mrm8488/mamba-chat.git
+cd mamba-chat
+
+pip install -r requirements.txt
+pip install -q gradio==4.8.0
+
+python app.py \
+--model clibrain/mamba-2.8b-ft-synthia-v1.3 \
+--share
+```
+## Evaluations
+
+Coming soon!
+
+
+## Acknowledgments
+
+Thanks to [mamba-chat](https://github.com/havenhq/mamba-chat/tree/main) for heavily inspiring our work