Upload folder using huggingface_hub

README.md

@@ -0,0 +1,86 @@
+---
+license: mit
+language:
+- en
+datasets:
+- nampdn-ai/tiny-textbooks
+---
+
+# Nuclues 1B Alpha1
+
+<p align="center">
+    <img src="https://github.com/prp-e/nucleus/raw/main/nucleus-logo.png" width=256 height=256>
+</p>
+
+## What is Nucleus?
+
+Nucleus is a small language model based on Mistral (actually, the trimmed untrained version you can find [here](https://huggingface.co/lmlab/lmlab-mistral-1b-untrained)) and trained in different steps. First, we've pretrained it on TinyStories dataset, then [TinyTextBooks](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks) to make it a more specific model. This model is just a _proof of concept_ at this point, but showed good promises in early tests. So with proper training, can be a good product over time!
+
+## Inference
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/prp-e/nucleus/blob/main/nucleus_1b_inference.ipynb)
+
+First you need to install `transformers` and `accelerate` libraries in order to run this model. Then, you basically have to run the following code:
+
+```python
+
+from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
+import torch
+
+model_name_or_id = "NucleusOrg/Nucleus-1B-alpha-1"
+
+model = AutoModelForCausalLM.from_pretrained(model_name_or_id, torch_dtype=torch.float16, device_map="cuda")
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_id)
+
+prompt = "### Lesson: Python Programming 101\n### Introduction\n"
+
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+generation_config = GenerationConfig(
+    do_sample=True,
+    top_k=1,
+    temperature=0.9,
+    max_new_tokens=500,
+    repetition_penalty=1.5,
+    pad_token_id=tokenizer.eos_token_id
+)
+
+outputs = model.generate(**inputs, generation_config=generation_config)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+```
+
+__Prompt Format__: This model does not have a specific prompt format, but the best results could be achieved with a _textbook_ type of format like:
+
+```
+### Chapter 1: Elon Musk and Iron Man
+Elon met Tony at a Cafe in Monaco, then they had a conversation about
+```
+
+You also can try something like this: 
+
+```
+Question: Who are you?
+Answer:
+```
+
+But since the model isn't made for chat/question answering, the result won't be good enough. 
+
+__Repetition Penalty__: Since most of these models like to repeat themselves, just keep that number there. You can increase or decrease it based on your liking,but keep in mind that a number lower than 1 makes the model _super repetitive_. 
+
+## Known Issues
+
+* Since we only had 420k rows of data, a lot of information are missing on this model. Since mentioned earlier in this very model card, it's a _proof of concept_ model.
+* You probably may test it with coding. Let's say that the model is terrible at coding. We may release a coding optimized model as soon as possible. 
+
+## Our Team
+
+* Muhammadreza Haghiri ([X (formerly Twitter)](https://twitter.com/haghiri_ai) - [Website](https://haghiri75.com/en) - [Github](https://github.com/prp-e) - [LinkedIn](https://www.linkedin.com/in/muhammadreza-haghiri-1761325b))
+* Mahi Mohrechi ([Website](https://mohrechi-portfolio.vercel.app/) - [Github](https://github.com/f-mohrechi) - [LinkedIn](https://www.linkedin.com/in/faeze-mohrechi/))
+
+## Special Thanks
+
+* LMLabs for providing 1B untrained model. 
+* Mistral Team for providing the best open source base model ever.
+* _Sina Rashidi_, who translated Alpaca dataset to Persian.
+* [Jupyto](https://jupyto.com) team for providing our infrastructure.

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "/jupyter/Nucleus-1B-alpha-1",
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.36.2",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.36.2"
+}

model.safetensors.index.json

@@ -0,0 +1,46 @@
+{
+  "metadata": {
+    "total_size": 2269192192
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00003-of-00003.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.norm.weight": "model-00003-of-00003.safetensors"
+  }
+}

output.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24307c16301af8031c1736a01abcd5fd1fa4ce623c0f3cff9e572b9ac95ab21d
+size 1020680752

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}