Duplicate from BlinkDL/rwkv-6-world

Co-authored-by: BlinkDL <BlinkDL@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,104 @@
+---
+language:
+- en
+- zh
+- fr
+- es
+- de
+- pt
+- ru
+- it
+- ja
+- ko
+- vi
+- ar
+tags:
+- pytorch
+- text-generation
+- causal-lm
+- rwkv
+license: apache-2.0
+datasets:
+- cerebras/SlimPajama-627B
+- EleutherAI/pile
+- bigcode/starcoderdata
+- oscar-corpus/OSCAR-2301
+---
+
+# RWKV-6 World
+
+Use rwkv pip package 0.8.24+ for RWKV-6 inference: https://pypi.org/project/rwkv/ (pipeline = PIPELINE(model, "rwkv_vocab_v20230424") for rwkv-world models)
+
+Online Demo 1: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2
+
+Online Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1
+
+GUI: https://github.com/josStorer/RWKV-Runner (see Releases)
+
+For developer: https://github.com/BlinkDL/ChatRWKV/blob/main/API_DEMO_CHAT.py
+
+https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_v6_demo.py
+
+How it works: https://twitter.com/BlinkDL_AI/status/1685230712247795713
+
+https://www.rwkv.com/
+
+## Model Description
+
+RWKV-6 trained on 100+ world languages (70% English, 15% multilang, 15% code).
+
+World = Some_Pile + Some_SlimPajama + Some_StarCoder + Some_OSCAR + All_Wikipedia + All_ChatGPT_Data_I_can_find
+
+World v1 = 0.59T tokens
+
+World v2 = 1.12T tokens
+
+World v2.1 = 1.42T tokens
+
+Recommended fine-tuning format (use \n for newlines):
+```
+User: xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+User: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+```
+
+A good chat prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
+```
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: xxx
+
+Assistant:
+```
+QA prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
+```
+Question: xxx
+
+Answer:
+```
+and
+```
+Instruction: xxx
+
+Input: xxx
+
+Response:
+```
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!

RWKV-x060-World-1B6-v2-20240208-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c9c877fb60a65cab269af175328b6aaf16d02b8b09738923254f9986e5dc440
+size 3199845663

RWKV-x060-World-1B6-v2.1-20240328-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cda1f0ebe802e2859bfa129546372fd1eac60319111f215f65ced67dd334db36
+size 3199845663

RWKV-x060-World-3B-v2-20240228-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1235079a07084472de86996846a6533e41e3964b153cdc1e8462cc138d8521d
+size 6199859158