Hugging Face's logo

BlinkDL
/
rwkv-4-pile-7b

Text Generation
PyTorch
English
causal-lm
rwkv
Model card Files Community
3
BlinkDL commited on
Commit
2bb4c10
•
1 Parent(s): 3f8e4ef

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -0
README.md CHANGED
@@ -27,6 +27,9 @@ n_embd = 4096
27
  RWKV-4-Pile-7B-20230109-ctx4096.pth : Fine-tuned to ctx_len 4096.
28
  * Likely better. Please test.
29
 
 
 
 
30
  RWKV-4-Pile-7B-20221115-8047.pth : Trained on the Pile for 332B tokens.
31
  * Pile loss 1.8415T
32
  * LAMBADA ppl 4.38, acc 67.18%
 
27
  RWKV-4-Pile-7B-20230109-ctx4096.pth : Fine-tuned to ctx_len 4096.
28
  * Likely better. Please test.
29
 
30
+ RWKV-4-Pile-7B-20230xxx-ctx8192-testxxx : Fine-tuned to ctx_len 8192.
31
+ * Slightly weaker than ctx4096 model when ctxlen < 3k.
32
+
33
  RWKV-4-Pile-7B-20221115-8047.pth : Trained on the Pile for 332B tokens.
34
  * Pile loss 1.8415T
35
  * LAMBADA ppl 4.38, acc 67.18%