Hugging Face's logo

BlinkDL
/
rwkv-4-pile-1b5

Text Generation
PyTorch
English
causal-lm
rwkv
Model card Files Community
BlinkDL commited on
Commit
933876b
•
1 Parent(s): b9622cb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -0
README.md CHANGED
@@ -26,6 +26,8 @@ ctx_len = 1024
26
  n_layer = 24
27
  n_embd = 2048
28
 
 
 
29
  Final checkpoint: RWKV-4-Pile-1B5-20220903-8040.pth : Trained on the Pile for 332B tokens.
30
  * Pile loss 2.0415
31
  * LAMBADA ppl 7.04, acc 56.43%
 
26
  n_layer = 24
27
  n_embd = 2048
28
 
29
+ New checkpoint: RWKV-4-Pile-1B5-20220929-ctx4096.pth : Fine-tuned to ctx_len = 4096
30
+
31
  Final checkpoint: RWKV-4-Pile-1B5-20220903-8040.pth : Trained on the Pile for 332B tokens.
32
  * Pile loss 2.0415
33
  * LAMBADA ppl 7.04, acc 56.43%