---
license: apache-2.0
---
# RWKV-x070-2B9-CJE-Instruct Model Card

## Model Overview
- **Model Name**: RWKV-x070-2B9-CJE-Instruct
- **Description**: An instruction-tuned model specialized for Japanese, Chinese, and English languages
- **Base Model**: rwkv-x070-2b9-world-v3-40%trained-20250113-ctx4k.pth
- **Architecture**: RWKV x070 "Goose"
- **Parameters**: 2.9B
- **Model Dimension**: 2560
- **Number of Layers**: 32

## Fine-tuning Details

### Training Configuration
- **Trainer**: RWKV-LM-RLHF (https://github.com/OpenMOSE/RWKV-LM-RLHF)
- **PEFT Mode**: Hybrid Training combining frozen embeddings and Bone (Block Affine Transformation) + full parameter training
- **SFT Method**: SmoothingLoss SFT
- **Context Window**: 5120 (trained with 1024 token overlap)
- **Compute Power**: AMD Instinct MI100 x 2 60hrs (100％ solar energy)

### Dataset Specifications
- **Size**: 800k pairs
- **Content**:
  - Mixed data in Japanese, Chinese, and English
  - Conversations
  - Programming code
  - Translation tasks
  - Chain-of-Thought reasoning tasks

### How to use
- Install latest RWKV-Infer (Linux,WSL) (https://github.com/OpenMOSE/RWKV-Infer)
- make folder 'models'
- move rwkv-x070-2b9-cje-instruct-1.pth to models folder
```
curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"models/rwkv-x070-2b9-cje-instruct-1.pth","model_viewname":"RWKV x070 2B9 CJE Instruct-1","model_strategy":"fp16","endtoken":"\\n\\n\\x17"}'
```
- Enjoy with openai compatible api http://127.0.0.1:9000/v1 :)

### Important Note
- Set the end token as '\n\n\x17'
```
User: who are you?\n\n\x17
Assistant: gooday i'm rwkv\n\n\x17
```

### Limitations and Considerations
- This is an experimental model; inference stability is not fully guaranteed
- Unexpected behaviors may occur
- Continuous improvements are being made; feedback is welcome

## License
Apache License 2.0

## Acknowledgments
We express our gratitude to the RWKV base model and the RWKV community for their support in developing this model.