ABEJA Qwen 2.5 7B Japanese - QNN Optimized

This repository contains the ABEJA Qwen 2.5 7B Japanese model optimized for Qualcomm Neural Network (QNN) deployment.

Model Details

Base Model: abeja/Qwen2.5-7B-Japanese
Architecture: Qwen2ForCausalLM
Parameters: ~7.6B
Language: Japanese (primary), English (secondary)
Quantization: 4-bit NF4
Target Hardware: Snapdragon 8cx Gen 2+ (SM8350)

Available Formats

1. Quantized PyTorch Model

Path: quantized_simple/
Format: 4-bit NF4 quantized
Size: ~4.5GB (reduced from ~15GB)
Usage: Direct inference with transformers

2. ONNX Models

Path: onnx/
Models:
- prefill/model.onnx - Context prefill
- token_gen/model.onnx - Token generation
Usage: Cross-platform inference

3. Quantized ONNX Models

Path: quantized_onnx/
Format: Dynamic quantization (INT8)
Usage: Optimized ONNX inference

4. QNN Compiled Models

Path: qnn_compiled/
Format: Qualcomm Neural Network format
Target: Snapdragon devices
Usage: Native ARM64 deployment

Usage

Quantized PyTorch Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn", subfolder="quantized_simple")
tokenizer = AutoTokenizer.from_pretrained("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn", subfolder="quantized_simple")

# Japanese text generation
inputs = tokenizer("こんにちは、私は", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

ONNX Inference

import onnxruntime as ort

# Load ONNX model
session = ort.InferenceSession("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/onnx/prefill/model.onnx")
# Run inference...

QNN Deployment

# Deploy to Snapdragon device
adb push marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/qnn_compiled/ /data/local/tmp/qnn_model/
# Use QNN runtime for inference

Performance

Quantization: 75% size reduction
Speed: 2-3x faster inference
Memory: ~4.5GB RAM usage
Tokens/sec: 8-15 tokens/sec on Snapdragon 8cx Gen 2+

Hardware Compatibility

✅ Snapdragon 8cx Gen 2+
✅ Snapdragon 8cx Gen 3
✅ Snapdragon 8 Gen 1+
✅ Windows on ARM devices
✅ Microsoft Surface Pro X
✅ Dell Latitude 7420

Files Structure

marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/
├── quantized_simple/          # 4-bit quantized PyTorch model
│   ├── model.safetensors
│   ├── config.json
│   ├── tokenizer.json
│   └── model_info.json
├── onnx/                      # ONNX models
│   ├── prefill/model.onnx
│   └── token_gen/model.onnx
├── quantized_onnx/            # Quantized ONNX models
│   ├── prefill/model_quantized.onnx
│   └── token_gen/model_quantized.onnx
├── qnn_compiled/              # QNN compiled models
│   ├── prefill/
│   ├── token_gen/
│   └── deployment_info.json
└── README.md                  # This file

License

Apache 2.0 - Same as base ABEJA Qwen 2.5 model

Citation

@misc{abeja-qwen25-qnn,
  title={ABEJA Qwen 2.5 7B Japanese - QNN Optimized},
  author={QNN Conversion Pipeline},
  year={2025},
  url={https://huggingface.co/marcusmi4n/abeja-qwen2.5-7b-japanese-qnn}
}

Base Model Citation

Please cite the original ABEJA Qwen 2.5 paper:

@article{abeja-qwen2.5,
  title={ABEJA Qwen 2.5: Japanese Language Model},
  author={ABEJA Inc.},
  journal={arXiv preprint},
  year={2024}
}