File size: 840 Bytes
955c824 d1e0141 955c824 bcbdc05 955c824 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# Bringing SOTA quantization to mobile LLM deployment: A practical Executorch integration guide
Article: https://blacksamorez.substack.com/p/aqlm-executorch-android
## Usage
- Download and install the `.apk` file on your Android phone. (`llama3-aqlm.apk` for ~1.1 tok/s at low power consumption. `llama3-aqlm-4cores.apk` for ~2.7 tok/s at high loads)
- Download the `.pte` and `.model` files and put them into the `/data/local/tmp/llama` folder on your Android phone.
- Running the app you will see the option to load the `.pte` and `.model` files. After loading them, you'll be able to chat with the model.
## Requirements
This app was tested on `Samsung S24 Ultra` running `Android 14`.
## Limitations
- Although the app looks like chat, generation requests are independent.
- Llama-3 chat template is hard-coded into the app. |