|
--- |
|
license: apache-2.0 |
|
base_model: |
|
- mistralai/Mistral-7B-Instruct-v0.3 |
|
--- |
|
|
|
|
|
# mistralai/Mistral-7B-Instruct-v0.3 |
|
- ## Introduction |
|
- Quantization Tool: Quark 0.6.0 |
|
- OGA Model Builder: v0.5.1 |
|
- Postprocess |
|
- ## Quantization Strategy |
|
- AWQ / Group 128 / Asymmetric / UINT4 Weights / FP16 activations |
|
- Excluded Layers: None |
|
``` |
|
python3 quantize_quark.py \ |
|
--model_dir "$model" \ |
|
--output_dir "$output_dir" \ |
|
--quant_scheme w_uint4_per_group_asym \ |
|
--num_calib_data 128 \ |
|
--quant_algo awq \ |
|
--dataset pileval_for_awq_benchmark \ |
|
--seq_len 512 \ |
|
--model_export quark_safetensors \ |
|
--data_type float16 \ |
|
--exclude_layers [] \ |
|
--custom_mode awq |
|
``` |
|
- ## OGA Model Builder |
|
``` |
|
python builder.py \ |
|
-i <quantized safetensor model dir> \ |
|
-o <oga model output dir> \ |
|
-p int4 \ |
|
-e dml |
|
``` |
|
- PostProcessed to generate Hybrid Model |
|
- |
|
- ## Quick Start |
|
For quickstart, refer to hybrid-llm-artifacts_1.3.0.zip available in [RyzenAI-SW-EA](https://account.amd.com/en/member/ryzenai-sw-ea.html) |
|
|
|
|
|
#### Evaluation scores |
|
The perplexity measurement is run on the wikitext-2-raw-v1 (raw data) dataset provided by Hugging Face. Perplexity score measured for prompt length 2k is 5.7053. |
|
|
|
#### License |
|
Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved. |
|
|
|
MIT License |
|
|
|
Copyright (c) 2024 Advanced Micro Devices, Inc |
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal |
|
in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell |
|
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: |
|
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. |
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, |
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER |
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE |
|
SOFTWARE. |
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); |
|
you may not use this file except in compliance with the License. |
|
You may obtain a copy of the License at |
|
|
|
http://www.apache.org/licenses/LICENSE-2.0 |
|
|
|
Unless required by applicable law or agreed to in writing, software |
|
distributed under the License is distributed on an "AS IS" BASIS, |
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
See the License for the specific language governing permissions and |
|
limitations under the License. |
|
|
