Create README.md

README.md

@@ -0,0 +1,55 @@
+---
+base_model: NousResearch/Nous-Hermes-2-Yi-34B
+inference: true
+model_type: yi
+quantized_by: robertgshaw2
+tags:
+- nm-vllm
+- marlin
+- int4
+---
+
+## Nous-Hermes-Yi-34B-marlin
+This repo contains model files for [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) optimized for [nm-vllm](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
+
+This model was quantized with [GPTQ](https://arxiv.org/abs/2210.17323) and saved in the Marlin format for efficient 4-bit inference. Marlin is a highly optimized inference kernel for 4 bit models.
+
+## Inference
+Install [nm-vllm](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
+```bash
+pip install nm-vllm[sparse]
+```
+
+Run in a Python pipeline for local inference:
+```python
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+
+model_id = "neuralmagic/Nous-Hermes-2-Yi-34B-marlin"
+model = LLM(model_id)
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+messages = [
+    {"role": "user", "content": "What is synthetic data in machine learning?"},
+]
+formatted_prompt =  tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+sampling_params = SamplingParams(max_tokens=200)
+outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+
+"""
+Synthetic data is data that has been artificially created or modified to serve the needs of machine learning and data analysis tasks. It can be generated either through title methods like stochastic simulations or through processes of data augmentation that take original data and modify/manipulate it to create new samples. Synthetic data is often used in machine learning when the available amount of real-world data is insufficient or in cases where the creation of real-world data can be dangerous, costly, or time-consuming.
+"""
+```
+
+## Quantization
+For details on how this model was quantized and converted to marlin format, run the `quantization/apply_gptq_save_marlin.py` script:
+
+```bash
+pip install -r quantization/requirements.txt
+python3 quantization/apply_gptq_save_marlin.py --model-id NousResearch/Nous-Hermes-2-Yi-34B --save-dir ./nous-hermes-2-yi-34b-marlin
+```
+
+## Slack
+
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)