--- license: apache-2.0 tags: - moe - mixtral --- # MetaModel_moe This model is a Mixure of Experts (MoE) made with [mergekit](https://github.com/cg123/mergekit) (mixtral branch). It uses the following base models: * [gagan3012/MetaModel](https://huggingface.co/gagan3012/MetaModel) * [jeonsworld/CarbonVillain-en-10.7B-v2](https://huggingface.co/jeonsworld/CarbonVillain-en-10.7B-v2) * [jeonsworld/CarbonVillain-en-10.7B-v4](https://huggingface.co/jeonsworld/CarbonVillain-en-10.7B-v4) * [TomGrc/FusionNet_linear](https://huggingface.co/TomGrc/FusionNet_linear) ## 🧩 Configuration ```yaml base_model: gagan3012/MetaModel gate_mode: hidden dtype: bfloat16 experts: - source_model: gagan3012/MetaModel - source_model: jeonsworld/CarbonVillain-en-10.7B-v2 - source_model: jeonsworld/CarbonVillain-en-10.7B-v4 - source_model: TomGrc/FusionNet_linear ``` ## 💻 Usage ```python !pip install -qU transformers bitsandbytes accelerate from transformers import AutoTokenizer import transformers import torch model = "gagan3012/MetaModel_moe" tokenizer = AutoTokenizer.from_pretrained(model) pipeline = transformers.pipeline( "text-generation", model=model, model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True}, ) messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}] prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) print(outputs[0]["generated_text"]) ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_gagan3012__MetaModel_moe) | Metric | Value | |-----------------------|---------------------------| | Avg. | 74.42 | | ARC (25-shot) | 71.25 | | HellaSwag (10-shot) | 88.4 | | MMLU (5-shot) | 66.26 | | TruthfulQA (0-shot) | 71.86 | | Winogrande (5-shot) | 83.35 | | GSM8K (5-shot) | 65.43 |