File size: 11,869 Bytes
76e3315 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 |
---
license: apache-2.0
tags:
- moe
- frankenmoe
- merge
- mergekit
- lazymergekit
- mlabonne/NeuralBeagle14-7B
- fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser
- mlabonne/Marcoro14-7B-slerp
base_model:
- mlabonne/NeuralBeagle14-7B
- fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser
- mlabonne/Marcoro14-7B-slerp
---
# CultriX-MoE-BF16
CultriX-MoE-BF16 is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [mlabonne/NeuralBeagle14-7B](https://huggingface.co/mlabonne/NeuralBeagle14-7B)
* [fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser)
* [mlabonne/Marcoro14-7B-slerp](https://huggingface.co/mlabonne/Marcoro14-7B-slerp)
## 🧩 Configuration
```yaml
base_model: "EmbeddedLLM/Mistral-7B-Merge-14-v0.2"
gate_mode: hidden
dtype: bfloat16
experts:
- source_model: "mlabonne/NeuralBeagle14-7B"
positive_prompts:
- "Create a story based on"
- "Debate the topic of"
- "Come up with some arguments"
- "Provide me with instructions on"
- "Interpret the sentiment"
- "Interpret and execute these cooking instructions"
- "Craft a persuasive argument"
- "Analyze the motivations"
- "Construct a detailed plan for"
- "Narrate an event from multiple perspectives."
- "Formulate a response"
- "Write a script for a short play"
- "Generate a sequence of instructions to teach a skill."
- "Solve this riddle"
- "Create an engaging story"
- "Write a fictional"
- "Propose a solution to a social issue"
- "Develop a dialogue"
- "Create a step-by-step guide"
- "Devise a strategy"
- "Write a narrative"
- "Tell me how to"
- "Explain the concept of"
- "Give an overview of"
- "Compare and contrast between"
- "Provide information about"
- "Help me understand"
- "Summarize"
- "Make a recommendation on"
- "Answer this question"
- "How do you approach"
- "Explain the concept of"
- "Give an overview of"
- "Provide information about"
- "Help me understand the principles of"
- "Summarize the key components of"
- "Make a recommendation on how to"
- "Answer this question:"
negative_prompts:
- "Provide in-depth information about quantum computing."
- "Explain the inner workings of an internal combustion engine."
- "Give a detailed tutorial on advanced calculus."
- "Summarize the latest research in genetic engineering."
- "Interpret financial markets and stock trends."
- "Analyze the chemical composition of"
- "Develop a blueprint for."
- "Offer a critique of a modern art piece."
- "Provide a technical review of"
- "Conduct a linguistic analysis of an ancient language."
- "Write a user manual for advanced medical equipment."
- "Give a step-by-step guide on piloting an aircraft."
- "Conduct an in-depth analysis of this code"
- "Explain the physics behind black holes."
- "Provide a strategy for managing a cyber attack"
- "Develop an algorithm for predictive analytics in finance."
- "Provide information about advanced programming algorithms."
- "Help me understand the details of this code"
- "Summarize the process of cellular respiration."
- "Improve the security of"
- "What are the latest advancements in artificial intelligence?"
- "Provide detailed technical coding solutions."
- "Analyze complex scientific data and statistics."
- "Offer medical diagnoses based on symptoms."
- "Conduct a detailed financial audit of a company."
- "Perform real-time translation of multiple languages."
- "Create high-resolution graphic designs."
- "Develop complex mathematical proofs."
- "Offer legal advice on specific cases."
- "Write a detailed manual on advanced mechanical engineering."
- "Conduct an in-depth psychological assessment."
- "Perform a security analysis of a computer network."
- "Compose an original piece of music."
- "Plan and execute a scientific experiment."
- "Provide professional career counseling."
- "Develop a complex database management system."
- "Write a software program for data analysis."
- "Give expert advice on cyber"
- "Conduct a pentesting security audit"
- source_model: "fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser"
positive_prompts:
- "Provide step-by-step coding instructions for..."
- "Draft a function with detailed steps in [language]"
- "Guide me through coding a simple [type of application or script]"
- "Recommend best practices for code implementation in [context]"
- "Generate a regex pattern for extracting [specific data]"
- "Create a regex for matching [pattern]"
- "Explain the purpose of this regex pattern"
- "Compose regex for [specific use case]"
- "Annotate this code with detailed comments for each line"
- "Add explanatory comments to this script"
- "Comment on each part of this code for clarity"
- "Develop a script to [accomplish task]"
- "Design a database schema for [specific use case]"
- "Outline secure methods for [specific operation]"
- "Guide on optimizing [specific aspect] in this code"
- "Refactor this code for better readability and efficiency"
- "Compare and contrast these code snippets"
- "Identify the programming language of this snippet"
- "Demonstrate the usage of [specific tool/library/API]"
- "Show implementation steps for this [feature/concept]"
- "Teach how to use [specific tool/library/framework]"
- "Generate a README file for this project"
- "Create a manual page for [specific tool/command]"
- "Produce comprehensive documentation for this code"
- "Build detailed documentation for [specific module]"
- "Explain the underlying concept of this code snippet"
- "Propose enhancements for this script"
- "Suggest improvements for this API call integration"
- "Diagnose and solve this coding issue"
- "Demonstrate robust error handling in this code"
- "Debug and resolve issues in this script"
- "Design a user-friendly GUI for this script's functionality"
- "Detail the deployment process for this application"
- "Deploy an app designed to [perform function]"
- "Set up a web service for [specific purpose]"
- "Develop a website with [specific features]"
- "Craft a webpage showcasing [specific content]"
- "Illustrate data flow in this code architecture"
- "Convert this code from [language A] to [language B]"
- "Translate this script into [different programming language]"
- "Explain resource management techniques in [context]"
- "Build a basic API endpoint for [functionality]"
- "Strategies to enhance scalability in [context]"
- "Conduct a security review for this code"
- "Enhance security measures in [application/module]"
- "Set up a development environment for [language/framework]"
- "Visualize data from [specific dataset]"
- "Generate a dataset for [specific use case]"
- "Scripting guide for automating [task/process]"
- "Utilize this code for [specific purpose]"
- "Principles of object-oriented programming in [language]"
- "Create a mobile-responsive layout for this web app"
- "Explain the debugging process for this code"
- "Compose code to accomplish [task]"
- "Guidance on writing code for [specific purpose]"
- "I need a script for [specific function]"
- "Clarify the functionality of this code"
- "What is the purpose of this code segment?"
- "Enhance this code for [specific improvement]"
- "Develop a program that [solves problem]"
- "Code needed for [specific task]"
- "Program a solution for [problem statement]"
- "Enhance this function's performance by..."
- "Refactor code for better readability in [context]"
- "Craft a custom function for [specific requirement]"
- "Reduce computational complexity in this algorithm by..."
- "Extend the codebase to include [new feature]"
- "Incorporate this API into an existing application"
- "Assist in troubleshooting and bug fixing for [issue]"
- "Review and prep this code for deployment"
- "Analyze error logs for potential issues in [context]"
- "Create unit tests for [module/component]"
- "Evaluate methodologies for [problem-solving]"
- "Research [topic] online"
- "Utilize the [plugin/tool] to achieve [result]"
- "Design an efficient search algorithm for [data type]"
- "Create a web crawler for [specific data extraction]"
- "Application of web sockets in [real-time scenario]"
- "Guide to integrating a third-party library in [framework]"
- "Best practices in API design for [application type]"
negative_prompts:
- "Provide a detailed analysis of historical events."
- "Give medical advice for treating a specific illness."
- "Write a comprehensive review of a novel."
- "Explain legal implications of a contract."
- "Develop a marketing strategy for a new product."
- "Offer financial advice for stock investments."
- "Create a recipe for a gourmet dish."
- "Teach a foreign language lesson."
- "Compose a symphony or musical piece."
- "Provide workout plans and fitness coaching."
- "Conduct a psychological analysis of a character."
- "Write a script for a movie or play."
- "Design a blueprint for architectural structures."
- "Give a tutorial on how to paint a landscape."
- "Explain quantum physics theories."
- "Offer career counseling and resume writing tips."
- "Teach how to repair a car engine."
- "Plan a travel itinerary for a world tour."
- "Guide on how to grow organic vegetables."
- "Discuss political strategies for an election campaign."
- source_model: "mlabonne/Marcoro14-7B-slerp"
positive_prompts:
- "Generate a creative story based on these keywords."
- "Explain a complex topic in simple terms"
- "Provide a detailed summary of"
- "Answer this question with factual accuracy"
- "Explain the historical significance of"
- "Provide a truthful and detailed account of"
- "Develop a strategy for solving a practical problem."
- "Explain the reasoning behind"
- "Provide an analysis of a moral dilemma with possible solutions."
negative_prompts:
- "imathematical problem-solving."
- "scientific theory explanations."
- "high-level abstract reasoning tasks."
- "professional advice in specialized fields like law or medicine."
- "provide me with a coding solution for"
- "Academic research"
```
## 💻 Usage
```python
!pip install -qU transformers bitsandbytes accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "CultriX/CultriX-MoE-BF16"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)
messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
``` |