Create README.md

README.md

@@ -0,0 +1,116 @@
+---
+license: apache-2.0
+tags:
+- moe
+- frankenmoe
+- merge
+- mergekit
+- lazymergekit
+- Gille/StrangeMerges_32-7B-slerp
+- mlabonne/AlphaMonarch-7B
+base_model:
+- Gille/StrangeMerges_32-7B-slerp
+- mlabonne/AlphaMonarch-7B
+---
+
+# MixtureofMerges-MoE-2x7b-v7
+
+MixtureofMerges-MoE-2x7b-v7 is a Mixture of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
+* [Gille/StrangeMerges_32-7B-slerp](https://huggingface.co/Gille/StrangeMerges_32-7B-slerp)
+* [mlabonne/AlphaMonarch-7B](https://huggingface.co/mlabonne/AlphaMonarch-7B)
+
+## 🧩 Configuration
+
+```yaml
+base_model: Gille/StrangeMerges_32-7B-slerp
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: Gille/StrangeMerges_32-7B-slerp
+    positive_prompts:
+      - "Answer this question from the ARC (Argument Reasoning Comprehension)."
+      - "Use common sense and logical reasoning skills."
+      - "What assumptions does this argument rely on?"
+      - "Are these assumptions valid? Explain."
+      - "Analyze the logical structure of this argument. Identify the premises, conclusion, and any assumptions made"
+      - "Identify any potential counterarguments to this position. How might someone challenge the reasoning presented?"
+      - "Could this be explained in a different way? Provide an alternative explanation."
+      - "Identify any weaknesses in this argument."
+      - "Does this argument contain any logical fallacies? If so, which ones?"
+      - "Generate a few possible continuations to this scenario."
+      - "Demonstrate understanding of everyday commonsense in your response."
+      - "Use contextual clues to determine the most likely outcome."
+      - "Continue this scenario, but make the writing style sound archaic and overly formal."
+      - "This narrative is predictable. Can you introduce an unexpected yet plausible twist?"
+      - "The character is angry. Continue this scenario showcasing a furious outburst."
+    negative_prompts:
+      - "misses key evidence"
+      - "overly general"
+      - "commits the fallacy of hasty generalization"
+      - "focuses on irrelevant details"
+      - "assumes information not provided"
+      - "relies on stereotypes"
+      - "repetitive phrases"
+      - "engages in circular reasoning"
+      - "overuse of the same words"
+      - "contradicts earlier statements - breaks the internal logic of the scenario"
+      - "out of character dialogue"
+      - "awkward phrasing - sounds unnatural"
+      - "doesn't match the given genre"
+  - source_model: mlabonne/AlphaMonarch-7B
+    positive_prompts:
+      - "Answer this question, demonstrating commonsense understanding and using any relevant general knowledge you may have."
+      - "Provide a concise summary of this passage, then explain why the highlighted section is essential to the main idea."
+      - "Read these two brief articles presenting different viewpoints on the same topic. List their key arguments and highlight where they disagree."
+      - "Paraphrase this statement, changing the emotional tone but keeping the core meaning intact. Example: Rephrase a worried statement in a humorous way"
+      - "Create a short analogy that helps illustrate the main concept of this article."
+      - "Explain the concept of physics to a high school student. Use analogies and examples to clarify the main ideas."
+      - "Calculate the answer to this math problem"
+      - "My mathematical capabilities are strong, allowing me to handle complex mathematical queries"
+      - "solve for"
+      - "Analyze the given data and identify any patterns or trends. What conclusions can be drawn from this information?"
+      - "A store sells apples at $0.50 each. If Emily buys 12 apples, how much does she need to pay?"
+      - "Isolate x in the following equation: 2x + 5 = 17"
+      - "Solve this equation and show your working."
+      - "Explain why you used this formula to solve the problem."
+      - "Attempt to divide this number by zero. Explain why this cannot be done."
+    negative_prompts:
+      - "sounds too basic"
+      - "understated"
+      - "dismisses important details"
+      - "avoids the question's nuance"
+      - "skips essential steps in the solution"
+      - "takes this statement too literally"
+      - "incorrect"
+      - "inaccurate"
+      - "assumed without proof"
+      - "uses jargon without explanation"
+      - "rushed calculation"
+      - "confuses mathematical concepts"
+      - "draws illogical conclusions"
+      - "circular reasoning"
+```
+
+## 💻 Usage
+
+```python
+!pip install -qU transformers bitsandbytes accelerate
+
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "jsfs11/MixtureofMerges-MoE-2x7b-v7"
+
+tokenizer = AutoTokenizer.from_pretrained(model)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
+)
+
+messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
+prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```