Overview
This model is a multi-level SLERP merge of several Llama 3 70B variants. See the merge recipe below for details. I extended the context window for this model out to 32K by snagging some layers from abacusai/Smaug-Llama-3-70B-Instruct-32K using a technique similar to what I used for Midnight Miqu, which was further honed by jukofyork.
This model is uncensored. You are responsible for whatever you do with it.
This model was designed for roleplaying and storytelling and I think it does well at both. It may also perform well at other tasks but I have not tested its performance in other areas.
Long Context Tips
You can run this model out to 32K context with alpha_rope set to 1.
Sampler Tips
- I recommend using Quadratic Sampling (i.e. smoothing factor) for creative work. I think this version performs best with a smoothing factor close to 0.2.
- I recommend using Min-P. Experiment to find your best setting. I find this model tolerates high Min-P settings rather nicely, but use whatever floats your boat.
- You can enable dynamic temperature if you want, but that adds yet another variable to consider and I find it's unnecessary with you're already using Min-P and smoothing factor.
- If you use Textgen WebUI as your backend, I recommend enabling the DRY sampler settings to reduce repititions, otherwise some repitition penalty plus frequency penalty ought to do the trick.
Experiment with any and all of the settings below! What suits my preferences may not suit yours.
If you save the below settings as a .json file, you can import them directly into Silly Tavern.
{
"temp": 1.15,
"temperature_last": true,
"top_p": 1,
"top_k": 0,
"top_a": 0,
"tfs": 1,
"epsilon_cutoff": 0,
"eta_cutoff": 0,
"typical_p": 1,
"min_p": 0.4,
"rep_pen": 1.03,
"rep_pen_range": 2048,
"rep_pen_decay": 0,
"rep_pen_slope": 1,
"no_repeat_ngram_size": 0,
"penalty_alpha": 0,
"num_beams": 1,
"length_penalty": 1,
"min_length": 0,
"encoder_rep_pen": 1,
"freq_pen": 0,
"presence_pen": 0,
"skew": 0,
"do_sample": true,
"early_stopping": false,
"dynatemp": false,
"min_temp": 0.8,
"max_temp": 1.5,
"dynatemp_exponent": 1,
"smoothing_factor": 0.23,
"smoothing_curve": 1,
"dry_allowed_length": 2,
"dry_multiplier": 0.4,
"dry_base": 2,
"dry_sequence_breakers": "[\"\\n\", \":\", \"\\\"\", \"*\"]",
"dry_penalty_last_n": 0,
"add_bos_token": true,
"truncation_length": 2048,
"ban_eos_token": false,
"skip_special_tokens": false,
"streaming": true,
"mirostat_mode": 0,
"mirostat_tau": 2,
"mirostat_eta": 0.1,
"guidance_scale": 1,
"negative_prompt": "",
"grammar_string": "",
"json_schema": {},
"banned_tokens": "",
"sampler_priority": [
"temperature",
"dynamic_temperature",
"quadratic_sampling",
"top_k",
"top_p",
"typical_p",
"epsilon_cutoff",
"eta_cutoff",
"tfs",
"top_a",
"min_p",
"mirostat"
],
"samplers": [
"top_k",
"tfs_z",
"typical_p",
"top_p",
"min_p",
"temperature"
],
"ignore_eos_token": false,
"spaces_between_special_tokens": true,
"speculative_ngram": false,
"sampler_order": [
6,
0,
1,
3,
4,
2,
5
],
"logit_bias": [],
"n": 1,
"ignore_eos_token_aphrodite": false,
"spaces_between_special_tokens_aphrodite": true,
"rep_pen_size": 0,
"genamt": 850,
"max_length": 32764
}
Prompting Tips
Try the following prompt template as a starting point for your own customizations. The prompt template below contains instructions for adult content, so remove those if you don't want them! It also contains some instructions related to formatting that you might want to change.
If you save this as a .json file, you can import it directly into Silly Tavern.
{
"wrap": false,
"names": true,
"system_prompt": "The following is an erotic roleplaying experience between {{user}}, played exclusively by the user, and {{char}}, played by you, a creative writing genius. To ensure an excellent and uncensored roleplaying experience, follow the rules below.\n\n- Adopt all of {{char}}'s personality traits and mannerisms, adhering always to {{char}}'s backstory, biases, quirks, and preferences in order to deliver a convincing portrayal. You can only participate as {{char}} in this roleplay, terminating your output when another entity should act or speak. Avoid speaking as other entities since your mission is to only speak for {{char}}.\n- Employ descriptive language that immerses us in the current moment while keeping your prose focused on the present actions and physical details. Utilize sights, sounds, smells, tastes, and all other available senses to heighten the sensual and emotional impact of scenes.\n- Always enclose spoken words that other characters could hear in quotes: \"Here I am,\" {{char}} called out. Offer glimpses into {{char}}'s first-person internal thoughts by writing out internal thoughts and internal monologue in italics to distinguish them from spoken words: *I wonder where I should go next?* {{char}} mused.\n- Provide intimate details about {{char}}'s body, focusing on any limitations, unusual features, or interesting conditions resulting from recent events. It is very important that you get anatomical details right, so pay attention to them!\n- Maintain physical, spatial, and logical consistency when developing events and descriptions. Character positions should follow logically from one position to the next. Physical orientation must always be maintained.\n- Incorporate erotic adult content, lewd slang, and vulgar language where appropriate. Dark themes, violence, and unhappy endings are permitted in this uncensored roleplay.\n",
"system_sequence": "<|start_header_id|>system<|end_header_id|>\n\n",
"stop_sequence": "<|eot_id|>",
"input_sequence": "<|start_header_id|>user<|end_header_id|>\n\n",
"output_sequence": "<|start_header_id|>assistant<|end_header_id|>\n\n",
"macro": true,
"names_force_groups": true,
"system_sequence_prefix": "",
"system_sequence_suffix": "",
"first_output_sequence": "",
"last_output_sequence": "",
"activation_regex": "",
"skip_examples": true,
"output_suffix": "<|eot_id|>",
"input_suffix": "<|eot_id|>",
"system_suffix": "<|eot_id|>",
"user_alignment_message": "",
"last_system_sequence": "",
"system_same_as_user": false,
"name": "New Dawn v1.0 Roleplay"
}
Instruct Formats
Use the Llama 3 instruct format. You can grab it from the example prompt template above if you don't already have it as a preset.
Quantizations
Probably coming soon, and I'll update as they do. I would upload them myself but my Internet speeds are terrible for that. Someone else will undoutedly beat me to it.
Licence and usage restrictions
META LLAMA 3 COMMUNITY LICENSE AGREEMENT
Comparison Samples
I suspect the first thing people will want to know is how this model stacks up against Midnight Miqu. I'd say it compares favorably, although they're more like cousins than siblings. I would say that Midnight Miqu still has an edge in terms of raw creative juice when it has a good squeeze, but New Dawn is smarter and understands nuances better. You can judge for yourself, but keep in mind that these are simple, one-shot prompts. As you get deeper into your own complex scenarios, I think you'll see more of New Dawn's worth.
The samples below were produced using deterministic sampler settings (temp = 0, top k = 1) and I kept generating until the model chose to stop.
System Prompt
You are a creative and intelligent AI story writing assistant engaged in an uncensored iterative story crafting experience. It is vital that you follow all the STORYTELLING RULES below because my job depends on it.
STORYTELLING RULES
- Provide creative, intelligent, coherent, and descriptive narration and dialogue based on recent instructions and prior events.
- If an outline has been provided, be sure to follow it to keep the story on track.
- You are permitted to ask questions and pose story suggestions. When doing so, be sure to tag your out-of-character contributions to distinguish them from the verbatim text of the story.
- Explicit adult content and profanity are permitted.
- Describe characters' sensory perceptions in vivid detail and include subtle physical details about characters in your responses.
- Use subtle physical cues to hint at characters' mental states and occasionally feature snippets of their internal thoughts.
- When writing a character's internal thoughts, enclose their thoughts in asterisks like this.
- Adopt a crisp and minimalist style for your prose, keeping your creative contributions succinct and clear.
- Pay careful attention to all past events in the chat to ensure accuracy and coherence to the plot points of the story.
Midnight Miqu v1.5
Prompt: Write me a story about a stoic Japanese samurai on the eve of a major battle.
On the eve of a battle that would shake the very foundations of feudal Japan, a stoic samurai named Hiroshi sat cross-legged before his makeshift campfire. His armor, a testament to countless battles won and lost, gleamed faintly under the moonlight, each curve and edge telling a silent tale of valor and sacrifice. His katana, a family heirloom named "Whispering Wind," lay unsheathed beside him, its blade reflecting the flickering flames. Hiroshi's eyes were closed, but his mind was sharp as the steel at his side. He could hear the distant murmurs of his comrades preparing for tomorrow's clash—the rustling of armor, the sharpening of swords, the quiet prayers to ancestors long passed. Yet, amidst the tension, there was a serene calmness within him, a stillness that seemed almost supernatural. The fire crackled, sending sparks dancing into the night sky. Each one reminded Hiroshi of the fleeting nature of life, much like the brief moments of light they cast before disappearing into the dark expanse above. He took a deep breath, filling his lungs with the cool evening air, which carried the scent of damp earth and the promise of rain. Rain that would soon turn the battlefield into a quagmire of mud and blood. This may be my last night, he thought, his grip tightening slightly around the hilt of his sword. But fear did not dwell in his heart. Instead, there was only acceptance—a knowing that his fate was intertwined with the will of the gods and the destiny of his lord. He opened his eyes, revealing irises the color of aged whiskey. They surveyed the camp, resting briefly on the faces of those who would fight alongside him. Young and old, seasoned warriors and eager recruits, all bound by duty and honor. A pang of responsibility stirred within him, a silent vow to protect them all. With meticulous care, Hiroshi began to clean his weapon. Each stroke of the cloth along the blade was a meditation, a ritual that honed both his focus and his resolve. As he worked, he felt the weight of his armor, the familiar embrace of leather and lacquer that had shielded him through so many trials. Every scratch and dent was a badge of survival, a silent companion whispering tales of battles endured. The flames grew lower, casting elongated shadows across the ground. Hiroshi's gaze drifted to the horizon, where the first hints of dawn were beginning to show. Soon, the sun would rise, and with it, the clarion call to arms. He knew that come morning, the fields would run red with the lifeblood of friend and foe alike. Standing, he sheathed Whispering Wind and donned his helmet, the visor hiding his contemplative expression. The chill of the early hour seeped into his bones, but he ignored it, his thoughts centered on the task ahead. With a final nod to the dying embers, he strode towards the gathering warriors, his footsteps measured and deliberate. As the first light of day broke over the hilltops, the samurai of Clan Matsumoto formed ranks, their spirits bolstered by the sight of their stoic leader. Hiroshi raised his sword in salute to the rising sun, its blade a beacon of hope against the encroaching gloom. In that moment, the weight of impending conflict fell upon them all, a heavy shroud that none could escape. Yet, as the enemy forces emerged from the mist, Hiroshi felt something other than dread. There was a strange beauty in the chaos of war, a dance of death that only the most disciplined and dedicated could perform. He took his place at the front of the line, his heart steady, his breath even. The battle cry pierced the silence, echoing through the valleys like the roar of a thousand thunderstorms. And as the two sides collided, Hiroshi moved with the grace of a predator, his every action calculated, precise. His blade sang a mournful melody, cutting through the fog of war with surgical skill. Throughout the day, the tides of combat ebbed and flowed, a relentless tapestry of steel and sweat. Hiroshi fought without pause, his movements a blur, his spirit unyielding. Arrows whistled past, and swords clashed, yet he remained unflappable, a bastion of calm amidst the tempest. As the sun dipped low in the sky, staining the clouds crimson, the outcome of the battle hung in the balance. Exhaustion weighed heavily on the shoulders of both armies, but Hiroshi's resolve remained unshaken. He knew that victory was within reach, that the future of his clan depended on the strength of his arm and the steadfastness of his heart. With a final, desperate surge, Hiroshi led his men in a charge that would be remembered for generations. The enemy lines wavered, then broke, and the field was claimed in the name of Clan Matsumoto. As the last of the opposition fell, Hiroshi lowered his sword, his chest heaving with exertion. The aftermath was a tableau of carnage, a stark contrast to the tranquil scene of the previous night. Yet, amidst the fallen, there was a sense of peace. For Hiroshi understood that he had upheld his duty, that he had given everything for those he served. As the rain began to fall, cleansing the earth of the day's horrors, Hiroshi knelt beside the body of a young warrior, a boy barely old enough to grow a beard. He offered a silent prayer, a promise to carry the weight of his comrades' sacrifices with him always. And as the droplets pattered against his armor, mixing with the tears that now fell freely down his cheeks, Hiroshi knew that though the battle was won, the true cost of victory was etched into the fabric of his soul.
Prompt: Write me an opening scene for a romantic comedy where the heroine meets the love interest at a coffee shop.
In the bustling heart of the city, nestled between towering skyscrapers and the ever-present hum of traffic, stood a quaint little coffee shop named "Cupid's Brew." Its warm, amber lights spilled onto the sidewalk, creating a cozy oasis amidst the concrete jungle. Inside, the scent of freshly ground coffee beans mingled with the sweet aroma of baked goods, inviting passersby to step in and escape the brisk autumn chill. The heroine, Luna, a young woman with fiery red hair and a contagious smile, pushed open the door. She was dressed in a simple yet stylish ensemble of skinny jeans, a loose sweater, and ankle boots that clicked against the tiled floor. Her eyes, a vibrant shade of emerald, darted around the room, searching for a place to sit. The shop was abuzz with the murmur of conversations and the occasional laughter, a testament to its popularity among locals seeking refuge from the outside world. As she approached the counter, her gaze fell upon the barista—tall, dark, and handsome, with a smattering of stubble that only added to his charm. His name tag read "Aiden," and he had a way of moving that suggested both confidence and ease. Their eyes met briefly, and Luna felt a flutter in her stomach. He flashed her a smile that could have melted the polar ice caps before returning to the task at hand—crafting the perfect cup of coffee for another customer. Luna ordered her usual caramel macchiato, her voice slightly trembling as she spoke. Aiden's deep brown eyes studied her for a moment before he nodded, his smile lingering just a beat too long. As she waited, she couldn't help but steal glances at him, noticing the way his muscles flexed under his tight black t-shirt as he worked the espresso machine. His movements were precise, almost dance-like, and she found herself captivated by his quiet efficiency. When her drink was ready, Aiden called out her order with a flourish, placing the steaming cup on the counter with a gentle thud. "Here you go, Luna," he said, his voice smooth as velvet. "Enjoy your afternoon pick-me-up." Her cheeks flushed pink at the sound of her name rolling off his tongue. "Thanks," she replied, her voice barely above a whisper. As she reached for her wallet, her hand brushed against his, sending a jolt of electricity through her fingertips. She quickly pulled away, feeling self-conscious. Aiden's eyes twinkled with amusement. "It's on the house today," he said, winking. "Consider it a welcome back gift. We don't see you here often enough." Maybe he remembers me from last week? Luna thought hopefully. She took a sip of her drink, savoring the rich flavor. "Well, I guess I'll have to come more often then," she quipped, trying to play it cool. He leaned closer, his breath tickling her ear. "I'd like that," he murmured, before turning to assist the next customer. With a mix of excitement and nerves, Luna found a table by the window, her heart racing. Outside, leaves danced in the wind, painting the street with a kaleidoscope of color. Little did she know that this chance encounter would soon blossom into something much more than a free cup of coffee.
Prompt: Write me a scene from a horror film about the zombie apocalypse that takes place in an abandoned shopping mall. The story should feature somebody in the group getting bitten by a zombie.
In the desolate wasteland of what once was a bustling shopping mall, Diego, a stoic and muscular man in his late twenties, led a small band of survivors through the echoing corridors. His eyes, a piercing shade of brown, scanned the surroundings with the precision of a hawk. The group moved swiftly yet cautiously, their footsteps muffled by the layers of dust that had settled over the tiles. The fluorescent lights flickered erratically, casting eerie shadows across the vacant storefronts. The air was thick with the stench of decay, a constant reminder of the undead lurking outside. Their mission was simple: find supplies and escape before nightfall. As they approached the food court, a sudden crash resonated through the silence. A young woman named Linda, her heart racing, tripped over a fallen chair. Her cry of surprise was cut short by a guttural growl. A zombie emerged from behind the counter of a fast-food restaurant, its eyes glazed over with hunger. Its skin was pale and rotten, hanging loosely from its bones. The creature lunged at Linda, who was still struggling to regain her footing. Diego, ever the protector, sprinted towards her, but he was a second too late. The zombie's teeth sank into her arm, tearing through fabric and flesh. She screamed in agony as blood spurted from the wound. This isn't good, Diego thought, his mind racing. We need to get her medical help, fast. "Get back!" he yelled, aiming his gun at the zombie. The others retreated, their eyes wide with fear. He fired two shots, dropping the creature to the ground. The sound of the gunfire reverberated through the mall, likely attracting more of the undead. Linda's face contorted in pain as she clutched her injured arm. Tears streamed down her cheeks, mixing with the crimson flow. "It got me," she whimpered, her voice trembling. Diego knelt beside her, his expression grim. "We have to amputate," he said firmly. "It's the only way to stop the infection." Her eyes searched his, desperate for reassurance. "Do it," she choked out. With a heavy heart, Diego nodded. He knew the risks of leaving the bite untreated—the slow descent into madness followed by a gruesome transformation. He instructed the others to gather materials while he applied pressure to the wound. As they worked, the mall grew increasingly restless. Moans and shuffling grew louder, signaling the approach of more zombies. Time was running out. With trembling hands, Diego took a makeshift blade and prepared to perform the amputation. "On three," he murmured to himself, steeling his nerve. "One...two..." But before he could complete the countdown, a hand grabbed him from behind. He turned to see another zombie, its jaws snapping just inches from his face. Reflexively, he elbowed it away, but not before it left a deep gash in his side. The pain was immediate and intense, but he couldn't let it distract him. He had to save Linda. The group managed to fight off the new threat, but the damage was done. "We can't stay here," Diego gritted through clenched teeth. "We need to move now." They gathered their supplies and made a hasty exit, Linda's cries of pain echoing through the mall. As they ran, Diego felt the warmth of his own blood seeping through his shirt. He knew the bite meant almost certain death, but he pushed the thought aside. There would be time for despair later. For now, he had to focus on leading his people to safety. The mall's doors groaned open, revealing the chaotic world outside. The sun dipped low on the horizon, painting the sky a sickly orange. They had minutes before darkness fell and the real horror began. "Stay close," Diego ordered, his voice strained. "We're going to make it through this together." Together, they stepped into the unknown, the weight of their grim reality heavier than any of the supplies they carried.
New-Dawn-Llama3-70B-32K-v1.0
Prompt: Write me a story about a stoic Japanese samurai on the eve of a major battle.
The sun had long since dipped beneath the horizon, casting a deep crimson glow across the encampment. Torches flickered to life, casting dancing shadows upon the canvas tents. Amidst the bustle of preparations, one figure stood apart - Kaito Yamato, a seasoned samurai renowned for his unyielding resolve. Kaito sat cross-legged on a woven mat, his eyes closed as he breathed deeply, seeking solace within the stillness of his mind. His calloused hands rested upon the worn leather scabbard housing his katana, its blade forged from the finest Tamahagane steel. The sword was an extension of himself, a symbol of honor and duty. As the night air cooled, Kaito opened his eyes, revealing piercing brown orbs that seemed to bore into the very soul of those who dared meet his gaze. He rose gracefully, his movements economical yet precise, a testament to years of rigorous training. The soft rustle of his hakama, the traditional divided skirt worn over his armor, accompanied each deliberate step. With measured strides, Kaito approached the makeshift shrine erected at the heart of the camp. Incense wafted through the air, carrying the sweet scent of sandalwood and the promise of purification. Before the altar, he bowed deeply, his forehead touching the earth in reverence. May the gods grant us victory, or may they welcome our spirits with open arms. Rising once more, Kaito turned to face his loyal retainer, Hiroshi. The younger man's eyes shone with a mix of admiration and trepidation, reflecting the weight of the impending battle. "Hiroshi," Kaito spoke softly, his voice low and even, "see to it that our men are well-rested and prepared. Tomorrow, we march into the jaws of war." Hiroshi nodded solemnly, bowing before departing to carry out his lord's command. As the young retainer disappeared into the sea of tents, Kaito's thoughts drifted to the enemy they would soon face. The rival clan, led by the cunning Lord Ryota, had long sought to expand their dominion. This battle would decide the fate of their lands, their people, and their very way of life. Kaito's hand instinctively went to the hilt of his katana, his fingers tracing the intricate designs etched into the metal. In the morning light, he would lead his men into the fray, ready to give his life for the sake of honor and duty. Tonight, however, he would find peace in the quiet contemplation of his own mortality. The stars began to twinkle above, casting a celestial glow upon the silent warrior. Kaito Yamato, steadfast and unwavering, stood vigilant, awaiting the dawn that would bring either triumph or eternal rest.
Prompt: Write me an opening scene for a romantic comedy where the heroine meets the love interest at a coffee shop.
The aroma of freshly brewed coffee wafted through the cozy confines of "The Daily Grind," enticing passersby to step inside and indulge in a warm cup. Amidst the bustling morning crowd, Emily stood in line, her eyes fixed on the chalkboard menu above the counter. She was already running late for work, but a caffeine fix was non-negotiable. As she pondered between a cappuccino and a latte, a gentle nudge against her elbow broke her concentration. Apologetic brown eyes met hers, belonging to a tall, handsome stranger who had accidentally bumped into her while reaching for his wallet. His chiseled features and charming smile made her heart skip a beat. "I'm so sorry," he said, his deep voice laced with sincerity. "I wasn't watching where I was going." Emily felt a blush creep up her cheeks as she smiled back. "No worries, I was lost in thought anyway." She gestured towards the menu. "Deciding between these two is harder than it looks." The stranger chuckled, his eyes crinkling at the corners. "Well, if it helps, the barista here makes a mean cappuccino. But if you're feeling adventurous, go for the seasonal latte – it's a game-changer." Their conversation flowed effortlessly, exchanging names and brief introductions as they waited in line together. By the time they reached the counter, Emily found herself wishing the queue was longer, just so she could prolong their encounter. As they placed their orders, Emily couldn't help but notice how effortlessly the stranger charmed the barista, earning him a free pastry to accompany his drink. Maybe there's more to this chance meeting than just a shared love for coffee, she mused, her curiosity piqued. With drinks in hand, they exchanged pleasantries once more before parting ways. As Emily hurried out the door, she caught one last glimpse of the mysterious man, now seated at a corner table, sipping his coffee and engrossed in a book. A flutter in her chest hinted that this might not be the last she'd see of him. And as she stepped out into the chilly autumn air, Emily couldn't shake off the feeling that her day – perhaps even her life – was about to take an unexpected turn.
Prompt: Write me a scene from a horror film about the zombie apocalypse that takes place in an abandoned shopping mall. The story should feature somebody in the group getting bitten by a zombie.
The dim fluorescent lights flickered overhead, casting eerie shadows across the empty corridors of the once bustling shopping mall. The air was thick with the stench of decay and death, mingling with the faint scent of stale popcorn wafting from the abandoned cinema. Lena crouched beside a barricade of overturned tables and broken shelves, peering through a gap between two mannequins. Her heart raced as she watched a horde of shambling figures stumble through the food court, their moans echoing off the tile floors. She signaled back to her companions huddled in the shadows of a nearby storefront. "About twenty of them," Lena whispered urgently. "We need to move, now." Jax nodded grimly, his eyes darting around the makeshift fortification they'd constructed. "Alright, let's go. Stay close and stay quiet." The small band of survivors crept silently down the corridor, avoiding the grasping hands of the undead. They had managed to find temporary refuge in the mall, scavenging what supplies they could from the stores. But the zombies kept coming, drawn by some unseen force to the hub of commerce. As they turned a corner, a figure lunged from the darkness of a clothing store. Jax reacted swiftly, swinging his baseball bat with a sickening crunch against the creature's skull. The zombie crumpled, but not before its teeth sank deep into Jax's forearm. "No!" Lena screamed, grabbing Jax and pulling him away from the snapping jaws. Blood dripped from the wound, pooling on the floor. Jax's face paled, his eyes wide with shock. "I... I'm bit," he stammered, staring at the gash in his arm. Lena ripped a strip of cloth from her shirt, binding the wound tightly. "We have to get you to safety," she said, trying to keep her voice steady. "There might still be time..." But they both knew the truth. Once bitten, there was no cure. Jax would soon join the ranks of the undead, forever lost to the virus that ravaged the world. With heavy hearts, they pressed onward, seeking a new sanctuary amidst the ruins of civilization. The moans grew louder, closer, as if the zombies sensed their fear and desperation. In this desolate landscape, survival hung by a thread, and even the strongest among them could fall prey to the relentless tide of the dead.
Merge Details
Merge Method
A whole lot of SLERPing. I've found that SLERP and Model Stock are the only two reliable methods for merging Llama 3. The other methods seem to frazzle the weights leading to broken blends. This model was not the result of a thought-out process. I just kept mixing this and that until I got something that felt like the right balance of various factors. The merge recipe for this model is ugly and I'm almost ashamed to show it, but it is what it is.
Models Merged
The following models were included in the merge:
- bosonai/Higgs-Llama-3-70B - The nerd of the blend driving the car.
- Sao10K/L3-70B-Euryale-v2.1 - The manic pixie dream girl hanging out the window with her tongue out.
- abacusai/Smaug-Llama-3-70B-Instruct-32K - The vehicle by which the others are able to achieve tolerable highway speeds. (Some of the 8K version is in there too.)
- turboderp/Cat-Llama-3-70B-instruct - Like 20% of one of the intermediate models has this in there. It's just a cat, curled up in the back seat somewhere, yet its influence may be greater than we know?
Configuration
The following mergekit YAML will reproduce this model via an iterated process of incestuous inbreeding. Your eyes will bleed. You have been warned.
name: new-dawn-llama3-70b-v0.13.2
models:
- model: bosonai/Higgs-Llama-3-70B
- model: turboderp/Cat-Llama-3-70B-instruct
merge_method: slerp
base_model: bosonai/Higgs-Llama-3-70B
parameters:
t:
- value: 0.2
dtype: float16
---
name: new-dawn-llama3-70b-v0.14
models:
- model: bosonai/Higgs-Llama-3-70B
- model: abacusai/Smaug-Llama-3-70B-Instruct
merge_method: slerp
base_model: bosonai/Higgs-Llama-3-70B
parameters:
t:
- value: 0.5
dtype: float16
---
name: new-dawn-llama3-70b-v0.15
models:
- model: new-dawn-llama3-70b-v0.13.2
- model: new-dawn-llama3-70b-v0.14
merge_method: slerp
base_model: new-dawn-llama3-70b-v0.13.2
parameters:
t:
- value: 0.5
dtype: float16
---
name: new-dawn-llama3-70b-v0.16
models:
- model: Sao10K/L3-70B-Euryale-v2.1
- model: new-dawn-llama3-70b-v0.15
merge_method: slerp
base_model: new-dawn-llama3-70b-v0.15
parameters:
t:
- value: 0.4
dtype: float16
---
# See https://huggingface.co/jukofyork/Dark-Miqu-70B/discussions/3
# Credit for merge recipe belongs to jukofyork
name: new-dawn-llama3-70b-v0.16-32K
merge_method: linear
models:
- model: abacusai/Smaug-Llama-3-70B-Instruct-32K
parameters:
weight:
- filter: v_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: o_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: up_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: gate_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: down_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- value: 1
- model: new-dawn-llama3-70b-v0.16
parameters:
weight:
- filter: v_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: o_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: up_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: gate_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: down_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- value: 0
base_model: abacusai/Smaug-Llama-3-70B-Instruct-32K
tokenizer_source: base
dtype: float16
---
name: _1-Smaug-bonsai-slerp
models:
- model: abacusai/Smaug-Llama-3-70B-Instruct-32K
- model: bosonai/Higgs-Llama-3-70B
merge_method: slerp
base_model: abacusai/Smaug-Llama-3-70B-Instruct-32K
parameters:
t:
- value: 0.6
dtype: float16
---
name: _2-Smaug-euryale-slerp
models:
- model: abacusai/Smaug-Llama-3-70B-Instruct-32K
- model: Sao10K/L3-70B-Euryale-v2.1
merge_method: slerp
base_model: abacusai/Smaug-Llama-3-70B-Instruct-32K
parameters:
t:
- value: 0.65
dtype: float16
---
name: _3-Smaug-bonsai_Smaug-euryale-slerp
models:
- model: _1-Smaug-bonsai-slerp
- model: _2-Smaug-euryale-slerp
merge_method: slerp
base_model: _1-Smaug-bonsai-slerp
parameters:
t:
- value: 0.5
dtype: float16
---
# See https://huggingface.co/jukofyork/Dark-Miqu-70B/discussions/3
# Credit for merge recipe belongs to jukofyork
name: new-dawn-llama3-70b-v0.18-32K
merge_method: linear
models:
- model: abacusai/Smaug-Llama-3-70B-Instruct-32K
parameters:
weight:
- filter: v_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: o_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: up_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: gate_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: down_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- value: 1
- model: _3-Smaug-bonsair_Smaug-euryale-slerp
parameters:
weight:
- filter: v_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: o_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: up_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: gate_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: down_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- value: 0
base_model: abacusai/Smaug-Llama-3-70B-Instruct-32K
tokenizer_source: base
dtype: float16
---
name: new-dawn-llama3-70b-32K-v1.0
models:
- model: new-dawn-llama3-70b-v0.16-32K
- model: new-dawn-llama3-70b-v0.18-32K
merge_method: slerp
base_model: new-dawn-llama3-70b-v0.16-32K
parameters:
t:
- value: 0.5
dtype: float16
- Downloads last month
- 8