Llama-3-Alpha-Centauri-v0.1-LoRA

Image generated with https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS.

---

Disclaimer

Note: All models and LoRAs from the Centaurus series were created with the sole purpose of research. The usage of this model and/or its related LoRA implies agreement with the following terms:

• The user is responsible for what they might do with it, including how the output of the model is interpreted and used;
• The user should not use the model and its outputs for any illegal purposes;
• The user is the only one resposible for any misuse or negative consequences from using this model and/or its related LoRA.

I do not endorse any particular perspectives presented in the training data.

---

Centaurus Series

This series aims to develop highly uncensored Large Language Models (LLMs) with the following focuses:

• Science, Technology, Engineering, and Mathematics (STEM)
• Computer Science (including programming)
• Social Sciences

And several key cognitive skills, including but not limited to:

• Reasoning and logical deduction
• Critical thinking
• Analysis

While maintaining strong overall knowledge and expertise, the models will undergo refinement through:

• Fine-tuning processes
• Model merging techniques including Mixture of Experts (MoE)

Please note that these models are experimental and may demonstrate varied levels of effectiveness. Your feedback, critique, or queries are most welcome for improvement purposes.

Base

This model and its related LoRA was fine-tuned on https://huggingface.co/failspy/Meta-Llama-3-8B-Instruct-abliterated-v3.

Merged Model

The merged model (base + LoRA) is available at https://huggingface.co/fearlessdots/Llama-3-Alpha-Centauri-v0.1.

GGUF

I provide some GGUF files for the merged model here: https://huggingface.co/fearlessdots/Llama-3-Alpha-Centauri-v0.1-GGUF.

Datasets

• https://huggingface.co/datasets/NobodyExistsOnTheInternet/ToxicQAFinal

Fine Tuning

- Quantization Configuration

• load_in_4bit=True
• bnb_4bit_quant_type="fp4"
• bnb_4bit_compute_dtype=compute_dtype
• bnb_4bit_use_double_quant=False

- PEFT Parameters

• lora_alpha=64
• lora_dropout=0.05
• r=128
• bias="none"

- Training Arguments

• num_train_epochs=1
• per_device_train_batch_size=1
• gradient_accumulation_steps=4
• optim="adamw_bnb_8bit"
• save_steps=25
• logging_steps=25
• learning_rate=2e-4
• weight_decay=0.001
• fp16=False
• bf16=False
• max_grad_norm=0.3
• max_steps=-1
• warmup_ratio=0.03
• group_by_length=True
• lr_scheduler_type="constant"

Credits

• Meta (https://huggingface.co/meta-llama): for the original Llama-3;
• HuggingFace: for hosting this model and for creating the fine-tuning tools used;
• failspy (https://huggingface.co/failspy): for the base model and the orthogonalization implementation;
• NobodyExistsOnTheInternet (https://huggingface.co/NobodyExistsOnTheInternet): for the incredible dataset;
• Undi95 (https://huggingface.co/Undi95) and Sao10k (https://huggingface.co/Sao10K): my main inspirations for doing these models =]

A huge thank you to all of them ☺️

About Alpha Centauri

Alpha Centauri is a triple star system located in the constellation of Centaurus. It includes three stars: Rigil Kentaurus (also known as α Centauri A), Toliman (or α Centauri B), and Proxima Centauri (α Centauri C). Proxima Centauri is the nearest star to the Sun, residing at approximately 4.25 light-years (1.3 parsecs) away.

The primary pair, α Centauri A and B, are both similar to our Sun - α Centauri A being a class G star with 1.1 solar masses and 1.5 times the Sun's luminosity; α Centauri B having 0.9 solar masses and under half the luminosity of the Sun. They revolve around their shared center every 79 years following an elliptical path, ranging from 35.6 astronomical units apart (nearly Pluto's distance from the Sun) to 11.2 astronomical units apart (around Saturn's distance from the Sun.)

Proxima Centauri, or α Centauri C, is a diminutive, dim red dwarf (a class M star) initially unseen to the naked eye. At roughly 4.24 light-years (1.3 parsecs) from us, it lies nearer than α Centauri AB, the binary system. Presently, the gap between Proxima Centauri and α Centauri AB amounts to around 13,000 Astronomical Units (0.21 light-years)—comparable to over 430 times Neptune's orbital radius.

Two confirmed exoplanets accompany Proxima Centauri: Proxima b, discovered in 2016, is Earth-sized within the habitable zone; Proxima d, revealed in 2022, is a potential sub-Earth close to its host star. Meanwhile, disputes surround Proxima c, a mini-Neptune detected in 2019. Intriguingly, hints suggest that α Centauri A might possess a Neptune-sized object in its habitable region, but further investigation is required before confirming whether it truly exists and qualifies as a planet. Regarding α Centauri B, although once thought to harbor a planet (named α Cen Bb), subsequent research invalidated this claim, leaving it currently devoid of identified planets.

Source: retrived from https://en.wikipedia.org/wiki/Alpha_Centauri and processed with https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1.