Matrix-Corp/Vortex-13b-V1

Vortex Scientific

Vortex Scientific is a from-scratch AI model family designed for deep scientific reasoning. Built from the ground up with a novel hybrid state-space + attention architecture, optimized for consumer laptop hardware (Apple Silicon MacBooks and Nvidia 4060 laptop GPUs).

🌟 Features

• Novel Architecture: Hybrid State-Space Model (SSM) + Local Attention blocks
• Science-Specialized: Custom tokenizer, domain-aware gating, and specialized modules for equations, numerical reasoning, citations, and molecular structures
• Hardware Optimized: Runs smoothly on 8GB VRAM (4060 laptop) and 16GB unified memory (MacBook Pro M2/M3)
• Two Model Sizes:
  • Vortex-7B: 7 billion parameters, fits in 8GB VRAM
  • Vortex-13B: 13 billion parameters, fits in 16GB VRAM with quantization
• HuggingFace Compatible: Full integration with transformers library
• From Scratch: No base model — everything built bottom-up including tokenizer and weights

🏗️ Architecture

Vortex uses a two-block hybrid architecture:

1. SSM-Only Blocks: State-space layers for efficient long-context processing (O(n) complexity)
2. Attention+Science Blocks: Local windowed attention + science modules + SciGate FFN

Layer ratios:

• 7B: 60% SSM, 40% Attention (pattern: SSM, SSM, Attn, ...)
• 13B: 50% SSM, 50% Attention (pattern: SSM, Attn, SSM, Attn, ...)

Science Modules

• EquationModule: LaTeX equation detection and structural understanding
• NumericalReasoningModule: Digit-level encoding, scientific notation, unit awareness
• CitationModule: Citation span detection, provenance tracking, confidence scoring
• MolecularModule: Element embeddings, SMILES understanding, amino acid sequences

📦 Project Structure

Vortex/
├── configs/
│   ├── vortex_7b_config.py      # 7B model configuration
│   ├── vortex_13b_config.py     # 13B model configuration
│   └── training_config.py       # Training hyperparameters
├── models/
│   ├── ssm_layer.py             # State-space layer
│   ├── attention_layer.py       # Local windowed attention
│   ├── scigate_ffn.py           # Science-gated feed-forward
│   ├── vortex_model.py          # Main model class
│   └── science_modules/         # Specialized science modules
├── tokenizer/
│   └── vortex_tokenizer.py      # Custom BPE tokenizer with science vocab
├── data/
│   ├── dataset_loader.py        # Open dataset loading (Pile, S2ORC, etc.)
│   ├── quality_filter.py        # Multi-stage quality filtering
│   ├── domain_classifier.py     # 7-domain classifier
│   ├── deduplication.py         # MinHash LSH deduplication
│   └── scraper.py               # Web scraping (arXiv, PubMed, etc.)
├── training/
│   ├── trainer.py               # Main training loop
│   ├── losses.py                # Science-aware loss functions
│   └── curriculum.py            # Curriculum learning scheduler
├── inference/
│   ├── cuda_optimize.py         # CUDA optimizations (Flash Attention, INT8)
│   └── mps_optimize.py          # MPS optimizations for Apple Silicon
├── evaluation/                  # Science benchmarks (coming soon)
├── configuration_vortex.py      # HF config class
├── tokenization_vortex.py       # HF tokenizer wrapper
├── modeling_vortex.py           # HF model integration
├── train.py                     # Training entry point
├── inference/inference.py       # Inference entry point
└── requirements.txt

🚀 Quick Start

Installation

# Clone and setup
cd Vortex
pip install -r requirements.txt

# For CUDA optimizations
pip install flash-attn
pip install bitsandbytes

Training

# Train 7B model on CUDA
python train.py \
    --model_size 7b \
    --device cuda \
    --data_dir ./data/processed \
    --output_dir ./checkpoints \
    --max_steps 100000

# Train 13B model with INT8 quantization (for 8GB VRAM)
python train.py \
    --model_size 13b \
    --device cuda \
    --quantization int8 \
    --data_dir ./data/processed \
    --output_dir ./checkpoints_13b

Inference

# Generate text with 7B model
python inference/inference.py \
    --model_path ./checkpoints/latest.pt \
    --model_size 7b \
    --device cuda \
    --prompt "The equation E = mc^2 describes" \
    --max_new_tokens 100

# Interactive mode
python inference/inference.py \
    --model_path ./checkpoints/latest.pt \
    --model_size 7b \
    --device cuda \
    --interactive

# On Apple Silicon (MPS)
python inference/inference.py \
    --model_path ./checkpoints/latest.pt \
    --model_size 7b \
    --use_mps \
    --prompt "Explain quantum mechanics"

HuggingFace Integration

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("./checkpoints")
tokenizer = AutoTokenizer.from_pretrained("./checkpoints")

# Generate
input_text = "The energy of a photon is given by"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

📊 Data Pipeline

1. Open Datasets: Automatically download from HuggingFace (Pile, S2ORC, Math datasets, PubMed QA)
2. Quality Filtering: Multi-stage checks (length, language, equations, repetition, citations)
3. Deduplication: MinHash LSH for near-duplicate detection
4. Domain Classification: Classify into 7 science domains
5. Tokenization: Custom science-aware BPE tokenizer
6. Sharding: Write to Parquet with statistics

from data.dataset_loader import VortexDatasetLoader
from data.quality_filter import ScienceQualityFilter
from data.deduplication import MinHashLSH

# Load and process data
loader = VortexDatasetLoader()
quality_filter = ScienceQualityFilter()
lsh = MinHashLSH()

# Stream datasets, filter, deduplicate, and shard
for sample in loader.load_multiple_datasets(["pile_scientific", "automath"]):
    if quality_filter.filter(sample["text"]):
        lsh.add_document(sample["id"], sample["text"])
        # Tokenize and save

🎯 Training Strategy

Curriculum Learning

Training progresses through 4 stages:

1. Foundation (0-20%): Basic science text, simple equations, definitions
2. Domain (20-50%): Domain-specific deep content per science area
3. Reasoning (50-80%): Scientific problem solving, multi-step derivations
4. Integration (80-100%): Cross-domain science, full dataset

Science-Aware Loss

total_loss = (
    lm_loss * 1.0              # Standard next token prediction
    + equation_loss * 0.3      # Equation reconstruction accuracy
    + domain_loss * 0.1        # Domain classification head
    + citation_loss * 0.1      # Citation detection accuracy
    + numerical_loss * 0.2     # Numerical reasoning accuracy
)

⚙️ Configuration

7B Config (VORTEX_7B_CONFIG)

• d_model: 4096
• num_layers: 32
• num_heads: 32
• d_state: 16
• ssm_ratio: 0.6
• vocab_size: 50000
• max_seq_len: 16384

13B Config (VORTEX_13B_CONFIG)

• d_model: 5120
• num_layers: 40
• num_heads: 40
• d_state: 32
• ssm_ratio: 0.5
• vocab_size: 50000
• max_seq_len: 16384

🔧 Hardware Targets

Nvidia 4060 Laptop (8GB VRAM)

• 7B: BF16, no quantization, Flash Attention 2, torch.compile
• 13B: INT8 quantization, Flash Attention 2, torch.compile
• Target TPS: 25-40 (7B), 15-25 (13B)

Apple Silicon (M2/M3)

• 7B on M3: BF16 (via float16), SDPA, no compile
• 13B on M3 Max: BF16, unified memory, SDPA
• Target TPS: 20-35 (7B), 12-20 (13B)

🧪 Science Domains

1. Physics ([PHYS])
2. Mathematics ([MATH])
3. Chemistry ([CHEM])
4. Biology ([BIO])
5. Earth Science ([EARTH])
6. Space Science ([SPACE])
7. Zoology ([ZOO])

Domain tags can be included in training data to guide the SciGate FFN routing.

📝 Tokenizer

Custom BPE tokenizer with:

• 40,000 base BPE tokens trained on scientific corpus
• 10,000 science-specific tokens:
  • 500 LaTeX math symbols (\alpha, \sum, \int, etc.)
  • 118 chemical element symbols
  • 200 SI and derived units
  • 300 scientific abbreviations (DNA, RNA, ATP, etc.)
  • 500 mathematical operators
  • Amino acid codes
  • Greek alphabet (α, β, γ, etc.)
• Special tokens: [EQUATION], [CITATION], [MOLECULE], [FIGURE], [TABLE], domain tags

🧪 Evaluation

Science benchmarks across all 7 domains will be added. Planned benchmarks:

• Physics: Feynman Questions, Physics GRE
• Math: MATH dataset, GSM8K
• Chemistry: Chemistry problem-solving, molecular property prediction
• Biology: PubMed QA, bioinformatics tasks
• Earth Science: Climate modeling questions
• Space Science: Astronomy problem sets
• Zoology: Species classification, ecological reasoning

📄 License

This is a school science project. Code is provided for educational purposes.

🙏 Acknowledgments

• Mamba (Gu et al.) for SSM architecture inspiration
• Flash Attention (Dao et al.) for efficient attention
• HuggingFace for transformers library
• All open scientific data sources: arXiv, PubMed, S2ORC, etc.

📧 Contact

For questions or issues, please open an issue on GitHub.

---

Built with ❤️ for scientific AI research