AbstractPhil/geolip-deep-embedding-analysis

GeoLIP Deep Embedding Analysis

The first battery is complete. The JSON shows all the embedding sizes that exist within the CV band.

Parse the sweep.json and find your model attention bands related to your embedding spaces. This can be used as a differentiation utility to determine how much downstream task is required to compensate for the embedding, how much should be reduced, how many layers your embeddings can propagate to, and the effective geometric range of those in conjunction.

What This Measures

The Cayley-Menger determinant computes the squared volume of a 4-simplex (pentachoron) formed by 5 randomly sampled embedding vectors. The coefficient of variation (CV) of these volumes across many random samples reveals the geometric operating regime of the embedding space.

• CV > 0.30: Volatile — simplex volumes vary wildly, geometric measurements are unstable
• 0.13 < CV < 0.30: Band-valid — volumes carry discriminative structural information
• CV < 0.13: Degenerate — all simplices look identical, the measurement is blind

The band exists as a function of embedding dimension only. Vocabulary size is irrelevant. Training signal does not move CV — it is a property of the ambient dimensionality.

Key Findings

Dimension	Avg CV	Band Status
D=8	0.605	Above band — volatile
D=16	0.383	Above band — entering
D=24	0.304	Phase boundary — binding constant 0.29154
D=32	0.257	Center of band
D=40	0.229	Center of band
D=48	0.207	Center of band
D=56	0.192	In band
D=64	0.180	In band
D=72	0.168	In band
D=80	0.159	In band
D=88	0.152	In band
D=96	0.144	In band
D=104	0.139	In band
D=112	0.134	In band
D=120	0.129	Below band — exiting
D=128	0.125	Below band
D=256	0.088	Degenerate
D=512	0.063	Degenerate
D=768	0.051	Degenerate

The standard MHA convention of 64 dims per head sits inside the band. This may be a direct causal relationship — the matmul scaling principle in attention operates at the dimensionality where simplex geometry remains discriminative.

Sweep Data

{
  "sweep": {"step": 8, "low": 8, "high": 2048},
  "band": {"lo": 0.13, "hi": 0.30},
  "band_results": [... 3014 entries sorted by CV ...],
  "all_results": [... 65536 entries ...]
}

Each entry: {"V": vocab_size, "D": dim, "CV": value, "in_band": bool}

Download and Nearest Dimensional Lookup

import json
import urllib.request

URL = "https://huggingface.co/AbstractPhil/geolip-deep-embedding-analysis/resolve/main/cv_sweep.json"

def load_sweep(path=None):
    """Load sweep from local path or download from HF."""
    if path:
        with open(path) as f:
            return json.load(f)
    with urllib.request.urlopen(URL) as r:
        return json.loads(r.read().decode())

def nearest_band_dim(target_dim, sweep=None):
    """Find the nearest band-valid dimension to your model's embedding dim.
    
    Returns the closest D where CV is in band, plus the expected CV range.
    Use this to determine compartment size for patchwork decomposition.
    
    Example: Your model uses D=768. This tells you to decompose into
    compartments of D=32 (24 compartments) or D=64 (12 compartments).
    """
    if sweep is None:
        sweep = load_sweep()

    # Build D -> CV stats from band_results
    by_dim = {}
    for r in sweep["band_results"]:
        d = r["D"]
        if d not in by_dim:
            by_dim[d] = []
        by_dim[d].append(r["CV"])

    band_dims = sorted(by_dim.keys())
    if not band_dims:
        return None

    # Find nearest
    nearest = min(band_dims, key=lambda d: abs(d - target_dim))

    # Also find best decompositions of target_dim
    decompositions = []
    for d in band_dims:
        if target_dim % d == 0:
            n_compartments = target_dim // d
            cvs = by_dim[d]
            decompositions.append({
                "compartment_dim": d,
                "n_compartments": n_compartments,
                "cv_min": round(min(cvs), 4),
                "cv_max": round(max(cvs), 4),
                "cv_avg": round(sum(cvs) / len(cvs), 4),
            })

    cvs = by_dim[nearest]
    return {
        "target_dim": target_dim,
        "nearest_band_dim": nearest,
        "cv_range": [round(min(cvs), 4), round(max(cvs), 4)],
        "cv_avg": round(sum(cvs) / len(cvs), 4),
        "valid_decompositions": sorted(decompositions, key=lambda x: x["compartment_dim"]),
    }


# ── Usage ──

if __name__ == "__main__":
    for model_dim in [768, 1024, 512, 384, 256, 128]:
        result = nearest_band_dim(model_dim)
        print(f"\n{'='*50}")
        print(f"Model dim: {model_dim}")
        print(f"Nearest band dim: D={result['nearest_band_dim']}  CV={result['cv_avg']:.4f}")
        if result["valid_decompositions"]:
            print(f"Valid decompositions:")
            for dec in result["valid_decompositions"]:
                print(f"  {dec['n_compartments']:3d} × D={dec['compartment_dim']:3d}  "
                      f"CV={dec['cv_avg']:.4f} [{dec['cv_min']:.4f}-{dec['cv_max']:.4f}]")
        else:
            print(f"  No exact decompositions — consider padding or truncating")

Parse and Filter

import json

with open("cv_sweep.json") as f:
    data = json.load(f)

# Filter for any CV range — example: binding constant region
lo, hi = 0.290, 0.292
hits = [e for e in data["band_results"] if lo <= e["CV"] <= hi]
hits.sort(key=lambda x: x["CV"])

print(f"CV in [{lo}, {hi}]: {len(hits)} entries")
for h in hits:
    print(f"  V={h['V']:6d}  D={h['D']:4d}  CV={h['CV']:.4f}")

# Group by D
dims = {}
for h in hits:
    dims.setdefault(h["D"], []).append(h)
for d in sorted(dims):
    entries = dims[d]
    print(f"  D={d:3d}: {len(entries)} entries  "
          f"CV={min(e['CV'] for e in entries):.4f}-{max(e['CV'] for e in entries):.4f}")

Rescale and Sort

def rescale_sort(sweep=None, group_by="dim"):
    """Sort and group sweep results for analysis.

    group_by: 'dim' groups by embedding dimension (recommended)
              'cv'  groups into cv quartiles within band
              'ratio' groups by V/D ratio
    """
    if sweep is None:
        sweep = load_sweep()

    band_lo = sweep["band"]["lo"]
    band_hi = sweep["band"]["hi"]
    results = [r for r in sweep["all_results"] if r["CV"] is not None]

    if group_by == "dim":
        # Group by D, show band status and CV statistics
        by_dim = {}
        for r in results:
            d = r["D"]
            if d not in by_dim:
                by_dim[d] = {"in_band": [], "below": [], "above": []}
            if r["CV"] > band_hi:
                by_dim[d]["above"].append(r["CV"])
            elif r["CV"] < band_lo:
                by_dim[d]["below"].append(r["CV"])
            else:
                by_dim[d]["in_band"].append(r["CV"])

        table = []
        for d in sorted(by_dim.keys()):
            g = by_dim[d]
            all_cvs = g["in_band"] + g["below"] + g["above"]
            avg = sum(all_cvs) / len(all_cvs)
            table.append({
                "D": d,
                "avg_cv": round(avg, 4),
                "in_band_pct": round(100 * len(g["in_band"]) / len(all_cvs), 1),
                "n_total": len(all_cvs),
                "n_in_band": len(g["in_band"]),
                "status": "IN_BAND" if band_lo < avg < band_hi else
                          "ABOVE" if avg >= band_hi else "BELOW",
            })
        return table

    elif group_by == "cv":
        # Quartile analysis within band
        band = [r for r in results if band_lo < r["CV"] < band_hi]
        if not band:
            return []
        band.sort(key=lambda r: r["CV"])
        n = len(band)
        return {
            "total_in_band": n,
            "q1_low":  [r for r in band[:n//4]],
            "q2_mid_low": [r for r in band[n//4:n//2]],
            "q3_mid_high": [r for r in band[n//2:3*n//4]],
            "q4_high": [r for r in band[3*n//4:]],
            "q1_cv_range": [round(band[0]["CV"], 4), round(band[n//4-1]["CV"], 4)],
            "q2_cv_range": [round(band[n//4]["CV"], 4), round(band[n//2-1]["CV"], 4)],
            "q3_cv_range": [round(band[n//2]["CV"], 4), round(band[3*n//4-1]["CV"], 4)],
            "q4_cv_range": [round(band[3*n//4]["CV"], 4), round(band[-1]["CV"], 4)],
        }

    elif group_by == "ratio":
        # Group by V/D ratio — demonstrates V irrelevance
        band = [r for r in results if band_lo < r["CV"] < band_hi]
        by_ratio = {}
        for r in band:
            ratio = round(r["V"] / r["D"], 1)
            if ratio not in by_ratio:
                by_ratio[ratio] = []
            by_ratio[ratio].append(r)
        return {k: {"count": len(v), "dims": sorted(set(r["D"] for r in v))}
                for k, v in sorted(by_ratio.items())}


# ── Usage ──

if __name__ == "__main__":
    table = rescale_sort(group_by="dim")
    print(f"{'D':>5}  {'Avg CV':>8}  {'Band%':>6}  {'Status'}")
    print("-" * 40)
    for row in table:
        if row["D"] <= 256:
            print(f"{row['D']:5d}  {row['avg_cv']:8.4f}  {row['in_band_pct']:5.1f}%  {row['status']}")

The Binding Constant is D=24

Filtering the sweep for CV in [0.290, 0.292] — the region around the empirically observed binding constant 0.29154 — returns 12 entries:

V	D	CV
24	16	0.2900
368	32	0.2903
1632	24	0.2906
208	24	0.2908
1096	24	0.2911
1992	24	0.2911
200	24	0.2914
1024	24	0.2916
760	24	0.2917
1232	24	0.2917
776	24	0.2919
904	24	0.2920

10 of 12 entries are D=24. The binding constant 0.29154 is the native CV of a 24-dimensional embedding space. It is not a learned value. It is not an empirical coincidence. It is the geometric fingerprint of D=24.

The Computational Boundary

D=24 is also the exact dimension where custom SVD kernels hit an 8x performance cliff and eigendecomposition (eigh) collapses. The binding constant marks a dual boundary:

• Geometric: the phase transition between volatile simplex volumes (above 0.30) and discriminative geometry (below 0.30)
• Computational: the resolution limit of compact spectral decomposition kernels

Every time the constant 0.29154 appeared across 17+ pretrained models, the system was measuring the dimensional fingerprint of its own computational ceiling. The constellation encoded this ceiling as a structural constant because it could not compute past it.

D=32 is the first dimension past this wall that remains in band (CV ~0.257). Operating there requires torch.linalg.det on a 6×6 CM matrix — which compiles regardless of embedding dimension, because the CM matrix is always 6×6 for five-point simplices. The pairwise distances are computed via gram matrix (batched matmul, compiles perfectly). Only the det call touches linalg, and 6×6 is well within kernel range.

MHA Activation Geometry

Measuring CV on per-head Q/K/V activations (not weights) after training reveals head_dim-dependent geometric behavior:

head_dim	Q activation CV	K activation CV	V activation CV
64	~0.32	~0.42	~0.41
32	~0.38	~0.45	~0.43
16	~0.48	~0.70	~0.53
8	~0.65	~0.77	~0.63

Key observations:

• Embedding activations are always in band (CV 0.19–0.30) regardless of nominal D — training compresses effective dimensionality into band
• K activations are asymmetrically volatile — keys spread further than queries to make attention discriminative
• Q activations track head_dim following the same curve as the embedding sweep — the 64-dim convention keeps Q near band edge
• The Q/K ratio measures selectivity pressure: too high = brittle attention, too close to 1.0 = uniform attention

These ratios can be used as a zero-cost diagnostic on any pretrained transformer: forward one batch, measure per-head activation CV, and immediately identify which heads are geometrically healthy vs collapsing.

Vocabulary Independence

CV at D=32 was verified from V=32 to V=13,000,000. The result is invariant:

V=        32  D=32  CV=0.2578
V=       512  D=32  CV=0.2615
V=     8,192  D=32  CV=0.2578
V=    65,536  D=32  CV=0.2663
V=   131,072  D=32  CV=0.2590
V=   500,000  D=32  CV=0.2745
V= 1,000,000  D=32  CV=0.2645
V= 4,000,000  D=32  CV=0.2541
V=13,000,000  D=32  CV=0.2681

Vocabulary size does not gate band membership. The CM determinant samples 5 points — the distribution of simplex volumes depends on ambient dimensionality, not on the number of points in the space.

Implications for Architecture Design

The band is not a training outcome. It is a geometric property of dimensionality. This means:

1. Embedding compartments must be D=32 to D=64 for Cayley-Menger volumes to carry discriminative information
2. A 768-dim model should decompose into 24×32 or 12×64 compartments, not operate as a monolithic vector
3. The standard 64-dim attention head may exist precisely because it sits inside this geometric band
4. Scaling comes from composing band-valid units with geometric linkages, not from widening dimensions beyond the band
5. D=24 (CV=0.29154) is the phase boundary — any component pushed above this threshold has crossed from structured into volatile geometry
6. The 6×6 CM determinant compiles at any embedding dimension — the computational bottleneck was in spectral decomposition, not in the geometric measurement itself

Reproducing

# The sweep script that generated this data
# Requires: torch

import torch, torch.nn as nn, torch.nn.functional as F, math, json

def cayley_menger_vol2(points):
    B, N, D = points.shape
    gram = torch.bmm(points, points.transpose(1, 2))
    norms = torch.diagonal(gram, dim1=1, dim2=2)
    d2 = F.relu(norms.unsqueeze(2) + norms.unsqueeze(1) - 2 * gram)
    cm = torch.zeros(B, N+1, N+1, device=points.device, dtype=points.dtype)
    cm[:, 0, 1:] = 1; cm[:, 1:, 0] = 1; cm[:, 1:, 1:] = d2
    k = N - 1
    return ((-1)**(k+1)) * torch.linalg.det(cm.float()).to(points.dtype) / ((2**k) * (math.factorial(k)**2))

def cv_metric(weight, n_samples=300):
    V, D = weight.shape
    pool = min(V, 512)
    idx = torch.stack([torch.randperm(pool)[:5] for _ in range(n_samples)])
    vol2 = cayley_menger_vol2(weight[:pool][idx])
    valid = vol2 > 1e-20
    if valid.sum() < 10: return None
    vols = vol2[valid].sqrt()
    return (vols.std() / (vols.mean() + 1e-8)).item()

Citation

Part of the GeoLIP geometric deep learning research.