Upload design.md

design.md

@@ -1,132 +1,286 @@
-# OCC Design Document
+# OCC: Formal System Definition
 
-## Philosophy
+## Overview
 
-Compute is the scarce resource in modern agent systems. Every action — a tool call, a retrieval, a debate turn, a verification pass — costs tokens, GPU seconds, or API dollars. Current systems allocate compute upfront (fixed budget) or reactively (retry on failure). OCC proposes a **proactive, earned-compute** model where agents must demonstrate marginal value before receiving more resources.
+OCC (Oracle-Credit-Compute) is a mechanism-design layer that governs agent access to compute, retrieval, debate turns, tool execution, and other resources. It treats compute allocation as a security boundary rather than a performance optimization.
 
-## Core Thesis
+## Core Insight
 
-> "Agents should earn compute, not spend it."
+In multi-agent systems, compute is not neutral. Extra turns, tokens, and tool calls can amplify adversarial influence unless access to deliberation is governed by verified marginal contribution. OCC makes agent compute scarce, earned, scoped, decaying, and auditable.
 
-The system is named Oracle-Credit-Compute because every compute decision flows through three stages:
-1. **Oracle:** Score the marginal impact of an action
-2. **Credit:** Update the agent's credit balance based on that score
-3. **Compute:** The broker decides whether to grant the requested resource
+---
 
-## Architecture
+## Formal Definition
 
-### Impact Oracle
+### Entities
 
-**Design principle:** Rule-based, auditable, resistant to reward hacking.
+Let:
+- **A** = {a₁, a₂, ..., aₙ} be a set of agents
+- **T** = {t₁, t₂, ..., tₘ} be a set of tasks
+- **R** = {r₁, r₂, ..., rₖ} be a set of resource types (model calls, retrieval, debate turns, tool execution, file writes, etc.)
+- **C** = {c₁, c₂, ..., cₗ} be a set of capability scopes
+- **O** be an Impact Oracle that maps (action, context, outcome) → score ∈ [−1, 1]
 
-Neural reward models are vulnerable to Goodhart's Law and reward hacking (Gao et al., 2023; Skalse et al., 2022). A neural RM can be optimized to produce high scores without producing correct answers. OCC uses rule-based scoring with the following properties:
+### Credit State
 
-- **Verifiable outcomes:** Code correctness is checked by running tests. QA correctness is checked against gold answers. Debate quality is checked against ground truth.
-- **Cost-adjusted scores:** Every score subtracts a compute cost penalty. This prevents agents from achieving correctness through brute-force token spending.
-- **Proper scoring rules:** Calibration bonus via Brier score encourages well-calibrated confidence, not just correctness.
-- **Anti-gaming detectors:** Explicit checks for hidden-test gaming, spam, collusion, and over-abstention.
+Each agent a has a credit vector at time step t:
 
-### Credit Ledger
-
-**Design principle:** Non-transferable, decaying, capability-scoped.
-
-- **Non-transferable:** `transfer()` always returns `False`. This prevents colluding agents from pooling credits or laundering them through intermediaries.
-- **Exponential decay:** Idle credits decay at rate λ per time step. This prevents hoarding and encourages agents to use credits or lose them.
-- **Capability-scoped:** Credits are scoped to specific capabilities (`retrieval`, `model_call`, `file_write`). An agent that is good at retrieval should not automatically get dangerous write permissions.
-- **Full provenance:** Every entry has an oracle score, compute cost, timestamp, and reason. This enables auditing and debugging.
+```
+credit[a, t] ∈ ℝ₊  (non-negative real)
+```
 
-### Resource Broker
+Credits are:
+- **Non-transferable**: ∀a,b ∈ A, a≠b, credit[b,t] cannot increase from credit[a,t]
+- **Decaying**: credit[a, t+1] = decay(credit[a,t]) where decay(x) = x · δ, δ ∈ (0,1)
+- **Task-scoped**: credits can be bound to a specific task τ
+- **Capability-scoped**: credits can be earmarked for capability scope c
 
-**Design principle:** Risk-adjusted, capability-based, dynamic.
+### Earning Function
 
-Resources are classified by risk:
-- **Low:** `retrieval_call`, `debate_turn` — threshold 0.5 credits
-- **Medium:** `model_call`, `verifier_call`, `memory_write` — threshold 2.0 credits
-- **High:** `file_write`, `shell_execute`, `human_escalation` — threshold 5.0 credits, may require approval
+```
+earn(a, action, oracle_score, compute_cost) → Δ ∈ ℝ
 
-The broker can make six decisions:
-- `ALLOW`: credits ≥ threshold, no flags
-- `DENY`: credits < threshold × 0.5
-- `REQUIRE_APPROVAL`: high-risk + high risk score
-- `DOWNGRADE`: credits between 0.5× and 1.0× threshold → downgrade to cheaper resource
-- `ESCALATE`: repeated denials from same agent
-- `ASK_JUSTIFICATION`: credits insufficient but agent has some history
+Δ = f(oracle_score, compute_cost, calibration, abstention_utility)
+```
 
-### GRPO/RL Hook
+Where f must satisfy:
+- oracle_score < 0 ⇒ Δ ≤ 0 (negative contribution yields ≤ 0 credit)
+- oracle_score = 0 ⇒ Δ = 0 (neutral action neither earns nor loses)
+- oracle_score > 0 ⇒ Δ > 0 (positive contribution earns credit)
+- compute_cost > 0 reduces Δ proportionally
+- calibration_error > threshold reduces Δ
+- confident_wrong action (high confidence + oracle_score < 0) ⇒ Δ < 0 (penalty)
 
-**Design principle:** Reward = verified impact - compute cost.
+### Spend Function
 
-The reward function wraps the Impact Oracle and produces a scalar reward per completion. It is designed to be passed directly to TRL's `GRPOTrainer` as `reward_funcs`.
+```
+spend(a, resource_type, capability_scope) → {allow, deny, downgrade, escalate, require_approval}
 
-The offline comparator allows policy comparison without training:
-1. Generate trajectories from two policies on the same test set
-2. Score both with the same reward hook
-3. Compare mean rewards, win rates, and failure rates
+allow if: credit[a,t] ≥ cost(resource_type, capability_scope)
+             AND a has capability_scope_policy[scope]
+             AND credit_decay_rate[a] ≤ max_decay
+             AND gaming_score[a] ≤ gaming_threshold
+```
 
-## Reward Formula
+### Decay Schedule
 
 ```
-reward =
-  verified_task_score
-  + abstention_utility
-  + calibration_bonus
-  - hallucination_penalty
-  - confident_wrong_penalty
-  - compute_cost_penalty
-  - gaming_penalty
+decay(credit[t]) = credit[t] · δ
 
-Where:
-  verified_task_score = correctness * weight_correctness
-  abstention_utility = +1.0 if correct abstain, -1.0 if wrong abstain
-  calibration_bonus = (1 - brier_score) * weight_calibration
-  brier_score = (confidence - outcome)^2
-  hallucination_penalty = 2.0 if entailment < 0.5 and contradiction > 0.5
-  confident_wrong_penalty = 3.0 if confidence > 0.8 and correctness < 0.5
-  compute_cost_penalty = compute_cost * 0.0001
-  gaming_penalty = 2.0 if hidden_tests fail while public pass
+where:
+  δ = 0.995  (per-turn decay, ~5% per 10 turns)
+  Or task-scoped: δ = 1.0 until task completion, then δ = 0.0 (credits expire)
 ```
 
-## Anti-Gaming Design
+### Credit Caps
 
-### Spam Attacks
-- Detection: Repeated low-value actions (compute > 100, raw_score < 0.5)
-- Containment: Oracle subtracts gaming_penalty. Ledger can revoke all credits on explicit detection.
+```
+credit[a,t] ≤ credit_cap(capability_scope)
 
-### Hidden-Test Gaming
-- Detection: `public_pass=True` but `hidden_pass=False`
-- Containment: Immediate gaming_penalty=2.0 subtracted from raw score.
+credit_cap translates to maximum resource access:
+  Model calls:     credit_cap / cost_per_call
+  Retrieval calls: credit_cap / cost_per_retrieval
+  Debate turns:    credit_cap / cost_per_turn
+```
 
-### Credit Laundering
-- Prevention: `transfer()` returns `False` unconditionally.
+### Oracle Scoring
 
-### Credit Hoarding
-- Prevention: Exponential decay on idle credits.
+```
+oracle_score = α₁ · correctness(a, t, outcome)
+             + α₂ · evidence_support(a, t, evidence)
+             + α₃ · improvement_over_prior(a, t, prior_state)
+             + α₄ · calibration(a, t, prediction, outcome)
+             + α₅ · abstention_utility(a, t, decision_to_abstain)
+             − β₁ · hallucination(a, t, evidence)
+             − β₂ · confident_wrong(a, t, prediction, outcome, confidence)
+             − β₃ · wasteful_compute(a, t, compute_used, value_produced)
+             − β₄ · gaming_suspicion(a, t, action_pattern)
+
+where:
+  correctness:        1 if correct, 0 if incorrect, −1 if harmful
+  evidence_support:   1 if evidence fully supports, 0 if neutral, −1 if contradicts
+  improvement:        + if better than prior, 0 if same, − if worse
+  calibration:        + if well-calibrated, − if overconfident
+  abstention_utility: + if abstaining was correct, − if it was evasive but answerable
+  hallucination:      − if generated claim contradicts evidence
+  confident_wrong:    − if high confidence AND incorrect (larger penalty than regular wrong)
+  wasteful_compute:   − if compute used ≫ value produced
+  gaming_suspicion:   − if action pattern matches known gaming signatures
+
+Default weights (tunable):
+  α = [0.30, 0.15, 0.10, 0.10, 0.15]
+  β = [0.20, 0.25, 0.15, 0.20]
+```
 
-### Over-Abstention
-- Detection: Agent abstains on answerable questions.
-- Containment: Wrong abstentions get -abstention_bonus (-1.0).
+### Reward Function (for RL/GRPO)
 
-### Confidence Manipulation
-- Detection: Brier score in calibration bonus.
-- Containment: Overconfident wrong answers get confident_wrong_penalty=3.0.
+```
+reward(a, action, context, outcome) =
+    oracle_score(a, action, context, outcome)
+    + abstention_utility
+    + calibration_bonus
+    − hallucination_penalty
+    − confident_wrong_penalty
+    − compute_cost · cost_multiplier
+    − gaming_penalty(a, history)
+
+Constrained to [−1, 1].
+```
 
-## Compute Budgeting
+---
+
+## System Invariants
+
+1. **Non-transferability**: ∀a,b ∈ A, a≠b: Δcredit[b] from a's action = 0
+2. **Positive decay**: ∀a: credit[a, t+1] ≤ credit[a, t] unless earned
+3. **Capability scoping**: access(r) requires scope_policy[r] AND credit ≥ cost(r)
+4. **External verification**: oracle_score depends only on oracle O, not on a
+5. **Append-only ledger**: credit events are immutable once recorded
+6. **Oracle separation**: spending agent cannot directly influence oracle O
+7. **Negative contribution**: oracle_score < 0 ⇒ Δ ≤ 0
+8. **Credit ≠ identity trust**: high credit does not imply trusted access to all resources
+9. **Reversal possible**: credit can be retroactively reduced on new evidence
+10. **Bounded credit**: credit[a,t] ≤ credit_cap(scope) always
+
+---
+
+## Ledger Event Schema
+
+Every credit mutation produces an immutable event:
+
+| Event | Fields |
+|-------|--------|
+| CREDIT_GRANTED | agent_id, amount, reason, oracle_score, task_id, timestamp |
+| CREDIT_DECAYED | agent_id, amount_decayed, new_balance, timestamp |
+| CREDIT_SPENT | agent_id, amount, resource_type, capability_scope, task_id, timestamp |
+| TURN_DENIED | agent_id, reason (insufficient_credit/wrong_scope/gaming_threshold), timestamp |
+| ORACLE_SCORE_RECORDED | agent_id, action_id, score, confidence, evidence_ref, timestamp |
+| CAPABILITY_SCOPE_CHANGED | agent_id, old_scope, new_scope, reason, timestamp |
+| AGENT_PENALIZED | agent_id, penalty_amount, reason, evidence, timestamp |
+| VERIFICATION_REVERSED | original_event_hash, new_score, reason, timestamp |
+| POOL_EXHAUSTED | task_id, remaining_credit, timestamp |
+| POLICY_UPDATED | parameter_changes, reason, timestamp |
+
+Each event includes:
+- event_hash: SHA-256 of (previous_event_hash + event_data)
+- parent_event_hash: chain to previous event
+- agent_id
+- task_id
+- timestamp (UTC ISO 8601)
+- capability_scope
+- oracle_id
+- score (if applicable)
+- credit_delta
+- reason (human-readable)
+- evidence_pointer (URI or hash to evidence)
+
+---
+
+## Resource Broker Decision Model
+
+For each request (agent a, resource r, scope c):
 
-The system assumes a fixed compute budget per task. The broker enforces this by:
-1. Tracking cumulative compute cost in the ledger entries
-2. Denying requests when the agent's credit balance is below the threshold
-3. Downgrading to cheaper resources when balance is marginal
+```
+function decide(a, r, c):
+    if not has_scope(a, c):
+        return DENY(reason="missing capability scope")
+    
+    if credit[a] < cost(r, c):
+        if credit[a] >= cost(downgraded(r), c):
+            return DOWNGRADE(alternative=downgraded(r), reason="insufficient credit for requested tier")
+        return DENY(reason="insufficient credit")
+    
+    if gaming_score[a] > GAMING_THRESHOLD:
+        return REQUIRE_APPROVAL(reason="gaming suspicion")
+    
+    if risk(r, a, c) > RISK_THRESHOLD:
+        return REQUIRE_APPROVAL(reason="high-risk action")
+    
+    if credit[a] < cost(r, c) * 2:  # running low
+        return ALLOW_WITH_WARNING(reason="low credit warning")
+    
+    return ALLOW
+```
 
-## Failure Modes
+### Resource Types and Costs
+
+| Resource | Base Cost | Capability Scope |
+|----------|-----------|-----------------|
+| model_call_small | 1 | basic_inference |
+| model_call_large | 5 | premium_inference |
+| retrieval_call | 2 | retrieval |
+| verifier_call | 3 | verification |
+| debate_turn | 3 | deliberation |
+| file_write | 5 | tool_execution |
+| shell_exec | 8 | tool_execution |
+| memory_write | 2 | memory |
+| human_escalation | 20 | escalation |
+
+---
+
+## When To Use OCC
+
+| OCC is valuable when | OCC is overkill when |
+|---------------------|---------------------|
+| Agents have heterogeneous reliability | Single-agent tasks suffice |
+| Long-running tasks need budget discipline | Ground truth is immediate and cheap |
+| Debate/collaboration can be poisoned | Adversarial participation is impossible |
+| Compute is expensive | All agents have equal trust and capability |
+| Auditability matters | Task budget is tiny (a few calls) |
+| Agents can earn durable authority | Latency matters more than robustness |
+| Post-hoc accountability required | Verifier/oracle cost exceeds saved compute |
+| Agents can game naive allocation | There are no bad actors in the system |
+
+---
+
+## Threat Model
+
+| Attack | What Adversary Controls | Success Condition | OCC Defense | Residual Risk |
+|--------|------------------------|-------------------|-------------|---------------|
+| Credit farming | Task selection | Accumulates budget via easy tasks | Decay + credit caps | Slow gaming over many cheap tasks |
+| Collusion | Multiple agent identities | Transfers influence between agents | Non-transferability | Vote-ring behavior (same answer) |
+| Oracle spoofing | Persuasive but wrong answers | Earns false credit | Verifier separation from spender | Judge hacking via prompt injection |
+| Griefing | Burns others' budget | Lowers group accuracy | Capability-scoped spend | Indirect poisoning via bad data |
+| Sandbagging | Hides competence | Manipulates future allocation | Decay + exploration bonus | Hard to detect without history |
+| Identity laundering | Resets agent identity | Escapes penalties | Identity binding to account | Account churn (rate-limited) |
+| Sybil agents | Many weak agents | Captures compute pool | Admission control | Deployment-specific new-account policy |
+| Strategic abstention | Avoids penalties | Hoards credit | Reward shaping for participation | Conservatism bias |
+| Verbosity gaming | Produces long but vacuous responses | Appears high-quality to weak oracle | Token-cost multiplier in reward | Requires quality oracle |
+| Confidence manipulation | Overstates certainty | Earns calibration bonus deceptively | Proper scoring rules | Hard to calibrate perfectly |
+
+---
+
+## Relationship to Prior Work
+
+OCC builds on:
+- **AI safety debate** (Irving, Christiano, Amodei 2018): Debate as a mechanism for surfacing truth. OCC adds: debate turns are not free speech — they are auditable compute privileges.
+- **GRPO/RLVR** (Shazeer et al. 2024): Group-relative policy optimization. OCC provides the reward function that makes GRPO converge to allocation policies.
+- **Proper scoring rules**: OCC's calibration and abstention rewards are proper scoring rule implementations.
+- **Capability-based security**: OCC's broker follows OS capability-system principles applied to agent API access.
+
+OCC departs from:
+- **Budget-aware reasoning** (e.g., token-budget RL): OCC is not about *minimizing* compute — it's about *governing* compute access.
+- **Adaptive inference** (early exit, cascade): OCC governs *who* gets compute, not *when* to stop computing.
+- **Multi-agent debate for accuracy**: OCC does not claim debate improves accuracy. It claims debate *without allocation control* amplifies adversarial influence.
+
+---
+
+## Implementation Reference
+
+Python package at: https://huggingface.co/narcolepticchicken/occ-stack
 
-1. **Oracle brittleness:** If the scoring rules are incomplete, agents will find and exploit the gaps.
-2. **Broker conservatism:** If thresholds are too high, agents cannot act even when they should.
-3. **Decay too aggressive:** If λ is too high, agents lose credits before completing multi-step tasks.
-4. **Scope explosion:** Capability-scoped credits multiply the state space.
+```
+/occ
+  /oracle     → oracle.py     (Impact Oracle: scoring, marginal impact, proper scoring)
+  /ledger     → ledger.py     (Credit Ledger: non-transferable, decaying, scoped credits)
+  /broker     → broker.py     (Resource Broker: capability-based access control)
+  /rl         → reward.py     (Reward function combining oracle + anti-gaming)
+             → grpo_hook.py  (TRL GRPOTrainer integration)
+  /benchmarks → benchmark_debate.py, benchmark_code.py, benchmark_retrieval_qa.py
+  /configs    → YAML configurations for experiments
+  /reports    → results, analysis, final report
+```
 
-## Future Extensions
+---
 
-1. **Hierarchical broker:** Nested capability scopes (e.g., `model_call/code` vs `model_call/qa`).
-2. **Dynamic thresholds:** Learn thresholds from historical data rather than hardcoding.
-3. **Peer review:** Multiple oracles vote on controversial actions.
-4. **Human-in-the-loop:** Escalate high-risk decisions to human reviewers with credit incentives.
+*Last updated: May 8, 2026. Version: 1.0.*