Documentation snapshot: This describes an internal system as documented, not its current live status. Configuration, counts, and operating state may have changed.
An unattended system is only as good as its ability to notice and fix its own problems. Archon Merkaba's reliability layer has three levels: observe (monitors), decide (fingerprinting + registry), and act (tiered remediation) — with a human escalation path that is used deliberately, not by default.
Observe: The Monitor Fleet
18+ independent monitors run on schedules on the cloud plane, each narrow by design:
- Resource monitors (CPU, disk, RAM, swap) with per-process attribution
- Trading-health monitors (daemon liveness, position risk, drawdown warnings)
- Data-freshness monitors (stale status files, stale candle data)
- Cron-health monitors (did scheduled jobs actually run?)
- Security monitors (threat scanning, secrets-in-tree, host artifact drift)
- Gateway watchdog (gateway + messaging + dashboard + lock cleanup + stuck sessions)
All monitor output is normalized onto a single event bus (append-only JSONL) with a shared alert-history stream — one place to answer "what has the system seen lately?"
Drift Detection
Two dedicated detectors watch for change rather than failure:
- Host artifact drift — daily hashing of critical host-side files (restart scripts, systemd units, firewall rules). Any change alerts by chat-ops, distinguishing sanctioned deployments from surprises.
- Decision drift — an auditor reviews recent agent outputs for compliance with the Evidence Gate protocol (are claims supported by cited code/data?), catching quality regression in the agents themselves, not just the infrastructure.
Decide: Fingerprints and the Registry
Every alert is reduced to a fingerprint — a canonical identity for a class of problem, extracted with strict source scoping so a trading-health signal can never be confused with a freshness signal. Fingerprints map to a remediation registry: a versioned JSON document pairing each known problem class with its canonical response, cooldowns, and safety flags.
Act: Tiered Remediation
The registry classifies responses by blast radius:
| Tier | Name | Response | Example |
|---|---|---|---|
| 0 | Static | Deterministic shell fix | Rotate an oversized log |
| 1 | Scripted | Recovery script / service restart | Restart the VPN when the trading API is unreachable |
| 2 | Agent | Dispatch a specialist agent to investigate | Stale daemon status → infrastructure agent investigates (no auto-restart) |
| 3 | Notify-only | Carve-out: never auto-touch | Anything affecting live positions or risk state |
The safety model is defense-in-depth:
- Global dry-run default — the executor simulates until explicitly enabled, and rollout proceeds one fingerprint at a time, ordered by blast radius, with a 24-hour observation window per activation.
- Shell allowlist — the executor can only run a short list of vetted commands.
- Cooldowns and daily caps — remediation storms are structurally impossible.
- Hard carve-outs — trading daemon state and risk configuration are tier-3 forever; no automation may touch them regardless of confidence.
Outcomes close the loop: every attempt is recorded to an outcomes database and audit mirror, the alerting layer skips issues the executor already handled, and a daily health audit by the self-healing agent reviews the whole pipeline. When a recent audit found the pipeline had been effectively idle (hundreds of attempts, all skipped by dry-run and caps), that finding itself became a remediation project — the monitoring monitors the remediator.
Restart Discipline
Process management follows a hard protocol learned from incidents:
- One sanctioned restart path (a host-level safe-restart script with hash-aware deployment tracking and notifications) — enforced by a pre-tool-use hook that blocks ad-hoc restart commands.
- Single kill → verify stopped → single start → verify healthy. Never
kill -9first, never multiple start methods racing each other. - Reboot resilience: autostart units are ordered so the daemon, its data feeds, the VPN, and the vault recover in dependency order — validated by an intentional reboot test rather than assumed.
Escalation Path
When automation can't or shouldn't act, alerts flow to the founder by chat-ops with severity-appropriate routing. The design goal is a quiet channel: tier 0–2 handles the mechanical noise, drift detection catches the surprising, and a human sees only what genuinely needs judgment. Alert triage itself is agent-work — an on-call agent investigates each escalation and journals its findings, so repeated alerts get faster answers over time.