Chaos, telemetry and determinism

Three questions for any agent harness: what faults does it survive, what records does it keep, and which parts behave the same way every time?

Related documents: 31-observability-alerting.md (alert routing), 32-security-red-team-log.md (guard testing method), 41-support-known-issues-faq.md. Path and role names follow 30-sre-runbooks.md.

Terms

• Chaos test: breaking something on purpose to see whether the system survives.
• Oracle: an independent source of truth used to grade a guard.
• Mutant: a deliberately broken copy of a guard. A good suite fails on it.
• Deterministic: the same input always gives the same output. Model judgement is not deterministic.

1. Summary

2. Chaos and stress

2.1 Fault classes a single-host fleet meets

Most serious stress on a single-host fleet is self-inflicted by fleet work: a scan, a model, a storm, a prune. External faults are rarer. Memory and disk pressure is the dominant failure mode.

2.2 Deliberate tests

The harness ships one suite: python3 harness/tests/run_all.py. The other rows describe tests to build for your own guards and sentinels; their scripts are not included in this repository.

2.3 Rehearsals worth running

These faults have no shipped automated test. Run each once on a quiet night with the owner's go, and record the result.

3. Telemetry

3.1 Where records go

3.2 Line shapes

Event log (the reference format), one JSON object per line, file named by UTC date:

{"id":"1767225600-orchestrator-a1b2c","agent":"orchestrator","org":"example","timestamp":"2026-01-01T00:00:00Z","category":"action","event":"task_dispatched","severity":"info","metadata":{"to":"ops","task":"rotate webhook secret"}}

id is epoch seconds, agent name and 5 random characters. timestamp is ISO 8601 UTC to the second. Writer: src/bus/event.ts.

Sentinels can write into the same family under a non-seat agent name such as sentinel:

{"id":"1767225630-sentinel-x9y8z","agent":"sentinel","timestamp":"2026-01-01T00:00:30Z","category":"action","event":"disk_sentinel_alert","severity":"info","metadata":{"avail_mb":11233,"kind":"sudden-step"}}

Outbound Telegram log:

{"timestamp":"2026-01-01T00:01:00Z","agent":"orchestrator","chat_id":"<chat>","text":"Deployed...","message_id":"1234","parse_mode":"html"}

Inbound Telegram log fields: message_id, from, from_name, chat_id, text, timestamp (milliseconds), archived_at (seconds), agent.

Crash and restart logs:

2026-01-01T00:02:00.000Z type=daemon-stop reason=daemon shutdown (SIGTERM) session=<uuid> last_task=...
[2026-01-01T00:03:00Z] HARD-RESTART: Context 80%; ...

stdout.log: raw terminal capture with ANSI escape codes. Strip escapes before parsing (strip-ansi or sed 's/\x1b\[[0-9;]*[A-Za-z]//g').

Sentinels (examples of the reference scripts):

2026-01-01T00:04:00Z avail=11233MB state=warn prev=warn orch=sudden-step
2026-01-01T00:05:00Z | proc=status | watched=4 | IDLE-LATENT=0 | WEDGE(alarms)=0 | K=3 | orchestrator:OK(0h0m,ctx) | ...
2026-01-01T00:06:00Z ok proc_up=1 ctx_age=0s ctx_basis=ctx-growth decided_by=primary (TTL 36000s)

Quota ledger (JSON, ts key):

{"ts":"2026-01-01T00:05:00Z","reading":{"five_hour":{"pct":11},"seven_day":{"pct":62},"source":"api"}}

Guard logs (~/.claude/hooks/<guard>.log), one line per verdict:

2026-01-01T00:07:00Z PASS Bash 0 issues: 'Deployed...'
2026-01-01T00:08:00Z ops BLOCK ['claude_config'] '<command>'

Guard logs are fleet-wide: one file holds verdicts for every seat that runs the guard. Not every guard writes a log; the brief guard and the future-time guard report only through their exit code and stderr.

3.3 Where the formats differ

Practical effect: a question like "what happened at 05:03" needs several greps with different patterns. If you build a merged timeline, normalise to the event-log shape: timestamp key, ISO 8601 seconds, UTC.

The command-line tool is named cortextos.

Recommended standard for new logs. JSON lines with timestamp (ISO 8601 UTC, seconds), agent, event, severity, and a metadata object. Write through cortextos bus log-event when a seat is the producer.

3.4 Retention

• The framework does not rotate stdout.log. Rotate it yourself (for example a launchd job that moves files over a size limit to stdout.log.1).
• Event files are split per day per seat, so size-based cleanups usually spare them.
• A cleanup that empties large log files erases their history. Archive before you truncate.
• Guard logs live outside $CTX_ROOT and need their own rotation.

4. Determinism: rule versus judgement

4.1 Hooks in an orchestrator seat

Shipped hooks install to <seat>/.claude/hooks/ogl/. Rows marked "optional, not included" describe components this repository does not ship.

All of these are type: command hooks. A command hook can still call a model; check the script, not the hook type.

4.2 Schedulers

Fires are not exact. A cron can run some minutes after its slot if the seat is busy. Design prompts to tolerate late fires.

4.3 The design pattern

Model judgement does the work. Deterministic guards fence its outputs: every send passes the send guard, every dispatch passes the brief guard, every write to a shared surface passes the surface guard.

Not fenced deterministically:

• The decision to call something "done". It rests on seat rules ("a return is a claim; run its BAR") and on the auditor, both model judgement. The brief audit script helps, but a seat must choose to run it.
• Memory written by consolidation and digest hooks.
• The auditor's verdict. The artefacts it checks are deterministic; the verdict is not.

5. Test suites

Two structural limits:

1. Install-time proof, not standing proof. Guard suites prove the guard once. Nothing re-runs them after install unless you schedule it. Freeze each gated version as <file>.gated-<sha> so the live file can be diffed against the version that passed.
2. No composition suite. Each guard is tested alone. Several PreToolUse hooks can fire on one Bash call. Interactions between them are where bugs hide; add a test that runs the full hook chain on a fixed set of calls.

6. Enterprise items and their equivalents

Design notes

• Run the framework suite on a schedule and page on red.
• Re-run each live guard's suite against the installed file nightly.
• Make model-calling hooks asynchronous where the seat does not need the result before it reads the message.

Source files

• src/bus/event.ts (event format)
• src/bus/hooks.ts (hook dispatch)
• src/bus/metrics.ts (event reader)
• src/daemon/cron-scheduler.ts (tick and catch-up)
• package.json, .github/workflows/ci.yml (framework tests)