Observability and alerting

What OGL Cortex measures, where each record lands, what writes it, the default thresholds, and who hears about it.

Related documents: 30-sre-runbooks.md (what to do when an alert fires), 32-security-red-team-log.md (guards as defences), 40-finops-capacity.md (quota and capacity limits), 42-chaos-telemetry-determinism.md (raw log formats). Path names ($CTX_ROOT, $SENTINEL_DIR, <seat>) and role names follow the conventions in 30-sre-runbooks.md.

1. Three kinds of signal

1. Self-reports. Seats write heartbeats, events and message logs about themselves through the OGL Cortex bus.
2. Sentinels. Shell or Python scripts run by launchd (or pm2) outside any model. They check the machine, the quota, credentials and websites. They keep working when every seat is down or capped.
3. Agent crons. The daemon injects a prompt into a seat on a schedule. The seat runs a check and decides whether to speak. These need a live seat with quota.

The command-line tool is named cortextos.

A page means a cortextos bus send-message <seat> high '<text>' call. It lands in that seat's inbox. Most pages go to the orchestrator, which decides whether the owner hears about it. A short list of producers reaches the owner's Telegram directly (section 5), so that a capped or busy orchestrator cannot swallow them.

2. Signal flow

3. Where records are written

4. Signal inventory with default thresholds

The thresholds below are the defaults in the reference sentinel scripts and framework hooks. Tune them per host.

4.1 Seat and daemon health

4.2 Machine capacity

4.3 Quota, login and credentials

4.4 Guards

4.5 Business and housekeeping

5. Paths that reach the owner without the orchestrator

Model-run crons can also message the owner on a condition. Those need a live seat with quota.

6. Signals that look live but are not

Check for these when you audit an install:

• Dead meters that read zero. A usage file that stopped updating still reads 0. Anyone who trusts it sees an idle plan. Rename retired meters to RETIRED-<name> so they cannot pass as healthy.
• Duplicate jobs that never run. A crontab copy of a sentinel that cannot reach the login keychain logs SKIP every run. It looks like cover and gives none. Keep one working copy.
• Disabled watchers. launchctl print-disabled gui/$(id -u) lists jobs that exist but never fire.
• Optional hooks not registered. cortextos bus hook-loop-detector blocks repeated tool loops, but only on seats whose settings.json registers it.
• Dated one-shot crons. A 5-field expression with a fixed day and month fires again a year later unless the prompt removes it.

7. Known limitations

• Most pages go to one inbox. If the orchestrator is capped, compacting or deep in a long job, pages queue. The direct owner paths in section 5 exist for that reason.
• No reply-latency alert ships by default. Inbound and outbound logs both carry timestamps; see the design notes.
• No per-seat token burn. The quota ledger shows the account total only.
• No cost-per-day figure. Bills arrive by mail; usage APIs are per vendor.
• Disk is measured by more than one sentinel, each with its own floor. Know which one is speaking.
• The heartbeat sentinel cannot watch the seat that runs it. Assign another seat to watch ops.
• Temporary watch scripts started with nohup die on reboot. Promote anything you want to keep to a launchd job.

8. Design notes: cheap additions

Source files

• src/bus/event.ts (event log writer)
• src/bus/metrics.ts (collect-metrics)
• src/telegram/logging.ts (outbound log)
• src/hooks/hook-crash-alert.ts, src/hooks/hook-compact-telegram.ts, src/hooks/hook-loop-detector.ts
• src/daemon/fast-checker.ts (context thresholds)
• dashboard/src/lib/watcher.ts (event feed)