# Orchestrator loop skeleton A copy-pasteable skeleton for the "one decider, many disposable workers" pattern. One agent wakes a few times a day, reads state, decides what matters, fans out a pool of throwaway workers to drain a shared queue, waits for them, then writes a single rolled-up note upward. Workers are cheap and mortal: a narrow mandate, a task cap, and a wall-clock timer. They claim work atomically so parallel workers never collide. This file is intentionally generic. It is pseudocode plus a real SQL claim pattern. Swap in your own queue table, your own worker runtime, and your own notification sink. Nothing here is tied to any product. --- ## Mental model - ONE orchestrator. It decides AND dispatches. Keep both hats on one head so the decider has the freshest read on state at the exact moment it sizes the fan-out. - MANY disposable workers. Each one drains a slice of the queue, then dies. - Hygiene runs FIRST, before any fan-out. Release stale claims and reap ghosts so fresh workers inherit a true picture of the queue. - A BARRIER at the end. Collect every worker result, reconcile, and write exactly one summary. Fan out to N workers without fanning out to N notifications. - Build the LIMITS before the fan-out. Caps are the safety interlock, not polish. --- ## 1. The queue table Minimum columns. Add your own payload fields. ```sql create table task_queue ( id bigint generated always as identity primary key, mode text not null, -- which kind of work status text not null default 'queued', -- queued | in_progress | done | failed payload jsonb not null default '{}', claimed_by text, -- worker id that owns the row claimed_at timestamptz, attempts int not null default 0, created_at timestamptz not null default now(), updated_at timestamptz not null default now() ); -- Index the hot path: find claimable rows for a given mode, oldest first. create index task_queue_claimable_idx on task_queue (mode, created_at) where status = 'queued'; ``` --- ## 2. The atomic claim (the part that makes parallelism safe) A claim is ONE transaction that selects the next free row and marks it owned in the same breath. `FOR UPDATE SKIP LOCKED` means a second worker reaching for the same row simply skips to the next free one. No external lock manager. No coordination chatter. The database arbitrates. ```sql -- Claim up to :batch rows for one worker, atomically. with next as ( select id from task_queue where status = 'queued' and mode = :mode order by created_at for update skip locked limit :batch ) update task_queue t set status = 'in_progress', claimed_by = :worker_id, claimed_at = now(), attempts = t.attempts + 1, updated_at = now() from next where t.id = next.id returning t.*; ``` --- ## 3. The worker (almost insultingly simple) Loop: claim, do, mark done. Exit the instant it hits its task cap, its clock, or an empty queue. When it dies, it dies clean. ```python def worker(mode, worker_id, task_cap, wall_clock_minutes): deadline = now() + minutes(wall_clock_minutes) handled = 0 while handled < task_cap and now() < deadline: rows = claim_atomic(mode=mode, worker_id=worker_id, batch=1) # SQL above if not rows: break # empty queue: nothing left to do, exit clean task = rows[0] try: result = do_the_work(task) # your actual labor mark(task.id, status="done", result=result) except Exception as e: # Fail the row but keep the worker alive for the next item. mark(task.id, status="failed", error=str(e)) handled += 1 return {"mode": mode, "worker_id": worker_id, "handled": handled} ``` --- ## 4. The orchestrator fire (the whole loop) This is the heart of the pattern. Read it top to bottom: hygiene, decide, fan out, barrier, synthesize. The ordering is load-bearing. ```python def orchestrator_fire(): # --- HYGIENE FIRST (before any fan-out) --------------------------------- # Agents crash mid-task and strand rows: owned by a ghost, never finished, # invisible to the atomic claim because they look taken. Reset them BEFORE # fanning out, so fresh workers inherit the resurrected work. release_stale_claims(in_progress_older_than_hours=6) reap_stranded_runs() # kill ghost processes still holding locks # --- DECIDE (the expensive judgment) ------------------------------------ # This is why this tier runs a stronger model. A dumb cron can recompute a # score on a timer, but it cannot look at the backlog and decide which few # things are worth doing right now. state = read_state(inbox, status_table, escalations) plan = prioritize(queue_snapshot(), state) # plan.modes_to_drain -> list of modes to work this fire # plan.parallelism[mode] -> how many workers per mode # plan sizing is bounded by MAX_POOL below. # --- FAN OUT disposable workers ----------------------------------------- pool = [] for mode in plan.modes_to_drain: for _ in range(plan.parallelism[mode]): if len(pool) >= MAX_POOL: # per-fire concurrency interlock break pool.append(spawn_worker( mode = mode, worker_id = new_worker_id(), task_cap = TASK_CAP, # at most N items, then exit wall_clock_minutes = WALL_CLOCK_MIN, # die after M minutes )) # --- BARRIER ------------------------------------------------------------ # Wait for the whole pool. Collect what each worker did. results = await_all(pool) # --- SYNTHESIZE ONE NOTE UPWARD ----------------------------------------- # Not forty worker logs. One brief: this many drained, these anomalies, the # one thing that needs a human. The orchestrator reads all N logs so the # human does not have to. update(status_table, results) write_rollup(inbox_up, summarize(results)) ``` --- ## 5. Hygiene helpers (recovery must be automatic) The governing principle: assume any agent can die at any moment, and make recovery automatic. Stranded work that needs a human to notice and requeue is, by definition, not a system that runs itself. ```sql -- Release stale claims: anything in_progress past a sane timeout with no sign -- of life goes back to claimable. update task_queue set status = 'queued', claimed_by = null, claimed_at = null, updated_at = now() where status = 'in_progress' and claimed_at < now() - interval '6 hours'; ``` Run the same sweep on a cheap deterministic cron BETWEEN fires too. Never assume the next orchestrator fire is close. --- ## 6. The limits (build these first, never last) Fan-out without limits is a denial-of-service attack on your own budget and your upstreams. These caps are the safety interlock. ```python TASK_CAP = 25 # max items one worker handles before it exits WALL_CLOCK_MIN = 15 # max minutes one worker runs before it dies MAX_POOL = 12 # max workers alive in a single fire DAILY_FIRE_CAP = 6 # max orchestrator fires per day (stops double-fire stampede) ``` Why each one exists: - TASK_CAP bounds the blast radius of one worker and forces the queue to be re-evaluated by the next fire instead of chewed through by a runaway process. - WALL_CLOCK_MIN kills a worker stuck on one pathological item before it runs away with your time and budget. - MAX_POOL bounds the whole fan-out so a bad prioritization cannot launch a swarm. - DAILY_FIRE_CAP stops a catch-up double-fire from launching two full fan-outs back to back. --- ## 7. Guard the one heartbeat This pattern is a single point of failure by design. The whole appeal (one heartbeat to watch) is also the whole risk: that one heartbeat IS the system now. A swarm of independent operators degrades gracefully when one dies. An orchestrated fleet stops cold if its one decider does. So the mitigation is blunt: treat the orchestrator's own firing as the single most important thing you monitor. Run a dedicated watcher whose only job is to scream if a scheduled fire did not happen. ```python def fire_watchdog(): last = last_successful_fire_time() if now() - last > expected_interval() + grace_period(): alert_human("Orchestrator did not fire. The fleet is dark.") ``` The narrower the critical surface, the more brutally you guard the one thing left. --- ## What to fold in, what to keep standing - FOLD INTO THE FAN-OUT: high-volume worker labor. Anything that is "drain a queue of routine items" belongs in the disposable pool. - KEEP ON ITS OWN SCHEDULE: anything that needs an independent cadence or its own resources. Supervisors that queue and roll up a lane keep their timers. Checks that need an environment the pool cannot borrow keep theirs. Cheap deterministic crons (recompute, promote, refresh, prune) keep humming underneath; they were never agents and should not be. Rule of thumb: fold the labor, keep the cadence-critical supervisors standing. --- ## Checklist before you ship this - [ ] Queue table has an atomic claim path (skip-locked), not app-level locking. - [ ] Hygiene (release stale, reap stranded) runs at the TOP of every fire. - [ ] A between-fires cron also sweeps stranded rows. - [ ] Every worker has a task cap AND a wall-clock limit. - [ ] The fan-out has a per-fire pool cap and the day has a fire cap. - [ ] The fire ends at a barrier that writes exactly ONE note upward. - [ ] A dedicated watcher alerts if a scheduled fire did not happen.