One decider, a pool of throwaway workers, no always-on swarm

I run a niche vertical-data product almost entirely on AI agents. The pattern that now does most of the actual labor isn't a clever model. It's a single decider that wakes a few times a day, picks what matters, and then spawns a pool of throwaway workers to drain the backlog. This is how that runtime works, and why it beat a fleet of always-on operators.

For a long time my system ran on the obvious design: one scheduled agent per job. A build worker on its own cron, a data worker on another, a research worker, an auditor, a content worker. Each a standalone process polling a shared queue on its own timer. Tidy in the way an org chart is tidy, and a disaster in the way a building full of unsupervised contractors is a disaster.

I tore most of it out for one runtime pattern that now drives the bulk of the fleet's labor: the orchestrator. A single agent that fires a handful of times a day, decides what matters right now, and then fans out a pool of ephemeral sub-agents to drain a shared task queue. The workers are disposable: born with a narrow mandate, a hard task cap, and a wall-clock timer, dead the moment any of those is hit. The orchestrator collects what they did, updates state, and writes one rolled-up note upward, so I still see a single brief, not forty worker logs.

This is the post worth reading if you've got a swarm of scheduled agents and a growing dread of which one broke today.

The evolution: from many heartbeats to a few

The first architecture had a heartbeat per job. Every domain had a long-lived scheduled operator, and most had two: a worker and the auditor that checks it. (The worker/auditor split is its own doctrine, covered in the org-chart piece; here it just means more schedules.) Call it a couple dozen recurring agent runs, each on its own timer. Three things made that untenable.

Idle burn. A scheduled agent wakes whether or not there's work, and every wake-up is a full context load (read the operating rules, the project state, the queue) even when the queue is empty. Multiply by a couple dozen timers and you're paying a standing tax for the privilege of mostly idling.

Collisions. Two operators with overlapping schedules wake together and both grab the same task. You've paid twice and possibly written conflicting results. You can paper over it with locking, but the deeper problem is many independent processes reaching into one queue with no coordination.

Drift you can't see. Every schedule can silently break. With two dozen timers, something is always quietly dark: a cron that stopped firing, an operator crashing on boot for a week. The monitoring surface is enormous, and the failure mode is absence, the hardest thing to alert on.

So I consolidated. The standalone per-domain operators got retired or folded into one orchestrator that fires a few times a day and, on each fire, decides, then drains. Consolidation won for a blunt reason: it collapsed "which of my two dozen schedules broke?" into "did the orchestrator fire?" One heartbeat instead of dozens: fewer moving parts, less idle burn, and decisions and execution finally in the same place.

The orchestrator wears two hats

The orchestrator does two genuinely different jobs on every fire.

Hat one: the decider. It reads the inbox, the status table, the open escalations, and the queue, and sets priorities: the call about what matters right now. Customer outputs stale and due for refresh? Drain that first. An auditor flagged an anomaly overnight? It jumps the line. The rest is routine and waits its turn. A scheduled SQL function can recompute a score on a timer, but it can't look at forty candidate tasks and decide which three are worth doing before lunch. That judgment is the entire reason this tier runs a stronger, more expensive model.

Hat two: the dispatcher. Having decided, the same agent spawns the workers that do the labor. It doesn't do the grunt work. It allocates it, launching a pool of cheaper sub-agents against the queue and letting them grind.

Keeping both hats on one head is the point: the decider has the freshest read on state at the exact moment it dispatches, so it sizes the fan-out to the actual backlog. Split them across two agents and you reintroduce the coordination problem you just deleted.

Disposable sub-agents, with caps and a clock

The workers are deliberately cheap and deliberately mortal. Each spawned drainer gets:

• A narrow mandate. One mode, one slice of the queue. A drainer doesn't roam.
• A task cap. Handle at most N items, then stop. This bounds the blast radius of one worker and forces the queue to be re-evaluated by the next fire rather than chewed through by a single runaway process.
• A wall-clock limit. Die after M minutes no matter what. A worker stuck on one pathological item can never run away with your budget or your time.

Several run in parallel against the same queue. What keeps them from colliding is atomic claiming: a claim is a single transaction that selects the next available row and marks it owned, skipping any row another worker already locked. (In Postgres this is the classic FOR UPDATE SKIP LOCKED pattern.) Two workers reaching for the same task is simply impossible; the loser skips to the next free row. No external lock manager, no coordination chatter. The database arbitrates.

Here's the shape of a fire, generalized and stripped of anything specific:

on orchestrator_fire():
    # --- hygiene first ---
    release_stale_claims(in_progress_older_than = 6h)   # un-strand abandoned work
    reap_stranded_runs()                                 # kill ghosts holding locks

    # --- decide ---
    state = read(inbox, status_table, escalations)
    plan  = prioritize(queue, state)                     # CEO-level call

    # --- fan out disposable workers ---
    pool = []
    for mode in plan.modes_to_drain:
        for _ in range(plan.parallelism[mode]):
            pool.append(spawn_worker(
                mode        = mode,
                task_cap    = N,      # at most N items, then exit
                wall_clock  = M_min,  # die after M minutes
                claim       = "atomic skip-locked",
            ))

    # --- barrier ---
    results = await_all(pool)                            # collect what they did

    # --- synthesize one note upward ---
    update(status_table, results)
    write_rollup(inbox_up, summarize(results))           # human sees ONE brief

A worker, in turn, is almost insultingly simple: loop claim, do, mark done, and exit the instant it hits its task cap, its clock, or an empty queue. When it dies, it dies clean. The full loop, atomic claims and caps included, is written up as an orchestrator loop skeleton on the downloads page.

Hygiene before the fan-out

The first thing the orchestrator does on every fire is not the work. It's cleanup, and the ordering is load-bearing.

Agents crash. A worker claims a row, sets it in_progress, and then its host falls over or its session is killed mid-task. Now that row is stranded: owned by a ghost, never to be finished, invisible to the atomic claim because it looks taken. Fan out before cleaning up and your fresh workers skip right past a pile of dead-but-claimed work, and it rots forever.

So the orchestrator opens every fire by releasing stale claims (anything in_progress past a sane timeout with no sign of life gets reset to claimable) and reaping stranded runs that are holding locks. Only then does it fan out, so the new pool inherits the resurrected work. Dumb database crons also sweep for stranded rows between fires, because you should never assume the next fire is close.

The governing principle: assume any agent can die at any moment, and make recovery automatic. Stranded work that needs a human to notice and requeue is, by definition, not a system that runs itself.

The barrier and the single note

The fan-out is only half the value; the other half is the synthesis at the end.

When the workers finish (or time out), the orchestrator hits a barrier: it waits for the pool, collects what each one did, reconciles against state, and writes one rolled-up note upward. Not forty worker logs. One summary: this many items drained in this mode, these anomalies surfaced, this is the one thing that needs a human decision.

This matters because the whole point of a single decider is to preserve a single throat to the human: the fleet escalates through one spine to one brief a day. If the disposable workers each reported independently, you'd shatter that the moment you introduced them. Barrier-and-synthesis is what lets you fan out to N parallel workers without fanning out to N notifications. The orchestrator reads all N logs so I don't have to.

What stays scheduled, what gets folded in

Consolidation is not "delete every schedule." The cut is by role, not reflex.

Folds into the fan-out: the high-volume worker labor. Anything that's "drain a queue of routine items" belongs in the disposable pool. The jobs that were burning idle cycles as standalone operators are exactly what a capped, parallel fan-out drains efficiently.

Stays on its own schedule: anything that needs a genuinely independent cadence or its own resources. The managers that queue and roll up a lane keep their timers. The auditors (especially the ones that drive a real browser session to walk the live product) keep theirs, because they need an environment the fan-out can't borrow. The cheap deterministic database crons (recompute, promote, refresh, prune) keep humming underneath all of it; they were never agents and shouldn't be. The rule of thumb: fold the labor, keep the cadence-critical supervisors standing.

The tradeoffs, honestly

This pattern is not free, and two of its costs are real enough to name up front.

It's a single point of failure. If the orchestrator doesn't fire, nothing drains. The whole appeal (one heartbeat to watch) is also the whole risk: that one heartbeat is the system now. A swarm of independent operators degrades gracefully when one dies; an orchestrated fleet stops cold if its one decider does. The mitigation is to treat the orchestrator's own firing as the single most important thing you monitor, with a dedicated watcher whose only job is to scream if a scheduled fire didn't happen. The narrower the critical surface, the more brutally you guard the one thing left.

It can stampede. Fan-out without limits is a denial-of-service attack on your own budget and upstreams. A bad prioritization, a duplicate fire, or a worker that ignores its bounds, and you've got a swarm hammering the queue and the model API at once. The caps aren't polish. They're the safety interlock. Task caps bound each worker; wall-clock limits kill the stuck ones; a per-fire concurrency limit bounds the whole pool; and the orchestrator needs a sane daily cap so a catch-up double-fire can't launch two full fan-outs back to back. Build the limits first, then the fan-out. Never the other way around.

Takeaways

• Consolidate many always-on operators into a few orchestrator fires. Fewer schedules, less idle burn, and one heartbeat to monitor instead of dozens.
• Let one agent decide and dispatch. The decider has the freshest state at the moment it fans out work; keep both hats on one head.
• Make workers disposable and bounded. A narrow mandate, a task cap, and a wall-clock limit so a stuck worker can never run away. Claim atomically (skip-locked) so parallel drainers never collide.
• Clean up before you fan out. Release stale claims and reap stranded runs first, so fresh workers inherit a true picture of the queue. Then synthesize one note upward so the human sees a single brief.
• Respect the costs. Orchestration is a single point of failure: guard the fire with a dedicated watcher. And it stampedes without caps: build the limits before the fan-out, not after.