Distributed task execution (@vzn/vx-dte), 2026-10

One run spread over workers that the user’s own CI spawns. The workers reach it through a relay that anyone can host. This page lists every decision VZN has made. Each one is marked shipped or planned, and this page changes in the same PR as the item it describes.

Workers

• Shipped: a worker is described by the machine it reports (cores, memory). max and adapter are optional, and ids look like small#1.
• Shipped: the orchestrator runs no tasks (local removed 2026-10-22). --parallel affects only the orchestrator.
• Shipped (2026-10-23, replacing the busy width and the retire rule): the run’s work is only what missed the cache, and what waits on a cached miss (missedWork); groups and hits need no worker. At each miss, finish, join and history read the planner (vx-dte/src/plan/) plays that work out, longest tail first, packed by the orchestrator’s own rule, on the workers online and starting and on N more joining after the warm-up, and starts the fewest N that end within 5% (5 s at least) of the best. Main 2163fff asked 6 workers for 109 tasks of which 85 hit; replayed, it asks 2 and ends 4 s after 6 would (tests/planner.test.ts).
• Shipped: a worker the plan first gives work later than its warm-up starts when that nears (againInMs): a graph that widens late starts its workers late (2026-10-23).
• Shipped: a worker that never joins is marked failed <reason>. When none can start, the run fails and says why.
• Shipped: an idle worker gets exit and leaves (left in the footer) only when the plan still ends in its margin with one started in its place after the warm-up: needed again sooner, it stays, so none is let go and started again (2026-10-23). The same signal again changes nothing (Orchestrator.release).

Packing and sizing (history only)

• Shipped: a worker fills itself by comparing its cores and memory with each task’s measured need: the cores it kept busy on average, unrounded (a quarter at least), and its peak memory. A task with no history gets a whole worker.
• Shipped: a task that reached at least 90% of its worker’s cores on average, or of its memory, moves up one size on the next run. It keeps moving up until it fits or no bigger size exists.
• There is no exec.resources: history is the only source.
• History keeps the placement label, the worker’s cores and memory, and the peak and average cores and memory. The orchestrator writes them (schema v44).
• Shipped: core seams only. vx-history carries each task’s measured need in the default branch’s timings blob and hands it to core (importTimings, schema v45); core’s resourcesFor answers it for a task never run on this machine, and vx-dte reads it from core. No plugin imports another.
• Shipped: vx-dte reads the needs again once the import lands: core’s ctx.onRefine calls it when another plugin refines (the schedule plugin’s fetched timings), and the tasks not yet placed pack by them.
• Shipped (2026-11-07): the waiting tasks also rank and book again then (Orchestrator.rerank). A rank was read once, as a task joined the queue: on CI the roots joined before the timings landed, each ranked an unknown task’s 30 s and booked unmeasured, and the 46 shards ready later ranked by history and went first; test.wire (73 s) began 129 s in (main 1c91c9d). The sim’s historyAt plays that: a CI-shaped run ended 165.5 s before, 149.8 s after, as with history from the start.

Heartbeat

• Shipped (2026-11-03): a side quiet 6 s (3 heartbeats) is lost, and the rest derives from it. A worker that said nothing for half of it (3 s) sends its report (live load, the tasks it runs); every message it sends carries its load, and what it has at one moment goes as one. The orchestrator speaks on no timer: orders, and the acks of what it heard. Nobody probes.
• Shipped: no lease. A task sent runs at once (no start); one of a batch the worker’s report says it read that it no longer names goes back to ready at once; a lost worker’s go back when it is lost.
• Shipped (2026-11-03): ran and end carry when the task began, its command ended and it saved, by the worker’s clock, read against the frame’s own time. ran is never sent alone: it rides the next frame, or the task’s end says it.

Retry on worker faults

• Shipped: a worker fault means the relay lost the worker (it left, died or froze) or its adapter saw its process or job end, its report no longer named the task, or it could not run the task (a full disk, a failed unpack, an upload no remote took after three tries: it sends fault). A task it began runs again on another worker, and the faulty worker gets no more work. Such a retry does not count against the task’s retries.
• Shipped: a lost worker frees its place under its size’s max, so the run starts another for what is left (a planned worker’s job is spent and is not started again). With none online, none starting and no share here, the run fails rather than wait: small#1 was lost and no worker is left. Workers that join on their own (no adapter) may still join, so such a run waits.
• Shipped: a run its cache answers whole starts no worker: workers start at the first task that reaches the executor.
• The limit is 2 workers. After faults on two workers that began it, the task fails and names both: t1: failed on 2 workers: small#1 lost, small#2 ENOSPC.
• A batch that could not be sent sends its tasks back to ready and does not count.
• Shipped (2026-10-23): no prefetch and no worker queue. Each frame a worker sends carries its load (cores busy over the last second, memory in use); a task is sent only when it fits that load and what was sent since, and begins at once. ran says a command ended, so its cores are the next task’s while its outputs save (M: “we orchestrate, not plan and execute”).
• Shipped: a failed checkout or install (git worktree add, prepare) fails that worker. The first one in a run frees its place, so another worker starts (a network blip); after a second, none more are started. The run fails when no worker is left: no worker could start: small#2 prepare exited 4: ….
• A non-zero exit is a real failure, where only the task’s retries apply.

Lost relay

• Shipped: the run fails, with no local fallback, and finished tasks keep their cache. A hung orchestrator is left to the CI timeout.

Relay

• No framework: one fetch-handler core over a small storage interface. Its one dependency, jose, checks the CI tokens (2026-10-22: proven libs over hand-rolled crypto).
• Shipped: two thin entries. bun.ts keeps state in memory. worker.ts holds each run in one Cloudflare Durable Object (wrangler.jsonc, deployed by wrangler: the Deploy to Cloudflare button parses no other config).
• Shipped (2026-10-22, owner): a run id is a channel the orchestrator claims. Its join names a P-256 key made for the run; the relay keeps it and takes each later request as the orchestrator only signed by it (once, within two minutes), so no other side acts as it, whatever adapter started it. The orchestrator checks each worker the moment the relay says it joined, before or after the claim, and removes one without the run’s key; the worker’s socket brings why, then closes.
• Shipped (2026-10-22, owner: “stores nothing, knows nothing, only relays”): a run is a hash of its id and claims, no message is stored (a restart loses a participant with a full queue), a worker’s info goes once told, and a closed run keeps only its hash a day. Next, after Tailscale’s DERP: per-participant rate limits and size padding.
• Shipped (2026-10-22): websockets on the Durable Object, hibernated between messages. The run is in the object’s storage, the runtime answers ping without waking it, and an alarm at the earliest participant’s three-heartbeat deadline, not a timer, finds the silent.
• Shipped: the package directory is the template of a Deploy to Cloudflare button. The user fills in only ALLOW, entries issuer:owner/repo:workflow: a repository per issuer, a workflow per repository (2026-10-22). The audience defaults to the URL the worker is reached at, which is unknown until the first deploy.
• Shipped (2026-10-22): the Bun relay offers websockets too, since Bun.serve has them for free.
• Shipped (2026-10-23, owner: “DTE should be clear light fast easy. relay as well”): the client is join over HTTP and a socket after it, nothing else: no polling fallback. A dropped socket is dialed again within half a heartbeat; one the relay lost or let go is let in and closed with code 4404, which a failed upgrade cannot say. The relay then dropped its HTTP send, poll, leave and remove: join and socket are its only requests.

Relay encryption

• Shipped: the Noise Protocol Framework (revision 34) on WebCrypto, no dependency, held to cacophony’s test vectors. One handshake per worker, the worker the initiator, prologue vx-dte\0<run>. With the run’s key: Noise_NNpsk0_25519_AESGCM_SHA256 (-> psk, e / <- e, ee), psk = SHA-256(key); the worker’s join info is message 1’s payload. Each body after is Noise transport messages naming their nonce (Noise §11.4, the relay may reorder): a changed body loses its worker, a replayed nonce is dropped. Ephemeral keys only (forward secrecy). VX_DTE_SECRET, when set, wins; else an adapter’s workers get one made per run (local(), docker()); with no adapter it is required. The full task env travels, encrypted.
• Shipped: GitHub, keyless (githubProof): Noise_NN_25519_AESGCM_SHA256, bound by OIDC tokens. The worker’s audience is vx-dte:<sha256 of join, run, message 1>, the orchestrator’s vx-dte:<sha256 of hello, run, handshake hash>; each side checks the other’s as GitHub’s and of the same repository_id. An adapter marks its workers keyless.
• Shipped (2026-10-28): a run’s id is a dispatch input, which anyone who reads the repository’s Actions sees, so a job of the same repository (a branch’s own dispatch) joined main’s run and got its task env and cache write. github() names the workflow runs it dispatched (workerRuns), and the orchestrator admits a worker only by a token of one (run_id). A dispatch that names no run (an API answering 204) leaves the run bound to its repository, as before.
• Shipped: the relay binds a run to its repository (repository_id, project_id), not its pipeline: workers run in a GitLab pipeline the run triggers or a dispatched GitHub workflow. The run’s own key admits workers.
• Shipped (2026-10-24): GitLab: the run makes its key and passes it as a trigger variable. A GitLab ID token’s audience is fixed when its job starts, so it admits a job to the relay but cannot bind the handshake.

Files

• Shipped: no file travels through the relay; it carries only orders and status. Every task’s files and output go through the workspace’s cache.
• Shipped (core seam): ExecutorContext.artifacts and CommandContext.artifacts (TaskArtifacts). save(key, task, run) keeps the files a task’s outputs match and its output as entry key, here and on each remote the workspace writes, masking the task env’s secrets. restore(key, task) lays them into the workspace from the local store or the first remote holding it, and resolves its output, both streams in the order written. A key restored and then saved is not uploaded again. A verb writes the remote under a run’s rules: on CI, or when the workspace’s cache.policy names it.
• Shipped: an order carries the task’s key and its dependencies’ keys. The order names every dependency; the worker restores each key once (tasks begun together wait on that one restore, and a task that clears its files makes it restore again), runs the task, and saves under the task’s key, or under a key of the run and task for an uncached or failed task, so no hit ever serves it. end carries the key and the result head (exit, time, usage, violations); the orchestrator restores by the key.
• Shipped: a worker reads the cache as the run does (2026-10-21 17:33-17:35). welcome carries CacheFacts: the run’s refs, so the worker works out the same read scope, and the access each remote lends (RemoteCacheLayer.lend, an OIDC exchange’s token, never a stored secret). The server’s token decides where it may write. A cache message lends again before the first token ends. A worker’s plugin uses a lent token over its own OIDC exchange, since the server files a write by the token (a worker’s own filed tasks where the run could not read them, 2026-10-22); only a token of its own set by hand wins. Each task gets only the env its config passes, not the worker’s whole env.

Display

Core renders, and plugins hand it data.

• Shipped: a task’s id carries its worker, app#build@small#1, the worker colored per worker. Local tasks get no suffix.
• Shipped: the running list shows workers. Plugin blocks, which can span several lines, render both live and in the summary.
• Shipped: a task waiting for a worker has no row and counts as scheduled on the tasks line; its row and clock start when it is sent, which is when its worker begins it. Worker cores and memory show only in the final summary.
• Decided (2026-10-22): no live log streaming. A task’s log replays in the orchestrator when the task ends: a CI orchestrator has no interactive terminal, so lines of tasks on many workers would interleave.
• Shipped: vx last shows where each task ran, and telemetry plugins get placement and worker stats.

Adapters

• Shipped: local() runs workers in git worktrees. Use prepare: 'bun install' when a worktree needs dependencies. The worktrees are removed by a detached shell after vx exits, so a stop is not held up by them (a 300k-file tree took 2.7 s).
• Shipped: github({ workflow? }). One vx-worker.yml (inputs: run, relay, sha, size, and the workers as JSON; runs-on by size; a matrix over the workers) is dispatched once per size for the workers one demand starts: live dispatch is the default. Workers check out the run’s commit. They are keyless (GitHub OIDC), and the relay token defaults to an OIDC token for the relay’s URL. The relay admits workers by repo and VX_DTE_RUN.
• Shipped (2026-10-22): a run id is a random (v4) UUID, which the relay and vx check, so no one can guess one: made by the run on GitHub and GitLab (runId).
• Shipped (2026-10-24, “tokenless trigger”): gitlab({ tags? }). A worker started before its run met no live run (410), so the run starts each one only once it is on the relay: a POST to /projects/:id/trigger/pipeline with the job’s CI_JOB_TOKEN, no stored token. Variables: VX_DTE_RUN, VX_DTE_WORKER, VX_DTE_RELAY, VX_DTE_SECRET, VX_DTE_SIZE, VX_DTE_SHA. The pipeline is of the project’s one .gitlab-ci.yml (vx dte init writes it): its rules run only vx-worker <size> when VX_DTE_SIZE is set, which fetches and checks out VX_DTE_SHA. One pipeline per wave and size: the job is a parallel: matrix of max slots (VX_DTE_SLOT, 200 at most), and $VX_DTE_SLOT =~ $VX_DTE_PICK runs the wave’s (GitLab reads matrix variables in rules:if, and a regex from a variable; 18.5 broke the first, 18.6 fixed it). A slot is the worker’s number wrapped at max: a wave’s numbers are consecutive and at most max, so they differ; a worker whose slot its wave holds starts a wave of its own. VX_DTE_IDS maps slots to names (workerId). A job token can trigger but not read or cancel jobs, so the run cancels none; an extra worker leaves when it joins or the run ends. Replaced vx dte plan, the child pipeline, when: manual jobs and VX_DTE_GITLAB_TOKEN.

Config

• Shipped: every property of an exported *Options and WorkerSize has its own JSDoc description (tests/jsdoc.test.ts).