@vzn/vx-dte

Website: https://vznjs.github.io/vx/ · source: https://github.com/vznjs/vx

Distributed task execution for vx: one vx run spread over workers your own CI starts, replayed here as one run. The run you start is the orchestrator. It runs no task itself, only restores cache hits: it starts workers and hands the ready tasks to the workers with room, critical path first, ranked by history and ranked again when fetched timings land. A worker runs it in its own checkout and saves the task’s output and the files its declared outputs left to the workspace’s cache, under the task’s key. They land here before the task ends, so the run reports the task as its own. No file travels through the relay: it carries only orders and status, so the workers and this machine need one cache they all read (a remote the workers write, or one shared store). A worker reads that cache as the run does: under the run’s refs, with the short-lived access the run’s remote lends it (an OIDC exchange’s token, never a stored secret) when it has none of its own.

Workers and the orchestrator meet at a relay (@vzn/vx-dte-server), since CI runners take no inbound connection. It speaks the Switchboard protocol: the run is a session, its id the run id, the orchestrator its host and each worker a peer. Host your own: its README has a Deploy to Cloudflare button, which asks only for the repositories whose runs it carries.

A run id is a channel. Workers join it before or after the orchestrator does, and the orchestrator claims it with a key made for that run; the relay keeps the public half and passes nothing to or from a side that cannot sign by it. The orchestrator checks each worker the moment it appears and removes any without the run’s key from the channel.

The relay reads nothing they say. Each worker and the orchestrator run a Noise handshake through it, one session per worker, with keys made for that session only. With the run’s key (VX_DTE_SECRET, or one an adapter made for its workers) it is Noise_NNpsk0_25519_AESGCM_SHA256: the key is the pre-shared key, so a side without it cannot finish the handshake, and the worker’s join info (cores, memory, tree) is already encrypted. On GitHub Actions with no key it is Noise_NN_25519_AESGCM_SHA256, and each side binds the handshake by an OIDC token of this repository (id-token: write): no secret is stored anywhere. With no adapter and no GitHub, set the same VX_DTE_SECRET on the run and each worker. A message the relay changes loses its worker; one it replays is dropped. The key never encrypts a message, so a leaked one opens no past run.

It runs workers with local(), docker(), github() and gitlab().

Bodies

What vx-dte puts in Switchboard’s info and body, for anyone writing a worker or orchestrator of their own. Each worker and the orchestrator run a Noise handshake through the relay, prologue vx-dte\0<run>, the worker the initiator.

• With the run’s key: Noise_NNpsk0_25519_AESGCM_SHA256, the psk the key’s SHA-256.
• Keyless on GitHub Actions: Noise_NN_25519_AESGCM_SHA256. The worker’s join carries an OIDC token whose audience is vx-dte:<base64url SHA-256 of "join\0<run>\0<message 1>">; the orchestrator’s hello one for "hello\0<run>\0<handshake hash>". Each side checks the other’s as a token of the same repository_id; with github(), the orchestrator also checks a worker’s run_id is a workflow run it dispatched, since the run’s id is a dispatch input anyone who reads the repository’s Actions sees. A dispatch GitHub answers without a run id (204) is refused, so its workers exit.

Who	info / body
worker join	info: { "noise": <message 1>, "token"? }
orchestrator	{ "type": "hello", "noise": <message 2>, "token"? }
orchestrator	{ "type": "refused", "reason" }, plain, then remove
either, then	["<n>.<ciphertext>", …], or ["+", …] with more to come

The orchestrator checks each worker the moment the relay says it joined, and removes one whose handshake fails. Message 1’s payload is the worker’s info (JSON), encrypted under the psk. After the handshake every body is Noise transport messages, base64, each at most 65535 bytes and naming its nonce n (Noise §11.4, since a relay may reorder); each is sealed with associated data <first n>.<parts> of its body, so no part opens in another body, and a body’s parts take consecutive nonces, so none is repeated or reordered in it. A body of more than 5 parts goes over several sends of 5, each but the last led by "+", so no frame passes the relay’s 1 MiB; the receiver holds them until the last and opens the body whole. A body whose first nonce was seen before is a replay and is dropped; one older than a body already opened (the relay held it back: the wire’s numbers are outside the seal) or that does not open loses its sender. The plaintext is a lead byte, the length of every body its sender sealed before (4 bytes, big-endian, mod 2³²), and then, 1, its JSON, an array of the messages sent together (up to 64 KiB), or 0, when smaller so, its zstd frame’s length (4 bytes) and the frame: compressed before sealing, since no relay or socket compresses what is sealed. The orchestrator’s are compressed against the last 16 KiB of its bodies’ JSON to that worker as their dictionary, so a batch’s env and paths said before cost a few bytes; a worker’s go without one, which would cost the orchestrator some 100 µs to open each. A body whose length is not the receiver’s own count was sealed after one it never opened, which the relay kept back, and loses its sender. It is padded with trailing spaces to its size class, a power of two from 64 bytes, then whole transport messages, so the relay learns no size within a class. Inside, each send is numbered and acknowledged end to end ({ "n", "b", "a"? }: its number, body and ack), and a quiet worker reports (Switchboard § Delivery, § Liveness).

Try it

bun add -d @vzn/vx-dte

.vx/workspace.ts

import { defineWorkspace } from '@vzn/vx/config'
import { dte, local } from '@vzn/vx-dte'

export default defineWorkspace({
  plugins: [
    dte({
      workers: { linux: { machines: { small: { max: 4, cpu: 2, memory: '8GB' } } } },
      adapter: local({ prepare: 'bun install' }),
    }),
  ],
})

VX_DTE_RUN=$(uuidgen) vx run build --all

VX_DTE_RUN names the run and turns distribution on; without it, vx run runs here as always (on GitHub Actions, github() with a relay names one itself). It is a random UUID, so no one can guess a run to open it first or join it; vx and the relay refuse any other. local() starts each worker as a process in its own git worktree of the tracked files as they are, uncommitted changes included, and a file mid-merge or flagged assume-unchanged or skip-worktree as it is on disk (untracked files are not: a task that keys one fails until it is git added), at the workspace’s place in it (a workspace in a subdirectory of its repository too), saving to the run’s store (a relative cache.dir or --cache-dir too). A run from a git hook works too: the worktree and its worker drop the hook’s repository variables (git rev-parse --local-env-vars), so the index being committed is left as it is, and a linked worktree’s hook GIT_DIR is dropped where git finds that repository anyway, as core does, so a workspace in a subdirectory is found there. When a worker fails, or cannot start, its row names the error line and the log that holds its whole stderr and all prepare printed (the error line alone when the log cannot be written). At the run’s end, or on Ctrl-C, vx exits without waiting for the worktrees to be removed: a detached shell waits up to 5 s for the workers to leave, kills any still running, removes the worktrees and prunes git’s records. A run whose vx was killed (kill -9) removes none; the next local() run on the machine removes them, once their run’s process is gone.

docker() starts each worker as a container held to its size: --cpus and --memory come from cpu and memory. The container clones this checkout (mounted read-only) into its own filesystem at its tracked files as they are, uncommitted changes too, runs prepare, then the workspace’s own node_modules/.bin/vx dte worker, so the image needs Bun and git, and prepare installs vx. This machine’s store (~/.vx) and a cache dir the workspace names (cache.dir, VX_CACHE_DIR, --cache-dir) are mounted at their own paths, and the clone keeps this checkout’s origin, so the workers save where the run reads, as local()’s do; no remote is needed. On Linux a container runs as this user (its home is /tmp, where the store is mounted), so what it writes is this user’s and git trusts the mounted checkout. A container that fails names its error line and the log of its whole stderr and all prepare printed:

dte({
  workers: { linux: { machines: { small: { max: 4, cpu: 2, memory: '4GB' } } } },
  adapter: docker({ image: 'my/bun-git', prepare: 'bun install' }),
})

The relay stays on loopback: on Linux a container joins the host’s network; elsewhere it reaches the relay through host.docker.internal. A container ends with its worker; one still up when the run ends is removed.

With no adapter, vx starts no workers: the run uses whichever join its relay. In three terminals on one checkout’s commit:

.vx/workspace.ts

export default defineWorkspace({ plugins: [dte({ relay: 'http://127.0.0.1:8080' })] })

bun /path/to/vx/packages/vx-dte-server/src/bun.ts --no-auth --port 8080   # the relay
export VX_DTE_SECRET=$(openssl rand -base64 32)                          # in each terminal: the same one
export VX_DTE_RUN=$(uuidgen)                                             # in each terminal: the same one
vx run build --all                                                       # the run, first
vx dte worker                                                             # a worker, as many as you like

A worker started before its run finds no run on the relay: it says so and exits.

A worker prints its name, the run, the relay and its machine, then a row as each task starts and ends:

─ vx dte worker ───────────────────────────────────────────────
  worker     worker#1
  run       5f0c7a52-…
  relay     http://127.0.0.1:8080
  machine   10 cores · 17.2 GB
› b#build  running
✓ b#build  308ms

On GitHub

The run is your own job’s vx run. When it needs workers it dispatches .github/workflows/vx-worker.yml once per size, with the workers of that size as a matrix, each job named for its worker (name: ${{ matrix.worker }}): a job that ends badly before its worker joins marks that worker failed with the step it failed at, and the run starts another in its place. When the run ends it cancels each worker run it dispatched, so one still queued does not hold a runner the next run waits for. Every job of a repository shares its GITHUB_TOKEN‘s API limit, 1000 requests an hour, so the run reads its workers’ jobs at most every 30 s, asks with each answer’s ETag (an unchanged answer is free), and asks nothing until a spent limit resets. A dispatch the limit refuses is sent again at the reset, if that comes within 5 minutes; past that it fails, naming when. A cancel the limit refuses is sent again at the reset, if that comes within a minute, unless a worker given up as late joined meanwhile: the run took it, so its workflow run goes on. A run that dies before it cancels (a cancelled or timed-out job) leaves it to the job’s last step, which cancels each worker run the run listed in RUNNER_TEMP. Past both, a worker that starts reads its run’s attempt (parent) first, and once that is over cancels its own workflow run, its queued fellows with it. On GitHub Actions with a relay, the run names itself: no VX_DTE_RUN to set. No secret: each side proves its key with the job’s OIDC token, and the relay’s token is an OIDC token for the relay’s URL. Such a token lasts 5 minutes, so each side asks for a new one a minute before its token expires.

.vx/workspace.ts

export default defineWorkspace({
  plugins: [
    dte({
      relay: 'https://relay.example.com',
      workers: { linux: { machines: { small: { max: 8, cpu: 4, memory: '16GB' } } } },
      adapter: github(),
    }),
  ],
})

.github/workflows/ci.yml

# Written by vx dte init: the run. main and a merge queue run every task;
# a pull request runs what it changed and its dependents. This job only
# orchestrates: every task runs on a worker (vx-worker.yml).
name: ci

on:
  pull_request:
  merge_group:
  push:
    branches: [main]

# A newer push to a pull request cancels its older run, and its workers.
concurrency:
  group: ci-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  vx:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    permissions:
      contents: read
      # The relay and the workers: each side proves itself by its OIDC token.
      id-token: write
      # Dispatching and cancelling the workers.
      actions: write
    env:
      GITHUB_TOKEN: ${{ github.token }}
    steps:
      # One commit: --affected fetches its base itself.
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
      - run: bun install --frozen-lockfile --filter './'
      - run: bun vx lock --check
      # Name the tasks to run, as 'bun vx run build test --affected': with
      # none named, vx fails, listing the workspace's tasks.
      - run: bun vx run ${{ github.event_name == 'pull_request' && '--affected' || '--all' }}
      - name: Cancel the workers
        if: cancelled()
        run: |
          [ -f "$RUNNER_TEMP/vx-dte-runs" ] || exit 0
          while read -r run; do curl -sS -o /dev/null -X POST -H "authorization: Bearer $GITHUB_TOKEN" "$GITHUB_API_URL$run/cancel" || true; done < "$RUNNER_TEMP/vx-dte-runs"

vx dte init writes both from dte(), the run’s above and the workers’ below, and prints what is left: a relay that admits them (its ALLOW, and how to deploy one), a shared remote cache, the commit. In a terminal it asks what main runs and what a pull request runs (main’s by default); without one it names none, and the run fails, listing the tasks, until you do. A workspace in a subdirectory of its repository gets them at the repository’s root, its jobs running in that directory. Each size runs on its platform’s GitHub-hosted runner (ubuntu-latest for none named, ubuntu-24.04-arm, macos-latest, macos-15-intel), or on the label a machine’s runsOn: 'ubuntu-latest-16-cores' names, as gitlab({ tags }) does. It installs with the package manager your lockfile names and runs vx as bun vx, which runs the repo’s own and never downloads one, as pnpm exec does (Yarn too: Bun reads no Plug’n’Play, so a Yarn workspace sets nodeLinker: node-modules). --dry prints it; a file you changed is kept unless --force. For the sizes above:

.github/workflows/vx-worker.yml

# Written by vx dte init from dte() in .vx/workspace.ts: the workers of a
# DTE run, which github() dispatches once per size. Run vx dte init
# again after changing the sizes.
name: vx-worker

on:
  workflow_dispatch:
    inputs:
      run: { required: true }
      relay: { required: true }
      sha: { required: true }
      size: { required: true }
      workers: { required: true }
      # The variables github({ env }) hands the workers, as a JSON object.
      env: { required: false, default: '{}' }
      # The run's attempt, which a worker reads first.
      parent: { required: false, default: '' }

permissions:
  contents: read
  id-token: write

jobs:
  worker:
    # The run reads a job that failed before its worker joined by this name.
    name: ${{ matrix.worker }}
    # A job's permissions replace the workflow's, so it names all three.
    permissions:
      contents: read
      id-token: write
      # A worker of a run that is over cancels its own run.
      actions: write
    runs-on: ${{ fromJSON('{"linux.small":"ubuntu-latest"}')[inputs.size] }}
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        worker: ${{ fromJSON(inputs.workers) }}
    env:
      VX_DTE_RUN: ${{ inputs.run }}
      VX_DTE_RELAY: ${{ inputs.relay }}
      VX_DTE_WORKER: ${{ matrix.worker }}
    steps:
      - name: Leave if the run is over
        if: inputs.parent != ''
        env: { GH_TOKEN: '${{ github.token }}', PARENT: '${{ inputs.parent }}' }
        run: |
          runs="$GITHUB_API_URL/repos/$GITHUB_REPOSITORY/actions/runs"
          status=$(curl -fsS -H "authorization: Bearer $GH_TOKEN" "$runs/$PARENT" | jq -r .status) || exit 0
          [ "$status" = completed ] || exit 0
          echo "::notice::run $PARENT is over: cancelling this one"
          curl -fsS -o /dev/null -X POST -H "authorization: Bearer $GH_TOKEN" "$runs/$GITHUB_RUN_ID/cancel"
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { ref: '${{ inputs.sha }}', persist-credentials: false }
      - env: { HANDED: '${{ inputs.env }}' }
        run: |
          jq -r 'to_entries[] | if (.key | test("^[A-Za-z_][A-Za-z0-9_]*$")) and (.value | type == "string") and (.value | test("[\r\n]") | not) then "\(.key)=\(.value)" else error("env \(.key): a name, and a string of one line") end' <<<"$HANDED" >> "$GITHUB_ENV"
      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
      # Every project: only pnpm selects by a git diff, as
      # `pnpm install --frozen-lockfile --filter "...[origin/main]"`
      # (fetch origin/main first: the checkout is one commit).
      - run: bun install --frozen-lockfile
      - run: bun vx dte worker

On GitLab

One pipeline file. Your vx job is the orchestrator; once it is on the relay, it triggers a pipeline of the same file for each wave of workers it asks of a size, with the job’s own CI_JOB_TOKEN, so no worker starts before the run and no token is stored. Each size’s job is a matrix of max slots (VX_DTE_SLOT); that pipeline runs only the slots the wave names (VX_DTE_PICK), at the run’s commit, and each takes its name from VX_DTE_IDS. The run starts workers as its work needs them, up to each size’s max, which gitlab() needs (at most 200, a matrix’s limit). GitLab lets a job token trigger but not cancel, so a worker the run no longer needs leaves when it joins or when the run ends. Each trigger passes the workers’ variables, so set Settings > CI/CD > Variables > “Minimum role to use pipeline variables” to Developer: a project made since GitLab 17.7 lets no one, and refuses every trigger.

.vx/workspace.ts

export default defineWorkspace({
  plugins: [
    dte({
      relay: 'https://relay.example.com',
      workers: { linux: { machines: { small: { max: 4, cpu: 2, memory: '8GB' } } } },
      adapter: gitlab(),
    }),
  ],
})

vx dte init writes it, from dte(), in the package manager your lockfile names, asking what main and a merge request run:

.gitlab-ci.yml

# Written by vx dte init from dte() in .vx/workspace.ts. The vx job
# runs no task: once it is on the relay it triggers a pipeline of this
# file for each wave of workers of a size, which runs only those slots
# of that size's job.
# Name the tasks in the vx job, as 'build test --all': with none named,
# vx fails, listing them.
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "pipeline" && $VX_DTE_SIZE

# The image needs Bun; a job adds git if it lacks it. By digest: a tag
# can move under the same name.
default:
  image: oven/bun:1.4.4@sha256:1d22548f177f4ff38e0c15baa3c2c46c6c2060ba2d1bc573bb4720b407548f1d

vx:
  rules:
    - if: $VX_DTE_SIZE
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables: { VX_TASKS: '--affected' }
    - when: on_success
      variables: { VX_TASKS: '--all' }
  id_tokens: { VX_DTE_TOKEN: { aud: https://relay.example.com } }
  before_script:
    - command -v git || (apt-get update -qq && apt-get install -qq -y --no-install-recommends git)
    - bun install --frozen-lockfile --filter './'
  script:
    - bun vx lock --check
    - bun vx run $VX_TASKS

# A worker runs at the run's commit, which its branch may have moved past.
.vx-worker:
  variables: { GIT_DEPTH: 1 }
  id_tokens: { VX_DTE_TOKEN: { aud: https://relay.example.com } }
  before_script:
    - command -v git || (apt-get update -qq && apt-get install -qq -y --no-install-recommends git)
    - git fetch -q --depth 1 origin $VX_DTE_SHA
    - git checkout -q FETCH_HEAD
    - bun install --frozen-lockfile
  script: [bun vx dte worker]

'vx-worker linux.small':
  extends: .vx-worker
  parallel:
    matrix:
      - VX_DTE_SLOT: ['1', '2', '3', '4']
  rules: [{ if: $VX_DTE_SIZE == "linux.small" && $VX_DTE_SLOT =~ $VX_DTE_PICK }]

gitlab({ tags }) names each size’s runner tags. Each job’s ID token (VX_DTE_TOKEN) admits it to the relay; the worker proves itself to the run by the run’s secret, made for that run only and passed as a trigger variable, since a GitLab ID token’s audience is fixed when its job starts and cannot bind the handshake as GitHub’s does. Set VX_DTE_SECRET to use your own instead.

What runs where

• A worker reports the cores and memory its machine has when it joins. One the run started is held to the size it was started as, so local() workers sharing a host split it rather than each take all of it. Each frame it sends (a task saved, its report, a fault) carries its machine’s load: cores busy over the last second and memory in use. A heartbeat with nothing else to say, it sends its report: that load, the tasks it runs and the last batch it read. A task of that batch or before its report no longer names goes back to the queue at once. A task goes to it only when it fits now: while one of its cores is free and the task’s memory fits. What a worker has in use is what its running tasks used on their last run (history’s measured CPU and peak RAM), plus the cores its machine spends on no task of the run while it runs some (each frame’s load less its tasks’, averaged: a GitHub runner’s own agent and the like, 0.6 to 1.1 of a 4-core small’s), or its live load plus what was sent since when that is more. Never less: load lags a task’s start, and a test shard idle for a second then uses all its cores. A command that ended frees its cores while its outputs save, and its worker’s load until the next frame: that load counted the task. Every task sent begins at once; none waits in a worker’s queue. A task asking a whole worker, or bigger than any, runs alone on an idle one (one that filled its worker last time). A task with history goes only on a size with room for its cores and memory when some size has it, so a 4-core task never runs at half speed on a 2-core worker while 8-core sizes go unasked. Workers its work needs within a warm-up of each other start as fewer, bigger ones with the cores of them all: four 2-core tasks ready at once start one 8-core worker, so fewer machines pay a warm-up. The first such task waiting holds the worker that empties soonest: no task after it is sent there, so it is not passed over again and again. Any ready task with history that fits no worker now keeps its room where it first frees: a task of lower rank goes there only beside it, and one with no history still goes. A task given keeps room for the tasks it makes ready before the rest of the pass takes it, so a chain’s next step never waits on fillers.
• A task naming exec.platforms runs only on a worker of the platform core placed it on: this machine’s when a size has it, else the first of the task’s a size has. A worker reports its platform when it joins: a size naming one refuses a worker of another; a size naming none is planned as this machine’s and takes any worker. When the sizes run this machine’s platform and another, a task naming none runs only on this machine’s: a worker of another platform runs only the tasks that name it, so they never wait behind the rest, and leaves when they are done. The run starts a worker of each platform its work needs. A task none of whose platforms any size has refuses the run before a worker starts.
• A task naming exec.worker runs only on machines of that worker type; any other on the first type, in declared order, with a machine of one of its platforms. So a type listed last (a costly Mac) gets only the tasks no other type can take. Core refuses the run before a worker starts when no type has the name, or none of its machines runs one of the task’s platforms. A worker that joins on its own is of the first type with its platform.
• Tasks waiting for room go out longest tail first: a task’s recorded duration and its longest chain of dependants, so a long chain that got ready late does not queue behind many short tasks ready at once.
• A task goes first to the worker the last plan gave it, where it fits now, while every task left has history; then to the worker that holds most of its dependencies’ outputs, so a chain stays on one machine; then to the one with fewest cores free that it fits, so workers fill before the next takes work and the ones the rest no longer needs fall idle and leave.
• A sandboxed task runs on a worker inside the worker’s own sandbox, with the same exec.sandbox, built there (sandboxRequestFor): its paths, walls and credential stores are the worker’s, so an orchestrator on Linux can run macOS workers. A violation there fails it here, named in this checkout’s paths. An empty file it wrote over a write grant’s placeholder is its output; only a placeholder it never touched goes. A sandbox the worker cannot host (no bwrap in its image) fails the task there, as it fails here; the worker stays in the run.
• A persistent task a task depends on starts on the worker that runs the task, once per worker, ready before it, and stops there once no task left in the run uses it (the run’s unneeded), or when the worker leaves; the orchestrator never starts it, and a worker no dependant reaches never does (2026-10-22). Its CPU is learned on the worker each second as cores = base + each running dependant's cost: the base from looks with no dependant running, each cost from looks with it (a look on a full machine only raises them). Each dependant’s cost is added to its CPU, so its history books it; the base is booked once on each worker that starts the server, in placement and in the plan, and kept in history for the next run, as warm-ups are.
• A task whose key hashes a runtime probe (inputs.runtime, such as bun --version) runs on a worker, which answers the probe first and fails the task when it answers otherwise than this machine did. A path under the worker’s checkout root reads as the same path under this machine’s, as the task’s environment does.
• A task’s command runs on a worker as written: a plugin changes where a command runs, never what it is. A path under this machine’s checkout in the environment is moved to the worker’s; one written into the command is not, so a local() worker would read and write this checkout through it. Name a file relative to the task’s directory, or through $PWD.
• No task runs on the orchestrator, only cache hits restore there (2026-10-24): a task with exec.remote: false, an interactive one and their dependants run on workers too (the executor is exclusive). A task no worker can take refuses the run before any starts, naming it: one whose exec.platforms no size offers, or a persistent task asked for on its own.
• An exec.remote: 'only' task’s files stay on the workers: this machine restores its output, never its files. Under --cache-download=toplevel or none, a task’s files it defers land here only if a task here needs them.
• A task’s output replays here as its worker read it: stdout and stderr in the order the task wrote them, bounded as core bounds a run’s.
• A result too long for one relay frame (thousands of sandbox violations, or of files a formatter changed) is sent in parts, none of it dropped. An order too long for one frame (a task with thousands of deps) sends its deps ahead of it the same way.
• A worker gets each dependency in full once. Later orders name it, so a deep graph’s orders stay small.
• A task runs with the orchestrator’s environment under the worker’s root. HOME, TMPDIR, USER and the like are the worker’s own, unless the task’s env.define sets them or its inputs.env keys them: a keyed one is the orchestrator’s value, or unset where it is unset there, so the worker runs on what the key saw; npm_execpath is the workspace’s package manager on the worker, as core sets it on a local task.
• A worker restores, by key, the declared outputs of every task the one it runs depends on, each after its own dependencies (an additive task’s rewrite of an upstream file stands), unless it ran or restored them before. Files a task writes without declaring them stay on the worker. An uncached task declares none, so no cache carries its files: its dependants go to the worker it ran on, unless it is sandboxed with no write grant and so leaves no file. One waits there up to 10 s; past that, with that worker gone or of another platform, or with its uncached upstreams on two workers, the upstream runs again where the dependant can go, and the dependant follows it. One whose exec.worker or exec.platforms rules out where its dependant runs refuses the run before a worker starts, naming both: cache it, or run both alike. A task whose upload no remote took after three tries (too big, a 429, a reset connection) is its worker’s fault: that worker takes no more tasks and the task runs on another; a second fault fails the run. Before a task runs, its worker removes the files under its outputs that an earlier task left, except committed files, nested projects’ files, and its dependencies’ outputs (an additive task builds on its upstream’s tree); before laying what it left, this machine removes the same, cached task or not, so its outputs end as the worker’s did. An upstream’s file the task leaves as it found it is not saved in its entry; one it removes, or a committed file under its own outputs it removes, is removed on this machine too, unless a link on its path leads elsewhere (that file is kept, with a warning), and the task saves no entry, as core saves none.
• A hit only this machine’s cache holds (the remote evicted it, or this run does not write the remote) is uploaded to the remotes the workers read before a task that depends on it is sent; with none written, its dependant fails on the worker, naming the entry. One whose remote copy does not read back on a worker (a truncated upload) is sent again from this machine over that copy, and the task runs again; once per entry.
• A failed or uncached task is saved under a key of the run, the task and the send (a retry gets its own), so no later lookup ever hits it; so is one core would not save (a --no-cache or read-only run, a dependant of a failure under --continue=always, one that left a process running that may still write its outputs, one a stopped run cancelled), and one that left a tracked file under its project changed on its worker (a formatter’s input), or that found an untracked, unignored file under its project there (its outputs and its dependencies’ aside), which this machine never sees: core withholds its own save when the key read that file or its inputs declare it, and says so.
• Under --force, a success is saved under its key (which keeps an entry it already holds, as core does) and under the run’s key, which this machine and the dependants’ workers restore: the bytes this run built, never the entry it bypassed.
• A worker whose tracked files differ from the orchestrator’s (another tree, or the same commit with a tracked file changed, such as by a prepare step) is refused, and the orchestrator says so as it happens: [vx] dte: refused worker <id>: <reason>. The reason names a project of the run git does not track first (git add it: a worktree lacks it, so its install rewrites the lockfile), then up to three tracked files the worker changed after checkout. Both sides read their tracked files as they are on disk, a file flagged assume-unchanged or skip-worktree too (one a sparse checkout left off disk is its index’s), so the same change on both is the same tree.
• A worker runs the commit’s bytes and modes, which are the ones the key read: the run’s snapshot stages each file’s executable bit as on disk (under core.fileMode=false too, as core keys it), and a worker writes back as the commit holds it each file a git filter of its own checked out otherwise (text, eol, ident, filter such as LFS, working-tree-encoding, core.autocrlf). A task whose key hashes a tracked file that is on disk other than the commit holds it (CRLF such a filter wrote, an expanded ident, an LFS file’s content) fails, naming the file: no worker builds on those bytes.
• A task whose key hashes a file the run’s commit lacks (untracked, or gitignored and named outright, or inside a submodule, whose files the commit holds only as a gitlink) fails, naming the file: no worker has it. git add puts it in the tree local() and docker() workers check out; workers on CI check out the pushed commit, so there it must be committed. A dependency’s declared output is not one: its worker restores it first, or, uncached, ran it there. A task whose project the commit lacks (no file under it tracked) fails, naming the project, and a run whose .vx/workspace.* is untracked is refused before a worker starts.
• A task’s exec.timeout counts from its start on its worker: the wait for a worker still booting is not its run (the executor declares timeouts).
• A worker that stops answering, stops saying it holds a task, or cannot run one (a full disk) gets no more work, and its tasks run again on another. A task two such workers were sent fails, naming both; one whose send failed does not count. With no worker of the run left to take it and none starting, it fails at once.
• A task that failed and retries (its retries, or the one retry a known-flaky task gets) runs on another worker than the attempt that failed, as core asks (ExecuteRequest.attempt): the worker, or what ran beside it, may be the flake. With no other worker of its platform in the run, it runs there again rather than wait.
• A relay that answers nothing for 10 s, every try counted, fails every task not done (the relay is unreachable); none runs here, which may be far smaller than the workers. What finished is saved, so a rerun resumes from it. The footer counts each worker that had joined lost, with what it ran. A fault (a 5xx, a dropped or refused connection) is tried again 3 times, by the orchestrator and workers alike, and the third failed retry ends them (no socket in 3 dials). The waits are random, from up to 250 ms doubling to up to half a heartbeat, about 3 s at most in all on the default, so a relay down or restarting is asked through only when it is back within them: one down under a second usually is, one down 2 s usually fails the run.
• The relay stores nothing, so a restart forgets the run. The orchestrator joins again as its host and each worker joins again as itself, for two heartbeats (4 s on the default); tasks keep running meanwhile, and what either side sent goes again once both are back, taken once. A worker keeps what it runs, and one sent a task it runs already does not run it again, so no work repeats. One not back by then ends: a worker’s tasks run elsewhere, an orchestrator’s run fails. Joining has the same 10 s in all and the same 3 retries, its waits spread over the 10 s (1.25 s doubling, jittered to half: 4.4 to 8.75 s in all), so a relay or network down a few seconds is still joined. A relay silent that long fails the run with cannot connect to the relay <origin>: no answer in 10 s; four failed tries say why and how long, as cannot connect to the relay <origin>: ENOTFOUND, 4 dials in 9.1 s.
• A worker that never joins says why on its row (failed exited 1: …); its job ending, a success too, is what says so. Once the first worker of a size joins, each other of that size not joined within twice that first join is given up (did not join within 60 s) and its job stopped; before one has, none is. One its CI started anyway that joins later is taken, as any worker the plan may let go. The first three such failures in a run each start another worker in its place, as a checkout, install or join can fail on the network; a fourth starts none. While a worker of a size is past its warm-up unjoined, or given up so beside one of its size online, the run asks no more of that size: its CI queued it, and one asked then would queue behind it. When none is online or starting and this machine has no share, the remaining tasks fail with no worker could start.
• Only leave ends a participant at once (2026-10-25): the orchestrator’s at its run’s end is the final call that closes the run, and a worker’s when it is done. Either whose socket drops without it dials again, 3 times as above and within 10 s, and the relay keeps it, and its run, for 10 s; past either the run is over. Either side hearing nothing from the other for 3 heartbeats (6 s on the default) loses it, and the rest follows from that: a worker that said nothing for half of it (3 s) sends its report; the orchestrator never speaks on a timer, only orders and the acks of what it heard; a send unacknowledged for 3 s goes again. A worker that loses its orchestrator ends, and says so. Nobody probes, the relay answers neither, and nobody pings. An orchestrator with no worker yet asks the relay an empty frame instead; a frame the relay leaves unanswered a heartbeat closes the socket, which is dialed again.
• The orchestrator’s socket, its acks and redials run on a thread of their own, so sync work on the run’s loop never holds them (one CI run went quiet 12 s that way and lost every worker). A loop held a second or more says so, with how long; one held 5 heartbeats (10 s) ends the run, as a dead orchestrator would.
• Each send is numbered for its receiver and held until the receiver acknowledges it, so one a socket or the relay dropped goes again after a heartbeat and a half, or at once on a new socket; the receiver takes each once, in order. An ack rides the next send back; the orchestrator’s, with no order to ride, go together after half a heartbeat, except a quiet worker’s beat, acked at once.
• A task sent runs at once: no message says it began. A command that ended (ran, with when it began and ended by the worker’s clock) rides the worker’s next frame, never alone; a task that saves first says it in its end (begun, ran, ended). The run reads those times against the clock of the frame that carried them.
• A run where no task began or ended for a minute warns what it waits on, a line a task: which worker, and whether it runs or saves.
• A message the relay would not take is a warning naming it, its receiver and the relay’s answer; one to a participant the relay already lost (404) is not, since its loss is reported on its own. Its number still reaches the receiver, carrying nothing, so the sends after it are not held up. A worker whose OIDC token could not be checked is refused by that cause, not as a key mismatch, and the run says so.

What the run shows

Each task a worker ran says which one last on its row and after the id on its frame, in that worker’s own color, by size and the order the run started it:

 ⏺︎   308ms success miss     c#build · linux.small#1

An idle worker the rest of the run cannot keep busy leaves (left): one beyond the peak of the cores the remaining tasks keep busy, so a lull between two wide parts keeps it. One a remaining task needs for its size stays, and so does one a later task needs when its size’s max is spent, since none could start in its place. With that max spent, the last worker of its platform also stays while any task of that platform has not ended: a hit whose restore fails runs after all. A lost worker’s tasks run again on another, and the run starts one in its place. A worker is lost when its socket closes, when it says nothing for 3 heartbeats (6 s), or at once when its adapter sees its process or job end. Workers start at the first task the cache does not answer, so a warm run starts none. Once one has joined, its warm-up is what the next of its size takes at the least: work that ends sooner starts no worker while others are online or starting, and new workers are started for the cores the run keeps busy after that warm-up, not before it. Each size’s worker is timed as a task is: @vzn/vx-dte#worker:<size>, defined by the size’s name, platform, cores, memory and runner, so a size changed starts over. Its warm-up is the task’s time, at most twice what history said (more is the CI’s queue), and its machine’s cores besides the run’s tasks its cores; its join spread is @vzn/vx-dte#join-spread:<size>, and each server’s base @vzn/vx-dte#server-base:<task>. Core keeps them in the run’s history, and a history plugin (@vzn/vx-history) shares them as it does task timings, so the next run plans by them before any worker of that size joins.

The workers rows, live while the run goes and again in the footer, give each worker its status (starting, online, left, lost, or failed with why, such as exited 1: Cannot find package '@vzn/vx-dte'; once the run is over, one still starting is not joined), the tasks it has done, failed or had aborted by a stop, and the ones it runs now; while the run goes, those that left are one line (3 workers left 9 done). A worker the run started is named by its size and the order it was started in; one that joined on its own is worker#N. A task waiting for a worker has no row; the tasks line counts it scheduled (30 success · 4 running · 12 scheduled · 62 total). Its row and clock start when it is sent: a worker runs what it is sent at once. A worker is online while the run goes; it ends left when the run let it go as not needed, lost when it crashed or went silent, and stopped when the run’s end stopped it. Once the run is over, each worker adds its warm-up (from asking the adapter to its join), and under it the cores and memory its size declares and its machine reports, green when they agree and red when not:

  workers    linux.small#1  online    3 done · 2 running
            linux.small#2  starting
            local    online    1 done

  workers    linux.small#1  stopped  5 done · warm-up 0.1s
                     declared 2c/8G · actual 4c/16G
            linux.small#2  lost     1 done · warm-up 0.2s
                     declared 2c/8G · actual 4c/16G
            local    online   1 done
                     actual 10c/32G

A run over that started none (every task a hit) says workers none started: no task needed one; one where a task waited on a worker and none joined says workers none joined: a task waited on one, or, when the run refused workers, workers none joined: 2 refused (<first reason>).

Under each worker, and local, the footer then spells out what it did: its time online, busy and idle; its CPU peak and average in cores and the cores the run booked for the tasks it was running, on average (a booked from a task’s start to its command’s end); its cpu utilization from its first task’s send to its last one’s end, the warm-up and the wait for a first task left out: its machine’s (any process’s busy core-time over the core-time it had, so what is left is idle cores), its tasks’ and its worker process’s; its memory peak and average, when its tasks reported any; and, for a worker, the messages and bytes the run sent it and received from it. The probe line says when core’s cache probe answered, how many tasks it left to run (held in no cache, or not answered for) and how many of them it found in none. The asked line says each time the run asked for workers: how many, when, for how many tasks that missed, how many of them with a measured need (an unmeasured one counts as unmeasured), and when its plan ended then. The ideal line is the run’s ideal end (below), so every log reads it beside core’s time row, and the least CPU a run ending then holds: on each platform, its tasks’ cores for as long as each held its worker, on as few workers of the least costly size as end that work by then, each worker’s cores less what its machine spends besides the run’s tasks, and each worker’s warm-up. The drift line says how far past it the run ended and what caused it, along the chain of tasks that ended the run: tasks waiting for a worker to join past its platform’s fastest warm-up, waiting for room on a joined one, their inputs reaching their worker, and the run’s time after its last task. The first ask waits for the needs another plugin fetches (ctx.refining()), such as @vzn/vx-history’s default-branch timings. The unmeasured line names the tasks the first ask had no measure of, and a worker that ran two tasks or fewer names them on its row. used avg is the cores its tasks used, machine avg the cores its machine was busy on average since the worker started, any process’s, as its last frame said. Where the rest went, the worker says in its own log as it leaves: cpu over 61.2s: machine 2.80 cores · tasks 2.00 cores · this worker 0.15 cores · its other children 0.05 cores · most besides: Runner.Worker 0.30 cores, the last from one ps as it starts and one as it leaves. The total line is the same cpu utilization of every machine together, the share of their history the run’s tasks used, and the CPU the workers held: each one’s cores from when the run asked for it until it left, was cancelled or the run ended, idle or not, so it reads against the ideal line’s least:

            linux.small#1  stopped  5 done · warm-up 0.2s
                     declared 4c/14G · actual 4c/14G
                     online 12.1s · busy 9.8s · idle 2.3s
                     cpu peak 3.6 · used avg 2.4 cores · machine avg 2.9 · booked avg 3.8
                     cpu utilization machine 78% · tasks 64% · worker 2%
                     memory peak 1.2 GB · avg 640.0 MB
                     sent 6 messages · 3.1 MB, received 15 messages · 8.4 MB
            probe    at 0.4s: 5 tasks left to run, 5 known misses
            asked    1 at 0.5s for 5 tasks (5 measured), plan 12.0s
            ideal    10.8s: each task as long as it took, placed perfectly, holding 34.5 core-s at least
            drift    +1.3s past ideal: workers joining 0.8s · waiting for room 0.3s · sending inputs 0.1s
            best     2 × linux.small (4c/14G): 11.4s, the fewest of its sizes
            more     3 × linux.small (4c/14G): 10.8s, past its sizes' max
            any      1 × (8c/28G): 10.9s, of any cores, the least core time
            total    cpu utilization machine 74% · tasks 60% · worker 2% · held 52.1 core-s

vx run --profile[=<path>] adds the run in time to the profile it writes (open it in https://ui.perfetto.dev or chrome://tracing). Each worker is a process: its first row holds its warm-up (split, when it says when its vx started, into its machine’s start and vx’s boot), its idle stretches, how it ended, a cores counter of the cores booked and used at once, and a machine counter of the cores its machine reported busy, its tasks’ or not (placement never fills a worker past it), beside the cores its running tasks’ process trees used, read each second as core’s task.sample reads them; each task takes the first lane free over its time there, from its send through restoring its dependencies, its command and its save, with its exit code, duration, CPU time and booked cores. The orchestrator process holds the cache probe, each ask for workers, each plan that started or let go of workers with what it saw of each worker (one that started workers also with each task’s planned time, cores and platform and each size’s warm-up, so it can be played back; the run’s first, and one that played its work because the packing ended later or left a task out, also with all plan() was given, as input), stalls, the time each task waited for room, and each restore. Times are the orchestrator’s: when it heard. Without --profile nothing is written. On CI, upload the file as an artifact; this repo’s ci.yml does and links it from the job summary.

After the run, dte() replays it perfectly: each task as long as it held its worker and with the cores it used meanwhile, on as many workers as the sizes allow, each joining after its platform’s fastest real warm-up. That ideal end (TaskExecutor.idealMs) sits beside the real time in the run summary and the footer’s ideal line, and @vzn/vx-ci prints it.

The same replay says which workers to declare next time (TaskExecutor.advice), each from an empty run with the plan’s own rule (the fewest that end it within 5% of the soonest): best is how many of each size; more, shown only when more than a size’s max would end the run sooner past that margin, is how many it would take; any is one shape for each platform, of 2 to 64 cores (memory a core as the platform’s sizes have most), the one ending as soon for the least CPU held, each worker from its ask to its last task’s end. A task goes only on a size with room for its cores and peak memory, when one is declared. Raise a size’s max, or change its cpu and memory, by them. They are in the footer and the run summary, and @vzn/vx-ci prints them under “Workers to declare”.

Environment

What @vzn/vx-dte reads from the CI job it runs in:

• GITHUB_SHA: the commit every worker checks out.
• GITHUB_HEAD_REF or GITHUB_REF_NAME, then GITHUB_BASE_REF: the ref github() dispatches the worker workflow on, the base when the head has no workflow file.
• GITHUB_RUN_ID and GITHUB_RUN_ATTEMPT: the run’s attempt, so a worker that starts after it ended cancels itself.
• ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN: where the job asks GitHub for its OIDC token; GITHUB_REPOSITORY_ID: the repository a token must name.
• GITLAB_CI and CI_JOB_TOKEN: gitlab() runs on GitLab CI.
• CI_MERGE_REQUEST_SOURCE_BRANCH_NAME or CI_COMMIT_REF_NAME, then CI_MERGE_REQUEST_TARGET_BRANCH_NAME: the ref worker pipelines run on.
• PATH: a worker keeps the task’s own bin directories, then its own machine’s PATH. A task whose exec.env.define sets PATH, or whose inputs.env names it, runs on that PATH as the orchestrator sent it.
• GIT_INDEX_FILE: local() clears it before git worktree add, so a run started from a git hook checks out the run’s commit, not the hook’s index.
• GIT_DIR: with no GIT_WORK_TREE, the run’s snapshot and local()’s and docker()’s git drop it when git finds that repository from the workspace anyway (a linked worktree’s hook sets it), as core does.

API

• dte(options?: DteOptions) is the plugin. DteOptions is { workers?, adapter?, relay?, local?, token?, scheduler?, unmeasured?, headers? }; a key it does not take, in a size or unmeasured too, is refused. headers go on every dial to the relay, the orchestrator’s and each worker’s (redials too), for a proxy in front of it, such as a Cloudflare Access service token ({ 'CF-Access-Client-Id': process.env.CF_ACCESS_CLIENT_ID, … }); an unset value is left out, and a name the dial sets itself (authorization, content-type, relay-proof) is refused. The rest:
  • workers names each worker type, a WorkerType { machines }, in order of preference (exec.worker picks one). Each machine is a size the adapter may start, named <type>.<machine> (linux.small; each name letters, digits, _ and -, as worker ids and log files carry them, and a machine’s . after its first character, c5.large) in the run’s output and an adapter’s options, and is a WorkerSize, { cpu, memory, max?, platform?, runsOn? } (runsOn: the runner label vx dte init writes for it): its cores and memory (bytes, or '8GB', '512MiB'), which the run counts on when it starts one, until the worker reports what its machine has, how many of it one run may start (absent, no cap), and its machines’ platform (linux-x64, linux-arm64, darwin-x64, darwin-arm64; absent, planned as this machine’s and not checked on join). With no adapter, the sizes’ platforms are the ones joining workers may have. Only work counts: a task that missed the cache, or waits on a cached one that did, and every uncached task, which runs every time; groups and hits need no worker. The run plays that work out on its workers and on N more (src/planner.ts), each new one joining after its size’s warm-up: this run’s first of that size, else an earlier run’s, else any size’s this run, else 20 s, and the best end is the one with all N. A worker counts its cores less what its machine spends besides the run’s tasks, and a new one of a size what this run’s of that size spent, else an earlier run’s (kept in history, as warm-ups are), else none; so does one just joined, until its own load says, which then moves it from there a step a frame, as tasks are placed on it. A new worker starts only when a task reaches its latest start (the best end less three times its chain, as history is a guess) with no room sooner on a worker running or starting: a task with slack waits for room the run pays for anyway, and the critical path never waits. A worker started for a task starts as soon as that task is ready, and tasks with slack fill its room. A chain through a task with no history has no slack the plan can prove: it takes room on a worker running or starting as it gets ready, and starts no worker for that alone. A task running past its history counts as running as long again. It plays only two warm-ups and the longest task ahead; past that, the rest ends no sooner than its longest chain, nor than its work spread on the cores it has, nor than a ready task with no room in the window can end from the window’s end, and the next plan, a warm-up on, sees further. A task on fewer cores than it asks is played at their share of its pace. A worker started in place of one let go is asked as its work nears, as any later start is. A big worker whose last work fits a smaller size stops taking tasks once its big work ends: the rest goes to the smallest size it fits, started to join then, when that holds less CPU and the run ends no later. The big one leaves once its running tasks end. While a task with no history is left, this happens only for the run’s last work. With every task’s time known, the plan packs the work instead, when that holds less CPU and ends as soon (src/pack.ts): each task, in order of its latest start, goes where a worker has room for it between its ready time and the start that ends its chain by the deadline run slow (each task’s slow run from history, else 15% over its usual time), on the worker whose time it lengthens least; with no such room, where one has room by its latest start; a new worker, of the smallest size it fits, starts for a task none has room for by then, or when keeping a worker idle until it costs more than a new one’s warm-up (two warm-ups for a worker the run has; never while one sits idle that could take it), asked to join as that task gets ready. The plan packs to the longest chain from now first, then past it by a warm-up, then to the best end; a packing that leaves out a task fits no worker and is not used. A task booking more cores than any size of its platform has is packed onto any worker whole, as the run books it. A task reading files an uncached upstream left on its worker is packed there, or anywhere else with that upstream’s time added, as the run makes it again first; the worker holding them stays while it has such a task to come. A server’s base is packed as the run books it: once on each worker that starts it, from its first dependant there until its last dependant anywhere ends. It then packs again in other orders, up to 64 times (fewer past 100 tasks, none past 6400), each new worker’s task a random way into its slack, and keeps the one holding least by the same end. It packs the whole run, past the window. Each next new worker of a size is counted as joining later by the spread this run’s joins of it showed (the slowest less the fastest, over the joins between), else an earlier run’s (kept in history, as warm-ups are); the play it is compared with counts its new workers so too. A worker past its warm-up is packed as if asked now. The run then gives each task only to the worker the plan gave it, once that joined, and not before the plan starts it: one the plan starts as it gets ready goes as soon as it is. A task whose worker has not joined by its start goes to any worker with room. A task the plan has not seen (its miss came after) waits for the plan its arrival asks: let go, it took the room the plan kept for a longer chain. The plan may end 5% (5 s at least) past the best; that margin is spent once: later plans keep its deadline until new work is known. A task waiting on one that runs waits until it ends. Then the run follows that plan: a start it wants later is asked when due, and a worker idle with its planned work done goes, once every task planned on it has started. It plays again only when reality leaves the plan: new work, work that leaves without running, a worker idle at its planned end with a task planned on it not started, a worker that joins, leaves or warms up off plan, a task past its time or ending sooner, one of no known time ending, a task starting with its time, CPU or memory more than 5% off (1 s, 0.05 cores, 64 MiB at least), or its window moving on. What is left, running tasks too, takes as many times its history as the run’s finished tasks took theirs (once they had 20 s of it; never less): a task past its forecast would otherwise be planned to end now, and a plan that thinks the run nearly done cancels the worker it asked. Before any worker can say what misses, the run counts as work each cached task whose inputs changed since HEAD’s parent (git diff HEAD^), and every task after one (src/forecast.ts): a lockfile or .vx/ change counts every cached task. A hit it guessed wrong costs a worker the plan cancels; with no parent, the run waits for its misses. Each task core’s up-front cache probe found in no store (TaskExecutor.demand’s missed) counts as work too, the moment the probe answers. A measured task books the cores it kept busy on average, and a whole core at least: eight test shards at 0.3 cores each shared one 4-core worker and ran 2-3 times their time alone, which their history then kept. It books those cores times the share of their history the run’s finished tasks used (once they used a minute of core-time; never under two thirds, nor a core), so a worker’s bookings may reach 1.5 times its cores when tasks use less than history says; plans book the same. What a task used counts divided by its share of the CPU: a Linux worker reads its run queue (procs_running in /proc/stat) every 200 ms, a macOS one its load average while all its cores are busy (a GitHub Mac reads a load of 49 with a core idle), and a task whose threads got a third of a core each used a third of its need, not its need. A task wanted at most what it used and the cores left free beside it, so its share is never read below that. The run history keeps that share (ExecuteResult.machine.cpuShare) and learns the need, at most the machine’s cores: learning the share packed the next run tighter, nine 1.1-core test shards on one 4-core worker. The need is the mean over runs, and a task fills its worker only when it does on average: one crowded run’s 2.84 cores for a build using 1.3 to 1.9 booked it a whole Mac. A worker the plan needs later starts when its warm-up nears. An idle worker leaves when the plan gives it no task (its warm-up is spent), or, when its next task is two warm-ups away, with one started in its place a warm-up after it left: none is let go and started again. An idle worker takes the work of one its CI has not started past its warm-up, which is cancelled. A worker still starting that the plan gives no task (asked, a worker stays while the plan has work for it, though the run would end no later without) is cancelled: the adapter stops it before its setup if it can (github() cancels its workflow run once no worker in it is needed and none of its jobs has a runner), and one that joins anyway is told to exit. One the adapter leaves running (its job had a runner, its run holds a worker still needed, or the adapter stops none) is counted again, and the run takes it when it joins: its setup is paid, and a cancel landing late could stop it mid-task. The same signal changes nothing. The run ends: a task the history has not measured books one free core on any worker (unmeasured sizes the run); one that used all of its worker’s cores or memory (90%) last time runs alone on the smallest bigger size, which the run starts for it. Each run’s history keeps where every task ran and that machine’s size, so sizing improves run by run.
  • adapter is the DteAdapter that starts workers, of the sizes in workers: at least one with max above 0, or dte() is refused. Absent, the run starts none and uses the workers that join relay.
  • relay is the relay’s base URL, http(s), without /v1, ?query or #fragment; absent, the adapter’s own.
  • token answers the relay’s bearer token (default VX_DTE_TOKEN). It is asked again a minute before the token’s exp, when it has one.
  • unmeasured, { cpu, memory } (default 2 cores, '4GB'), is what a task with no measure counts for when the run decides how many workers to start: a new task, or one over before the Linux sandbox’s first look (250 ms), which leaves no CPU. It books one free core on any worker, so it waits for no worker of its own, and goes where the most cores are free, room to use more than it booked.
  • scheduler (a partial Strategy) picks each layer of how the run schedules; a layer left out is the default (first named):
    • order, which waiting task goes first: critical-path (the longest chain after it), graphene (the long and the wide first), fifo;
    • place, which worker takes a measured task: best-fit (fewest cores left), tetris (needs aligned with what is free), worst-fit, first-fit;
    • book, what a task holds: measured (its history), margin (a fifth more), whole (a worker each);
    • scale, how many workers: plan (above), backlog (cores, of each platform, for what is ready or running, Spark’s dynamic allocation). tests/sim.test.ts plays main’s runs with each.
• vx dte worker is a worker. It reads VX_DTE_RUN, VX_DTE_RELAY (unset or empty, relay), VX_DTE_SECRET (the run’s) and, when the run started it, VX_DTE_WORKER (its name). It is described only by the cores and memory its machine reports. It exits 0 when the run ended before it joined. On SIGINT or SIGTERM it stops its tasks, leaves the run, which runs them again on another worker, and exits 130 or 143; a second signal ends it at once.
• DteAdapter is { spawn(worker: WorkerSpawn), relay?(), close?(), cancel?(ids), init?(setup: DteSetup), runId?(), adopt?(worker), report?(warn), keyless?, workerRuns?() }; init gets a DteSetup (workspace root, its directory in the repository, relay, sizes, the PackageManager read from the lockfile, the tasks asked for) and returns a DteInit: the CI files vx dte init writes and the steps it prints; report hands it the run’s warning channel as the run starts, where github() says a status poll or a cancel the CI API refused, by status and body; adopt hands each worker the CI started before the run to worker(id, size), which counts it and returns its spawn; cancel stops workers the run no longer needs that have not joined, and may answer with those it left running, which the run takes (with late, workers given up on joining, stopped whatever their job holds); runId names the run, in the adapter’s own CI, when VX_DTE_RUN is unset; workerId names a worker in a job its CI started for several, when VX_DTE_WORKER is unset; keyless workers prove themselves by GitHub OIDC, and workerRuns names the workflow runs whose jobs alone may join as them (undefined: any job of the repository). WorkerSpawn is { run, relay, size, max, cpu, memory, index, id, workspaceRoot, commit, cacheDir?, env }, with memory in bytes and max its size’s (Infinity uncapped), where id is the worker’s name (linux.small#2, numbered within its size), commit the tracked files as the run saw them at its start (uncommitted changes included), whose tree a worker must check out, cacheDir the run’s cache dir (absolute) when the workspace names one, for workers on this machine to save where the run reads, and env holds the three variables the worker reads.
• docker(options: DockerOptions) starts workers as containers and a relay of its own on loopback. DockerOptions is { image, prepare? }.
• github(options?: GithubOptions) dispatches workers as matrix jobs of one workflow, one dispatch per size and demand (at most 256 workers, a matrix’s limit). GithubOptions is { workflow?, env? }: workflow defaults to vx-worker.yml, a file name in .github/workflows other than ci.yml, the run’s own; env is variables every worker gets (one-line strings, no secret-named one: the worker job’s log shows them), sent as the dispatch’s env input only when set, which vx dte init’s workflow exports. Its workers are keyless, and only jobs of the workflow runs it dispatched join as them. It needs GITHUB_TOKEN with actions: write. See On GitHub.
• gitlab(options?: GitlabOptions) triggers a pipeline of the project’s .gitlab-ci.yml per wave and size, with CI_JOB_TOKEN. GitlabOptions is { tags? }: runner tags by size, which vx dte init writes. See On GitLab.
• local(options?: LocalOptions) starts workers on this machine and a relay of its own on loopback. LocalOptions is { prepare? }: a shell line each worktree runs before its worker starts.

@vzn/vx-dte-server is a dependency for local()’s and docker()’s relay, loaded only when a DTE run starts.