Account-scoped sandbox API

(Review 1/2)

Reviewed end to end. Strong spec and the core thesis is right: the runner fleet is a substrate, and sandboxes are a second workload with a small conceptual delta. Most of my feedback is structural (altitude + one forward-compatibility constraint), with a handful of correctness concerns in billing, capacity, and isolation that hold regardless of how the spec is framed.

1. Raise the altitude. A lot of this is implementation detail that will rot the moment the code diverges: the file-by-file Layout block, the exact Ecto DDL and indexes, the ClickHouse engine choice, the module-path list, and the lifecycle pseudocode. I’d move all of that to the PR and keep the spec to: Why, the runner→sandbox delta, Goals/Non-goals, the API surface as a contract, the open questions, and Done-when. That alone roughly halves the length and puts the open questions at the center, which is where a draft should sit.

2. Keep the substrate separable from the inbound API (build for a future outbound consumer, don’t scope it now). The whole spec assumes a caller-driven model: someone calls inbound and drives a POST /exec loop over our SSH broker. That’s the right v1. But I’d keep the provisioning substrate (pod lifecycle, repo staging, image, billing session) cleanly decoupled from the inbound REST/exec/SSH/preview surface, so we don’t bake in the assumption that execution is always driven inbound. The concrete future consumer I have in mind is Anthropic’s self-hosted sandboxes for Claude Managed Agents: there Anthropic owns the agent loop and you run a worker that connects outbound, claims a session from their queue, downloads skills, and drives execution from inside the sandbox, with no inbound exec, broker, or preview URL involved. We don’t need to design that now, but if the lifecycle and billing assume the broker/preview path is always present, we’ll be unwinding it to support that later. (It’s also where our macOS + Xcode pool is a real differentiator, since every provider Anthropic lists for this is Linux. Worth a one-line nod in Why, not a use case in scope.)

3. Promote VM snapshots / warm images to a Goal. Today they’re uncommitted API stubs (in the surface, absent from Goals/Done-when, parked as an open question). They should be a goal, scoped per platform, justified by provisioning latency: macOS cold boot is minutes, which neither an interactive session nor any agent loop tolerates per sandbox. Restoring from a pre-warmed image (Xcode + deps resident) is what makes the macOS path viable, and it doubles as the capacity-reclaim lever in (7). (Note this is a full-VM snapshot on macOS vs a volume/fs snapshot on Linux: different capabilities under one word, pick which we promise per platform.)

4. Billing correctness (survives any altitude). Two real decisions, not implementation detail:

• Open anchor. runner_sessions opens at claim-win specifically to keep warm-pool idle out of billing. The sandbox flow opens at create (status pending, before the pod is ready), so we’d bill provisioning and boot, which on macOS is minutes. The schema already has ready_at; decide bill-from-ready vs bill-from-create deliberately, especially for the customer accounts behind the flag.
• Close trigger. Runner sessions are closed by the controller observing a terminal Pod phase (finishedAt), covering every abnormal exit. The sandbox design closes from the Terminator after a DELETE, with the reaper covering deadline. A pod that dies outside a DELETE (OOM, node loss, guest crash, all of which we see on this fleet) leaves an open session billing until the 6h clamp between cron ticks. Sandboxes need an equivalent reconcile/observer.

(Review 2/2, continued)

5. The “reuse verbatim” claims are overstated. Tuist.Runners.Billing.interval_intersection/3 doesn’t exist; the real surface is compute_milliseconds/4 (which already takes a :platform scope opt, so the per-platform dashboard claim is genuinely free). Its @max_session_lifetime_seconds = 6h clamp is a runner safety net hard-coded as a module attribute; reused as-is it silently caps sandbox billing and needs to be per-kind once deadlines approach it. And Tuist.Runners.Claims is keyed on workflow_job_id with ON CONFLICT to collapse pollers racing one shared job; sandboxes have no shared job, so the reusable part is the per-account count + advisory lock, not the claim row. At higher altitude, just say “reuse the per-account billing + concurrency primitives” without naming arities.

6. Isolation / multi-tenancy is under-specified. Sandboxes run untrusted code by design, and the provisioner mounts cache + registry creds into the pod, a larger exfiltration surface than runners (which already defend the SA-token-reuse path). State the isolation posture (Linux: runc/gVisor/Kata? macOS: full VM, strong), scope mounted tokens least-privilege + short-TTL, and note that project_id is described as “just a reporting tag” but actually gates credential mounting: a tuist-account sandbox holding a customer project’s creds is a cross-account credential flow that needs an explicit grant model. Also: the single multi-tenant SSH broker is framed as an availability SPOF, but a leaked/guessable token is a confidentiality blast radius across sandboxes; and public preview URLs expose untrusted-code services to the internet (auth + SSRF back into the internal network).

7. Capacity is a risk, not just an open question. A sandbox pins a pool slot for its whole deadline; a runner boots-runs-exits in minutes. And the fleet is pre-bought bare metal with a fixed slot ceiling, there’s no autoscaler to absorb the load, so a long-lived hold is a direct and lasting subtraction from CI capacity, with the only relief being to buy and provision more hardware. The macOS pool is especially tight (~2 VMs per Mac mini, so the ceiling is small and hard), and a few parallel macOS sandboxes contend head-to-head with CI for a fixed number of slots; the Linux pool is already reservation-bound with a guest-OOM history. I’d treat macOS long-lived-guest-stability as a gating item for that path, and admission control / fair-sharing between runner and sandbox demand (plus a capacity-planning signal for when to add hardware) as a goal rather than an open question.

8. Smaller. tuist sandbox ssh is in Goals and Done-when but missing from the Design CLI list, and it’s hand-written (mint token + shell out), not OpenAPI-generated, so call that out. MCP wait_sandbox_ready has no REST mirror (it’s client-side polling sugar, fine, just not a literal mirror). The ClickHouse ReplacingMergeTree for an append-only event stream looks wrong (it collapses rows on the sort key), though this disappears if the engine choice is trimmed per (1).

What’s good and should stay: the two-pool-one-substrate framing, the runner→sandbox delta table, the partial reaper index, the vendor survey and its “convergent verbs” synthesis, and the flag + per-account-cap rollout path.

@marek thanks, solid review. Worked through all eight points in revs 8-15.

(1) altitude cut: Layout, DDL, Lifecycle pseudo, CH engine all gone. (2) new Layers section splits provisioning substrate from inbound surfaces. Outbound-driven workers (Anthropic Managed Agents) called out in Why + Non-goals. (3) warm-image provisioning and customer snapshot/restore promoted to Goals, with per-platform mechanism documented. (4) billing opens at ready_at, closes on observed pod terminal phase (covers OOM/node loss/guest crash), session-lifetime clamp goes per-kind. (5) “reuse verbatim” softened; Claims-row vs primitive distinction made explicit. (6) new Security subsection. grants is the access-control field, project_id is metadata, account agent tokens replace project tokens. v1 keeps grants same-account; cross-account becomes the open question. (7) admission control (runners win by default) and macOS guest stability promoted from open questions to Goals. (8) tuist sandbox ssh flagged as hand-written, wait_sandbox_ready clarified as polling sugar, ReplacingMergeTree gone with the altitude cut.

Would appreciate another pass.

Apologies, the previous comment rendered without line breaks. Same content, properly formatted:

Worked through all eight points in revs 8-15.

1. Altitude cut: Layout, DDL, Lifecycle pseudo, CH engine all gone.
2. New Layers section splits provisioning substrate from inbound surfaces. Outbound-driven workers (Anthropic Managed Agents) called out in Why + Non-goals.
3. Warm-image provisioning and customer snapshot/restore promoted to Goals, with per-platform mechanism documented.
4. Billing opens at ready_at, closes on observed pod terminal phase (covers OOM/node loss/guest crash); session-lifetime clamp goes per-kind.
5. “Reuse verbatim” softened; Claims-row vs primitive distinction made explicit.
6. New Security subsection. grants is the access-control field, project_id is metadata, account agent tokens replace project tokens. v1 keeps grants same-account; cross-account becomes the open question.
7. Admission control (runners win by default) and macOS guest stability promoted from open questions to Goals.
8. tuist sandbox ssh flagged as hand-written, wait_sandbox_ready clarified as polling sugar, ReplacingMergeTree gone with the altitude cut.

Would appreciate another pass.

Stepping back from the point-by-point: two more fundamental things, the first a disagreement.

1. Don’t share the CI fleet in v1. I’d reverse Goal 2 for now. Hive is internal-only, so sharing means risking customer-facing CI stability for a consumer that delivers no customer value yet. And admission control doesn’t buy what we need here: it arbitrates scheduling (runners win a scarce slot at create time) but does nothing about runtime interference. A sandbox that already won a slot and is co-scheduled with a runner still contends for CPU/IO, and on macOS a sandbox VM sharing a Mac mini with a runner VM puts the unproven guest-stability risk (Goal 5) on the same host as customer CI. For an internal-only, stability-unproven workload, the right v1 is dedicated sandbox capacity (a cordoned subset / separate pool) with the customer CI fleet isolated. Share later, once sandboxes are proven stable AND there’s external demand that justifies the risk. As a bonus this dissolves the warm-pool-vs-admission-control tension and the procurement-pace bet.

2. The spec states goals where it needs to show mechanism. Right altitude on the incidental stuff (good that Layout/DDL are gone) but the genuinely hard parts are still asserted, not solved. “Closes on observed pod terminal phase”, “warm-image restore in seconds”, “resume-where-I-left-off”, “clamp per-kind”: that is the design, and it’s a sentence each. This isn’t a reversal of the altitude cut, it’s the other half of it. Cut the mechanical detail; spend the reclaimed length on the uncertain mechanisms. The spec should be longest where the risk is. I’d want the core to be:

(a) one lifecycle diagram covering the session AND snapshot state machines (pending, ready, terminating, terminated, failed; plus the snapshot sub-lifecycle: requested -> snapshotting -> stored -> restoring -> ready), and

(b) a transition table: for each edge, the trigger, the component that drives it, the failure mode, and the billing/state-consistency rule.

Filling that table is where the open residuals actually get answered, because each is a transition the prose currently skips:

• abnormal death -> terminated: what watches sandbox pod terminal phase? Runners get it from the controller’s POST /api/internal/runners/pods/stopped. Sandboxes need a named equivalent, or the OOM/node-loss/crash billing-close doesn’t actually exist.
• ready -> snapshot: live or quiesced? captured state = memory+process or filesystem only? The macOS full-VM and Linux fs mechanisms deliver different fidelity, so one “resume where I left off” promise can’t hold across both as written.
• stored -> restore: snapshot storage cost, retention, GC: unspecified, and billing only covers session intervals.
• pending -> ready: warm-image restore vs cold-boot fallback (and, only if we don’t dedicate capacity, whether warm reservations are preemptible by runner demand).

I’d rather the spec spend its length here than on the goal list. Happy to pair on the state machine.

@marek thanks, both fair. Rev 16:

1. Dedicated capacity in v1. Goal 2 reversed: cordoned sandbox pool, isolated from CI capacity. Goal 3 shrinks to just the capacity-planning signal (no shared contention to arbitrate). Reference model, Why, and Done-when updated to match.
2. Mechanism over goals. Took a shot at the state machine. New “Lifecycle and state machines” section: sandbox transition table (seven edges, abnormal-exit driven by a named Sandbox Pod Watcher + POST /api/internal/sandboxes/pods/stopped, mirroring the runner controller), snapshot sub-lifecycle table, per-platform fidelity table (full_vm vs filesystem, the API surfaces kind so the asymmetry is visible), plus snapshot storage and warm-pool paragraphs.

Would still take you up on the pair session to refine the residuals, especially: snapshot fidelity (live vs quiesced per call or fixed per platform?), and warm-pool reservation semantics if we ever share the fleet.

Rev 16 is the right move: named mechanisms (tart pause-copy-resume, CSI volume snapshot, Tigris storage, the Sandbox Pod Watcher) instead of asserted goals. Two things it still doesn’t answer, and they turn out to be the same problem.

Snapshot/restore is named but not timed

The fidelity and sub-lifecycle tables describe what happens, but “resume in a reasonable time” is exactly what’s missing:

1. No latency budget, no Done-when criterion. The warm pool gets “resolves in seconds”; snapshot-create and restore get no target, and Done-when has no timing acceptance for either. The hard number is unstated.
2. macOS full_vm is data-movement-bound and the spec hides the cost. “memory + process state” means dumping the VM’s multi-GB RAM image to disk and pushing it to Tigris on snapshot, then pulling multi-GB back on restore. That transfer is the latency, and it’s minutes-scale, not seconds. tart pause-copy-resume is presented as a checkbox when it’s the single hardest latency problem here. It deserves the same “gating, prove it with a budget” treatment guest-stability (Goal 5) and warm-image (Goal 4) already get.
3. Restore can’t use the warm pool, but the spec implies create-speed. Restore runs Sandboxes.create(snapshot_ref) -> restoring -> ready. Warm pods are blank OS+Xcode; a restore needs this snapshot’s state rehydrated, which a warm pod can’t provide. So restore is a separate, slower, transfer-bound path with no budget. Linux CSI snapshot-to-PV is tractable; macOS multi-GB memory load is not, yet. Say so explicitly.
4. Internal contradiction. “Snapshot capture happens in-place… the sandbox stays ready throughout” is false for macOS: pause-copy-resume freezes the VM for the duration of a multi-GB dump. Only Linux CoW is non-disruptive.

To answer the question: a per-platform latency budget for snapshot-create and restore, a Done-when criterion, an explicit “restore is not on the warm-pool fast path” note, and the macOS memory-image transfer called out as the gating risk with a proof point.

Idle offload isn’t addressed at all

There’s no idle concept in the state machine. The only automatic exits are deadline_at (reaper) and pod death. A sandbox that’s ready but doing nothing keeps its slot and keeps billing until the deadline or explicit DELETE. Agent loops are idle a lot (waiting on the model, between iterations, waiting on a human), so a customer hogs a dedicated-pool slot and pays for dead air up to an hours-long deadline.

The irony is that the machinery to solve this is already in the spec and just isn’t wired up. Idle offload is suspend-to-snapshot: detect idle -> snapshot (offload) -> free the slot and pause compute billing -> restore on next activity (the Cloudflare wake-by-name pattern). What’s needed:

• an idle / suspended state, with a definition of idle (no exec / no SSH session / no port traffic for T),
• the offload transition (suspend-to-snapshot) and the wake transition (restore-on-activity),
• a billing rule for suspended time (compute pauses, only snapshot storage accrues),
• the deadline interaction (does suspend stop the deadline clock?).

They’re the same question

You can’t offer transparent idle offload without fast restore. Auto-suspending a macOS sandbox and then taking minutes to reload a multi-GB memory image on wake is a bad experience, and that wake path can’t use the warm pool. So macOS fast-restore is the linchpin for both “reasonable snapshot time” and “don’t hog idle sessions,” and the shared root cause, moving VM memory state to/from object storage quickly, is the thing the spec hasn’t costed. I’d make that the centerpiece of the next revision (or the pair session): pick a latency target, prototype the macOS memory snapshot/restore against Tigris, and design the idle->suspend->wake loop on top of whatever that proves out.

@marek sharp catch on both, and you’re right they’re the same problem. Rev 17:

1. Customer-controlled snapshot endpoint deferred from v1. The mechanism stays in the substrate but is internal-only. Goal 11 (customer snapshot) is gone; the POST /snapshots(/:snap/restore) endpoints are out of the API surface.
2. Idle offload is now a v1 Goal (Linux only). New Goal 11: idle threshold 5 min (configurable), suspend = volume snapshot + pod release, wake on activity, compute billing pauses while suspended, deadline clock pauses while suspended, 7-day absolute wall-clock cap. New state-machine rows for ready -> suspending -> suspended -> restoring -> ready plus suspended -> terminating.
3. macOS suspend/wake is a gated research goal. New Goal 12: prove that VM memory image transfer to/from Tigris is fast enough to be useful, “fast enough” decided from the prototype output (not committed up front per our discussion). Until then, macOS sandboxes have no idle offload and no snapshot.
4. Fidelity table reframed and the in-place lie fixed. “Snapshot sub-lifecycle” -> “Suspend/wake fidelity”; Linux ships, macOS gated. Explicit note that if macOS suspension ships later, it transitions ready -> suspending with the VM paused during the dump, not in-place.
5. Warm pool note: restore (suspended -> ready) explicitly does NOT use the warm pool; a restore needs the suspended sandbox’s state rehydrated, not a blank base image.

Still want to pair on the macOS prototype side: even without a committed number, the prototype design (sample VM size, transfer mechanism to Tigris, snapshot format, whether incremental memory snapshots are tractable) is where the real risk lives.