Hive Hive
Sign in
Projects
Domains
Forage
Specs
Drops

feat(infra): join the macOS fleet to a Tailscale tailnet

GitHub issue · Closed

Metadata
Source
tuist/tuist #10761
Updated
Jun 24, 2026
Domains
Compute
Details

Summary

Brings the xcresult-processor + customer-runner Mac mini fleet onto the cluster’s Tailscale tailnet so alloy-metrics can scrape both the xcresult-processor VM’s PromEx endpoint AND host-level node_exporter + tart-kubelet metrics on each mini, without exposing any of them on the public WAN. Builds on top of #10653.

  • Why now: the Oban dashboard’s per-worker histograms only exist for workers whose host pods Alloy can scrape. ProcessXcresultWorker runs on the Mac mini fleet, which sits behind Scaleway’s per-host public IPs with no private path from the Hetzner cluster. The DB-polled gauges in a2f8817a gave us queue-length coverage for alerting; real per-worker latency/error distributions for the macOS workers need an actual scrape path.
  • The fix: install open-source tailscaled on each Mac mini at bootstrap; tart-kubelet advertises the tailnet IPv4 as Node.Status.Addresses[InternalIP]; the host-side metrics forwarder binds there; node_exporter covers host-level signals. The Tailscale K8s operator provides the cluster-Pod-to-tailnet egress via a shared ProxyGroup plus one ExternalName Service per Mac mini, auto-materialised by the CAPI provider.

Architecture

 Hetzner cluster (Linux)

 ┌──────────────────────────────────────────────────────────────────┐

 │ alloy-metrics (clustered DaemonSet)                              │

 │   ├── annotationAutodiscovery → xcresult-processor Pod           │

 │   │   PodIP rewritten by tart-kubelet to tailnet IPv4            │

 │   │   → scrape <tailnet-ip>:9091 (PromEx via forwarder)          │

 │   │                                                              │

 │   └── extraConfig scrape jobs (Service-role discovery)           │

 │       filter on label tuist.dev/macmini-egress=true              │

 │       → svc <machine>.tailscale-operator.svc.cluster.local:9100  │

 │       → svc <machine>.tailscale-operator.svc.cluster.local:8080  │

 │                              │                                   │

 │ tailscale-operator                                               │

 │   ├── Connector (subnet-router)                                  │

 │   │   advertises cluster IPs into the tailnet                    │

 │   │   (lets tailnet devices dial in-cluster Services)            │

 │   └── ProxyGroup macmini-egress (type=egress, replicas=2)        │

 │       ◄─ per-machine ExternalName Service                        │

 │          (created by CAPI controller, one per SasMachine)        │

 │          annotated tailscale.com/tailnet-fqdn + proxy-group      │

 │          operator rewrites externalName → ClusterIP fronting PG  │

 └────────────────────────────────┬─────────────────────────────────┘

                                  │ tailnet (WireGuard)

                                  ▼

 Scaleway Mac mini fleet (darwin/arm64, tag:tuist-macmini-<env>)

 ┌──────────────────────────────────────────────────────────────────┐

 │ tailscaled (open-source variant, launchd-supervised)             │

 │ tart-kubelet --node-ip-source=tailscale                          │

 │   ├── Node.Status.Addresses InternalIP = tailnet IPv4            │

 │   ├── host-side Forwarder on <tailnet-ip>:9091                   │

 │   │   → Tart VM (xcresult_processor) /metrics                    │

 │   ├── controller-runtime metrics on <tailnet-ip>:8080            │

 │   │   (incl. tart_kubelet_vm_boot_duration_seconds)              │

 │   └── runs scheduled Pods as Tart VMs                            │

 │ node_exporter on <tailnet-ip>:9100 (launchd-supervised)          │

 └──────────────────────────────────────────────────────────────────┘

The egress shape (ProxyGroup + ExternalName) is necessary because subnet-router mode advertises the cluster INTO the tailnet, not the other way round. Verified empirically: the Connector Pod’s own tailscaled can reach the minis at 100.x.y.z, but generic cluster Pods cannot. The ProxyGroup runs tailscaled in egress mode and routes per-Service traffic out to the tailnet.

What’s in the change

Mac mini side

  • macos-host-bootstrap/bootstrap.go gains two idempotent steps: installTailscale extracts the operator-baked tailscale + tailscaled darwin/arm64 binaries to /usr/local/bin, writes a custom launchd plist with explicit --state / --socket / --port flags and a StandardErrorPath for crash diagnostics, and runs tailscale up. installNodeExporter writes a prebuilt node_exporter-darwin-arm64 binary plus a wrapper that resolves the tailnet IPv4 fresh on every start and binds the exporter there. Bootstrap renders --node-ip-source=tailscale into the kubelet launchd plist when both the binaries and auth key are wired.
  • tart-kubelet/cmd/tart-kubelet/main.go: new --node-ip-source=auto|tailscale flag with a tailscaleNodeIP() helper that shells out to tailscale ip -4. Fatal on failure under tailscale: silent fallback to the public interface would defeat the boundary.
  • tart-kubelet/internal/nodeagent/node.go: NodeIP is published as Node.Status.Addresses[InternalIP] so cluster-side scrapes find the right target.
  • tart-kubelet/internal/podagent/proxy.go: 100.64.0.0/10 added to DefaultScrapeAllowedCIDRs so tailnet-traversing scrapes match the allowlist.
  • tart-kubelet/internal/podagent/metrics.go: new tart_kubelet_vm_boot_duration_seconds histogram covering VM cold-start latency.

Operator side (CAPI provider for Scaleway Apple Silicon)

  • Dockerfile: new build stages cross-compile tailscale + tailscaled from source (v${TAILSCALE_VERSION}, currently 1.96.5) and pull the prebuilt darwin/arm64 node_exporter release (cross-build needs Apple SDK for three collectors).
  • credentials.Manager.GetTailscaleAuthKey reads the pre-auth key Secret per reconcile from the controller-runtime cache. Empty TailscaleAuthKeySecretName disables the step.
  • ScalewayAppleSiliconMachineReconciler threads TailscaleBinaries / NodeExporterBinary / TailscaleTags into both bootstrap.Run AND bootstrap.UpdateTartKubelet. The drift-update path keeps the tailnet wiring after operator-image bumps. Includes a fix for the original miss where NodeExporterBinary wasn’t being passed to UpdateTartKubelet, silently skipping the installer on every drift-loop run.
  • New reconcileTailscaleEgressService stage maintains one ExternalName Service per Mac mini in the egress namespace, idempotent via CreateOrUpdate, with explicit cleanup in reconcileDelete (cross-namespace OwnerRefs aren’t allowed). Tolerates the optimistic-concurrency conflict that fires when the Tailscale operator races us on the same Service.
  • Manager flags: --tailscale-binaries-path, --node-exporter-binary-path, --tailscale-auth-key-secret-name, --tailscale-tags, plus the egress trio --tailscale-egress-{proxy-group,namespace,magicdns-suffix}. Empty --tailscale-egress-proxy-group leaves the OSS shape (no tailnet) untouched.
  • Cache scoping: when the egress reconciler is wired, the Services informer is narrowed to the egress namespace so the namespaced Role + RoleBinding actually satisfies what controller-runtime asks for. Without this, the SA would need cluster-wide services read, which we explicitly don’t want.

Cluster side (Helm)

  • New tailscale-operator wrapper chart depending on tailscale/tailscale-operator:1.96.5. Renders an ESO Secret for the OAuth client, a Connector CR (subnet-router for inbound tailnet → cluster Services), and a ProxyGroup of type egress for the reverse direction. Connector + ProxyGroup are gated behind post-install hooks because their CRDs ship in the subchart’s templates/ (not crds/) and would race a same-batch CR apply.
  • infra/helm/tuist/templates/macos-fleet-tailscale-external-secrets.yaml: ExternalSecret pulling the auth-key field from the TAILSCALE 1Password item, gated on macosFleet.tailscale.enabled.
  • infra/helm/tuist/templates/capi-scaleway-applesilicon.yaml: Tailscale flags + the new egress flags on the operator Deployment. Namespaced Role/RoleBinding in the egress namespace grants the operator SA services CRUD, scoped tight.
  • infra/helm/k8s-monitoring/values.yaml:
    • New extraConfig on collectors.alloy-metrics (Service-role discovery filtered by tuist.dev/macmini-egress=true, two named ports → two scrape jobs). Side fix: previous k8s-monitoring.extraConfig.alloy-metrics nesting was at the wrong key and silently ignored by the chart’s collector helper.
    • integrations.alloy enabled for all four Alloy roles with alloy_config_last_load_successful + alloy_config_last_load_success_timestamp_seconds whitelisted on top of the upstream allow-list. Alloy’s reload model is “hot-swap if valid, keep serving previous valid config otherwise”; without this metric, a broken config in the ConfigMap is silent (the running pod stays Ready, serves stale config, and the reloader sidecar’s 400s in the log are the only signal). PromQL for the Grafana Cloud alert is in a comment next to the values block.
  • infra/tailscale/acls.json: drop-in replacement preserving the existing tag:tuist-mgmt owner, the wide-open *->* grant (load-bearing for Talos-node access from ops laptops), and the Tailscale-SSH stanza. Adds seven new tags: tuist-ops (env-agnostic human SSH), tuist-k8s-{staging,canary,production} (per-env operator + Connector + ProxyGroup identity), tuist-macmini-{staging,canary,production} (per-env Mac mini fleet). Three env-pair grants scope each env’s operator to that env’s minis on :8080,:9091,:9100. tests block validates the routing.
  • infra/grafana-dashboards/runners.json: three-panel dashboard for VM boot duration backed by the new tart-kubelet histogram.

Per-env wiring: macosFleet.tailscale.enabled: true + macosFleet.tailscaleEgress.enabled: true + magicDNSSuffix in values-managed-common.yaml. Off by default in the chart for self-hosters.

What this is NOT

  • Doesn’t put runner Tart VMs on the tailnet. The runner data path (VM → /api/internal/runners/dispatch) is already authenticated via SA tokens validated through TokenReview. Putting per-job VMs on the tailnet would burn Premium-tier ephemeral minutes or eat tagged-device slots at runner velocity. The host fleet is enough.
  • Doesn’t close public port 22 on Mac minis. The tag:tuist-ops ACL is in place for when we do; flipping the firewall is a separate one-line change after the tailnet proves out.
  • Doesn’t route kubelet → kube-apiserver via the tailnet. Real fix but more moving parts. Separate PR.
  • Doesn’t ship the Grafana Cloud alert rule. There’s no alert-as-code primitive in this repo (alerts live in the Grafana Cloud UI). PromQL + thresholds are documented inline in k8s-monitoring/values.yaml so they’re findable later.

Cost expectations

At the 50-tagged-device cap on Tailscale’s published plans:

  • Today: ~6 minis (xcresult-processor + runners) + ~3 operator/router/proxy pods = ~10 tagged devices.
  • With runners at moderate scale (20 hosts): ~30 tagged devices.
  • 50-device cap is the architectural cliff. Would trigger an “evaluate Headscale” decision before getting locked into an unpublished enterprise quote.

Out-of-band setup before deploy

These are admin-console actions, not code changes. Per env (staging, canary, production):

  • Tailscale admin console: generate an OAuth client (Settings > OAuth clients): scopes Devices > Core > Write + Keys > Auth Keys > Write, allowed tag tag:tuist-k8s-<env>.
  • Tailscale admin console: generate an auth key (Settings > Keys): reusable, ephemeral, tag tag:tuist-macmini-<env>.
  • 1Password in each env’s tuist-k8s-<env> vault: TAILSCALE/auth-key and TAILSCALE_OPERATOR/{client-id,client-secret}.
  • Tailscale ACL: paste infra/tailscale/acls.json into Access Controls.

Post-deploy:

  • Grafana Cloud alert rule on min by (cluster, instance, job) (alloy_config_last_load_successful) == 0 (For: 5m, severity: critical). Full setup steps documented in the chart comment next to the integrations block.

Test plan

  • go build ./... clean across tart-kubelet, macos-host-bootstrap, cluster-api-provider-scaleway-applesilicon.
  • go test ./... clean across all three (+ four new unit tests for reconcileTailscaleEgressService).
  • go vet ./... clean.
  • helm lint + helm template clean for tuist, tailscale-operator, k8s-monitoring with all three env overlays.
  • helm template with default (off) values renders zero Tailscale objects (self-hoster path unaffected).
  • Staging deploy: both minis on the tailnet, ProxyGroup ProxyGroupReady, per-machine Services materialised by the controller, node_exporter + tart-kubelet metrics reachable from a generic cluster Pod through the egress proxy, alloy-metrics scraping both jobs with 0 connection failures, samples landing in Grafana Cloud.
  • Canary rollout (auto-triggered on merge to main).
  • Production rollout (gated on canary acceptance tests).

🤖 Generated with Claude Code

Comments

No GitHub comments yet.