Pomerium kubectl gateway doesn’t proxy exec/attach/port-forward (connection upgrade dropped)

Symptom

kubectl exec (and by extension attach, port-forward, cp, logs -f) through the per-env Pomerium kubectl gateway (tuist-k8s-<env> contexts → kube-<env>.tuist.dev) fails:

error: unable to upgrade connection: empty server response

Plain REST verbs (get, describe, non-follow logs, and writes under the edit tier) work fine. Only the streaming/upgrade subresources fail.

What it’s not

• Not RBAC / permissions. A permissions gap returns 403. This was reproduced under an active group:tuist-canary-write (edit-tier) JIT elevation, so the identity is authorized — the request reaches the connection-upgrade step and gets an empty (non-101) response.
• Not a SPDY-vs-WebSocket protocol mismatch. The cluster is v1.34 (WebSocket exec available). Forcing WebSocket on the client (KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true) fails identically. So the gateway drops the Connection: Upgrade for both the legacy SPDY and the newer WebSocket exec protocols.

Likely root cause

The exec/attach/portforward subresources require the proxy to hijack the connection and stream bidirectionally. Somewhere in the gateway chain (Pomerium all-in-one → the kube-impersonator reverse-proxy sidecar → apiserver) the upgrade isn’t being forwarded/hijacked. The prime suspect is the kube-impersonator Go sidecar: a stock httputil.ReverseProxy does not hijack connections for arbitrary upgrades, so unless it explicitly handles Connection: Upgrade (or delegates to a streaming proxy) for these subresource paths, the upgrade dies there. Pomerium’s handling of the upgrade is the other place to check.

Impact

No one — human or agent — can kubectl exec / port-forward / attach through the gateway. The only working path today is the break-glass admin kubeconfig (direct to the apiserver, bypassing Pomerium), which is 1Password-biometric-gated and human-only. So routine non-destructive operator work (open a remote console, flip a FunWithFlags, debug a pod) requires break-glass — heavier friction than the design intends for read/console access.

Suggested next steps

1. Trace where the upgrade dies: Pomerium access log (does the exec request arrive? what status?), then the kube-impersonator sidecar (does it see the upgrade request? does it forward it?), then the apiserver.
2. Make the gateway forward connection upgrades for the .../exec, .../attach, .../portforward subresource paths — likely teaching kube-impersonator to detect Connection: Upgrade and hand off to a streaming/hijacking proxy (Go httputil.ReverseProxy forwards WebSocket upgrades since 1.12, but SPDY exec needs explicit handling; alternatively force WS exec end-to-end and ensure the proxy forwards the WS upgrade).
3. Until fixed, document the break-glass kubeconfig as the path for exec/port-forward in infra/k8s/onboarding.md.

Surfaced while trying to flip a FunWithFlags flag on canary to validate the kura runner-cache StorageClass work.

No GitHub comments yet.