Registry: archive construction is not deterministic, so any resync of a non-passthrough package breaks pins

Rebuilding a registry version is not a safe operation today: archive construction is not reproducible, so any resync of an affected package publishes a new checksum and breaks every Package.resolved pinned to it.

This is independent of any incident. It is the mechanism that turns a routine resync into a broken pin.

Scope corrected. This issue originally attributed the problem to the clone path only, on the strength of two builds of one package through the repack path coming out byte-identical. Production has since shown the repack path is also non-deterministic. Both paths share zip_directory/2, so the proposed fix is unchanged — but the affected population is much larger than “packages with submodules”.

Measurement

Only passthrough is reproducible, because those bytes are GitHub’s rather than ours.

Clone path, measured directly: two clones three seconds apart produced different archives. git clone writes checkout time to the filesystem and zip -r records what it finds. The divergence is at second granularity, so a unzip -l listing can show identical minutes while the bytes differ.

Repack path, observed in production on Adyen/adyen-3ds2-ios@2.3.0 — no .gitmodules, 5 symlink entries in the zipball, so it routes through unzip → zip -r -y:

15:23:33  attempt 1 uploads  etag fd0480f1…-4  17,613,886 bytes  → verification fails
15:23:59  attempt 2 uploads  etag db59e41c…-4  17,613,886 bytes  → verification passes

Same size, different bytes, 26 seconds apart, same input. Restoring mtimes from the zipball — which is why an earlier two-build test of a simpler package came out stable — is not sufficient on its own. The remaining variance is most likely entry ordering: zip -r walks the directory, and after resolve_symlinks/1 rewrites entries the traversal order need not be stable between runs.

Why this is worse than one changed checksum

A verification failure triggers an Oban retry, and the retry rebuilds from scratch. Because the build is not reproducible, the retry produces different bytes again rather than reproducing the ones that failed to verify. So a transient failure does not converge — each attempt uploads a fresh archive and leaves the previous one behind.

2.3.0 did not fail once. It failed, rebuilt differently, uploaded again, and left two distinct objects in storage seconds apart. Any retry-based recovery on a non-reproducible build has this shape.

Fix

Normalize the inputs to zip in zip_directory/2, which both non-passthrough paths already share:

• mtimes normalized to a fixed timestamp
• entry order fixed by feeding a sorted file list rather than relying on directory traversal order
• -X to drop the extended-timestamp and uid/gid extra fields

Verified on the clone path: two clones three seconds apart produced byte-identical archives with those three changes. The repack path shares the same function, so it inherits the fix — worth re-verifying against a symlink-bearing package before closing, since that is exactly the assumption that made the original scope wrong.

Rollout consideration

Switching to deterministic packing changes the bytes for every non-passthrough package one final time. That is a checksum change, so it has to go out under the explicit override added in #12193 rather than silently — accepting one last break in order to stop all future ones.

Why this is a prerequisite for repairs

The repair in #12194 regenerates archives. Regenerating any non-passthrough package before this lands bakes in another arbitrary checksum that the next rebuild will change again — and, as above, a verification retry can leave several arbitrary checksums behind rather than one.

Determining whether a package is affected is a cheap check: it is not affected only if its zipball contains no symlink entries and the repository has no .gitmodules.

Context and the full census are in #12191.

No GitHub comments yet.