kura: mmap residency tests fail on virtio-fs (Kata) runners — mincore reports sub-page files non-resident

Summary

Three of Kura’s mmap unit tests fail when run under bazel test on the tuist-linux runners but pass on ubuntu-latest. The root cause is not the Bazel sandbox, the compiler, memory pressure, or virtio-fs in general. It is a specific virtio-fs behavior:

On virtio-fs, mincore(2) reports a file’s final partial page (a page that extends past EOF because the file is smaller than one page) as not resident, even immediately after write + fsync. Pages that are fully backed by file data report resident normally.

tuist-linux runners are Kata Containers QEMU/Firecracker microVMs whose container filesystem is virtio-fs (infra/runners-controller, infra/linux-runner-image). ubuntu-latest is a normal VM with ext4.

Kura’s map_file_region (kura/src/mmap.rs) gates mmap serving on mincore residency (mapping_is_resident). The three failing tests map sub-page files, so on virtio-fs their single mapped page (which extends past EOF) reports non-resident → map_file_region returns Ok(None) → the residency assertions panic.

Affected tests

All three map files smaller than a 4 KiB page.

---- mmap::tests::maps_unaligned_file_regions stdout ----
panicked at src/mmap.rs: freshly written region should be page-cache resident
---- store::tests::mmap_artifact_bytes_is_opportunistic_under_memory_pressure stdout ----
panicked at src/store.rs: normal memory pressure should permit mmap serving
test result: FAILED. 257 passed; 3 failed

Reproduction

A standalone Rust app using Kura’s exact residency code (memmap2 0.9.10 + libc::mincore, mapping_is_resident copied verbatim) reproduces it deterministically.

Cargo.toml:

[package]
name = "mincore_probe_rs"
version = "0.0.0"
edition = "2021"
publish = false

[dependencies]
libc = "0.2.177"
memmap2 = "0.9.10"

src/main.rs (residency check is verbatim from kura/src/mmap.rs):

use std::fs::OpenOptions;
use std::io::Write;
use std::os::raw::c_void;
use memmap2::{Mmap, MmapOptions};

fn page_size() -> Option<usize> {
    let size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
    usize::try_from(size).ok().filter(|s| s.is_power_of_two())
}

fn mapping_residency(mmap: &Mmap) -> (i32, usize, usize) {
    let len = mmap.len();
    if len == 0 { return (0, 0, 0); }
    let page_size = page_size().unwrap();
    let addr = mmap.as_ptr() as usize;
    let aligned = addr & !(page_size - 1);
    let total = (addr - aligned) + len;
    let pages = total.div_ceil(page_size);
    let mut vec = vec![0u8; pages];
    let rc = unsafe { libc::mincore(aligned as *mut c_void, total, vec.as_mut_ptr().cast()) };
    (rc, vec.iter().filter(|p| *p & 1 == 1).count(), pages)
}

fn probe(dir: &str, file_size: usize, offset: u64, len: u64) {
    let path = format!("{dir}/.probe.bin");
    let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(&path).unwrap();
    (&file).write_all(&vec![0xABu8; file_size]).unwrap();
    file.sync_all().unwrap();
    let mmap = unsafe { MmapOptions::new().offset(offset).len(len as usize).map(&file).unwrap() };
    let (rc, resident, pages) = mapping_residency(&mmap);
    println!("file_size={file_size:>7} offset={offset} len={len} -> rc={rc} resident={resident}/{pages} {}",
        if rc == 0 && resident == pages { "Some(RESIDENT)" } else { "None(NOT resident -> PANIC)" });
    drop(mmap);
    let _ = std::fs::remove_file(&path);
}

fn main() {
    let dir = std::env::args().nth(1).unwrap_or_else(|| "/tmp".into());
    probe(&dir, 16, 3, 8);          // the exact failing case
    probe(&dir, 1, 0, 1);
    probe(&dir, 4096, 3, 8);
    probe(&dir, 4096, 0, 4096);
    probe(&dir, 256 * 1024, 0, 256 * 1024);
}

Run it on both runner types (cargo run --release -- /tmp). Workflow matrix snippet:

jobs:
  mincore-probe:
    runs-on: ${{ matrix.runner }}
    strategy:
      matrix:
        runner: [ubuntu-latest, tuist-linux]
    steps:
      - uses: actions/checkout@v4
      - uses: jdx/mise-action@v4.0.1
        with: { install_args: "rust" }
      - run: |
          cd path/to/mincore_probe_rs
          findmnt -no FSTYPE --target /tmp
          cargo run --quiet --release -- /tmp

Observed output

The threshold is exactly at the page boundary: file < 4096 B → reported non-resident on virtio-fs; file >= 4096 B → resident. mincore returns rc=0 (success) in all cases — it just clears the resident bit for the partial-past-EOF page on virtio-fs.

Ruled out (with evidence)

• virtio-fs in general — full-page files report resident on virtio-fs.
• Bazel sandbox / user namespace — a cc_test running the probe inside the real Bazel linux-sandbox on virtio-fs reported resident; running under unshare --user did too.
• Memory pressure — deterministic on tiny files; reproduces with a warm cache and no compilation.
• Compiler / libclang — rustc is pinned identically; the pure mmap test executes no C-compiled code.
• jemalloc — set only in src/main.rs, so the lib test binary uses the system allocator.

Impact

• Production is unaffected (not a correctness bug). Kura’s data dir is a real block-storage PVC (/var/cache/kura), where mincore works, and map_file_region returning None falls back to the always-correct streaming reader. The only effect, if Kura ran with a virtio-fs data dir, is that mmap serving would silently disable for sub-page artifacts — a performance characteristic, never wrong bytes.
• The tests are non-hermetic for sub-page files on virtio-fs. CI was moved to ubuntu-latest to unblock, but the underlying gate behavior on virtio-fs remains.

Suggested fixes

1. Harden the tests (lowest risk). Use >= 1 page fixtures in the three tests, or gate the hard residency assertion behind a runtime capability probe, so they don’t depend on partial-page mincore behavior. Doesn’t change production behavior.

2. Make mapping_is_resident robust to the partial last page (deeper fix). Don’t require the file’s final partial (past-EOF) page to be reported resident — only require the fully-backed pages. For a sub-page file that means the gate would permit mmap serving even when mincore is silent on that page; at worst this risks a single-page fault, which is negligible. This makes the gate behave correctly on virtio-fs too.

3. Skip the mmap fast path for very small artifacts. mmap serving for a handful of bytes is pure overhead; serving sub-page artifacts inline/through the reader sidesteps the quirk and is arguably better regardless of filesystem.

4. Document that the mmap-serving fast path is disabled for sub-page artifacts on virtio-fs-backed data dirs, and keep block storage as the recommended/default KURA_DATA_DIR backing (already the production default).

5. CI: keep Kura’s Bazel jobs on ubuntu-latest (already done), or mount a non-virtio-fs volume (tmpfs/block) for $TEST_TMPDIR on the Kata runners if these are ever moved back.

A combination of (1) or (2) plus (4) addresses both the test flakiness and the underlying behavior.