kura: anon RSS grows monotonically — RocksDB allocates through glibc, whose per-thread arenas retain freed memory

Summary

kura_process_resident_anon_bytes (and therefore kura_process_resident_memory_bytes) grows monotonically on write-active Kura nodes and never comes back down. The growth is entirely outside the Rust global allocator — it is RocksDB’s C++ allocations landing in glibc malloc, where per-thread arenas retain freed memory and effectively never return it to the OS.

Nothing is currently failing, but the growth is invisible to Kura’s own memory accounting and is not reclaimable by Kura’s memory-pressure trims, so the failure mode when it does bite is bad (see Impact).

Evidence

The growth is not in jemalloc

Kura sets jemalloc as the global allocator and tunes it aggressively for reclamation in kura/src/main.rs:

b"background_thread:true,max_background_threads:1,dirty_decay_ms:4000,muzzy_decay_ms:4000\0"

That works. On a node whose anon RSS climbed several hundred MB over 36h, kura_jemalloc_resident_bytes stayed flat within a few MB the whole time, oscillating up and down. The quantity anon_rss - jemalloc_resident accounted for 100% of the rise.

Across nodes spanning a ~45x range in total anon RSS, the jemalloc figure varies only within a narrow band. Every Rust-side cache is bounded and steady over the same window — manifest cache pinned at its cap, snapshot cache empty, manifest index flat in entry count, RocksDB block cache at its cap, write buffer stable.

The growth is glibc arenas

Parsing /proc/<pid>/smaps on a production node shows the textbook glibc new_heap() signature: 64 MiB-aligned mappings, each split into rw-p <used> + ---p <remainder> summing to exactly 64 MiB.

On the node inspected:

• 42 such arenas, holding ~79% of total anonymous RSS
• plus [heap] (main arena, sbrk)
• everything else was jemalloc’s extents, matching kura_jemalloc_resident_bytes

The accounting closes to within a few MB.

It is retention, not live data

Comparing several nodes:

Two details make this conclusive:

1. Arena count barely varies between idle and busy nodes (~30–63 in both cases). What explodes is per-arena residency — roughly 0.1 MB/arena when idle vs ~15 MB/arena on the busiest node.
2. The busiest node by anon RSS had the lowest RocksDB self-reported live memory of the write-active group. Residency tracks peak allocation-burst history, not live data.

Root cause

kura/Cargo.toml builds RocksDB without the jemalloc feature:

rocksdb = { version = "0.24.0", default-features = false, features = ["bindgen-runtime", "lz4"] }

So RocksDB’s C++ allocations go to glibc malloc, outside both:

• kura_jemalloc_* metrics (hence invisible to our own accounting), and
• the dirty_decay_ms / muzzy_decay_ms tuning in kura/src/main.rs.

glibc creates per-thread arenas on lock contention, capped at 8 * nproc. Each arena retains its own free lists; only the top of an arena’s heap is ever trimmed, so memory freed after a compaction or flush burst mostly stays resident.

Compounding factors:

• The container observes nproc = 32 (the pod has a CPU request but no cgroup CPU quota, and glibc reads _SC_NPROCESSORS_ONLN, not the cgroup). That gives a cap of 256 arenas x 64 MiB. Only a fraction are in use so far, so there is a lot of room left to grow.
• MALLOC_ARENA_MAX and MALLOC_TRIM_THRESHOLD are not set anywhere in the repo.

Worth noting RocksDB itself is configured sensibly — store.rs sets cache_index_and_filter_blocks(true) and pin_l0_filter_and_index_blocks_in_cache(true), and max_open_files is bounded — so the classic unbounded table-reader growth is already mitigated. This is allocator retention, not a logical RocksDB leak.

Impact

Nothing is broken today. kura_memory_pressure_state has been Normal on every production node for the last 14 days, and there have been no OOM kills.

The reason is that the fleet rolls often enough to mask it — no node currently has more than ~36h of uptime. That is luck, not design.

The real hazard is what happens if a write-heavy node lives long enough to reach the soft limit:

• Kura’s pressure trims free Rust-side objects. jemalloc already returns those promptly — kura_jemalloc_resident_bytes is flat and healthy.
• The trims cannot touch the glibc arenas, which is where the memory actually is.
• Result: degraded cache hit rate and continued RSS growth, ending in an OOM kill.

This is the same unreclaimable-memory-latches-pressure-high shape as the silent manifest-index zeroing issue.

We have already observed a node reach ~96% of its soft limit before being recycled.

Proposed fixes

1. MALLOC_ARENA_MAX=2 (or 4) on the Kura container — size:XS

One env var, no rebuild, immediately caps the fragmentation surface. Cheapest possible A/B against the rest of the fleet, and it confirms the diagnosis outright.

2. Enable the jemalloc feature on the rocksdb dependency — size:S

rocksdb = { version = "0.24.0", default-features = false, features = ["bindgen-runtime", "lz4", "jemalloc"] }

The real fix: RocksDB then allocates through the same jemalloc that already has background reclaim and 4s decay. Bonus, kura_jemalloc_resident_bytes becomes a true measure of process memory instead of a minority slice of it. Needs care that the prefixed/unprefixed jemalloc symbol setup in kura/src/main.rs still resolves correctly.

3. Alert on kura_process_resident_anon_bytes / kura_memory_soft_limit_bytes — size:XS

kura_memory_pressure_state only moves once a node is already at the soft limit, so there is currently zero lead time. A warn at ~60% would give hours of warning.

Note for other services

Sizing allocator arenas from host core count while the pod requests a fraction of a CPU is the generic container/glibc mismatch. Any other glibc-linked service in these clusters is exposed to the same pattern and is worth a look.

No GitHub comments yet.