A Python health check that slowly grew container memory: bytecode, negative dentries and slab

Today I learned why the memory metric of our txclf_serve container grew linearly from 29.8% to 44% over three days. The cause was not a leak in the Python code. It was a health check that started a new Python process every 30 seconds.

The chain of causes

1. The official python:3.12-slim image ships the standard library without .pyc files (compiled bytecode).
2. Python normally compiles a module once and stores the result in a pycache directory next to the source. Later processes check the source's modification time and size, then load the .pyc and skip the compile.
3. Our image runs as a non-root app user. The pycache directory of the standard library belongs to root, so the user cannot write there. It can write elsewhere, for example in /tmp.
4. The write fails quietly. Python runs the freshly compiled code anyway and tries again in the next process. The health check therefore compiled the standard library from source every time, and took about 3× longer.
5. Python writes bytecode to a temporary name first, os.cpython-312.pyc.<id>, and then renames it. The <id> is a memory address, so it differs in every process. Each failed attempt asked the kernel for a name it had never seen.

The kernel keeps a dentry cache. A dentry maps (parent directory, name) to an inode, so the kernel does not ask the filesystem again for every path lookup. A negative dentry records that a name does not exist, which speeds up repeated lookups of missing files.

Each failed temporary name left one negative dentry, about 30 KiB per process in total. Nobody ever asked for that name again, so nothing used or aged out the entry. The dentries live in slab memory, the kernel's object cache. They are reclaimable under memory pressure, so this was not a true leak and probably would not have caused an OOM. It still raised the memory metric, which could have tripped the 80% alarm.

I first thought this was the page cache. It is not. The page cache holds file contents, and no file was ever written.

The fix: I compile the standard library once at build time, as root:

RUN python -m compileall -q /usr/local/lib/python3.12

A check in scripts/image-smoke.sh fails the build if the image has no .pyc files. After the deploy the memory graph stayed flat.

Look at it on a host

cat /proc/sys/fs/dentry-state   # 2nd value: unused entries, 5th: negative dentries
slabtop                         # look for the dentry line
grep -E 'anon|file|slab' /sys/fs/cgroup/memory.stat   # inside a cgroup v2 container

A related option is the PYTHONPYCACHEPREFIX variable, which sends bytecode to a writable path. I chose to compile in the image, because the bytecode is then there from the first start.

Linear growth with no change in load usually points to something periodic. Here it was the health check, so the first question next time is "what starts a new process on a timer?"