Linux madvise() advice values incorrectly combined via bitwise OR in GCToOSInterface::VirtualReset

[BenV]

Description

As part of our (@neiljohari and @andyblox) investigation into GC heap corruption we discovered that VirtualReset in gcenv.unix.cpp combines MADV_FREE and MADV_DONTDUMP via bitwise OR and passes the result to a single madvise() call. madvise() takes a single advice constant, not a bitmask. The combined value is either rejected or misinterpreted by the kernel.

This change was made here: #95643 in an effort to reduce the number of
syscalls (#87173)

Reproduction Steps

The bug is in this code path:

// gcenv.unix.cpp — GCToOSInterface::VirtualReset

int madviseFlags = 0;

#ifdef MADV_DONTDUMP
    madviseFlags |= MADV_DONTDUMP;   // 16
#endif

#ifdef HAVE_MADV_FREE
    madviseFlags |= MADV_FREE;       // 8
#endif

st = madvise(address, size, madviseFlags);  // passes 24

When both are defined (standard on modern glibc/x86_64 Linux), madviseFlags = 16 | 8 = 24.

From asm-generic/mman-common.h:

Constant	Value
MADV_FREE	8
MADV_DONTDUMP	16
MADV_DONTNEED_LOCKED	24

Note that VirtualReserveInner in the same file correctly uses a standalone madvise(pRetVal, size, MADV_DONTDUMP) call.

Expected behavior

VirtualReset should apply MADV_FREE and MADV_DONTDUMP independently via separate madvise() calls.

Actual behavior

The combined value 24 is passed as a single advice:

• Kernel < 5.18: 24 is unrecognized, madvise() returns EINVAL. Neither advice takes effect. (We were able to confirm this with Ubuntu 22 + Kernel 5.15)
• Kernel >= 5.18: 24 matches MADV_DONTNEED_LOCKED, which immediately discards pages (including locked pages) — different semantics from both MADV_FREE and MADV_DONTDUMP.

This also has the side effect of dropping MADV_DONTDUMP which will lead to extra pages being included in dumps.

Regression?

The change was introduced with #95643

Known Workarounds

No response

Configuration

• .NET 10.0.5
• Affected: All Linux builds where both MADV_DONTDUMP and HAVE_MADV_FREE are defined (tested on Ubuntu 22 with kernel version 5.15)
• Source: src/coreclr/gc/unix/gcenv.unix.cpp — GCToOSInterface::VirtualReset

Other information

The VirtualReset madvise calls should likely be reverted to the behavior in the previous version where two individual calls are made.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[janvorli]

Oh, @BenV thank you for finding this! I was convinced that these were flags.

[BenV]

No problem at all, it took us a while to spot this as well, especially since calls like mmap do use flags 😆

[Static-Codes]

This is very eye opening! I know your research was done on Linux + glibc, however, I was wondering if you believed it was plausible that this behavior would have cascading effects on unix-like systems?

For context, I was having issues with cross-platform sys calls between Linux 6.X + glibc and Darwin 22.X + libc, that I suspect may be caused by this implementation flaw.

[BenV]

This is very eye opening! I know your research was done on Linux + glibc, however, I was wondering if you believed it was plausible that this behavior would have cascading effects on unix-like systems?

For context, I was having issues with cross-platform sys calls between Linux 6.X + glibc and Darwin 22.X + libc, that I suspect may be caused by this implementation flaw.

Quite possibly, as the code paths are the same for Linux/Unix and variants (although they differ based on some predefines such as HAVE_MADV_FREE etc)