Fix Linux madvise() advice values incorrectly combined via bitwise OR in GCToOSInterface::VirtualReset

[Copilot]

Description

VirtualReset was combining MADV_FREE (8) and MADV_DONTDUMP (16) via bitwise OR and passing the result (24) to a single madvise() call. madvise() takes a single advice constant — not a bitmask — so this was either rejected as EINVAL (kernels < 5.18) or silently matched MADV_DONTNEED_LOCKED (kernels ≥ 5.18), a completely different operation.

Fix: Issue two separate madvise() calls, and use posix_madvise(POSIX_MADV_DONTNEED) as a proper fallback only when MADV_FREE is not available:

// Before (broken)
int madviseFlags = 0;
madviseFlags |= MADV_DONTDUMP;  // 16
madviseFlags |= MADV_FREE;      // 8
st = madvise(address, size, madviseFlags);  // passes 24 — invalid

// After (fixed)
#ifdef MADV_DONTDUMP
st = madvise(address, size, MADV_DONTDUMP);  // coredump hint (independent)
#endif

#ifdef HAVE_MADV_FREE
st = madvise(address, size, MADV_FREE);      // primary: page reclaim
#elif defined(HAVE_POSIX_MADVISE)
st = posix_madvise(address, size, POSIX_MADV_DONTNEED);  // fallback when MADV_FREE unavailable
#endif

The posix_madvise(POSIX_MADV_DONTNEED) path is now an #elif chained to #ifdef HAVE_MADV_FREE, making it a proper fallback that only executes when MADV_FREE is unavailable. The previous #if defined(HAVE_POSIX_MADVISE) && !defined(MADV_DONTDUMP) condition was semantically unrelated to that fallback decision. This matches the pattern already used by VirtualReserveInner and VirtualCommitInner in the same file. Introduced by #95643.

Customer Impact

On all Linux builds where both MADV_DONTDUMP and MADV_FREE are defined (i.e., all modern glibc/x86-64 Linux):

• Kernels < 5.18: GC never successfully resets pages to the OS — memory pressure behavior is broken and MADV_DONTDUMP is never applied (reset memory appears in coredumps).
• Kernels ≥ 5.18: MADV_DONTNEED_LOCKED is invoked instead — immediately discarding pages including locked pages, which is semantically incorrect and potentially dangerous.

Regression

Yes — introduced in .NET 9 by #95643 (an optimization to reduce syscall count).

Testing

The bug is a straightforward misuse of a syscall interface. The fix has been manually verified against the Linux madvise(2) man page and kernel headers. No existing automated test infrastructure covers madvise() advice values directly.

Risk

Low. The change is a minimal, targeted fix: replace one combined madvise() call with two separate calls, and restructure the posix_madvise fallback as a proper #elif. No logic changes, no new APIs, no structural changes.

Package authoring no longer needed in .NET 9

IMPORTANT: Starting with .NET 9, you no longer need to edit a NuGet package's csproj to enable building and bump the version.
Keep in mind that we still need package authoring in .NET 8 and older versions.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[janvorli]

LGTM

[Copilot]

Pull request overview

Fixes Linux GCToOSInterface::VirtualReset to avoid passing an invalid/incorrectly-combined advice value to madvise(), ensuring the GC’s “reset pages” hint behaves correctly across kernel versions.

Changes:

• Replace the single madvise() call (with OR-combined advice values) with separate calls for MADV_DONTDUMP and the reclaim hint (MADV_FREE or POSIX fallback).
• Restructure the POSIX fallback as a proper #elif so it is only used when the MADV_FREE path is unavailable.

[Copilot]

HAVE_MADV_FREE doesn't appear to be defined by the GC unix CMake configure step (it isn't in config.gc.h.in, and gc/unix/configure.cmake doesn't check for MADV_FREE). As a result, the MADV_FREE path may be compiled out on all platforms, and this change may not actually exercise the intended madvise(..., MADV_FREE) behavior. Consider either (1) keying this branch off MADV_FREE directly, or (2) adding a configure check and #cmakedefine01 HAVE_MADV_FREE entry so the macro is reliably set when available.

[jkotas]

Valid feedback?

[janvorli]

Yes, apparently this was never defined :-(. @copilot can you please replace HAVE_MADV_FREE by MADV_FREE?