Apply mitigations to System.Security.Cryptography.Xml components

[eiriktsarpalis]

Apply depth checks to a number of recursive components.
Opt out of using unsafe transforms in EncryptedXml by default.

Co-Authored-By: @PranavSenthilnathan

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR strengthens System.Security.Cryptography.Xml defenses against maliciously deep or recursively constructed XML by adding a configurable recursion-depth limit across several recursive code paths, and by disallowing “dangerous” EncryptedXml transforms by default (with an opt-out AppContext switch).

Changes:

• Add a new DangerousMaxRecursionDepth AppContext setting (default 64; 0 = unlimited) and enforce it in multiple recursive XML processing components.
• Introduce an allow-list of “safe” transform algorithms for EncryptedXml CipherReference processing, defaulting to deny unsafe transforms unless explicitly enabled via AppContext switch.
• Add/adjust tests (including RemoteExecutor-based tests) and add a new embedded XML resource to validate the mitigations.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/libraries/System.Security.Cryptography.Xml/tests/XmlDsigExcC14NTransformTest.cs	Adds depth-limit test coverage for exclusive C14N transform.
src/libraries/System.Security.Cryptography.Xml/tests/XmlDsigC14NTransformTest.cs	Adds depth-limit test coverage for C14N transform (plus RemoteExecutor boundary test).
src/libraries/System.Security.Cryptography.Xml/tests/XmlDecryptionTransformTest.cs	Adds nested EncryptedData depth-limit tests using RemoteExecutor.
src/libraries/System.Security.Cryptography.Xml/tests/System.Security.Cryptography.Xml.Tests.csproj	Embeds a new deep encrypted XML sample; minor formatting fix.
src/libraries/System.Security.Cryptography.Xml/tests/SignedXmlTest.cs	Adds tests for unsafe transforms in EncryptedXml during signature verification and for recursion depth during signing/verifying.
src/libraries/System.Security.Cryptography.Xml/tests/ReferenceTest.cs	Makes digest-method assertions conditional on NET.
src/libraries/System.Security.Cryptography.Xml/tests/EncryptedXmlTests.cs	Adds tests for default-deny unsafe transforms and deep-file depth failures; uses new embedded sample.
src/libraries/System.Security.Cryptography.Xml/tests/EncryptedXmlTest.cs	Updates an existing CipherReference test to validate new transform allow-list behavior (default + AppContext opt-in).
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/XmlDecryptionTransform.cs	Reworks recursive processing to iterative queue + enforces max depth.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/LocalAppContextSwitches.cs	Adds new AppContext switches + helper config readers.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/KeyInfo.cs	Adds recursion-depth accounting around LoadXml.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedXml.cs	Adds safe-transform allow-list enforcement for CipherReference transforms.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedType.cs	Adds thread-static recursion-depth tracking helpers for LoadXml paths.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedKey.cs	Wraps LoadXml in depth increment/decrement.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedData.cs	Wraps LoadXml in depth increment/decrement.
src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/CanonicalizationDispatcher.cs	Adds recursion depth checks during canonicalization writes/hashes.
src/libraries/System.Security.Cryptography.Xml/src/Resources/Strings.resx	Adds a new depth-exceeded error message resource.

[Copilot]

This null-argument test is in XmlDsigC14NTransformTest but it constructs XmlDsigExcC14NTransform. This looks like a copy/paste mistake and means the null behavior of XmlDsigC14NTransform isn’t actually being covered here. Consider changing the constructed type to XmlDsigC14NTransform (and cleaning up the extra space in the generic type argument).

Suggested change

	Assert.Throws< NullReferenceException>(() => new XmlDsigExcC14NTransform().GetDigestedOutput(null));
	Assert.Throws<NullReferenceException>(() => new XmlDsigC14NTransform().GetDigestedOutput(null));

[Copilot]

RSA implements IDisposable; this test creates an RSA instance without disposing it. Please wrap the RSA in a using declaration/statement (or dispose it in a finally) to avoid leaking native handles across the test run.

Suggested change

	RSA key = RSA.Create();
	using RSA key = RSA.Create();

[Copilot]

RSA implements IDisposable; this test assigns SignedXml.SigningKey = RSA.Create() without disposing the key. Please use a using declaration for the RSA and assign the created instance, or otherwise ensure it’s disposed after the test completes.

Suggested change

	signedXml.SigningKey = RSA.Create();
	using RSA rsa = RSA.Create();
	signedXml.SigningKey = rsa;

[Copilot]

RSA implements IDisposable; this helper creates an RSA instance without disposing it. Since these helpers can be called multiple times within a RemoteExecutor process, please ensure the RSA is disposed (e.g., using RSA rsa = RSA.Create(); signedXml.SigningKey = rsa;).

[Copilot]

RSA implements IDisposable; this helper creates an RSA instance without disposing it. Please dispose the RSA (e.g., via using declaration) to avoid leaking handles in the test process.

[jeffhandley]

The remaining test feedback from Copilot looks like it could all be valid, but otherwise looks good to me.