Skip to content

[Version 11.0] Feature support for UTF-8 string literals #1610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
RexJaeschke wants to merge 4 commits into
base: draft-v11
Choose a base branch
from
Draft

[Version 11.0] Feature support for UTF-8 string literals #1610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5047856
Support UTF-8 string literals
RexJaeschke Mar 18, 2026
38d5138
Support UTF-8 string literals
RexJaeschke Mar 18, 2026
efd0095
Support UTF-8 string literals
RexJaeschke Mar 18, 2026
5f9225a
fix md formatting
RexJaeschke Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions standard/attributes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -826,7 +826,7 @@ The line number may be affected by `#line` directives ([§6.5.8](lexical-structu

The attribute `System.Runtime.CompilerServices.CallerFilePathAttribute` is allowed on optional parameters when there is a standard implicit conversion ([§10.4.2](conversions.md#1042-standard-implicit-conversions)) from `string` to the parameter’s type.

If a function invocation from a location in source code omits an optional parameter with the `CallerFilePathAttribute`, then a string literal representing that location’s file path is used as an argument to the invocation instead of the default parameter value.
If a function invocation from a location in source code omits an optional parameter with the `CallerFilePathAttribute`, then a UTF-16 string literal representing that location’s file path is used as an argument to the invocation instead of the default parameter value.

The format of the file path is implementation-dependent.

Expand All @@ -836,7 +836,7 @@ The file path may be affected by `#line` directives ([§6.5.8](lexical-structure

The attribute `System.Runtime.CompilerServices.CallerMemberNameAttribute` is allowed on optional parameters when there is a standard implicit conversion ([§10.4.2](conversions.md#1042-standard-implicit-conversions)) from `string` to the parameter’s type.

If a function invocation from a location within the body of a function member or within an attribute applied to the function member itself or its return type, parameters or type parameters in source code omits an optional parameter with the `CallerMemberNameAttribute`, then a string literal representing the name of that member is used as an argument to the invocation instead of the default parameter value. (In the case of a function invocation from a top-level statement ([§7.1.3](basic-concepts.md#713-using-top-level-statements)), the member name is that generated by the implementation.)
If a function invocation from a location within the body of a function member or within an attribute applied to the function member itself or its return type, parameters or type parameters in source code omits an optional parameter with the `CallerMemberNameAttribute`, then a UTF-16 string literal representing the name of that member is used as an argument to the invocation instead of the default parameter value. (In the case of a function invocation from a top-level statement ([§7.1.3](basic-concepts.md#713-using-top-level-statements)), the member name is that generated by the implementation.)

For invocations that occur within generic methods, only the method name itself is used, without the type parameter list.

Expand Down
31 changes: 26 additions & 5 deletions standard/expressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@

> *Example*: For the operation `b * s`, where `b` is a `byte` and `s` is a `short`, overload resolution selects `operator *(int, int)` as the best operator. Thus, the effect is that `b` and `s` are converted to `int`, and the type of the result is `int`. Likewise, for the operation `i * d`, where `i` is an `int` and `d` is a `double`, `overload` resolution selects `operator *(double, double)` as the best operator. *end example*


Check failure on line 301 in standard/expressions.md

View workflow job for this annotation

GitHub Actions / lint

Multiple consecutive blank lines 

standard/expressions.md:301 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2] https://github.com/DavidAnson/markdownlint/blob/v0.40.0/doc/md012.md
**End of informative text.**

#### 12.4.7.2 Unary numeric promotions
Expand Down Expand Up @@ -1542,7 +1542,7 @@

1. If the target of an assignment or method-call argument has type `string`, the expression is processed by the default interpolated string handler, `System.Runtime.CompilerServices.DefaultInterpolatedStringHandler`, and the result has type `string`.
1. If the target of an assignment or method-call argument has type `System.IFormattable` or `System.FormattableString`, a string value is not composed from the interpolated string. Instead an instance of `System.FormattableString` is created.
1. If the target of an assignment or method-call argument has a custom interpolated string handler (§custInterpStrExpHandler) type, then

Check failure on line 1545 in standard/expressions.md

View workflow job for this annotation

GitHub Actions / TOC and Anchor updater

standard/expressions.md#L1545 

TOC002::`§custInterpStrExpHandler` not found

- If the interpolated string contains no interpolations, the expression is processed as if the target type was `string`.
- Otherwise, the expression is processed by the custom interpolated string handler and the result has that custom interpolated string handler’s type.
Expand Down Expand Up @@ -4282,7 +4282,7 @@

For an operation of the form `x + y`, binary operator overload resolution ([§12.4.5](expressions.md#1245-binary-operator-overload-resolution)) is applied to select a specific operator implementation. The operands are converted to the parameter types of the selected operator, and the type of the result is the return type of the operator.

The predefined addition operators are listed below. For numeric and enumeration types, the predefined addition operators compute the sum of the two operands. When one or both operands are of type `string`, the predefined addition operators concatenate the string representation of the operands.
The predefined addition operators are listed below. For numeric and enumeration types, the predefined addition operators compute the sum of the two operands. When one or both operands are of type `string`, or both are of type `ReadOnlySpan<byte>`, the predefined addition operators concatenate the string representation of the operands.

- Integer addition:

Expand Down Expand Up @@ -4332,15 +4332,15 @@
```

At run-time these operators are evaluated exactly as `(E)((U)x + (U)y`).
- String concatenation:
- UTF-16 string concatenation:

```csharp
string operator +(string x, string y);
string operator +(string x, object y);
string operator +(object x, string y);
```

These overloads of the binary `+` operator perform string concatenation. If an operand of string concatenation is `null`, an empty string is substituted. Otherwise, any non-`string` operand is converted to its string representation by invoking the virtual `ToString` method inherited from type `object`. If `ToString` returns `null`, an empty string is substituted.
These overloads of the binary `+` operator perform concatenation of UTF-16 strings. If an operand is `null`, an empty UTF-16 string is substituted. Otherwise, any non-`string` operand that is not a ref struct ([§16.2.3]( structs.md#1623-ref-modifier)) is converted to its UTF-16 string representation by invoking the virtual `ToString` method inherited from type `object`. If `ToString` returns `null`, an empty UTF-16 string is substituted.

> *Example*:
>
Expand Down Expand Up @@ -4369,7 +4369,28 @@
>
> *end example*

The result of the string concatenation operator is a `string` that consists of the characters of the left operand followed by the characters of the right operand. The string concatenation operator never returns a `null` value. A `System.OutOfMemoryException` may be thrown if there is not enough memory available to allocate the resulting string.
The result of the operator is a `string` that consists of the characters of the left operand followed by the characters of the right operand. The string concatenation operator never returns a `null` value. A `System.OutOfMemoryException` may be thrown if there is not enough memory available to allocate the resulting string.
- UTF-8 string concatenation:

```csharp
ReadOnlySpan<byte> operator +(ReadOnlySpan<byte> x, ReadOnlySpan<byte> y);
```

This overload of the binary `+` operator performs concatenation of UTF-8 string literals and the concatenated results thereof (which is much more restrictive than for UTF-16 string concatenation). The operands shall be UTF-8-encoded values.
The result of the operator is a ReadOnlySpan<byte> that consists of the bytes of the left operand followed by the bytes of the right operand. The result may be used directly as an operand to the UTF-8 string concatenation operator.
Copy link
Copy Markdown
Contributor

@KalleOlaviNiemitalo KalleOlaviNiemitalo Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this normative text, it is not clear whether redundant parentheses are allowed.

using System;
public class C {
    public void M() {
        ReadOnlySpan<byte> a
            = (( "A"u8 + "a"u8 ))
            + (( "B"u8 + "b"u8 ));
    }
}


> *Example*:
>
> <!-- Example: {template:"standalone-console", name:"AdditionOperator2", expectedErrors:["CS9047","CS9047"]} -->
> ```csharp
> ReadOnlySpan<byte> sp1 = "ABC"u8 + "DEF"u8; // OK
> ReadOnlySpan<byte> sp2 = sp1 + "DEF"u8; // error
> ReadOnlySpan<byte> sp3 = "ABC"u8 + "DEF"u8 + "123"u8; // OK
> ReadOnlySpan<byte> sp4 = "ABC"u8 + (ReadOnlySpan<byte>)stackalloc byte[]
> { (byte)'D', (byte)'E', (byte)'F', (byte)'\x0' }; // error
> ```
>
> In the case of `sp1`, both operands are UTF-8 string literals. However, once `sp1` is initialized, that UTF-8 pedigree is no longer tracked. That is, `sp1` itself is not seen as being UTF-8 encoded. As such, it is not permitted to be an operand in the case of the initialization of `sp2`. In the initializer for `sp3`, the left pair of operands is evaluated, and as they are both UTF-8 string literals, the result is deemed to also be UTF-8 encoded, so it can further be used as the left operand of the right operator. In the case of `sp4`, while both operands are `ReadOnlySpan<byte>`s, only the left operand is UTF-8 encoded, even though the `Span<byte>` returned by `stackalloc` has the internal form of a UTF-8 string literal (that is, an array of bytes with a null-byte terminator). See [§6.4.5.6](lexical-structure.md#6456-string-literals). *end example*
- Delegate combination. Every delegate type implicitly provides the following predefined operator, where `D` is the delegate type:

```csharp
Expand Down Expand Up @@ -7491,7 +7512,7 @@

Only the following constructs are permitted in constant expressions:

- Literals (including the `null` literal).
- Literals (including the `null` literal, but excluding UTF-8 string literals).
- Constant interpolated strings.
- References to `const` members of class, struct, and interface types.
- References to members of enumeration types.
Expand Down
31 changes: 28 additions & 3 deletions standard/lexical-structure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -906,14 +906,16 @@ A verbatim string literal consists of an `@` character followed by a double-quo

In a verbatim string literal, the characters between the delimiters are interpreted verbatim, with the only exception being a *Quote_Escape_Sequence*, which represents one double-quote character. In particular, simple escape sequences, and hexadecimal and Unicode escape sequences are not processed in verbatim string literals. A verbatim string literal may span multiple lines.

All string literal forms may optionally have a trailing *Utf8_Suffix*. The representation of each form is discussed below.

```ANTLR
String_Literal
: Regular_String_Literal
| Verbatim_String_Literal
;

fragment Regular_String_Literal
: '"' Regular_String_Literal_Character* '"'
: '"' Regular_String_Literal_Character* '"' Utf8_Suffix?
;

fragment Regular_String_Literal_Character
Expand All @@ -929,7 +931,7 @@ fragment Single_Regular_String_Literal_Character
;

fragment Verbatim_String_Literal
: '@"' Verbatim_String_Literal_Character* '"'
: '@"' Verbatim_String_Literal_Character* '"' Utf8_Suffix?
;

fragment Verbatim_String_Literal_Character
Expand All @@ -944,6 +946,10 @@ fragment Single_Verbatim_String_Literal_Character
fragment Quote_Escape_Sequence
: '""'
;

fragment Utf8_Suffix
: 'u8' | 'U8'
;
```

> *Example*: The example
Expand Down Expand Up @@ -976,7 +982,26 @@ fragment Quote_Escape_Sequence
<!-- markdownlint-enable MD028 -->
> *Note*: Since a hexadecimal escape sequence can have a variable number of hex digits, the string literal `"\x123"` contains a single character with hex value `123`. To create a string containing the character with hex value `12` followed by the character `3`, one could write `"\x00123"` or `"\x12"` + `"3"` instead. *end note*

The type of a *String_Literal* is `string`.
A *String_Literal* that does not contain a *Utf8_Suffix* is a ***UTF-16 string literal***, whose type is `string`.

A *String_Literal* that contains a *Utf8_Suffix* is a ***UTF-8 string literal***, whose type is `System.ReadOnlySpan<byte>` (an indexable collection type), and whose value contains a UTF-8 byte representation of the string. A null terminator (a byte with value zero) is placed beyond the last byte in memory (and outside the length of the `ReadOnlySpan<byte>`) in order to support scenarios that expect null-terminated byte strings. A UTF-8 string literal is not a constant. A UTF-8 string literal without its *Utf8_Suffix* shall be valid UTF-16. (For example, `"\uDC00\uDD00"u8` is ill-formed as one low surrogate cannot be followed by another.)
Copy link
Copy Markdown
Contributor

@KalleOlaviNiemitalo KalleOlaviNiemitalo Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add a new subclause within 16.5.15 Safe context constraint, saying that when a UTF-8 string literal has a ReadOnlySpan<byte> value, the safe-context of that value is caller-context. That would allow it to be returned out of the function, like so:

using System;
public class C {
    public ReadOnlySpan<byte> M() {
        return "xyz"u8;
    }
}

Concatenation of UTF-8 string literals would then follow the safe-context rule in 16.5.15.5 Operators. As the safe-context of both operands would be caller-context, the safe-context of the result would be the same, without having to be specified separately for this case.

Copy link
Copy Markdown
Contributor

@KalleOlaviNiemitalo KalleOlaviNiemitalo Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following would then likewise be valid:

public class C {
    public ref readonly byte M() {
        return ref "xyz"u8[0];
    }
}


> *Note*: While every UTF-8 string literal is a `ReadOnlySpan<byte>`, not every `ReadOnlySpan<byte>` represents a UTF-8 string literal. See the description of UTF-8 string concatenation in [§12.13.5](expressions.md#12135-addition-operator). *end note*
<!-- markdownlint-disable MD028 -->

<!-- markdownlint-enable MD028 -->
> *Note*: As `ReadOnlySpan<byte>` is a ref struct type, a UTF-8 string literal cannot be converted to `object` or used as a type parameter ([§16.2.3]( structs.md#1623-ref-modifier)). *end note*
Copy link
Copy Markdown
Contributor

@KalleOlaviNiemitalo KalleOlaviNiemitalo Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allows ref struct is added in C# 13, and this PR is for C# 11, so that part is OK…

But the wording is imprecise. A string literal is syntax rather than a type so it couldn't be used as a type parameter (or rather a type argument) anyway.

<!-- markdownlint-disable MD028 -->

<!-- markdownlint-enable MD028 -->
> *Example*: Here are examples of each form of string literal:
>
> | **Encoding** | **Type** | **Regular String Literal** | **Verbatim String Literal** | **Raw String Literal** |
> |--------------|----------------------|---------------------|--------------------|--------------------|
> | UTF-16 | `string` | `"Hello"` | `@"Hello"` | `"""Hello"""` |
> | UTF-8 | `ReadOnlySpan<byte>` | `"Hello"u8` | `@"Hello"u8` | `"""Hello"""u8` |
>
> *end example*

Each string literal does not necessarily result in a new string instance. When two or more string literals that are equivalent according to the string equality operator ([§12.15.8](expressions.md#12158-string-equality-operators)), appear in the same assembly, these string literals refer to the same string instance.

Expand Down
Loading