Errors resulting from unicode normalisation

[gertjanvanzwieten]

Bug report

Bug description:

The CPython interpreter appears to apply some form of unicode normalization to variable names, but not to strings, leading to surprising errors as demonstrated in the following code. Though the code contains only the combination d, U+0307 (COMBINING DOT ABOVE) to form a dotted ḋ, some of these get altered to U+1E0B (LATIN SMALL LETTER D WITH DOT ABOVE) resulting in key and type errors. Is this behaviour as intended?

good_dict = {'ḋ': 1}
# {'ḋ': 1} <--- not changed

good_dict['ḋ']
# ok

bad_dict = dict(ḋ = 1)
# {'ḋ': 1} <--- changed

#bad_dict['ḋ'] # uncomment to trigger error
# KeyError: 'ḋ'

def f(ḋ):
    print(ḋ)

f(ḋ=1)
# ok

#f(**good_dict) # uncomment to trigger error
# TypeError: f() got an unexpected keyword argument 'ḋ'

f(**bad_dict)
# ok

CPython versions tested on:

3.13, 3.14, 3.12, 3.11

Operating systems tested on:

Linux

[StanFromIreland]

This is intentional, it is also too late to change this.

Per the Identifiers and keywords section of the Lexical analysis:

All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC.

For more historical information, see PEP 3131. Strings, however, are not subject to normalization, by design. You can see the grammar specification.

[StanFromIreland]

@encukou Since you are working on the lexical analysis section at the moment (or 13 minutes ago at the time of writing IIRC ;-), I think explicitly stating this would avoid future confusion, thoughts?

[gertjanvanzwieten]

Thanks for the quick repy!
Right, I now understand that this is documented behaviour that cannot be changed, but out of curiosity, do you happen to know why normalization was deemed necessary at all? PEP 3131 doesn't mention this to have been a point of discussion. I imaging it is to avoid mysterious errors caused by visually equal but different characters in the code, but it appears to me that in return we now have mysterious errors caused by identical characters in the code.
Anyway, it's easy to work around it once you know. Thanks again!

[encukou]

out of curiosity, do you happen to know why normalization was deemed necessary at all?

Sorry, that predates my involvement. If you want to dig, at that time discussions were on python-list.

[gertjanvanzwieten]

Thanks!