PEP 782: Add PyBytesWriter C API

[vstinner]

Hi,

After multiple iterations on the API, I decided to write down a PEP for the PyBytesWriter C API. It's easier to understand the API with its documentation, examples, and discussions around it. A single document (the PEP) should help the discussion.

-> Read PEP 782: Add PyBytesWriter C API <-

The API is now based on sizes rather than pointers, even if two functions using pointers are provided for convenience.

I plan to submit the PEP to the Steering Council, but first I would like to get the opinion of the C API Working Group.

The API was discussed in length, see the PEP for all links to prior discussions. The most recent thread: https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617.

Abstract

Add a new PyBytesWriter C API to create bytes objects.

Soft deprecate PyBytes_FromStringAndSize(NULL, size) and _PyBytes_Resize() APIs. These APIs treat an immutable bytes object as a mutable object. They remain available and maintained, don't emit deprecation warning, but are no longer recommended when writing new code.

Vote

• @encukou (abstains for now)
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @zooba

I abstain from voting on my own PEP :-)

[zooba]

I'm happy with this proposal.

Every time I look at the high level APIs I feel like I want them to return the number of bytes written. I know it's easy to get it another way, but it just feels right to me, particularly for Format. But I'm not intending to block the proposal for it.

[vstinner]

Every time I look at the high level APIs I feel like I want them to return the number of bytes written. I know it's easy to get it another way, but it just feels right to me, particularly for Format. But I'm not intending to block the proposal for it.

I would prefer to return int (-1 on error, 0 on success) instead of Py_ssize_t (-1 on error, number of written bytes on success) for PyBytesWriter_WriteBytes() and PyBytesWriter_Format(). The API doesn't require to know the written bytes.

IMO the API is easier to use and more "regular" (with other recently added functions) using int (-1 on error, 0 on success). For example, the PyUnicodeWriter API also returns an int, rather that the number of written bytes.

I know it's easy to get it another way

Yeah, you can easily compute the number of written bytes by calling PyBytesWriter_GetSize() before and after calling these functions.

Using the PyUnicodeWriter API, it's not possible to compute the number of written bytes and I'm not aware of any use case which needs this number.

[encukou]

I read through it quickly and didn't find a major issue. I abstain from the vote, so I don't block it in case I don't manage to review in time.

[zooba]

@vstinner I probably wouldn't hold up submitting to the SC, they might decide to accept it without necessarily getting another WG vote on it (since we've discussed it pretty thoroughly, and I'm sure the SC knows C as well as we do).

If they pass it back to the WG, then we can aggressively ping the silent members until they review and respond 😄

[vstinner]

Ping @erlend-aasland @mdboom @serhiy-storchaka

[vstinner]

@serhiy-storchaka asked me to run benchmarks:

• Old private API vs new public API: there is basically no performance change
• PyBytes_FromStringAndSize(NULL, length) versus PyBytesWriter_Create(): PyBytesWriter_Create() is around 10 ns slower which means between 1.01x and 1.32x slower depending on the buffer size and the function. PyBytesWriter is a wrapper built on top of PyBytes_FromStringAndSize(NULL, length) internally, so it's hard to beat its performance.

[zooba]

Those differences don't worry me. (I'm assuming it's a flat 10ns difference, independent of buffer size.)

[serhiy-storchaka]

Well, it looks like the overhead is having less of an impact than I feared. Please do some additional benchmarks (especially for encoding), and if the result doesn't bring any unpleasant surprises, I have no objections.

[vstinner]

Please do some additional benchmarks (especially for encoding)

Hi, you can find more benchmarks at: python/cpython#131681 (comment)

[vstinner]

Thanks for voting. I submitted PEP 782 to the steering council: python/steering-council#290.