[FLINK-39422] [REST][docs] Fix OpenAPI spec for `SlotSharingGroupId`, `MetricCollectionResponseBody`, and `SerializedThrowable` custom serializers by nikhilsu · Pull Request #27903 · apache/flink

nikhilsu · 2026-04-08T07:56:35Z

What is the purpose of the change

This PR fixes three Flink request/response DTOs that use custom Jackson serializers, while their OpenAPI specs are generated from the DTO schema. This leads to a mismatch between the actual wire format and what the generated spec describes, breaking downstream clients:

SlotSharingGroupId: Serialized as a hex string by SlotSharingGroupIDSerializer (added in FLINK-20090), but the spec incorrectly defines it as an object with bytes, lowerPart, upperPart fields because overrideIdSchemas() does not include SlotSharingGroupId.
MetricCollectionResponseBody: Serialized as a raw JSON array [{"id": "metricName", "value": "1"}] by a custom Serializer (annotated with @JsonSerialize), but the spec incorrectly defines it as an object with a metrics property.
SerializedThrowable: Serialized with three fields (class, stack-trace, serialized-throwable) by SerializedThrowableSerializer, but the spec override only included serialized-throwable, omitting the class and stack-trace string fields.

None of these changes break backward compatibility - the previous schemas never matched the actual wire format, so any client generated from the spec would have already failed to decode these fields correctly.

Affected endpoints:

GET /jobs/:jobid (SlotSharingGroupId in JobDetailsInfo.JobVertexDetailsInfo)
GET /jobmanager/metrics
GET /jobs/:jobid/metrics
GET /jobs/:jobid/vertices/:vertexid/metrics
GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
GET /jobs/:jobid/vertices/:vertexid/watermarks
GET /taskmanagers/:taskmanagerid/metrics
GET /taskmanagers/metrics
Any endpoint returning SerializedThrowable (savepoint/checkpoint failure responses)

Brief change log

Added SlotSharingGroupId to OpenApiSpecGenerator.overrideIdSchemas() as type: string, pattern: [0-9a-f]{32}
Refactored MetricCollectionResponseBody to extend ArrayList<Metric>, removing the custom serializer/deserializer. The wire format is identical (JSON array), but the class structure now matches, so the spec generator produces the correct schema without a manual override.
Added class and stack-trace fields to the SerializedThrowable schema override in OpenApiSpecGenerator, using public constants from SerializedThrowableSerializer.

Verifying this change

This change is already covered by existing tests, such as OpenApiSpecGeneratorTest and MetricCollectionResponseBodyTest.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: yes (MetricCollectionResponseBody custom serializer/deserializer removed, replaced by extending ArrayList<Metric> which Jackson serializes identically as a JSON array)
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

flinkbot · 2026-04-08T08:05:49Z

CI report:

a5e9660 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

nikhilsu · 2026-04-08T08:07:20Z

cc: @zentol @gaborgsomogyi

nikhilsu · 2026-04-10T23:21:32Z

@RocMarshal do you know who can review this PR?

RocMarshal

Hi, @nikhilsu Thanks a lot for the contribution.
Left a few of comments.
Pls take a look~

RocMarshal · 2026-04-11T03:36:49Z

+     * serializer that writes the metrics collection as a raw JSON array, not as an object with a
+     * "metrics" field.
+     */
+    private static void overrideMetricCollectionSchema(final OpenAPI openApi) {


IIUC, The method is not need.

Because the MetricCollectionResponseBody is not a collection like array or list.

If the definition of MetricCollectionResponseBody like follows, maybe the change is need.

Pls let me know your opinion.

Hey @RocMarshal thanks for the review!

The reason you need this is because MetricCollectionResponseBody uses a custom Serializer which omits the metric key and serializes a response object as a JSON array.

Here's how a sample response looks like if you call the GET /v1/jobs/${jobid}/metrics endpoint. Notice, that there is not metric key here, MetricCollectionResponseBody is serialized to a JSON array:

$ curl 'http://localhost:8081/v1/jobs/bc6df938e80006c05639a139e6cf716f/metrics?get=numRecordsInPerSecond,numRecordsOutPerSecond,numRestarts' \ -H 'Accept: application/json, text/plain, */*' \ -H 'Accept-Language: en-US,en;q=0.9' \ --insecure | jq . [ { "id": "numRestarts", "value": "0" } ]

However, the problem is that the OpenAPI Spec Generator uses the class schema (through reflections) and generates API specs where the response includes the metric key: https://github.com/apache/flink/blob/release-1.20/docs/static/generated/rest_v1_dispatcher.yml#L2721-L2727

This causes a mismatch in OpenAPI spec generated Flink clients. The client expects a metric key in the response but it actually does not exist.

Sorry for ignoring the details and Thank you @nikhilsu for your clarification.
Got it~

RocMarshal · 2026-04-11T15:22:24Z

+     * serializer that writes the metrics collection as a raw JSON array, not as an object with a
+     * "metrics" field.
+     */
+    private static void overrideMetricCollectionSchema(final OpenAPI openApi) {


Hi, @nikhilsu
In my limited reading,Perhaps implementing the MetricCollectionResponseBody class in the style of
org.apache.flink.runtime.rest.messages.ConfigurationInfo would be a good choice.
Main considerations are as follows:

We wouldn’t need to modify the OpenApiSpecGenerator class when introducing similar new classes in the future

There is already a precedent with the implementation of ConfigurationInfo, and we wouldn’t need to explicitly introduce serialization and deserialization processors

In the REST API’s HTML and YML documentation, the definition of the return type would be relatively more detailed. For example, an object type would be refined to something like Array<Metric>

this should not break the existing JSON field structure.

Hey @RocMarshal, this is a good suggestion. It would be much cleaner to extend MetricCollectionResponseBody from an ArrayList<Metric>.

However, after this change, endpoints no longer reference MetricCollectionResponseBody - they inline the array. Generated clients would need regeneration. The wire format is the same (JSON array), but the type in generated code changes from MetricCollectionResponseBody to List[Metric].

Here is the diff of the spec.

This would break downstream client code. But it might be ok as this endpoint would have never worked in the first place.

I've pushed the change. Let me know what you think.

Hi, @nikhilsu In my limited reading,Perhaps implementing the MetricCollectionResponseBody class in the style of org.apache.flink.runtime.rest.messages.ConfigurationInfo would be a good choice. Main considerations are as follows:

We wouldn’t need to modify the OpenApiSpecGenerator class when introducing similar new classes in the future

There is already a precedent with the implementation of ConfigurationInfo, and we wouldn’t need to explicitly introduce serialization and deserialization processors

In the REST API’s HTML and YML documentation, the definition of the return type would be relatively more detailed. For example, an object type would be refined to something like Array<Metric>

Hi, @1996fanrui Could you help take a look for this strategy ?

I guess the generation of the REST API documentation might be related to the release quality of 2.3, although the impact is probably relatively small.

Thank you very much.

Btw, @RocMarshal now that we've updated MetricCollectionResponseBody to mirror the structure of ConfigurationInfo, for the sake of consistency we should also do the same in case of:
AggregatedMetricsResponseBody.java

Hi， @nikhilsu
If the current implementation in this AggregatedMetricsResponseBody.java file does not cause the previous issues when generating the corresponding documentation, then I would prefer to keep it as is.

WDYTA?

I'm pretty sure it does as it uses a custom serializer and deserializer.

Yup, you can see it's being generated as an object when response will be an array:
https://github.com/apache/flink/blob/release-2.2/docs/static/generated/rest_v1_dispatcher.yml#L1620-L1621

nikhilsu · 2026-04-12T06:06:36Z

FYI: changes to *.html and *.yml were made as part of running ./mvnw package -Dgenerate-rest-docs -pl flink-docs -am -nsu -DskipTests 2>&1

RocMarshal

Thanks @nikhilsu .
LGTM on the whole, just left a few of comments.
BTW, IIUC,
RuntimeRestAPIStabilityTest will be failure, maybe you could have a try on running it.
then do some adjustments based on the hint output of the unit-test result.
like running RuntimeRestAPIStabilityTest with -Dgenerate-rest-snapshot

RocMarshal · 2026-04-15T02:33:14Z

-
-            jsonGenerator.writeObject(metricCollectionResponseBody.getMetrics());
-        }
+        return this;


Suggested change

return this;

return List.copyOf(this);

Actually let me make it simpler and remove this method altogether. It's mostly used only in tests.

@nikhilsu One of the reasons for keeping this method is to maintain compatibility with previous behavior as much as possible.

Ok sounds good.

@RocMarshal IMHO we should return this. Returning a copy would be a behavior change - the old implementation returned this.metrics which was the same collection reference passed to the constructor, not a defensive copy. Callers that mutated the result would have mutated the internal state before too.

More importantly, the class itself is an ArrayList now. Returning a copy from getMetrics() while the object itself is mutable via add()/remove() would be inconsistent - you'd have two ways to access the same data with different mutability guarantees.

…hema overrides

nikhilsu · 2026-04-15T05:36:37Z

Thanks @nikhilsu . LGTM on the whole, just left a few of comments. BTW, IIUC, RuntimeRestAPIStabilityTest will be failure, maybe you could have a try on running it. then do some adjustments based on the hint output of the unit-test result. like running RuntimeRestAPIStabilityTest with -Dgenerate-rest-snapshot

Ran mvn test -pl flink-runtime-web -Dtest="RuntimeRestAPIStabilityTest" -Dgenerate-rest-snapshot -Denforcer.skip=true and mvn test -pl flink-runtime-web -Dtest="RuntimeRestAPIStabilityTest" -Denforcer.skip=true and both pass.

The test pass because the REST snapshots records "type": "any" for metric endpoints because Jackson's JsonSchemaGenerator (used by CompatibilityRoutines) couldn't introspect the old custom serializer. The compatibility check https://github.com/apache/flink/blob/release-2.2/flink-runtime/src/test/java/org/apache/flink/runtime/rest/compatibility/CompatibilityRoutines.java#L218 when the old type is any, so the snapshot doesn't need updating.

nikhilsu marked this pull request as ready for review April 8, 2026 07:59

github-actions bot added the community-reviewed PR has been reviewed by the community. label Apr 8, 2026

nikhilsu changed the title ~~[hotfix][docs] Add SlotSharingGroupId to OpenAPI spec generator ID schema overrides~~ [hotfix][docs] Fix OpenAPI spec for SlotSharingGroupId and MetricCollectionResponseBody custom serializers Apr 8, 2026

nikhilsu force-pushed the fix-slotsharinggroupid-openapi-schema branch from a89e2bd to 6b7ff1e Compare April 8, 2026 20:58

nikhilsu changed the title ~~[hotfix][docs] Fix OpenAPI spec for SlotSharingGroupId and MetricCollectionResponseBody custom serializers~~ [hotfix][docs] Fix OpenAPI spec for SlotSharingGroupId, MetricCollectionResponseBody, and SerializedThrowable custom serializers Apr 8, 2026

RocMarshal self-assigned this Apr 11, 2026

RocMarshal reviewed Apr 11, 2026

View reviewed changes

nikhilsu force-pushed the fix-slotsharinggroupid-openapi-schema branch from e684820 to 5beedd0 Compare April 12, 2026 06:05

nikhilsu commented Apr 12, 2026

View reviewed changes

RocMarshal requested changes Apr 15, 2026

View reviewed changes

nikhilsu added 9 commits April 14, 2026 22:18

[hotfix][docs] Add SlotSharingGroupId to OpenAPI spec generator ID sc…

1dbc4a9

…hema overrides

Regen docs

f056835

Override MetricCollectionSchema

baabd9a

Regen Docs

cbe596a

Add spec override for SerializedThrowable

4930eac

Refactor MetricCollectionResponseBody to inherit from an ArrayList

1f22ed5

Regen docs

46d50b0

Minor tweaks

acdfde9

Update AggregatedMetricsResponseBody to extend from ArrayList

27a3ea4

nikhilsu force-pushed the fix-slotsharinggroupid-openapi-schema branch from e8fc938 to 27a3ea4 Compare April 15, 2026 05:18

Randomize

a5e9660

nikhilsu requested a review from RocMarshal April 15, 2026 06:04

Conversation

nikhilsu commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Affected endpoints:

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

nikhilsu commented Apr 8, 2026

Uh oh!

nikhilsu commented Apr 10, 2026

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RocMarshal Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikhilsu Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RocMarshal Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikhilsu commented Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nikhilsu commented Apr 8, 2026 •

edited

Loading

flinkbot commented Apr 8, 2026 •

edited

Loading

RocMarshal Apr 11, 2026 •

edited

Loading

nikhilsu Apr 15, 2026 •

edited

Loading

RocMarshal Apr 15, 2026 •

edited

Loading