You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Plugin EP docs] Split development and usage docs into separate pages, document some 1.24 API additions (#27196)
### Description
<!-- Describe your changes. -->
Split development-and-usage.md into development.md and usage.md.
Add some initial documentation for plugin EP APIs introduced in 1.24.
Preview:
https://edgchen1.github.io/onnxruntime/docs/execution-providers/plugin-ep-libraries/
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
More reorganizing. Start documenting 1.24 additions.
# Plugin Execution Provider Library Development and Usage
9
+
# Developing a Plugin Execution Provider Library
11
10
{: .no_toc }
12
11
13
-
This page provides a reference for the APIs necessary to develop and use plugin EP libraries with ONNX Runtime.
12
+
This page provides a reference for how to develop plugin EP libraries with ONNX Runtime.
14
13
15
14
## Contents
16
15
{: .no_toc }
@@ -19,13 +18,15 @@ This page provides a reference for the APIs necessary to develop and use plugin
19
18
{:toc}
20
19
21
20
## Creating a plugin EP library
22
-
A plugin EP is built as a dynamic/shared library that exports the functions `CreateEpFactories()` and `ReleaseEpFactory()`. ONNX Runtime calls `CreateEpFactories()` to obtain one or more instances of `OrtEpFactory`. An `OrtEpFactory` creates `OrtEp` instances and specifies the hardware devices supported by the EPs it creates.
21
+
A plugin EP is built as a dynamic/shared library that exports the C functions `CreateEpFactories()` and `ReleaseEpFactory()`. ONNX Runtime calls `CreateEpFactories()` to obtain one or more instances of `OrtEpFactory`. An `OrtEpFactory` creates `OrtEp` instances and specifies the hardware devices supported by the EPs it creates.
23
22
24
-
The ONNX Runtime repository includes a [sample plugin EP library](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/autoep/library), which is referenced in the following sections.
23
+
The ONNX Runtime repository includes a few [sample plugin EP libraries](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/autoep/library), which are referenced in the following sections.
25
24
26
25
### Defining an OrtEp
27
26
An `OrtEp` represents an instance of an EP that is used by an ONNX Runtime session to identify and execute the model operations supported by the EP.
28
27
28
+
The [`OrtEp`](https://onnxruntime.ai/docs/api/c/struct_ort_ep.html) API struct is defined in [`onnxruntime_ep_c_api.h`](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_ep_c_api.h).
29
+
29
30
The following table lists the **required** variables and functions that an implementer must define for an `OrtEp`.
30
31
31
32
<table>
@@ -53,6 +54,18 @@ The following table lists the **required** variables and functions that an imple
The following table lists the **optional** functions that an implementor may define for an `OrtEp`. If an optional `OrtEp` function is not defined, ONNX Runtime uses a default implementation.
88
+
The following table lists the **required** functions that an implementer must define for an `OrtEp` that provides operator kernels with a kernel registry.
89
+
90
+
<table>
91
+
<tr>
92
+
<th>Field</th>
93
+
<th>Summary</th>
94
+
<th>Example implementation</th>
95
+
</tr>
96
+
97
+
<tr>
98
+
<td>GetKernelRegistry</td>
99
+
<td>
100
+
101
+
Gets the execution provider's kernel registry, if any.<br/><br/>
102
+
A kernel registry contains kernel creation information for operator kernels supported by an EP.
The following table lists the **optional** functions that an implementer may define for an `OrtEp`. If an optional `OrtEp` function is not defined, ONNX Runtime uses a default implementation.
76
111
77
112
<table>
78
113
<tr>
@@ -153,11 +188,22 @@ The compatibility information string can be used with <code>OrtEpFactory::Valida
153
188
<td></td>
154
189
</tr>
155
190
191
+
<tr>
192
+
<td>IsConcurrentRunSupported</td>
193
+
<td>
194
+
Gets whether the execution provider supports concurrent run calls made on the session.<br/><br/>
195
+
If not implemented, ORT assumes that concurrent runs are supported.
196
+
</td>
197
+
<td></td>
198
+
</tr>
199
+
156
200
</table>
157
201
158
202
### Defining an OrtEpFactory
159
203
An `OrtEpFactory` represents an instance of an EP factory that is used by an ONNX Runtime session to query device support, create allocators, create data transfer objects, and create instances of an EP (i.e., an `OrtEp`).
160
204
205
+
The [`OrtEpFactory`](https://onnxruntime.ai/docs/api/c/struct_ort_ep_factory.html) API struct is defined in [`onnxruntime_ep_c_api.h`](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_ep_c_api.h).
206
+
161
207
The following table lists the **required** variables and functions that an implementer must define for an `OrtEpFactory`.
162
208
163
209
<table>
@@ -261,6 +307,41 @@ This is use to create a synchronization stream for the <code>OrtMemoryDevice</co
Check for known incompatibility reasons between a hardware device and this execution provider.<br/><br/>
314
+
This function allows an execution provider to check if a specific hardware device is compatible with the execution provider. The EP can set specific incompatibility reasons via the <code>OrtDeviceEpIncompatibilityDetails</code> parameter using <code><ahref="https://onnxruntime.ai/docs/api/c/struct_ort_ep_api.html#a6f089e88e76dc58c9bf1e7f5102fb6d7">OrtEpApi::DeviceEpIncompatibilityDetails_SetDetails</a></code>.
Create an <code>OrtExternalResourceImporterImpl</code> for external resource import.<br/><br/>
323
+
This is used to create an external resource importer that enables zero-copy import of external GPU memory (e.g., D3D12 shared resources) and synchronization primitives (e.g., D3D12 timeline fences).<br/><br/>
324
+
EPs that support external resource import (via CUDA, HIP, Vulkan, or D3D12 APIs) can implement this to allow applications to share GPU resources without copies.
As shown in the following sequence diagram, registering a plugin EP library causes ONNX Runtime to load the library and
320
-
call the library's `CreateEpFactories()` function. During the call to `CreateEpFactories()`, ONNX Runtime determines the subset
321
-
of hardware devices supported by each factory by calling `OrtEpFactory::GetSupportedDevices()` with all hardware devices that
322
-
ONNX Runtime discovered during initialization.
323
-
324
-
The factory returns `OrtEpDevice` instances from `OrtEpFactory::GetSupportedDevices()`.
325
-
Each `OrtEpDevice` instance pairs a factory with a hardware device that the factory supports.
326
-
For example, if a single factory instance supports both CPU and NPU, then the call to `OrtEpFactory::GetSupportedDevices()` returns two `OrtEpDevice` instances:
327
-
- ep_device_0: (factory_0, CPU)
328
-
- ep_device_1: (factory_0, NPU)
329
-
330
-
<br/>
331
-
<palign="center"><imgwidth="100%"src="../../../images/plugin_ep_sd_lib_reg.png"alt="Sequence diagram showing registration and unregistration of a plugin EP library"/></p>
332
-
333
-
### Session creation with explicit OrtEpDevice(s)
334
-
The application code below uses the API function [SessionOptionsAppendExecutionProvider_V2](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a285a5da8c9a63eff55dc48e4cf3b56f6) to add an EP from a library to an ONNX Runtime session.
335
-
336
-
The application first calls [GetEpDevices](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a52107386ff1be870f55a0140e6add8dd) to get a list of `OrtEpDevices`
337
-
available to the application. Each `OrtEpDevice` represents a hardware device supported by an `OrtEpFactory`.
338
-
The `SessionOptionsAppendExecutionProvider_V2` function takes an array of `OrtEpDevice` instances as input, where all `OrtEpDevice` instances refer to the same `OrtEpFactory`.
As shown in the following sequence diagram, ONNX Runtime calls `OrtEpFactory::CreateEp()` during session creation in order to create an instance of the plugin EP.
373
-
374
-
<br/>
375
-
<palign="center"><imgwidth="100%"src="../../../images/plugin_ep_sd_appendv2.png"alt="Sequence diagram showing session creation with explicit ep devices"/></p>
376
-
377
-
### Session creation with automatic EP selection
378
-
The application code below uses the API function [SessionOptionsSetEpSelectionPolicy](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a2ae116df2c6293e4094a6742a6c46f7e) to have ONNX Runtime automatically select an EP based on the user's policy (e.g., PREFER_NPU).
379
-
If the plugin EP library registered with ONNX Runtime has a factory that supports NPU, then ONNX Runtime may select an EP from that factory to run the model.
<palign="center"><imgwidth="100%"src="../../../images/plugin_ep_sd_autoep.png"alt="Sequence diagram showing session creation with automatic EP selection"/></p>
Unregister an EP library with ORT. Caller <b>MUST</b> ensure there are no <code>OrtSession</code> instances using the EPs created by the library before calling this function.
0 commit comments