You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This article describes how to configure the [change event streaming (CES)](overview.md) feature introduced in [!INCLUDE [sssql25-md](../../../includes/sssql25-md.md)], Azure SQL Database, and Azure SQL Managed Instance.
20
20
@@ -515,6 +515,29 @@ The following table lists the stored procedures, system functions, and DMVs that
515
515
|[sys.sp_help_change_feed_table_groups](../../system-stored-procedures/sp-help-change-feed-table-groups.md)| Returns metadata that is used to configure change event streaming groups. |
516
516
|[sys.sp_help_change_feed_table](../../system-stored-procedures/sp-help-change-feed-table.md)| Provides the status and information of the streaming group and table metadata for change event streaming. |
517
517
518
+
## Transaction log growth
519
+
520
+
Because message delivery is guaranteed, the transaction log for a database that has CES enabled can continue to grow. Log truncation is prevented as long as there are CES changes to stream from the log. Once the transaction log size reaches the max defined limit, writes to the database fail.
521
+
522
+
To prevent this in Azure SQL Database and Azure SQL Managed Instance, when the transaction log size approaches the max defined limit, Microsoft might disable CES or kill long running transactions. For unmanaged SQL Server instances such as on-premises or on SQL Server on Azure VMs, you're responsible for monitoring the transaction log size and ensuring it doesn't reach the max defined limit, and manually disabling CES or killing long running transactions if needed.
523
+
524
+
Once CES is disabled, or a long running transaction is killed, the transaction log is truncated to free up space. You must manually enable CES again after it has been disabled, or retry any long running transactions that were killed. Data changes made while CES was disabled aren't captured. Only changes made after CES is restarted are streamed.
525
+
526
+
The following is a list of typical scenarios that can lead to transaction log growth with CES enabled:
527
+
528
+
- Persistent errors. CES retries to send a message that is rejected repeatedly and CES can't continue. Reasons for persistent errors that can lead to rejected messages include:
529
+
- Network issues or misconfiguration.
530
+
- Credential misconfiguration.
531
+
- Misconfigured max message size that the destination rejects.
532
+
- The destination throttles the incoming events. For example, Azure Event Hubs rate limits based on its SKU.
533
+
- Long running transactions that generate a lot of log records and prevent log truncation.
534
+
535
+
To ensure smooth operations, monitor the size of the transaction log and [CES delivery errors](../../system-dynamic-management-views/sys-dm-change-feed-errors.md) regularly.
536
+
537
+
## Performance
538
+
539
+
On SQL Server, Azure SQL Managed Instance and Azure SQL Database elastic pools, you can enable CES on multiple databases. Each CES-enabled database consumes server resources and competes with other server workload. Make sure that your server is adequately resourced for the expected workload and monitor the performance of your server and databases regularly.
540
+
518
541
## Limitations
519
542
520
543
Change event streaming (CES) has the following limitations:
@@ -524,7 +547,8 @@ Change event streaming (CES) has the following limitations:
-[Permissions in the source database](#permissions-in-the-source-database)
550
+
-[Permissions in the source database](#permissions-in-the-source-database-and-data-residency)
551
+
-[Networking and connectivity](#networking-and-connectivity)
528
552
529
553
### Azure SQL Database
530
554
@@ -548,7 +572,7 @@ The following limitations apply when using CES with Azure SQL Database:
548
572
- CES only supports databases configured with the full recovery model.
549
573
- CES doesn't support databases configured with [Fabric Mirrored Databases for SQL Server](/fabric/database/mirrored-database/sql-server), [transactional replication](../../replication/transactional/transactional-replication.md), [change data capture](../about-change-data-capture-sql-server.md), or [Azure Synapse Link](/azure/synapse-analytics/synapse-link/sql-synapse-link-overview). [Change tracking](../about-change-tracking-sql-server.md) is supported on databases configured with CES.
550
574
- CES can only stream from writable primary databases. Secondary databases that are part of Always On availability groups or that use the [Managed Instance link](/azure/azure-sql/managed-instance/managed-instance-link-feature-overview) can't be configured as streaming sources.
551
-
- You can't enable CES on views or materialized views.
575
+
- You can't enable CES on views or indexed views.
552
576
553
577
### Table-level limitations
554
578
@@ -584,11 +608,16 @@ The following limitations apply when using CES with Azure SQL Database:
584
608
-**xml**
585
609
- User-defined types (UDT)
586
610
587
-
### Permissions in the source database
611
+
### Permissions in the source database and data residency
588
612
589
613
- For row-level security, CES emits changes from all rows, regardless of user permissions.
590
614
- Dynamic data masking doesn't apply to data sent through CES. Data is streamed unmasked, even if masking is configured.
591
615
- CES doesn't emit events related to object-level permission changes (for example, granting permissions to specific columns).
616
+
- CES streams data to the configured destination if the network configuration allows it. If the destination is in a different region, CES streams the data across regions. Ensure this complies with your data residency and compliance requirements.
617
+
618
+
### Networking and connectivity
619
+
620
+
- Currently, CES can only stream to Azure Event Hubs public endpoints. Service endpoints and private endpoints aren't currently supported.
The following are answers to questions about the Change Event Streaming (CES) feature for [!INCLUDE [sssql25-md](../../../includes/sssql25-md.md)], Azure SQL Database, and Azure SQL Managed Instance.
answer: No. CES can't be enabled on a database enabled with CDC.
69
69
- question: Can CES be used with Fabric Mirrored Databases for SQL Server?
70
70
answer: No. CES isn't supported with [Fabric Mirrored Databases for SQL Server](/fabric/database/mirrored-database/sql-server).
71
-
- question: If CES stops, does it resume from where it left off?
71
+
- question: What happens if CES is disabled? Does it resume from where it left off when reenabled?
72
72
answer: |
73
-
If CES is manually stopped, it doesn't resume automatically. You must manually restart CES to continue streaming events. Changes made while CES was stopped won't be captured. Only changes made after CES is restarted will be streamed.
74
-
CES guarantees *at least once* delivery. If CES fails to deliver a message, it continues to retry until the message is successfully delivered, or CES is manually stopped. If CES isn't stopped, failed deliveries prevent log truncation until the error condition is resolved, and the message is successfully delivered. If CES is stopped manually, log truncation resumes, and events that occur during the downtime aren't captured.
73
+
If CES is disabled, it can't be resumed. You must manually reconfigure CES to continue streaming events. Data changes made while CES was disabled aren't captured. Only changes made after CES is restarted are streamed.
74
+
CES guarantees *at least once* delivery. If CES fails to deliver a message, it continues to retry until the message is successfully delivered, or CES is disabled. If CES isn't disabled, failed deliveries prevent log truncation until the error condition is resolved, and the message is successfully delivered. If CES is disabled, log truncation resumes, and events that occur during the downtime aren't captured.
75
75
76
+
- question: What happens if I do a restore (or another type of migration) of a database that was configured for CES?
77
+
answer: After restore (or other types of database migration), CES is disabled and stopped for the database. You have to reconfigure it to start streaming again.
76
78
- question: Can CES do an initial snapshot ("seed") of existing data in tables when CES is enabled?
77
79
answer: No. CES currently only streams changes to data that happen after CES is enabled. It doesn't stream data in a table that exists before CES is enabled.
78
80
- question: How can I monitor CES activity (for example, events processed, errors)?
This article describes the JSON format of a CloudEvents message that is streamed from SQL Server to Azure Event Hubs when using the [change event streaming (CES)](overview.md) feature introduced in [!INCLUDE [sssql25-md](../../../includes/sssql25-md.md)], Azure SQL Database, and Azure SQL Managed Instance
19
19
@@ -25,14 +25,14 @@ Events emitted by change event streaming follow the [CloudEvents](https://github
25
25
26
26
## Related specifications and resources
27
27
28
-
When applicable, the descriptions in this section are taken from [CloudEvent specification](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md), which includes additional details.
28
+
When applicable, the descriptions in this section are taken from [CloudEvent specification](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md), which includes more details.
29
29
30
30
## Attributes
31
31
32
32
-**`specversion`**:
33
33
- Data type: String
34
34
- Required CloudEvent attribute
35
-
- The version of the CloudEvents specification which the event uses. This enables the interpretation of the context.
35
+
- The version of the CloudEvents specification that the event uses. This enables the interpretation of the context.
36
36
37
37
-**`type`**
38
38
- Data type: String
@@ -82,7 +82,7 @@ When applicable, the descriptions in this section are taken from [CloudEvent spe
82
82
-**`finalsegment`**
83
83
- Data type: Boolean
84
84
- Extension attribute
85
-
- Tells if this segment is final segment of the sequence. This field is always present and helps to identify if a SQL event that was too large for configured max message size was split into sub-events.
85
+
- Tells if this segment is final segment of the sequence. This field is always present and helps to identify if a SQL event that was too large for configured max message size was split into subevents.
0 commit comments