K8SPSMDB-1643: prevent concurrent backup and restore operations#2318
Open
Vinh1507 wants to merge 1 commit intopercona:mainfrom
Open
K8SPSMDB-1643: prevent concurrent backup and restore operations#2318Vinh1507 wants to merge 1 commit intopercona:mainfrom
Vinh1507 wants to merge 1 commit intopercona:mainfrom
Conversation
Vinh1507
requested review from
egegunes,
gkech,
hors,
mayankshah1607,
nmarukovich,
oksana-grishchenko and
pooknull
as code owners
|
|
Collaborator
commit: 8c4144e |
mayankshah1607
requested changes
Apr 24, 2026
Member
mayankshah1607
left a comment
mayankshah1607
left a comment
There was a problem hiding this comment.
Thanks for the quick fix. I believe we need to perform the same checks for physical restore as well
| return status, nil | ||
| } | ||
|
|
||
| hasActiveLocks, err := pbmc.HasLocks(ctx) |
Member
There was a problem hiding this comment.
Lets add a new backup.IsBackupLock predicate like we do with the previous pbmc.HasLocks call
Contributor
There was a problem hiding this comment.
yeah otherwise restores would be blocked if oplog slicer for PiTR is running
egegunes
requested changes
Apr 24, 2026
Comment on lines
+64
to
+82
| leaseActive, err := k8s.IsLeaseActive(ctx, r.client, naming.BackupLeaseName(cluster.Name), cluster.Namespace) | ||
| if err != nil { | ||
| return status, errors.Wrap(err, "check backup lease") | ||
| } | ||
| if leaseActive { | ||
| log.Info("Waiting for active backup to complete before starting restore.") | ||
| status.State = psmdbv1.RestoreStateWaiting | ||
| return status, nil | ||
| } | ||
|
|
||
| hasActiveLocks, err := pbmc.HasLocks(ctx) | ||
| if err != nil { | ||
| return status, errors.Wrap(err, "checking pbm locks") | ||
| } | ||
| if hasActiveLocks { | ||
| log.Info("Waiting for active PBM operation to complete.") | ||
| status.State = psmdbv1.RestoreStateWaiting | ||
| return status, nil | ||
| } |
Contributor
There was a problem hiding this comment.
like @mayankshah1607 said, this should be moved to high level restore Reconcile function so it applies to all kinds of restores
| return status, nil | ||
| } | ||
|
|
||
| hasActiveLocks, err := pbmc.HasLocks(ctx) |
Contributor
There was a problem hiding this comment.
yeah otherwise restores would be blocked if oplog slicer for PiTR is running
| return status, errors.Wrap(err, "check backup lease") | ||
| } | ||
| if leaseActive { | ||
| log.Info("Waiting for active backup to complete before starting restore.") |
Contributor
There was a problem hiding this comment.
I would put the lease name in this log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGE DESCRIPTION
Problem:
When a restore CR is created while a backup is running (or just starting), the operator
blocks indefinitely on "Waiting for restore metadata".
Cause:
The original lock guard only checked for a PITR lock (
IsPITRLock). A running backupholds a different lock type, so the check always returned
falseduring an activebackup and
CmdRestorewas sent unconditionally. PBM returns aConcurrentOpError,writes nothing to
pbmRestores, and the operator waits forever.Solution:
Three guards are applied in sequence in
pkg/controller/perconaservermongodbrestore/logical.gobeforeCmdRestoreis sent:CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability
coordination.k8s.io/v1Lease which has been stable since Kubernetes 1.14.