Skip to content

K8SPSMDB-1643: prevent concurrent backup and restore operations #2318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Vinh1507 wants to merge 1 commit into
base: main
Choose a base branch
from
+36 −0
Open

K8SPSMDB-1643: prevent concurrent backup and restore operations #2318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions pkg/controller/perconaservermongodbrestore/logical.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import (
pbmErrors "github.com/percona/percona-backup-mongodb/pbm/errors"

psmdbv1 "github.com/percona/percona-server-mongodb-operator/pkg/apis/psmdb/v1"
"github.com/percona/percona-server-mongodb-operator/pkg/k8s"
"github.com/percona/percona-server-mongodb-operator/pkg/naming"
"github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup"
)

Expand Down Expand Up @@ -59,6 +61,26 @@ func (r *ReconcilePerconaServerMongoDBRestore) reconcileLogicalRestore(
return status, nil
}

leaseActive, err := k8s.IsLeaseActive(ctx, r.client, naming.BackupLeaseName(cluster.Name), cluster.Namespace)
if err != nil {
return status, errors.Wrap(err, "check backup lease")
}
if leaseActive {
log.Info("Waiting for active backup to complete before starting restore.")
Copy link
Copy Markdown
Contributor

@egegunes egegunes Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put the lease name in this log

status.State = psmdbv1.RestoreStateWaiting
return status, nil
}

hasActiveLocks, err := pbmc.HasLocks(ctx)
Copy link
Copy Markdown
Member

@mayankshah1607 mayankshah1607 Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add a new backup.IsBackupLock predicate like we do with the previous pbmc.HasLocks call

Copy link
Copy Markdown
Contributor

@egegunes egegunes Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah otherwise restores would be blocked if oplog slicer for PiTR is running

if err != nil {
return status, errors.Wrap(err, "checking pbm locks")
}
if hasActiveLocks {
log.Info("Waiting for active PBM operation to complete.")
status.State = psmdbv1.RestoreStateWaiting
return status, nil
}
Comment on lines +64 to +82
Copy link
Copy Markdown
Contributor

@egegunes egegunes Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like @mayankshah1607 said, this should be moved to high level restore Reconcile function so it applies to all kinds of restores


log.Info("Starting restore", "backup", backupName)
status.PBMname, err = runRestore(ctx, backupName, pbmc, cr)
status.State = psmdbv1.RestoreStateRequested
Expand Down
14 changes: 14 additions & 0 deletions pkg/k8s/lease.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,20 @@ func AcquireLease(ctx context.Context, c client.Client, name, namespace, holder
return lease, nil
}

func IsLeaseActive(ctx context.Context, c client.Client, name, namespace string) (bool, error) {
lease := new(coordv1.Lease)

if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, lease); err != nil {
if k8serrors.IsNotFound(err) {
return false, nil
}

return false, errors.Wrap(err, "get lease")
}

return lease.Spec.HolderIdentity != nil && *lease.Spec.HolderIdentity != "", nil
}

func ReleaseLease(ctx context.Context, c client.Client, name, namespace, holder string) error {
lease := new(coordv1.Lease)

Expand Down