Skip to content

Delete the old environment before setting the status to NEW#4414

Open
lubaihua33 wants to merge 3 commits intomainfrom
baihua/rg_clean
Open

Delete the old environment before setting the status to NEW#4414
lubaihua33 wants to merge 3 commits intomainfrom
baihua/rg_clean

Conversation

@lubaihua33
Copy link
Copy Markdown
Collaborator

@lubaihua33 lubaihua33 commented Apr 14, 2026

This change addresses the issue where environments were not cleaned up when the Environment retry count was greater than zero.

Description

This change addresses the issue where environments were not cleaned up when the Environment retry count was greater than zero.
Previously, when a retry was triggered, only the latest environment was deleted, while older environments remained uncleared. This resulted in leftover resources after multiple retries.
To resolve this issue, the _delete_environment_task has been added during retry operations to ensure that all previously created environments are properly cleaned up.

Related Issue

Before the fix
The related logs are:

2026-04-14 02:23:26.056[67164][INFO] lisa.env[generated_0].deploy creating or updating resource group: [lisa-None-20260414-022033-118-e0]
2026-04-14 02:23:28.331[67164][INFO] lisa.env[generated_0].deploy Creating Resource group: 'lisa-None-20260414-022033-118-e0'
2026-04-14 02:23:31.994[67164][DEBUG] lisa.env[generated_0].deploy validating deployment
2026-04-14 02:23:43.276[67164][DEBUG] lisa.env[generated_0].deploy found storage account: lisaswestus36a28e896
2026-04-14 02:23:43.277[67164][INFO] lisa.env[generated_0].deploy resource group 'lisa-None-20260414-022033-118-e0' deployment is in progress...
2026-04-14 02:44:36.079[67164][DEBUG] lisa.env[generated_0].deploy checking panic...
2026-04-14 02:44:36.083[67164][DEBUG] lisa.env[generated_0].deploy checking root filesystem mount failure...
2026-04-14 02:44:36.089[67164][INFO] lisa.runner[0] Retrying... (Attempt 1/2)
2026-04-14 02:44:36.096[64088][DEBUG] lisa.[azure].prepare[generated_0] allowed locations: ['westus3']
...
2026-04-14 02:44:41.205[67164][INFO] lisa.env[generated_0].deploy creating or updating resource group: [lisa-None-20260414-022033-118-e0-1]
2026-04-14 02:44:43.946[67164][INFO] lisa.env[generated_0].deploy Creating Resource group: 'lisa-None-20260414-022033-118-e0-1'
2026-04-14 02:44:47.818[67164][DEBUG] lisa.env[generated_0].deploy validating deployment
2026-04-14 02:44:59.625[67164][DEBUG] lisa.env[generated_0].deploy found storage account: lisaswestus36a28e896
2026-04-14 02:44:59.625[67164][INFO] lisa.env[generated_0].deploy resource group 'lisa-None-20260414-022033-118-e0-1' deployment is in progress...
2026-04-14 03:05:57.405[67164][DEBUG] lisa.env[generated_0].deploy checking panic...
2026-04-14 03:05:57.411[67164][DEBUG] lisa.env[generated_0].deploy checking root filesystem mount failure...

Only the newest environment was deleted, while older environment lisa-None-20260414-022033-118-e0 is leftover.
2026-04-14 03:27:54.743[67164][INFO] lisa.[azure].del[generated_0] deleting resource group: lisa-None-20260414-022033-118-e0-1, wait: False

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Refactoring
  • Documentation update

Checklist

  • Description is filled in above
  • No credentials, secrets, or internal details are included
  • Peer review requested (if not, add required peer reviewers after raising PR)
  • Tests executed and results posted below

Test Validation

After the fix, all old environments are deleted as expected.

2026-04-14 04:01:19.710[67552][INFO] lisa.[azure].del[generated_0] deleting resource group: lisa-None-20260414-033738-838-e0, wait: False
2026-04-14 04:23:43.158[67552][INFO] lisa.[azure].del[generated_0] deleting resource group: lisa-None-20260414-033738-838-e0-1, wait: False

Key Test Cases:

Impacted LISA Features:

Tested Azure Marketplace Images:

Test Results

Image VM Size Result
PASSED / FAILED / SKIPPED

This is to fix the issue that the resources are not cleaned when
retrying to deploy a new environment
Copilot AI review requested due to automatic review settings April 14, 2026 05:58
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes retry cleanup so that previously created environments are deleted before resetting status to NEW, preventing leaked resources across multiple retry attempts.

Changes:

  • Trigger _delete_environment_task when _need_retry(environment) is true (before resetting status to NEW).
  • Update the env-retry selftest expectations to reflect additional delete operations during retries.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
lisa/runners/lisa_runner.py Adds environment deletion during retry handling prior to resetting status to NEW.
selftests/runners/test_lisa_runner.py Updates expected deleted-environment assertions for the retry flow.

Comment thread lisa/runners/lisa_runner.py
Comment thread selftests/runners/test_lisa_runner.py Outdated
@LiliDeng LiliDeng closed this Apr 15, 2026
@LiliDeng LiliDeng reopened this Apr 15, 2026
@github-actions
Copy link
Copy Markdown

✅ AI Test Selection — PASSED

1 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1
Test case details
Test Case Status Time (s) Message
smoke_test (lisa_0_0) ✅ PASSED 29.914

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 16, 2026 00:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread selftests/runners/test_lisa_runner.py Outdated
@github-actions
Copy link
Copy Markdown

✅ AI Test Selection — PASSED

1 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1
Test case details
Test Case Status Time (s) Message
smoke_test (lisa_0_0) ✅ PASSED 39.122

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 16, 2026 00:39
@github-actions
Copy link
Copy Markdown

✅ AI Test Selection — PASSED

1 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1
Test case details
Test Case Status Time (s) Message
smoke_test (lisa_0_0) ✅ PASSED 35.824

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants