Skip to content

Add stale GCS test bucket cleaner - #40386

Open
goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:add-gcs-temp-bucket-cleaner
Open

goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:add-gcs-temp-bucket-cleaner

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Adds cleanup for GCS buckets leaked by GcsUtil integration tests.

Before:

  • Tests created randomized apache-beam-temp-bucket-<UUID> buckets.
  • A failed or interrupted teardown could leave those buckets indefinitely.
  • The scheduled cleanup workflow did not handle these buckets.

After:

  • Lists buckets only from apache-beam-testing using the dedicated prefix.
  • Revalidates the complete apache-beam-temp-bucket-<UUIDv4> name before deletion.
  • Deletes only buckets older than 24 hours.
  • Defaults to dry-run; the scheduled Gradle task explicitly passes --delete.
  • Removes objects from leaked buckets before deleting them.
  • Continues processing after individual failures and reports failures at the end.
  • Runs the regression tests before the scheduled cleanup task.

Validation:

  • python3 -m unittest test_stale_cleaner.py test_stale_gcs_buckets_cleaner.py — 29 tests passed.
  • ./gradlew :beam-test-tools:testStaleGcsBucketsCleaner — passed.
  • ./gradlew :beam-test-tools:removeStaleGcsBuckets --dry-run — Gradle wiring validated.
  • Python compilation and git diff --check passed.
  • Independent correctness and security reviews found no remaining P0, P1, or P2 issues.

CHANGES.md is not updated because this only changes internal test infrastructure.

Fixes #40347

@github-actions github-actions Bot added the infra label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @kennknowles added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Add a GCS temp bucket cleaner to .test-infra/tools to prevent bucket leaks

1 participant