mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-06-13 23:36:45 +03:00
S3: delay empty folder cleanup to prevent Spark write failures (#8970)
* S3: delay empty folder cleanup to prevent Spark write failures (#8963) Empty folders were being cleaned up within seconds, causing Apache Spark (s3a) writes to fail when temporary directories like _temporary/0/task_xxx/ were briefly empty. - Increase default cleanup delay from 5s to 2 minutes - Only process queue items that have individually aged past the delay (previously the entire queue was drained once any item triggered) - Make the delay configurable via filer.toml: [filer.options] s3.empty_folder_cleanup_delay = "2m" * test: increase cleanup wait timeout to match 2m delay The empty folder cleanup delay was increased to 2 minutes, so the Spark integration test needs to wait longer for temporary directories to disappear. * fix: eagerly clean parent directories after empty folder deletion After deleting an empty folder, immediately try to clean its parent rather than relying on cascading metadata events that each re-enter the 2-minute delay queue. This prevents multi-minute waits when cleaning nested temporary directory trees (e.g. Spark's _temporary hierarchy with 3+ levels would take 6m+ vs near-instant). Fixes the CI failure where lingering _temporary parent directories were not cleaned within the test's 3-minute timeout.
This commit is contained in:
@@ -70,7 +70,8 @@ print("WRITE_COUNT=" + str(count))
|
||||
"issue-8285/output/_temporary/0/",
|
||||
"issue-8285/output/_temporary/0/_temporary/",
|
||||
}
|
||||
lingering := waitForObjectsToDisappear(t, env, "test", temporaryCandidates, 35*time.Second)
|
||||
// Empty folder cleanup has a 2m default delay + 30s processor interval
|
||||
lingering := waitForObjectsToDisappear(t, env, "test", temporaryCandidates, 3*time.Minute)
|
||||
if len(lingering) > 0 {
|
||||
t.Fatalf("issue #8285 regression detected: lingering temporary directories: %v", lingering)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user