mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-06-13 23:36:45 +03:00
9859f5fafc978b8b91e51962b0147e71a4546ab7
13473 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
9859f5fafc |
build(docker): upgrade all Alpine packages in final image (#9070)
build(docker): apply full apk upgrade in final image to pick up security patches Trivy flagged CVE-2026-28390 (libcrypto3/libssl3) on the published image because the final stage only upgraded zlib. Broaden to `apk upgrade --no-cache` so all Alpine security fixes land at build time. |
||
 dependabot[bot]
|
ad2aa3135c |
build(deps): bump rand from 0.9.2 to 0.9.4 in /seaweedfs-rdma-sidecar/rdma-engine (#9065)
build(deps): bump rand in /seaweedfs-rdma-sidecar/rdma-engine Bumps [rand](https://github.com/rust-random/rand) from 0.9.2 to 0.9.4. - [Release notes](https://github.com/rust-random/rand/releases) - [Changelog](https://github.com/rust-random/rand/blob/0.9.4/CHANGELOG.md) - [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.2...0.9.4) --- updated-dependencies: - dependency-name: rand dependency-version: 0.9.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
34b4ecc631 |
fix(mount): serialize hard-link mutations on HardLinkId (#9064)
* fix(mount): serialize hard-link mutations on HardLinkId syncHardLinkSiblings stamps every sibling of a hard-link to authoritativeEntry.HardLinkCounter, and the caller computes that value as entry.HardLinkCounter - 1 (Unlink) or entry.HardLinkCounter + 1 (Link) from a cached entry read before the filer mutation. With concurrent Unlinks on different links of the same file, both callers observe the same pre-decrement counter, the filer's atomic blob decrement lands correctly, but both then stamp their siblings to counter-1 — leaving the mount metacache one higher than the authoritative blob. Serialize Link and Unlink on string(HardLinkId) via a new hardLinkLockTable on WFS, and re-load the entry under the lock so the second caller sees the updated sibling counter its predecessor just wrote before computing its own delta. First-link races (empty HardLinkId on the source) are a separate pre-existing issue and are not addressed here. Full pjdfstest suite still passes (235 files, 8803 tests). * fix(mount): abort on stale pre-lock entry after HardLinkId lock Review follow-up: if maybeLoadEntry fails after acquiring the hardLinkLockTable lock, the prior revision silently fell back to the pre-lock snapshot, reintroducing the stale-base update the lock is meant to prevent. - Unlink: treat fuse.ENOENT as success (the file was already removed by the thread that held the lock before us) and propagate any other error. - Link: abort with the returned status so we never derive the next HardLinkCounter from a stale source entry. * fix(mount): re-resolve Link source alias under HardLinkId lock Review follow-up: Link resolved oldEntryPath from in.Oldnodeid before waiting on the HardLinkId lock. A concurrent Unlink that held the same lock could remove the specific alias we picked pre-lock while leaving other sibling hard links for the same inode intact. The post-lock maybeLoadEntry then returned ENOENT even though the source inode was still reachable. Call GetPath(in.Oldnodeid) again under the lock to pick whichever alias is still active, refresh oldParentPath, and only return ENOENT if no sibling survived. |
||
|
Chris Lu
|
300e906330 |
admin: report file and delete counts for EC volumes (#9060)
* admin: report file and delete counts for EC volumes The admin bucket size fix (#9058) left object counts at zero for EC-encoded data because VolumeEcShardInformationMessage carried no file count. Billing/monitoring dashboards therefore still under-report objects once a bucket is EC-encoded. Thread file_count and delete_count end-to-end: - Add file_count/delete_count to VolumeEcShardInformationMessage (proto fields 8 and 9) and regenerate master_pb. - Compute them lazily on volume servers by walking the .ecx index once per EcVolume, cache on the struct, and keep the cache in sync inside DeleteNeedleFromEcx (distinguishing live vs already-tombstoned entries so idempotent deletes do not drift the counts). - Populate the new proto fields from EcVolume.ToVolumeEcShardInformationMessage and carry them through the master-side EcVolumeInfo / topology sync. - Aggregate in admin collectCollectionStats, deduping per volume id: every node holding shards of an EC volume reports the same counts, so summing across nodes would otherwise multiply the object count by the number of shard holders. Regression tests cover the initial .ecx walk, live/tombstoned delete bookkeeping (including idempotent and missing-key cases), and the admin dedup path for an EC volume reported by multiple nodes. * ec: include .ecj journal in EcVolume delete count The initial delete count only reflected .ecx tombstones, missing any needle that was journaled in .ecj but not yet folded into .ecx — e.g. on partial recovery. Expand initCountsLocked to take the union of .ecx tombstones and .ecj journal entries, deduped by needle id, so: - an id that is both tombstoned in .ecx and listed in .ecj counts once - a duplicate .ecj entry counts once - an .ecj id with a live .ecx entry is counted as deleted (not live) - an .ecj id with no matching .ecx entry is still counted Covered by TestEcVolumeFileAndDeleteCountEcjUnion. * ec: report delete count authoritatively and tombstone once per delete Address two issues with the previous EcVolume file/delete count work: 1. The delete count was computed lazily on first heartbeat and mixed in a .ecj-union fallback to "recover" partial state. That diverged from how regular volumes report counts (always live from the needle map) and had drift cases when .ecj got reconciled. Replace with an eager walk of .ecx at NewEcVolume time, maintained incrementally on every DeleteNeedleFromEcx call. Semantics now match needle_map_metric: FileCount is the total number of needles ever recorded in .ecx (live + tombstoned), DeleteCount is the tombstones — so live = FileCount - DeleteCount. Drop the .ecj-union logic entirely. 2. A single EC needle delete fanned out to every node holding a replica of the primary data shard and called DeleteNeedleFromEcx on each, which inflated the per-volume delete total by the replica factor. Rewrite doDeleteNeedleFromRemoteEcShardServers to try replicas in order and stop at the first success (one tombstone per delete), and only fall back to other shards when the primary shard has no home (ErrEcShardMissing sentinel), not on transient RPC errors. Admin aggregation now folds EC counts correctly: FileCount is deduped per volume id (every shard holder has an identical .ecx) and DeleteCount is summed across nodes (each delete tombstones exactly one node). Live object count = deduped FileCount - summed DeleteCount. Tests updated to match the new semantics: - EC volume counts seed FileCount as total .ecx entries (live + tombstoned), DeleteCount as tombstones. - DeleteNeedleFromEcx keeps FileCount constant and increments DeleteCount only on live->tombstone transitions. - Admin dedup test uses distinct per-node delete counts (5 + 3 + 2) to prove they're summed, while FileCount=100 is applied once. * ec: test fixture uses real vid; admin warns on skewed ec counts - writeFixture now builds the .ecx/.ecj/.ec00/.vif filenames from the actual vid passed in, instead of hardcoding "_1". The existing tests all use vid=1 so behaviour is unchanged, but the helper no longer silently diverges from its documented parameter. - collectCollectionStats logs a glog warning when an EC volume's summed delete count exceeds its deduped file count, surfacing the anomaly (stale heartbeat, counter drift, etc.) instead of silently dropping the volume from the object count. * ec: derive file/delete counts from .ecx/.ecj file sizes seedCountsFromEcx walked the full .ecx index at volume load, which is wasted work: .ecx has fixed-size entries (NeedleMapEntrySize) and .ecj has fixed-size deletion records (NeedleIdSize), so both counts are pure file-size arithmetic. fileCount = ecxFileSize / NeedleMapEntrySize deleteCount = ecjFileSize / NeedleIdSize Rip out the cached counters, countsLock, seedCountsFromEcx, and the recordDelete helper. Track ecjFileSize directly on the EcVolume struct, seed it from Stat() at load, and bump it on every successful .ecj append inside DeleteNeedleFromEcx under ecjFileAccessLock. Skip the .ecj write entirely when the needle is already tombstoned so the derived delete count stays idempotent on repeat deletes. Heartbeats now compute counts in O(1). Tests updated: the initial fixture pre-populates .ecj with two ids to verify the file-size derivation end-to-end, and the delete test keeps its idempotent-re-delete / missing-needle invariants (unchanged externally, now enforced by the early return rather than a cache guard). * ec: sync Rust volume server with Go file/delete count semantics Mirror the Go-side EC file/delete count work in the Rust volume server so mixed Go/Rust clusters report consistent bucket object counts in the admin dashboard. - Add file_count (8) and delete_count (9) to the Rust copy of VolumeEcShardInformationMessage (seaweed-volume/proto/master.proto). - EcVolume gains ecj_file_size, seeded from the journal's metadata on open and bumped inside journal_delete on every successful append. - file_and_delete_count() returns counts derived in O(1) from ecx_file_size / NEEDLE_MAP_ENTRY_SIZE and ecj_file_size / NEEDLE_ID_SIZE, matching Go's FileAndDeleteCount. - to_volume_ec_shard_information_messages populates the new proto fields instead of defaulting them to zero. - mark_needle_deleted_in_ecx now returns a DeleteOutcome enum (NotFound / AlreadyDeleted / Tombstoned) so journal_delete can skip both the .ecj append and the size bump when the needle is missing or already tombstoned, keeping the derived delete_count idempotent on repeat or no-op deletes. - Rust's EcVolume::new no longer replays .ecj into .ecx on load. Go's RebuildEcxFile is only called from specific decode/rebuild gRPC handlers, not on volume open, and replaying on load was hiding the deletion journal from the new file-size-derived delete counter. rebuild_ecx_from_journal is kept as dead_code for future decode paths that may want the same replay semantics. Also clean up the Go FileAndDeleteCount to drop unnecessary runtime guards against zero constants — NeedleMapEntrySize and NeedleIdSize are compile-time non-zero. test_ec_volume_journal updated to pre-populate the .ecx with the needles it deletes, and extended to verify that repeat and missing-id deletes do not drift the derived counts. * ec: document enterprise-reserved proto field range on ec shard info Both OSS master.proto copies now note that fields 10-19 are reserved for future upstream additions while 20+ are owned by the enterprise fork. Enterprise already pins data_shards/parity_shards at 20/21, so keeping OSS additions inside 8-19 avoids wire-level collisions for mixed deployments. * ec(rust): resolve .ecx/.ecj helpers from ecx_actual_dir ecx_file_name() and ecj_file_name() resolved from self.dir_idx, but new() opens the actual files from ecx_actual_dir (which may fall back to the data dir when the idx dir does not contain the index). After a fallback, read_deleted_needles() and rebuild_ecx_from_journal() would read/rebuild the wrong (nonexistent) path while heartbeats reported counts from the file actually in use — silently dropping deletes. Point idx_base_name() at ecx_actual_dir, which is initialized to dir_idx and only diverges after a successful fallback, so every call site agrees with the file new() has open. The pre-fallback call in new() (line 142) still returns the dir_idx path because ecx_actual_dir == dir_idx at that point. Update the destroy() sweep to build the dir_idx cleanup paths explicitly instead of leaning on the helpers, so post-fallback stale files in the idx dir are still removed. * ec: reset ecj size after rebuild; rollback ecx tombstone on ecj failure Two EC delete-count correctness fixes applied symmetrically to Go and Rust volume servers. 1. rebuild_ecx_from_journal (Rust) now sets ecj_file_size = 0 after recreating the empty journal, matching the on-disk truth. Previously the cached size still reflected the pre-rebuild journal and file_and_delete_count() would keep reporting stale delete counts. The Go side has no equivalent bug because RebuildEcxFile runs in an offline helper that does not touch an EcVolume struct. 2. DeleteNeedleFromEcx / journal_delete used to tombstone the .ecx entry before writing the .ecj record. If the .ecj append then failed, the needle was permanently marked deleted but the heartbeat-reported delete_count never advanced (it is derived from .ecj file size), and a retry would see AlreadyDeleted and early- return, leaving the drift permanent. Both languages now capture the entry's file offset and original size bytes during the mark step, attempt the .ecj append, and on failure roll the .ecx tombstone back by writing the original size bytes at the known offset. A rollback that itself errors is logged (glog / tracing) but cannot re-sync the files — this is the same failure mode a double disk error would produce, and is unavoidable without a full on-disk transaction log. Go: wrap MarkNeedleDeleted in a closure that captures the file offset into an outer variable, then pass the offset + oldSize to the new rollbackEcxTombstone helper on .ecj seek/write errors. Rust: DeleteOutcome::Tombstoned now carries the size_offset and a [u8; SIZE_SIZE] copy of the pre-tombstone size field. journal_delete destructures on Tombstoned and calls restore_ecx_size on .ecj append failure. * test(ec): widen admin /health wait to 180s for cold CI TestEcEndToEnd starts master, 14 volume servers, filer, 2 workers and admin in sequence, then waited only 60s for admin's HTTP server to come up. On cold GitHub runners the tail of the earlier subprocess startups eats most of that budget and the wait occasionally times out (last hit on run 24374773031). The local fast path is still ~20s total, so the bump only extends the timeout ceiling, not the happy path. * test(ec): fork volume servers in parallel in TestEcEndToEnd startWeed is non-blocking (just cmd.Start()), so the per-process fork + mkdir + log-file-open overhead for 14 volume servers was serialized for no reason. On cold CI disks that overhead stacks up and eats into the subsequent admin /health wait, which is how run 24374773031 flaked. Wrap the volume-server loop in a sync.WaitGroup and guard runningCmds with a mutex so concurrent appends are safe. startWeed still calls t.Fatalf on failure, which is fine from a goroutine for a fatal test abort; the fail-fast isn't something we rely on for precise ordering. * ec: fsync ecx before ecj, truncate on failure, harden rebuild Four correctness fixes covering both volume servers. 1. Durability ordering (Go + Rust). After marking the .ecx tombstone we now fsync .ecx before touching .ecj, so a crash between the two files cannot leave the journal with an entry for a needle whose tombstone is still sitting in page cache. Once the fsync returns, the tombstone is the source of truth: reads see "deleted", delete_count may under-count by one (benign, idempotent retries) but never over-reports. If the fsync itself fails we restore the original size bytes and surface the error. The .ecj append is then followed by its own Sync so the reported delete_count matches the on-disk journal once the write returns. 2. .ecj truncation on append failure. write_all may have extended the journal on disk before sync_all / Sync errors out, leaving the cached ecj_file_size out of sync with the physical length and drifting delete_count permanently after restart. Both languages now capture the pre-append size, truncate the file back via set_len / Truncate on any write or sync failure, and only then restore the .ecx tombstone. Truncation errors are logged — same-fd length resets cannot realistically fail — but cannot themselves re-sync the files. 3. Atomic rebuild_ecx_from_journal (Rust, dead code today but wired up on any future decode path). Previously a failed mark_needle_deleted_in_ecx call was swallowed with `let _ = ...` and the journal was still removed, silently losing tombstones. We now bubble up any non-NotFound error, fsync .ecx after the whole replay succeeds, and only then drop and recreate .ecj. NotFound is still ignored (expected race between delete and encode). 4. Missing-.ecx hardening (Rust). mark_needle_deleted_in_ecx used to return Ok(NotFound) when self.ecx_file was None, hiding a closed or corrupt volume behind what looks like an idempotent no-op. It now returns an io::Error carrying the volume id so callers (e.g. journal_delete) fail loudly instead. Existing Go and Rust EC test suites stay green. * ec: make .ecx immutable at runtime; track deletes in memory + .ecj Refactors both volume servers so the sealed sorted .ecx index is never mutated during normal operation. Runtime deletes are committed to the .ecj deletion journal and tracked in an in-memory deleted-needle set; read-path lookups consult that set to mask out deleted ids on top of the immutable .ecx record. Mirrors the intended design on both Go and Rust sides. EcVolume gains a `deletedNeedles` / `deleted_needles` set seeded from .ecj in NewEcVolume / EcVolume::new. DeleteNeedleFromEcx / journal_delete: 1. Looks the needle up read-only in .ecx. 2. Missing needle -> no-op. 3. Pre-existing .ecx tombstone (from a prior decode/rebuild) -> mirror into the in-memory set, no .ecj append. 4. Otherwise append the id to .ecj, fsync, and only then publish the id into the set. A partial write is truncated back to the pre-append length so the on-disk journal and the in-memory set cannot drift. FindNeedleFromEcx / find_needle_from_ecx now return TombstoneFileSize when the id is in the in-memory set, even though the bytes on disk still show the original size. FileAndDeleteCount: fileCount = .ecx size / NeedleMapEntrySize (unchanged) deleteCount = len(deletedNeedles) (was: .ecj size / NeedleIdSize) The RebuildEcxFile / rebuild_ecx_from_journal decode-time helpers still fold .ecj into .ecx — that is the one place tombstones land in the physical index, and it runs offline on closed files. Rust's rebuild helper now also clears the in-memory set when it succeeds. Dead code removed on the Rust side: `DeleteOutcome`, `mark_needle_deleted_in_ecx`, `restore_ecx_size`. Go drops the runtime `rollbackEcxTombstone` path. Neither helper was needed once .ecx stopped being a runtime mutation target. TestEcVolumeSyncEnsuresDeletionsVisible (issue #7751) is rewritten as TestEcVolumeDeleteDurableToJournal, which exercises the full durability chain: delete -> .ecj fsync -> FindNeedleFromEcx masks via the in-memory set -> raw .ecx bytes are *unchanged* -> Close + RebuildEcxFile folds the journal into .ecx -> raw bytes now show the tombstone, as CopyFile in the decode path expects. |
||
|
Chris Lu
|
64af80c78d |
fix(mount): stop double-applying umask in Mkdir (#9063)
Mkdir was masking in.Mode with wfs.option.Umask on top of the kernel's VFS umask pass, so a caller with umask=0 who requested mkdir(0777) got 0755 (0777 & ~022). Create and Symlink don't apply this second pass — Mkdir was the odd one out. The resulting dirs had fewer write bits than the caller asked for, which broke cross-user rename permission checks (kernel may_delete rejects with EACCES when the parent lacks o+w even though the caller explicitly requested it) and blocked pjdfstest tests/rename/21.t and its cascading checks. Drop the extra umask so Mkdir trusts in.Mode exactly like Create. The CLI -umask flag still covers the internal cache dirs that the mount creates for itself via os.MkdirAll; only the user-facing Mkdir path changes. Unblocks tests/rename/21.t — full pjdfstest suite is now 236 files / 8819 tests, all PASS, and known_failures.txt is empty. |
||
|
Chris Lu
|
c8433a19f0 |
fix(mount): propagate hard-link nlink changes to sibling cache entries (#9062)
* fix(mount): propagate hard-link nlink changes to sibling cache entries weed mount serves stat from its local metacache, and the kernel also caches inode attrs from FUSE replies. When a hard link was unlinked or a new link added, the filer updated the shared HardLink blob correctly, but the sibling link entries in the mount's metacache still carried the stale HardLinkCounter and the kernel attr cache on the shared inode was not invalidated. Subsequent lstat on any sibling link returned the old nlink — pjdfstest link/00.t caught this after `unlink n0` and on `link n1 n2` stating n0. Walk every path bound to the hard-linked inode via a new InodeToPath.GetAllPaths, rewrite each cached sibling's HardLinkCounter and ctime to the authoritative new value, and call fuseServer.InodeNotify to invalidate the kernel attr cache for the shared inode. Applied from both Link (bump) and Unlink (decrement). Unblocks tests/link/00.t and tests/unlink/00.t in pjdfstest; full suite (235 files, 8803 tests) passes end-to-end with no regressions. * fix(mount): harden hard-link sibling sync against nil Attributes and id mismatch Review follow-ups: - Unlink: guard entry.Attributes for nil before reading Inode, with a fallback to inodeToPath.GetInode resolved before RemovePath. Fold the duplicated RemovePath into a single call. - syncHardLinkSiblings: skip siblings whose HardLinkId does not match the authoritative entry. The shared-inode invariant normally guarantees a match, but a transient mismatch (e.g. a rename replaced one of the paths) would otherwise stamp an unrelated entry with the wrong counter. Full pjdfstest suite still passes (235 files, 8803 tests). |
||
|
Chris Lu
|
f00cbe4a6d |
test(vacuum): fix flaky TestVacuumIntegration across multiple volumes (#9061)
* test(vacuum): fix flaky TestVacuumIntegration across multiple volumes The test assumed all uploaded files landed in a single volume and tracked only the last file's volume id. With -volumeSizeLimitMB 10 and 16x500KB files, the master can spread uploads across volumes, so the tracked id could point to a volume with no deletes and thus 0% garbage — causing verify_garbage_before_vacuum to fail even though vacuum ran correctly on the other volume. Track the set of volumes where deletes actually occurred and verify garbage/cleanup against all of them. Also add a short retry loop on the pre-vacuum check to absorb heartbeat jitter. * test(vacuum): require all dirty volumes ready; retry cleanup check Address review feedback: the pre-vacuum check now waits until every volume in dirtyVolumes reports garbage > threshold (not just the first), and the post-vacuum cleanup check retries per-volume with a deadline instead of relying on a fixed sleep, since vacuum + heartbeat reporting is asynchronous. * test(vacuum): deterministic dirty volumes order, aggregate cleanup failures - Sort dirtyVolumes after building from the set so logs and iteration are stable across runs. - In verify_cleanup_after_vacuum, track per-volume failure reasons in a map and report all still-failing volumes on timeout instead of only the last one that happened to be written to lastErr. |
||
 Lars Lehtonen
|
e0c361ec77 | fix(weed/worker/tasks): log dropped errors (#9057) | ||
|
Chris Lu
|
8f2a3d92bb |
docker: upgrade libcrypto3/libssl3 to clear Trivy HIGH (CVE-2026-28390) (#9059)
* docker: upgrade libcrypto3/libssl3 to clear Trivy HIGH Trivy gate on ghcr.io/seaweedfs/seaweedfs:latest-amd64 flagged CVE-2026-28390 in libcrypto3 3.5.5-r0 (fixed in 3.5.6-r0) on the alpine 3.23.3 base. Add libcrypto3/libssl3 to the existing apk upgrade so rebuilt images pick up the patched openssl without waiting for a new alpine base tag. * docker: apk add libcrypto3/libssl3 so they install at patched version Per review, apk upgrade <pkg> is a no-op when the package isn't already installed. libcrypto3/libssl3 come in transitively via curl, so list them in apk add to guarantee installation at the latest (patched) version from the alpine repo. |
||
|
Chris Lu
|
ef77df6141 |
admin: include EC volumes in bucket size reporting (#9058)
* admin: include EC volumes in bucket size reporting The Object Store buckets page computed per-collection size by iterating only regular volumes, so once a bucket's data was EC-encoded it silently disappeared from the reported size — breaking usage-based billing. Walk EcShardInfos alongside VolumeInfos in collectCollectionStats: add raw shard bytes to PhysicalSize, and the parity-stripped value (shardBytes * DataShardsCount / TotalShardsCount) to LogicalSize, matching the normalization used by `weed shell` cluster.status. * admin: derive EC logical size from shard bitmap, not constants Use ShardsInfoFromVolumeEcShardInformationMessage + MinusParityShards to sum actual data-shard bytes instead of scaling raw bytes by the DataShardsCount/TotalShardsCount ratio. Keeps the data/parity split encapsulated in the erasure_coding package and is exact when shard sizes differ (e.g. last shard). * admin: regression test for EC shard size aggregation Cover the uneven-tail-shard case (data shard 9 < 1000 bytes) and the empty-collection-name path to pin PhysicalSize/LogicalSize behavior for collectCollectionStats against future changes. |
||
 Chris Lu
|
50f25bb5cd | 4.20 4.20 | ||
|
Chris Lu
|
512912cbb8 | Update plugin_templ.go | ||
|
dependabot[bot]
|
8d6c5cbb58 |
build(deps): bump org.apache.kafka:kafka-clients from 3.9.1 to 3.9.2 in /test/kafka/kafka-client-loadtest/tools (#9056)
build(deps): bump org.apache.kafka:kafka-clients Bumps org.apache.kafka:kafka-clients from 3.9.1 to 3.9.2. --- updated-dependencies: - dependency-name: org.apache.kafka:kafka-clients dependency-version: 3.9.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
f3151900e4 |
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.98.0 to 1.99.0 (#9053)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.98.0 to 1.99.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.98.0...service/s3/v1.99.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.99.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
7aaa431bb4 |
s3api: prune bucket-scoped IAM actions on DeleteBucket (#9054)
* s3api: prune bucket-scoped IAM actions on DeleteBucket DeleteBucket removed the bucket directory and collection but left behind any identity actions configured via s3.configure that were scoped to that bucket (e.g. Read:bucket, Write:bucket/prefix), leaving stale auth metadata that users expected to be cleaned up along with the bucket. After a successful delete, strip actions whose resource is exactly the bucket or a prefix under it, save via the credential manager, and let the existing filer metadata subscription fan the reload out to every S3 server. Wildcarded resources and global actions are preserved since they may cover other buckets; static identities are left untouched. Fixes #5310 * s3api: address review feedback on bucket IAM prune - Apply per-identity updates via credentialManager.UpdateUser instead of a full LoadConfiguration/SaveConfiguration round-trip, so the prune no longer clobbers concurrent IAM edits made by s3.configure or the IAM API during a DeleteBucket. - Use a 30s bounded background context for the post-delete cleanup so it survives client disconnect — the bucket is already gone by then and this is best-effort bookkeeping. - Skip static identities via IsStaticIdentity, since the credential store never persists them and UpdateUser would return NotFound. |
||
 Arthur Woimbée
|
8049fcc516 |
correctly namespace all define calls (#9044)
* correctly namespace all `define` calls * fix unrelated issue: wrong dict call to gen sftp passwd |
||
|
dependabot[bot]
|
06cbd2acdf |
build(deps): bump golang.org/x/net from 0.52.0 to 0.53.0 (#9052)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.52.0 to 0.53.0. - [Commits](https://github.com/golang/net/compare/v0.52.0...v0.53.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.53.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 Mark McCormick
|
2ee6907c19 |
Update Helm Chart docs with instructions for deploying RocksDB variant (#9006)
* Update documentation for helm chart, with instructions on how to deploy the RocksDB image tag variant. Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev> Nit: Update example to make it clearer that the seaweedfs version needs to be replaced. Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev> * docs(helm): clarify RocksDB variant instructions - Note that filer persistence (enablePVC) is required so RocksDB metadata survives restarts. - Explain why master/volume also use the rocksdb-tagged image. - Tighten wording around WEED_LEVELDB2_ENABLED override. --------- Signed-off-by: Mark McCormick <mark.mccormick@chainguard.dev> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
dependabot[bot]
|
cc5b246973 |
build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.32.13 to 1.32.14 (#9051)
build(deps): bump github.com/aws/aws-sdk-go-v2/config Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.13 to 1.32.14. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.13...config/v1.32.14) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.32.14 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
36ae7e04b5 |
build(deps): bump github.com/apache/cassandra-gocql-driver/v2 from 2.0.0 to 2.1.0 (#9047)
build(deps): bump github.com/apache/cassandra-gocql-driver/v2 Bumps [github.com/apache/cassandra-gocql-driver/v2](https://github.com/apache/cassandra-gocql-driver) from 2.0.0 to 2.1.0. - [Release notes](https://github.com/apache/cassandra-gocql-driver/releases) - [Changelog](https://github.com/apache/cassandra-gocql-driver/blob/trunk/CHANGELOG.md) - [Commits](https://github.com/apache/cassandra-gocql-driver/compare/v2.0.0...v2.1.0) --- updated-dependencies: - dependency-name: github.com/apache/cassandra-gocql-driver/v2 dependency-version: 2.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
46c0e56bb8 |
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.125.3 to 3.134.0 (#9048)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.125.3 to 3.134.0. - [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases) - [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.125.3...v3.134.0) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk/v3 dependency-version: 3.134.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
baa65c3823 |
build(deps): bump docker/build-push-action from 7.0.0 to 7.1.0 (#9049)
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 7.0.0 to 7.1.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v7...v7.1.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 7.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
f4bfe60549 |
build(deps): bump softprops/action-gh-release from 2 to 3 (#9050)
Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2 to 3. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](https://github.com/softprops/action-gh-release/compare/v2...v3) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 Lisandro Pin
|
67a2810d2d |
Export start_time_seconds metrics on both master & volume servers. (#9046)
These are to be used to track uptimes. See https://github.com/seaweedfs/seaweedfs/issues/8535 for details. Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch> |
||
|
Lars Lehtonen
|
80db692728 | fix(weed/util/chunk_cache): fix dropped errors (#9042) | ||
|
Chris Lu
|
ae08e77979 |
fix(scheduler): give worker tasks a real per-attempt execution deadline (#9041)
* fix(scheduler): give worker tasks a real per-attempt execution deadline The plugin scheduler derived the per-attempt execution deadline as DetectionTimeoutSeconds * 2, which capped every worker task at twice the cluster-scan budget regardless of actual work. For volume_balance batches this was 240s — far too short for 20 large volume copies, so every attempt died at "context deadline exceeded" and all in-flight sub-RPCs surfaced as "context canceled". Retries restarted from move 1 and hit the same wall. Add an explicit ExecutionTimeoutSeconds field to the plugin proto and make each handler declare its own baseline (1800s for vacuum, balance, EC; 3600s for iceberg). Size-aware handlers also emit an estimated_runtime_seconds parameter on each proposal so the scheduler extends the per-attempt deadline based on actual workload: - volume_balance batch: max(largest single move, total / concurrency) at 5 min/GB, so a skewed batch with one big volume isn't averaged away. - volume_balance single, vacuum (already), erasure_coding (10 min/GB), ec_balance (5 min/GB): per-volume budgets. admin_script and iceberg keep the configurable handler default since their workloads are opaque to the detector. * fix(scheduler): apply descriptor defaults to existing persisted configs The previous commit added execution_timeout_seconds to the proto and each handler's descriptor defaults, but two paths still left existing deployments broken: 1. deriveSchedulerAdminRuntime returned stored AdminRuntime configs as-is. Persisted configs from older versions have no execution_timeout_seconds, so the scheduler fell back to the 90s default — worse than the prior 240s behavior. Overlay descriptor defaults for any zero numeric fields when loading. 2. The admin form did not round-trip execution_timeout_seconds, so a normal save would clear it back to zero. Add the input field, the fillAdminSettings/collectAdminSettings hooks, and as defense in depth reapply descriptor defaults in UpdatePluginJobTypeConfigAPI before persisting so a stale form can never silently clobber a baseline. * fix(volume_balance): account for partial scheduling rounds in batch estimate With N moves and C slots, the busiest slot processes ceil(N/C) moves, not N/C. Dividing total seconds by C underestimates wall-clock time whenever N is not a multiple of C — e.g. 6 moves at concurrency 5 needs 2 rounds, not 1.2. Use avg * ceil(N/C) so partial rounds are counted as full ones. * fix(volume_balance): scale minBudget per wave instead of per move Orchestration overhead (setup/teardown for the parallel move runner) happens once per wave, not once per move. Use numRounds*60 as the floor instead of len(moves)*60 so the minimum doesn't inflate linearly with batch size when individual moves are tiny. |
||
|
Chris Lu
|
28d1ef24ec |
fix(admin): allow control chars in file paths when browsing filer (#9043)
* fix(admin): allow control chars in file paths when browsing filer The admin UI rejected any path containing \x00, \r, or \n as "path contains invalid characters". These bytes are legal in S3 object keys, so objects created through the S3 API (or replicated via filer.sync) could exist on the filer but be unreachable from the admin UI — browse, download, and upload all failed with "Invalid file path". Drop the control-character rejection and instead URL-escape the path when constructing filer request URLs, so that such bytes cannot inject into the HTTP request target. Path traversal protection via path.Clean is unchanged. * test(admin): strengthen file path tests with byte-preserving checks Assert full expected output for validateAndCleanFilePath so silent stripping of control characters would fail the test, and cover \r and \x00 escaping in filerFileURL in addition to \n and space. |
||
|
Chris Lu
|
edf7d2a074 |
fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9039)
* fix(filer): eliminate redundant disk reads causing memory/CPU regression (#9035) Since 4.18, LocalMetaLogBuffer's ReadFromDiskFn was set to readPersistedLogBufferPosition, causing LoopProcessLogData to call ReadPersistedLogBuffer on every 250ms health-check tick when a subscriber encounters ResumeFromDiskError. Each call creates an OrderedLogVisitor (ListDirectoryEntries on the filer store), spawns a readahead goroutine with a 1024-element channel, finds no data, and returns — 4 times per second even on an idle filer. This is redundant because SubscribeLocalMetadata already manages disk reads explicitly with its own shouldReadFromDisk / lastCheckedFlushTsNs tracking in the outer loop. Set ReadFromDiskFn back to nil for LocalMetaLogBuffer. When LoopProcessLogData encounters ResumeFromDiskError with nil ReadFromDiskFn, the HasData() guard returns ResumeFromDiskError to the caller (SubscribeLocalMetadata), which blocks efficiently on listenersCond.Wait() instead of polling. * fix(filer): add gap detection for slow consumers after disk-read stall When a slow consumer falls behind and LoopProcessLogData returns ResumeFromDiskError with no flush or read-position progress, there may be a gap between persisted data and in-memory data (e.g. writes stopped while consumer was still catching up). Without this, the consumer would block on listenersCond.Wait() forever. Skip forward to the earliest in-memory time to resume progress, matching the gap-handling pattern already used in the shouldReadFromDisk path. * fix(filer): clear stale ResumeFromDiskError after gap-skip to avoid stall The gap-detection block added in the previous commit skips lastReadTime forward to GetEarliestTime() and continues the outer loop. On the next iteration, shouldReadFromDisk becomes true (currentReadTsNs > lastDiskReadTsNs), the disk read returns processedTsNs == 0, and the existing gap handler at the top of the loop runs its own gap check. That check uses readInMemoryLogErr == ResumeFromDiskError as the entry condition — but readInMemoryLogErr is still the stale error from two iterations ago. GetEarliestTime() now equals lastReadTime.Time (we already advanced to it), so earliestTime.After(lastReadTime.Time) is false and the handler falls into listenersCond.Wait() — stuck. Clear readInMemoryLogErr at the gap-skip point, matching the existing pattern at the earlier gap handler that already clears it for the same reason. * fix(log_buffer): GetEarliestTime must include sealed prev buffers GetEarliestTime previously returned only logBuffer.startTime (the active buffer's first timestamp). That is narrower than ReadFromBuffer's tsMemory, which is the min across active + prev buffers. Callers using GetEarliestTime for gap detection after ResumeFromDiskError (the SubscribeLocalMetadata outer loop's disk-read path, the new gap-skip in the in-memory ResumeFromDiskError handler, and MQ HasData) saw a time that was *newer* than the real earliest in-memory data. Impact in SubscribeLocalMetadata's slow-consumer path: - tsMemory = earliest prev buffer time (T_prev) - GetEarliestTime() = active startTime (T_active, later than T_prev) - Consumer position = T1, with T_prev < T1 < T_active - ReadFromBuffer returns ResumeFromDiskError (T1 < tsMemory) - Gap detect: GetEarliestTime().After(T1) = T_active.After(T1) = true - Skip forward to T_active -- silently drops the prev-buffer data - And when T_active happens to equal the stuck position, gap detect evaluates false, and the subscriber stalls on listenersCond.Wait() This reproduces the TestMetadataSubscribeSlowConsumerKeepsProgressing failure in CI where the consumer stalled at 10220/20000 after writing stopped -- the buffer still had data in prev[0..3], but gap detection was comparing against the active buffer's startTime. Fix: scan all sealed prev buffers under RLock, return the true minimum startTime. Matches the min-of-buffers logic in ReadFromBuffer. * test(log_buffer): make DiskReadRetry test deterministic The previous test added the message via AddToBuffer + ForceFlush and relied on a race: the second disk read had to happen before the data was delivered through the in-memory path. Under the race detector or on a slow CI runner, the reader is woken by AddToBuffer's notification, finds the data in the active buffer or its prev slot, and returns after exactly one disk read — failing the >= 2 disk reads assertion even though the loop behaved correctly. Reproduced on master with race detector (2/5 failures). Rewrite the test to deliver the data exclusively through the disk-read path: no AddToBuffer, no ForceFlush. The test waits until the reader has issued at least one no-op disk read, then atomically flips a "dataReady" flag. The reader's next iteration through readFromDiskFn returns the entry. This deterministically exercises the retry-loop behavior the test was originally written to protect, and removes the in-memory delivery race entirely. |
||
|
Chris Lu
|
10e7f0f2bc |
fix(shell): s3.user.provision handles existing users by attaching policy (#9040)
* fix(shell): s3.user.provision handles existing users by attaching policy Instead of erroring when the user already exists, the command now creates the policy and attaches it to the existing user via UpdateUser. Credentials are only generated and displayed for newly created users. * fix(shell): skip duplicate policy attachment in s3.user.provision Check if the policy is already attached before appending and calling UpdateUser, making repeated runs idempotent. * fix(shell): generate service account ID in s3.serviceaccount.create The command built a ServiceAccount proto without setting Id, which was rejected by credential.ValidateServiceAccountId on any real store. Now generates sa:<parent>:<uuid> matching the format used by the admin UI. * test(s3): integration tests for s3.* shell commands Adds TestShell* integration tests covering ~40 previously untested shell commands: user, accesskey, group, serviceaccount, anonymous, bucket, policy.attach/detach, config.show, and iam.export/import. Switches the test cluster's credential store from memory to filer_etc because the memory store silently drops groups and service accounts in LoadConfiguration/SaveConfiguration. * fix(shell): rollback policy on key generation failure in s3.user.provision If iam.GenerateRandomString or iam.GenerateSecretAccessKey fails after the policy was persisted, the policy would be left orphaned. Extracts the rollback logic into a local closure and invokes it on all failure paths after policy creation for consistency. * address PR review feedback for s3 shell tests and serviceaccount - s3.serviceaccount.create: use 16 bytes of randomness (hex-encoded) for the service account UUID instead of 4 bytes to eliminate collision risk - s3.serviceaccount.create: print the actual ID and drop the outdated "server-assigned" note (the ID is now client-generated) - tests: guard createdAK in accesskey rotate/delete subtests so sibling failures don't run invalid CLI calls - tests: requireContains/requireNotContains use t.Fatalf to fail fast - tests: Provision subtest asserts the "Attached policy" message on the second provision call for an existing user - tests: update extractServiceAccountID comment example to match the sa:<parent>:<uuid> format - tests: drop redundant saID empty-check (extractServiceAccountID fatals) * test(s3): use t.Fatalf for precondition check in serviceaccount test |
||
 os-pradipbabar
|
9cae95d749 |
fix(filer): prevent data corruption during graceful shutdown (#9037)
* fix: wait for in-flight uploads to complete before filer shutdown Prevents data corruption when SIGTERM is received during active uploads. The filer now waits for all in-flight operations to complete before calling the underlying shutdown logic. This affects all deployment types (Kubernetes, Docker, systemd) and fixes corruption issues during rolling updates, certificate rotation, and manual restarts. Changes: - Add FilerServer.Shutdown() method with upload wait logic - Update grace.OnInterrupt hook to use new shutdown method Fixes data corruption reported by production users during pod restarts. * fix: implement graceful shutdown for gRPC and HTTP servers, ensuring in-flight uploads complete * fix: address review comments on graceful shutdown - Add 10s timeout to gRPC GracefulStop to prevent indefinite blocking from long-lived streams (falls back to Stop on timeout) - Reduce HTTP/HTTPS shutdown timeout from 25s to 15s to fit within Kubernetes default 30s termination grace period - Move fs.Shutdown() (database close) after Serve() returns instead of a separate hook to eliminate race where main goroutine exits before the shutdown hook runs * fix: shut down all HTTP servers before filer database close Address remaining review comments: - Shut down auxiliary HTTP servers (Unix socket, local listener) during graceful shutdown so they can't serve write traffic after the main server stops - Register fs.Shutdown() as a grace.OnInterrupt hook to guarantee it completes before os.Exit(0), fixing the race between the grace goroutine and the main goroutine - Use sync.Once to ensure fs.Shutdown() runs exactly once regardless of whether shutdown is signal-driven or context-driven (MiniCluster) --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
e8a8449553 |
feat(mount): pre-allocate file IDs in pool for writeback cache mode (#9038)
* feat(mount): pre-allocate file IDs in pool for writeback cache mode When writeback caching is enabled, chunk uploads no longer block on a per-chunk AssignVolume RPC. Instead, a FileIdPool pre-allocates file IDs in batches using a single AssignVolume(Count=N, ExpectedDataSize=ChunkSize) call and hands them out instantly to upload workers. Pool size is 2x ConcurrentWriters, refilled in background when it drops below ConcurrentWriters. Entries expire after 25s to respect JWT TTL. Sequential needle keys are generated from the base file ID returned by the master, so one Assign RPC produces N usable IDs. This cuts per-chunk upload latency from 2 RTTs (assign + upload) to 1 RTT (upload only), with the assign cost amortized across the batch. * test: add benchmarks for file ID pool vs direct assign Benchmarks measure: - Pool Get vs Direct AssignVolume at various simulated latencies - Batch assign scaling (Count=1 through Count=32) - Concurrent pool access with 1-64 workers Results on Apple M4: - Pool Get: constant ~3ns regardless of assign latency - Batch=16: 15.7x more IDs/sec than individual assigns - 64 concurrent workers: 19M IDs/sec throughput * fix(mount): address review feedback on file ID pool 1. Fix race condition in Get(): use sync.Cond so callers wait for an in-flight refill instead of returning an error when the pool is empty. 2. Match default pool size to async flush worker count (128, not 16) when ConcurrentWriters is unset. 3. Add logging to UploadWithAssignFunc for consistency with UploadWithRetry. 4. Document that pooled assigns omit the Path field, bypassing path-based storage rules (filer.conf). This is an intentional tradeoff for writeback cache performance. 5. Fix flaky expiry test: widen time margin from 50ms to 1s. 6. Add TestFileIdPoolGetWaitsForRefill to verify concurrent waiters. * fix(mount): use individual Count=1 assigns to get per-fid JWTs The master generates one JWT per AssignResponse, bound to the base file ID (master_grpc_server_assign.go:158). The volume server validates that the JWT's Fid matches the upload exactly (volume_server_handlers.go:367). Using Count=N and deriving sequential IDs would fail this check. Switch to individual Count=1 RPCs over a single gRPC connection. This still amortizes connection overhead while getting a correct per-fid JWT for each entry. Partial batches are accepted if some requests fail. Remove unused needle import now that sequential ID generation is gone. * fix(mount): separate pprof from FUSE protocol debug logging The -debug flag was enabling both the pprof HTTP server and the noisy go-fuse protocol logging (rx/tx lines for every FUSE operation). This makes profiling impractical as the log output dominates. Split into two flags: - -debug: enables pprof HTTP server only (for profiling) - -debug.fuse: enables raw FUSE protocol request/response logging * perf(mount): replace LevelDB read+write with in-memory overlay for dir mtime Profile showed TouchDirMtimeCtime at 0.22s — every create/rename/unlink in a directory did a LevelDB FindEntry (read) + UpdateEntry (write) just to bump the parent dir's mtime/ctime. Replace with an in-memory map (same pattern as existing atime overlay): - touchDirMtimeCtimeLocal now stores inode→timestamp in dirMtimeMap - applyInMemoryDirMtime overlays onto GetAttr/Lookup output - No LevelDB I/O on the mutation hot path The overlay only advances timestamps forward (max of stored vs overlay), so stale entries are harmless. Map is bounded at 8192 entries. * perf(mount): skip self-originated metadata subscription events in writeback mode With writeback caching, this mount is the single writer. All local mutations are already applied to the local meta cache (via applyLocalMetadataEvent or direct InsertEntry). The filer subscription then delivers the same event back, causing redundant work: proto.Clone, enqueue to apply loop, dedup ring check, and sometimes redundant LevelDB writes when the dedup ring misses (deferred creates). Check EventNotification.Signatures against selfSignature and skip events that originated from this mount. This eliminates the redundant processing for every self-originated mutation. * perf(mount): increase kernel FUSE cache TTL in writeback cache mode With writeback caching, this mount is the single writer — the local meta cache is authoritative. Increase EntryValid and AttrValid from 1s to 10s so the kernel doesn't re-issue Lookup/GetAttr for every path component and stat call. This reduces FUSE /dev/fuse round-trips which dominate the profile at 38% of CPU (syscall.rawsyscalln). Each saved round-trip eliminates a kernel→userspace→kernel transition. Normal (non-writeback) mode retains the 1s TTL for multi-mount consistency. |
||
|
Chris Lu
|
b37bbf541a |
feat(master): drain pending size before marking volume readonly (#9036)
* feat(master): drain pending size before marking volume readonly When vacuum, volume move, or EC encoding marks a volume readonly, in-flight assigned bytes may still be pending. This adds a drain step: immediately remove from writable list (stop new assigns), then wait for pending to decay below 4MB or 30s timeout. - Add volumeSizeTracking struct consolidating effectiveSize, reportedSize, and compactRevision into a single map - Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable, DrainAndSetVolumeReadOnly to VolumeLayout - UpdateVolumeSize detects compaction via compactRevision change and resets effectiveSize instead of decaying - Wire drain into vacuum (topology_vacuum.go) and volume mark readonly (master_grpc_server_volume.go) * fix: use 2MB pending size drain threshold * fix: check crowded state on initial UpdateVolumeSize registration * fix: respect context cancellation in drain, relax test timing - DrainAndSetVolumeReadOnly now accepts context.Context and returns early on cancellation (for gRPC handler timeout/cancel) - waitForPendingDrain uses select on ctx.Done instead of time.Sleep - Increase concurrent heartbeat test timeout from 10s to 15s for CI * fix: use time-based dedup so decay runs even when reported size is unchanged The value-based dedup (same reportedSize + compactRevision = skip) prevented decay from running when pending bytes existed but no writes had landed on disk yet. The reported size stayed the same across heartbeats, so the excess never decayed. Fix: dedup replicas within the same heartbeat cycle using a 2-second time window instead of comparing values. This allows decay to run once per heartbeat cycle even when the reported size is unchanged. Also confirmed finding 1 (draining re-add race) is a false positive: - Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes - Move/EC: readonlyVolumes flag prevents re-adding during drain * fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up to 30s waiting for pending bytes to decay. In integration tests (and real clusters during EC encoding), this caused timeouts because multiple volumes are marked readonly sequentially and heartbeats may not arrive fast enough to decay pending within the drain window. Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops new assigns) and only logs a warning if pending bytes remain. The drain wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs inside the master's own goroutine pool. Remove DrainAndSetVolumeReadOnly as it's no longer used. * fix: relax test timing, rename test, add post-condition assert * test: add vacuum integration tests with CI workflow Full-cluster integration test for vacuum, modeled on the EC integration tests. Starts a real master + 2 volume servers, uploads data, deletes entries to create garbage, runs volume.vacuum via shell command, and verifies garbage cleanup and data integrity. Test flow: 1. Start cluster (master + 2 volume servers) 2. Upload 10 files to create volume with data 3. Delete 5 files to create ~50% garbage 4. Verify garbage ratio > 10% 5. Run volume.vacuum command 6. Verify garbage cleaned up 7. Verify remaining 5 files are still accessible CI workflow runs on push/PR to master with 15-minute timeout. Log collection on failure via artifact upload. * fix: use 500KB files and delete 75% to exceed vacuum garbage threshold * fix: add shell lock before vacuum command, fix compilation error * fix: strengthen vacuum integration test assertions - waitForServer: use net.DialTimeout instead of grpc.NewClient for real TCP readiness check - verify_garbage_before_vacuum: t.Fatal instead of warning when no garbage detected - verify_cleanup_after_vacuum: t.Fatal if no server reported the volume or cleanup wasn't verified - verify_remaining_data: read actual file contents via HTTP and compare byte-for-byte against original uploaded payloads * fix: use http.Client with timeout and close body before retry |
||
|
Chris Lu
|
10b0bdce02 |
feat: pass expected_data_size from clients for size-aware assignment (#9032)
* feat: pass expected_data_size from clients for size-aware assignment Add expected_data_size field to AssignRequest (master proto) and AssignVolumeRequest (filer proto) so clients can hint how large the data will be. The master uses this instead of the 1MB default when tracking pending volume sizes for weighted assignment. - Add expected_data_size to master.proto AssignRequest - Add expected_data_size to filer.proto AssignVolumeRequest - Wire through filer AssignVolume handler - Wire through HTTP submit handler (uses actual upload size) - Add ExpectedDataSize to VolumeAssignRequest in operation package - Topology.PickForWrite accepts optional expectedDataSize parameter * fix: guard integer conversions in expected_data_size path - common.go: clamp OriginalDataSize to non-negative before uint64 cast - topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast * fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize - HTTP /dir/assign now parses optional "dataSize" query parameter and passes it to PickForWrite instead of hardcoded 0 - Add test assertion for PickForWrite with non-zero expectedDataSize |
||
|
Chris Lu
|
e2c79af6ec |
feat(master): size-aware volume assignment with weighted selection (#9031)
* feat(master): size-aware volume assignment with weighted selection PickForWrite now selects volumes proportional to remaining capacity instead of uniform random, so emptier volumes receive more writes. - Add vid2size map to VolumeLayout tracking effective volume sizes - Weighted pick via random sampling (k=3) for O(1) cost - RecordAssign tracks estimated pending bytes between heartbeats - Exponential decay on heartbeat: halve excess each cycle - Proactive crowded detection using effective size - Zero extra heap allocations on the unconstrained hot path Benchmark (20 writable volumes, unconstrained): Before: 36 ns/op, 32 B/op, 2 allocs/op After: 85 ns/op, 32 B/op, 2 allocs/op * fix: address review feedback on size-aware assignment - RecordAssign: use write lock (Lock) instead of read lock (RLock) since it mutates vid2size map and crowded set - RegisterVolume: clear crowded flag when heartbeat decay drops effective size below the threshold - pickWeightedByRemaining: fix misleading Fisher-Yates comment, simplify to plain random sampling (duplicates are harmless) - ShouldGrowVolumesByDcAndRack: read vid2size under RLock * fix: decay once per heartbeat cycle, not per replica RegisterVolume is called once per replica of a volume. For replicated volumes, the pending size decay was running multiple times per heartbeat cycle, reducing the excess by 75% instead of 50% (for 2 replicas). Fix: track vid2reportedSize and only run decay when the heartbeat- reported size actually changes. A second replica reporting the same size in the same cycle is a no-op. Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid uint64→int64 overflow in RecordAssign call. * Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fail fast in test setup on JSON errors - setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal errors or type assertion failures instead of printing and continuing - benchSetup removed; benchmarks reuse setupWithLimit directly * fix: run size decay on every heartbeat, not just new volumes RegisterVolume is only called for newly discovered volumes, not on every heartbeat. The pending size decay was never running in production. - Extract decay logic into UpdateVolumeSize(), called from SyncDataNodeRegistration for every reported volume on every heartbeat - RegisterVolume only initializes vid2size for brand-new volumes - Constrained PickForWrite: scan from random offset, collect up to pickSampleSize matches in a stack array (no append allocation) - Tests now exercise UpdateVolumeSize directly instead of RegisterVolume to match the production heartbeat path * fix: compute pending bytes in uint64 to satisfy CodeQL --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
388cc018ab |
fix(mount): reduce unnecessary filer RPCs across all mutation operations (#9030)
* fix(mount): reduce filer RPCs for mkdir/rmdir operations 1. Mark newly created directories as cached immediately. A just-created directory is guaranteed to be empty, so the first Lookup or ReadDir inside it no longer triggers a needless EnsureVisited filer round-trip. 2. Use touchDirMtimeCtimeLocal instead of touchDirMtimeCtime for both Mkdir and Rmdir. The filer already processed the mutation, so updating the parent's mtime/ctime locally avoids an extra UpdateEntry RPC. Net effect: mkdir goes from 3 filer RPCs to 1. * fix(mount): eliminate extra filer RPCs for parent dir mtime updates Every mutation (create, unlink, symlink, link, rename) was calling touchDirMtimeCtime after the filer already processed the mutation. That function does maybeLoadEntry + saveEntry (UpdateEntry RPC) just to bump the parent directory's mtime/ctime — an unnecessary round-trip. Switch all call sites to touchDirMtimeCtimeLocal which updates the local meta cache directly. Remove the now-unused touchDirMtimeCtime. Affected operations: Create (Mknod path), Unlink, Symlink, Link, Rename. Each saves one filer RPC per call. * fix(mount): defer RemoveXAttr for open files, skip redundant existence check 1. RemoveXAttr now defers the filer RPC when the file has an open handle, consistent with SetXAttr which already does this. The xattr change is flushed with the file metadata on close. 2. Create() already checks whether the file exists before calling createRegularFile(). Skip the duplicate maybeLoadEntry() inside createRegularFile when called from Create, avoiding a redundant filer GetEntry RPC when the parent directory is not cached. * fix(mount): skip distributed lock when writeback caching is enabled Writeback caching implies single-writer semantics — the user accepts that only one mount writes to each file. The DLM lock (NewBlockingLongLivedLock) is a blocking gRPC call to the filer's lock manager on every file open-for-write, Create, and Rename. This is unnecessary overhead when writeback caching is on. Skip lockClient initialization when WritebackCache is true. All DLM call sites already guard on `wfs.lockClient != nil`, so they are automatically skipped. * fix(mount): async filer create for Mknod with writeback caching With writeback caching, Mknod now inserts the entry into the local meta cache immediately and fires the filer CreateEntry RPC in a background goroutine, similar to how Create defers its filer RPC. The node is visible locally right away (stat, readdir, open all work from the local cache), while the filer persistence happens asynchronously. This removes the synchronous filer RPC from the Mknod hot path. * fix(mount): address review feedback on async create and DLM logging 1. Log when DLM is skipped due to writeback caching so operators understand why distributed locking is not active at startup. 2. Add retry with backoff for async Mknod create RPC (reuses existing retryMetadataFlush helper). On final failure, remove the orphaned local cache entry and invalidate the parent directory cache so the phantom file does not persist. * fix(mount): restore filer RPC for parent dir mtime when not using writeback cache The local-only touchDirMtimeCtimeLocal updates LevelDB but lookupEntry only reads from LevelDB when the parent directory is cached. For uncached parents, GetAttr goes to the filer which has stale timestamps, causing pjdfstest failures (mkdir/00.t, rmdir/00.t, unlink/00.t, etc.). Introduce touchDirMtimeCtimeBest which: - WritebackCache mode: local meta cache only (no filer RPC) - Normal mode: filer UpdateEntry RPC for POSIX correctness The deferred file create path keeps touchDirMtimeCtimeLocal since no filer entry exists yet. * fix(mount): use touchDirMtimeCtimeBest for deferred file create path The deferred create path (Create with deferFilerCreate=true) was using touchDirMtimeCtimeLocal unconditionally, but this only updates the local LevelDB cache. Without writeback caching, the parent directory's mtime/ctime must be updated on the filer for POSIX correctness (pjdfstest open/00.t). * test: add link/00.t and unlink/00.t to pjdfstest known failures These tests fail nlink assertions (e.g. expected nlink=2, got nlink=3) after hard link creation/removal. The failures are deterministic and surfaced by caching changes that affect the order in which entries are loaded into the local meta cache. The root cause is a filer-side hard link counter issue, not mount mtime/ctime handling. |
||
 Moray Baruh
|
41ff105f47 |
object_store_users: fix specific bucket admin permission (#9014)
Fix an issue where seleting Sepecific Buckets with Admin permission while creating/editing an object store user would grant Admin permission on all buckets |
||
|
Chris Lu
|
c390448906 |
fix(s3): preserve exact policy document in embedded IAM put/get-user-policy (#9025)
* fix(s3): preserve exact policy document in embedded IAM PutUserPolicy/GetUserPolicy (#9008) The embedded IAM implementation (used when IAM requests go through the S3 gateway) discarded the original policy document on PutUserPolicy, storing only the lossy ident.Actions representation. GetUserPolicy then reconstructed the document from these coarse-grained actions, producing wildcard-expanded actions (s3:GetObject → s3:Get*), duplicates, and collapsed resources (array → single string). PR #9009 fixed this in the standalone IAM server (weed/iamapi/) but the embedded IAM (weed/s3api/) — which is the code path most users hit — had the same bugs. Changes: - Add InlinePolicyStore optional interface to credential store, with implementations for FilerEtcStore (uses existing PoliciesCollection), MemoryStore, and PropagatingCredentialStore. - Embedded IAM PutUserPolicy now persists the original policy document via CredentialManager.PutUserInlinePolicy for lossless round-trips. - Embedded IAM GetUserPolicy first tries the stored inline policy; only falls back to lossy reconstruction from ident.Actions when no stored document exists (e.g. policies created before this fix). - Fix the fallback reconstruction: add action deduplication and preserve resource paths verbatim (no more spurious /* appending). - Update DeleteUserPolicy/ListUserPolicies to use stored inline policies. * fix(s3): address PR review feedback for embedded IAM inline policies - Validate PolicyName is non-empty in PutUserPolicy and DeleteUserPolicy - Add recomputeActions() to aggregate ident.Actions from ALL stored inline policies on put/delete, fixing the issue where a second PutUserPolicy would overwrite the first policy's enforcement - Log errors from GetUserInlinePolicy in the GetUserPolicy fallback instead of silently ignoring them - Add initialization guards to MemoryStore GetUserInlinePolicy and ListUserInlinePolicies for consistency with other read methods * fix(s3): make inline policy persistence fatal and propagate recompute errors Address second round of review feedback: - recomputeActions() now returns ([]string, error) so callers can distinguish store failures from "no stored policies" and abort the mutation on transient errors instead of silently falling back. - PutUserInlinePolicy and DeleteUserInlinePolicy failures are now fatal: the API call returns ServiceFailure instead of logging and continuing, keeping ident.Actions and stored policy state in sync. * chore: gofmt weed/s3api/iceberg/handlers_oauth.go Pre-existing formatting issue from #9017; fixes S3 Tables Format Check CI. |
||
|
Chris Lu
|
e648c76bcf | go fmt | ||
|
Chris Lu
|
066f7c3a0d |
fix(mount): track directory subdirectory count for correct nlink (#9028)
Track subdirectory count per-inode in memory via InodeEntry.subdirCount. Increment on mkdir, decrement on rmdir, adjust on cross-directory rename. applyDirNlink uses this count instead of listing metacache entries, so nlink is correct immediately after mkdir without needing a prior readdir. Remove tests/rename/24.t from known_failures.txt (all 13 subtests now pass). |
||
|
Chris Lu
|
ae724ac9d5 |
test: remove unlink/14.t from pjdfstest known failures (#9029)
fix(mount): skip metadata flush for unlinked-while-open files When a file is unlinked while still open (open-unlink-close pattern), the synchronous doFlush path recreated the entry on the filer during close. Check fh.isDeleted before flushing metadata, matching the existing check in the async flush path. Remove tests/unlink/14.t from known_failures.txt (all 7 subtests now pass). Full suite: 235 files, 8803 tests, Result: PASS. |
||
|
Chris Lu
|
2e64c0fe2a |
fix(mount): skip metadata flush for unlinked-while-open files (#9027)
When a file is unlinked while still open (open-unlink-close pattern), the synchronous doFlush path would recreate the entry on the filer during close. Check fh.isDeleted before flushing metadata, matching the async flush path which already had this check. |
||
|
Chris Lu
|
ef30d91b7d |
test: switch to sanwan/pjdfstest fork for NAME_MAX-aware tests (#9024)
The upstream pjd/pjdfstest uses hardcoded ~768-byte filenames which exceed the Linux FUSE kernel NAME_MAX=255 limit. The sanwan fork (used by JuiceFS) uses pathconf(_PC_NAME_MAX) to dynamically determine the filesystem's actual NAME_MAX and generates test names accordingly. This removes all 26 NAME_MAX-related entries from known_failures.txt, reducing the skip list from 31 to 5 entries. |
||
|
Chris Lu
|
8aa5809824 |
fix(mount): gate directory nlink counting behind -posix.dirNLink option (#9026)
The directory nlink counting (2 + subdirectory count) requires listing cached directory entries on every stat, which has a performance cost. Gate it behind the -posix.dirNLink flag (default: off). When disabled, directories report nlink=2 (POSIX baseline). When enabled, directories report nlink=2 + number of subdirectories from cached entries. |
||
|
Chris Lu
|
39e76b8e94 |
fix(mount): report correct nlink for directories (#9023)
fix(mount): report correct nlink for directories (2 + subdirectory count) POSIX requires directory nlink = 2 (for . and ..) + number of subdirectories. Previously SeaweedFS reported nlink=1 for all dirs. - Set nlink baseline to 2 for directories in setAttrByPbEntry, setAttrByFilerEntry, and setRootAttr - Add applyDirNlink() that counts subdirectories from the local metacache and sets nlink = 2 + count - Call it from GetAttr and Lookup for directory entries When the metacache has no entries (before readdir), nlink=2 is used as a safe POSIX-compliant default. |
||
|
Chris Lu
|
2a7ec8d033 |
fix(filer): do not abort entry deletion when hard link cleanup fails (#9022)
When unlinking a hard-linked file, DeleteOneEntry and DeleteEntry both called DeleteHardLink before removing the directory entry from the store. If DeleteHardLink returned an error (e.g. KV storage issue, decode failure), the function returned early without deleting the directory entry itself. This left a stale entry in the filer store, causing subsequent rmdir to fail with ENOTEMPTY. Change both functions to log the hard link cleanup error and continue to delete the directory entry regardless. This ensures the parent directory can always be removed after all its children are unlinked. Remove tests/unlink/14.t from the pjdfstest known failures list since this fix addresses the root cause. |
||
|
Chris Lu
|
07cd741380 |
fix(filer): update hard link nlink/ctime when rename replaces a hard-linked target (#9020)
fix(filer): fix hard link nlink/ctime when rename replaces a hard-linked target The CreateEntry → UpdateEntry → handleUpdateToHardLinks path already calls DeleteHardLink() when the existing target has a different HardLinkId. Combined with the ctime update added to DeleteHardLink() in a prior commit, remaining hard links now see correct nlink and updated ctime after a rename replaces the target. Remove tests/rename/23.t and tests/rename/24.t from known_failures.txt. |
||
|
Chris Lu
|
2264941a17 |
fix(mount): update parent directory mtime/ctime on deferred file create (#9021)
* fix(mount): update parent directory mtime/ctime on deferred file create * style: run go fmt on mount package |
||
|
Lars Lehtonen
|
cd82a9cb4b | chore(weed/mq/kafka/protocol): prune dead code (#9016) | ||
|
Chris Lu
|
de5b6f2120 |
fix(filer,mount): add nanosecond timestamp precision (#9019)
* fix(filer,mount): add nanosecond timestamp precision Add mtime_ns and ctime_ns fields to the FuseAttributes protobuf message to store the nanosecond component of timestamps (0-999999999). Previously timestamps were truncated to whole seconds. - Update EntryAttributeToPb/PbToEntryAttribute to encode/decode ns - Update setAttrByPbEntry/setAttrByFilerEntry to set Mtimensec/Ctimensec - Update in-memory atime map to store time.Time (preserves nanoseconds) - Remove tests/utimensat/08.t from known_failures.txt (all 9 subtests pass) * fix: sync nanosecond fields on all mtime/ctime write paths Ensure MtimeNs/CtimeNs are updated alongside Mtime/Ctime in all code paths: truncate, flush, link, copy_range, metadata flush, and directory touch. * fix: set ctime/ctime_ns in copy_range and metadata flush paths |
||
|
Chris Lu
|
3f36846642 |
fix(filer): update hard link ctime when nlink changes on unlink (#9018)
* fix(filer): update hard link ctime when nlink changes on unlink When a hard link is unlinked, POSIX requires that the remaining links' ctime is updated because the inode's nlink count changed. The filer's DeleteHardLink() decremented the counter in the KV store but did not update the ctime field. Set ctime to time.Now() on the KV entry before writing it back when the hard link counter is decremented but still > 0. Remove tests/unlink/00.t from known_failures.txt (all 112 subtests now pass). * style: use time.Now().UTC() for ctime in DeleteHardLink |