7 Commits

Author SHA1 Message Date
qzhello 4f9393889c feat(shell): Support batched EC encode and multi-volume selection in ec.encode (#10030)
* fix(shell): correct volume.list -writable filter unit and comparison

* fix(shell): correct volume.list -writable filter unit and comparison

* chore(shell): fix typo in EC shard helper param names

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* fix(shell): use exact match for volume.balance -racks/-nodes filter

The old strings.Contains-based filter quietly included any id that was a
  substring of the user-supplied flag value (e.g. -racks=rack10 also matched
  rack1). Replace it with an exact-match set parsed from the comma-separated
  flag value, and add regression tests for both -racks and -nodes paths.

  Also fix a small typo in the "remote storage" error returned by
  maybeMoveOneVolume.

* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers

* feat(shell): support batched EC encode and multi-volume selection

Add -volumeIds (comma-separated) and -batchSize flags to ec.encode.

When -batchSize > 0, volumes are processed in independent batches, each
committed separately: encode -> rebalance -> verify -> delete originals.
This bounds the working set and lets source volumes be reclaimed without
waiting for the entire set to finish, at the cost of per-batch rebalancing.
Because each batch deletes its originals, a failure in a later batch is
unrecoverable for already-completed batches.

To let the single-volume, multi-volume, and collection paths share one
per-batch routine, the re-balance scope is now always derived from the
volumes actually selected for encoding (collectCollectionsForVolumeIds),
rather than every collection matching the -collection regex. Practical
effect: with -collection, a collection that matches the pattern but
contributes no encodable volumes is no longer re-balanced as a side effect.
The -volumeId path is unchanged; -batchSize=0 (default) preserves the
original single-pass flow.

The per-batch routine reuses the existing assertEncodableRegularVolumes
guard, doEcEncode skipped-node handling, and verifyEcShardsBeforeDelete
retry loop. The capacity pre-flight check takes the already-fetched
topology instead of issuing another VolumeList to the master per batch.

Also clarify the -collection flag description to note it accepts a regex
pattern, matching the existing command help.

-volumeId and -volumeIds are mutually exclusive; ids in -volumeIds are
validated and de-duplicated.
2026-06-22 01:22:20 -07:00
Chris Lu 2ed5a8f65c add tests 2026-02-09 01:37:56 -08:00
Lisandro Pin f2db746690 Introduce logic to resolve volume replica placement within EC rebalancing. (#6254)
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.

All tests defined in this file are now for `command_ec_common.go`.

* Minor code cleanups.

- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.

* Introduce logic to resolve volume replica placement within EC rebalancing.

This will be used to make rebalancing logic topology-aware.

* Give shell.EcNode.dc a dedicated DataCenterId type.
2024-11-18 18:05:06 -08:00
Ryan Russell bd2dc6d641 refactor(shell): Decending -> Descending (#3675)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:06:48 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 95717d1006 simpler output 2022-02-24 13:50:08 -08:00
chrislu 13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00