41aedaa687
Shell: support regular expression for collection selection ( #7158 )
...
* support regular expression for collection selection
* refactor
* ordering
* fix exact match
* Update command_volume_balance_test.go
* simplify
* Update command_volume_balance.go
* comment
2025-08-23 11:04:24 -07:00
535985adb6
Shell: add verbose ec encoding mode ( #7105 )
...
* add verbose ec encoding mode
* address comments
2025-08-07 00:12:05 -07:00
365d03ff32
mount ec shards correctly ( #7079 )
2025-08-03 23:10:28 -07:00
69553e5ba6
convert error fromating to %w everywhere ( #6995 )
2025-07-16 23:39:27 -07:00
44dfa793d5
Collecting volume locations for volumes before EC encoding
...
fix https://github.com/seaweedfs/seaweedfs/issues/6963
2025-07-14 12:17:33 -07:00
f894e7b7a5
Support filtering source disk type in volume.tier.upload ( #6868 )
2025-06-15 20:30:04 -07:00
0be020b0fa
Nit: unify the default --maxParallelization value for weed shell commands supporting this option ( #6788 )
2025-05-13 07:59:26 -07:00
848d1f7c34
Improve safety for weed shell's ec.encode. ( #6773 )
...
Improve safety for weed shells `ec.encode`.
The current process for `ec.encode` is:
1. EC shards for a volume are generated and added to a single server
2. The original volume is deleted
3. EC shards get re-balanced across the entire topology
It is then possible to lose data between #2 and #3 , if the underlying volume storage/server/rack/DC
happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so:
* Newly created EC shards are spread across all locations for the source volume.
* Source volumes are deleted only after EC shards are converted and balanced.
2025-05-09 09:01:32 -07:00
97dad06ed8
Improve parallelization for ec.encode ( #6769 )
...
Improve parallelization for `ec.encode`.
Instead of processing one volume at at time, perform all EC conversion
steps (mark readonly -> generate EC shards -> delete volume -> remount) in
parallel for all of them.
This should substantially improve performance when EC encoding
entire collections.
2025-05-08 17:14:14 -07:00
c07596691c
ec.encode: Fix resolution of target collections. (#6585 )
...
* Don't ignore empty (`""`) collection names when computing collections for a given volume ID.
* `ec.encode`: Fix resolution of target collections.
When no `volumeId` parameter is provided, compute volumes
based on the provided collection name, even if it's empty (`""`).
This restores behavior to before recent EC rebalancing rework. See also
https://github.com/seaweedfs/seaweedfs/blob/ec30a504bae6cad75f859964e14c60d39cc43709/weed/shell/command_ec_encode.go#L99 .
2025-02-28 11:42:19 -08:00
392656d59e
ec.encode: Explictly mount EC shards after volume conversion. (#6528 )
...
This guarantees EC shards are immediately available after encoding,
even if not affected by subsequent re-balancing.
2025-02-10 09:49:58 -08:00
eab2e0e112
ec.encode: Fix bug causing source volumes not being deleted after EC conversion. (#6447 )
...
This logic was originally part of `spreadEcShards()`, which got removed during
the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344 ),
accidentally breaking functionality in the process.
The commit restores the deletion code for EC'd volumes - with parallelization support.
2025-01-17 01:02:30 -08:00
4d91ec359b
Fix volume replica parallelization within ec.encode. ( #6377 )
...
See 826edd5d
2024-12-19 17:46:11 -08:00
ba0707af64
Allow configuring the maximum number of concurrent tasks for EC parallelization. ( #6376 )
...
Follow-up to b0210df0
2024-12-18 13:26:26 -08:00
44c48c929a
Parallelize volume replica operations within ec.encode. ( #6374 )
2024-12-18 11:59:48 -08:00
b0210df081
Begin implementing EC balancing parallelization support. ( #6342 )
...
* Begin implementing EC balancing parallelization support.
Impacts both `ec.encode` and `ec.balance`,
* Nit: improve type naming.
* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.
* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.
* Fix concurrency bug.
* Fix whitespace after Gitlab automerge.
* Delete stray TODO.
2024-12-12 09:14:44 -08:00
23ffbb083c
Limit EC re-balancing for ec.encode to relevant collections when a volume ID argument is provided. ( #6347 )
...
Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
2024-12-12 08:41:33 -08:00
6320036c56
Delete legacy balancing code for ec.encode. ( #6344 )
2024-12-12 07:42:03 -08:00
8c82c037b9
Unify the re-balancing logic for ec.encode with ec.balance. ( #6339 )
...
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
2024-12-10 13:30:13 -08:00
0d5393641e
Unify usage of shell.EcNode.dc as DataCenterId. ( #6258 )
2024-11-19 06:33:18 -08:00
72b14a451e
delete aborted ec shards from both source and target servers ( #6221 )
...
fix https://github.com/seaweedfs/seaweedfs/issues/6205#issuecomment-2465004586
2024-11-09 11:32:08 -08:00
c29c912bdc
fix format ( #6185 )
...
unitest weed/shell fail
2024-10-31 08:30:35 -07:00
9105c6bdd1
fix format
2024-10-28 11:29:08 -07:00
089d4316ef
ensure 2 volume space since actual need 1.4x volume size empty space
2024-10-24 22:44:53 -07:00
6e388e29c9
correcting free volume count, factor it during ec encoding to ensure enough disk space available
...
fix https://github.com/seaweedfs/seaweedfs/issues/6163
2024-10-24 22:42:38 -07:00
ec30a504ba
refactor
2024-09-29 10:38:22 -07:00
701abbb9df
add IsResourceHeavy() to command interface
2024-09-28 20:23:01 -07:00
d056c0ddf2
fix(volume): don't persist RO state in specific cases ( #6058 )
...
* fix(volume): don't persist RO state in specific cases
* fix(volume): writable always persist
2024-09-24 16:15:54 -07:00
0c62d591e2
Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) ( #5635 )
2024-06-02 14:16:05 -07:00
2bc05f70e7
log full percentage
2023-10-22 12:59:34 -07:00
0fd7222d65
default to skip if less than 4 nodes
2023-10-05 11:13:48 -07:00
31b2751aff
clone volume locations in case they are changed
...
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-07-06 00:32:58 -07:00
25535e9c36
Delete volume is empty ( #4561 )
...
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co >
2023-06-12 10:42:44 -07:00
31bb91583f
fix bug when vid not found
...
fix https://github.com/seaweedfs/seaweedfs/issues/4193
2023-02-09 17:30:44 -08:00
676e27c589
shell: stop long running jobs if lock is lost
2022-08-22 14:12:23 -07:00
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
f18803424a
volume.balance: add delay during tight loop
...
fix https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:53:55 -08:00
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
a2d3f89c7b
add lock messages
2021-12-10 13:24:38 -08:00
00ae965d8d
randomize a bit for ec shards distribution
2021-11-04 09:23:40 -07:00
794375ca0a
adjust help message since both fullPercent and quietFor are needed.
2021-11-01 17:22:47 -07:00
119d5908dd
shell: do not need to lock to see volume -h
2021-09-13 22:13:34 -07:00
6cd1ce8b74
erasure coding: add cleanup step if anything goes wrong
2021-09-13 01:55:49 -07:00
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
0f7d4556d8
shell: volume.tier.move makes up changes if volume move failed
2021-08-13 03:09:28 -07:00
76c48ffe27
optional parallel copy ec shards
...
fix https://github.com/chrislusf/seaweedfs/issues/2048
2021-05-06 01:53:35 -07:00
1b1c018165
adjust text
2021-02-28 19:02:43 -08:00
1c233ad986
refactoring
2021-02-22 00:28:42 -08:00
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
chrislu
NyaMisty
Lisandro Pin
wyang
Max Denushev
Konstantin Lebedev
chrislu