mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-06-13 23:36:45 +03:00
docs: document -metadataStrategy and -listingCacheTTL flags for remote.mount
Add metadata strategies section (eager/lazy/lazy+listingCacheTTL) to Mount-Remote-Storage with comparison table. Update Cloud-Drive-Architecture to mention lazy mounts and listing cache.
@@ -43,7 +43,7 @@ The remote storage, e.g., AWS S3, can be [[configured|Configure Remote Storage]]
|
||||
> remote.mount -dir=/path/to/xxx -remote=cloud1/bucket
|
||||
```
|
||||
|
||||
On mount, all the metadata will be pulled down and cached to the local filer store.
|
||||
By default (eager strategy), all the metadata will be pulled down and cached to the local filer store on mount. You can also use `-metadataStrategy=lazy` to skip the upfront pull and fetch metadata on demand, and `-listingCacheTTL=<seconds>` to enable automatic directory listing cache with a TTL.
|
||||
|
||||
The metadata will be used for all metadata operations, such as listing, directory traversal, read file size, compare file modification time, etc, which will be free and fast as usual, without any API calls to the cloud.
|
||||
|
||||
@@ -55,6 +55,8 @@ To fetch metadata changes in the remote storage, just sync the whole mounted dir
|
||||
```
|
||||
The latest cloud metadata will be saved(except deletions). It is a light weight process and you can run it regularly.
|
||||
|
||||
See [[Mount Remote Storage]] for details on metadata strategies and listing cache.
|
||||
|
||||
## Cache/Uncache File Content
|
||||
|
||||
By default, the file content is [[cached|Cache Remote Storage]] to local volume servers on the first read.
|
||||
|
||||
+52
-6
@@ -11,6 +11,10 @@ Usage of remote.mount:
|
||||
allows the mounting over a non-empty directory
|
||||
-remote string
|
||||
a directory in remote storage, ex. <storageName>/<bucket>/path/to/dir
|
||||
-metadataStrategy string
|
||||
lazy: skip upfront metadata pull; eager: full metadata pull (default "eager")
|
||||
-listingCacheTTL int
|
||||
seconds to cache remote directory listings (0 = disabled)
|
||||
|
||||
> help remote.mount
|
||||
remote.mount # mount remote storage and pull its metadata
|
||||
@@ -18,11 +22,16 @@ Usage of remote.mount:
|
||||
# assume a remote storage is configured to name "s3_1"
|
||||
remote.configure -name=cloud1 -type=s3 -s3.access_key=xyz -s3.secret_key=yyy
|
||||
|
||||
# mount and pull one bucket
|
||||
# mount and pull one bucket (eager metadata pull, the default)
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket
|
||||
# mount and pull one directory in the bucket
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket/dir1
|
||||
|
||||
# lazy mount: skip upfront metadata pull, fetch on demand
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy
|
||||
# lazy mount with on-demand directory listing cached for 5 minutes
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy -listingCacheTTL=300
|
||||
|
||||
# after mount, start a separate process to write updates to remote storage
|
||||
weed filer.remote.sync -filer=<filerHost>:<filerPort> -dir=/xxx
|
||||
|
||||
@@ -30,17 +39,52 @@ Usage of remote.mount:
|
||||
|
||||
With `remote.mount`, you can mount one bucket or any directory in the bucket.
|
||||
|
||||
## Cache Metadata
|
||||
## Metadata Strategies
|
||||
|
||||
This `remote.mount` will also pull down all metadata from the remote storage.
|
||||
The `-metadataStrategy` flag controls how metadata is cached when mounting:
|
||||
|
||||
Later, any metadata operations will be fast to just read local metadata.
|
||||
### Eager (default)
|
||||
|
||||
```
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=eager
|
||||
```
|
||||
|
||||
Pulls down **all** metadata from the remote storage at mount time. All subsequent metadata operations (listing, stat, etc.) read from the local filer store with no remote API calls.
|
||||
|
||||
Best for: buckets with a manageable number of objects where you want instant local access.
|
||||
|
||||
### Lazy
|
||||
|
||||
```
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy
|
||||
```
|
||||
|
||||
Skips the upfront metadata pull. Individual file metadata is fetched on demand when a file is first accessed (stat, read, etc.) and cached locally.
|
||||
|
||||
Best for: very large buckets where pulling all metadata upfront would be too slow or expensive.
|
||||
|
||||
### Lazy with Directory Listing Cache
|
||||
|
||||
```
|
||||
remote.mount -dir=/xxx -remote=cloud1/bucket -metadataStrategy=lazy -listingCacheTTL=300
|
||||
```
|
||||
|
||||
With `-listingCacheTTL=<seconds>`, directory listings (`ls`, S3 `ListObjects`) will automatically fetch from remote and cache results. The cached listing expires after the specified TTL, at which point the next listing triggers a fresh remote fetch.
|
||||
|
||||
This can also be used with eager mounts to keep listings up-to-date when remote data changes over time.
|
||||
|
||||
| Strategy | Upfront cost | File access | Directory listing | Stale data |
|
||||
|----------|-------------|-------------|-------------------|------------|
|
||||
| Eager | High (pulls all metadata) | Instant | Instant | Use `remote.meta.sync` |
|
||||
| Lazy | None | First access fetches | Empty without `-listingCacheTTL` | TTL-based refresh |
|
||||
| Lazy + listingCacheTTL | None | First access fetches | Auto-fetched, cached per TTL | Auto-refreshed |
|
||||
|
||||
`remote.unmount` will drop all local metadata and cached file content.
|
||||
|
||||
### Repeatedly Update Metadata
|
||||
|
||||
Sometimes the data on the cloud has changes and local metadata becomes stale. To `unmount` first and `mount` again can work but costly, since all data has to be cached again.
|
||||
Sometimes the data on the cloud changes and local metadata becomes stale. To `unmount` first and `mount` again can work but is costly, since all data has to be cached again.
|
||||
|
||||
To refresh the metadata changes, you can run this on the mounted directory or any sub directories, e.g.:
|
||||
```
|
||||
@@ -49,7 +93,9 @@ To refresh the metadata changes, you can run this on the mounted directory or an
|
||||
```
|
||||
This will update local metadata accordingly and still keep file contents that are not changed.
|
||||
|
||||
If the data on the cloud can changed often, you can create a cronjob to run it. Or you can add this command to the admin scripts defined in `master.toml`, to run it regularly.
|
||||
If the data on the cloud changes often, you can create a cronjob to run it. Or you can add this command to the admin scripts defined in `master.toml`, to run it regularly.
|
||||
|
||||
With `-listingCacheTTL`, directory listings refresh automatically, reducing the need for manual `remote.meta.sync` for listing purposes.
|
||||
|
||||
## Write Back Changes to Remote Storage
|
||||
|
||||
|
||||
Reference in New Issue
Block a user