mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-06-13 23:36:45 +03:00
s3 APIs
Chris Lu
+208
-459
@@ -1,14 +1,13 @@
|
||||
To be compatible with Amazon S3 API, a separate "weed s3" command is provided. This provides much faster access when reading or writing files, compared to operating files on the cloud.
|
||||
|
||||
# How it works?
|
||||
`weed s3` will start a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer.
|
||||
For convenience, `weed server -s3` will start a master, a volume server, a filer, and the S3 gateway. And `weed filer -s3` can start a filer and the S3 gateway together also.
|
||||
# How It Works
|
||||
|
||||
Each bucket is stored in one collection, and mapped to folder `/buckets/<bucket_name>` by default.
|
||||
`weed s3` starts a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer.
|
||||
For convenience, `weed server -s3` starts a master, a volume server, a filer, and the S3 gateway. `weed filer -s3` can start a filer and the S3 gateway together also.
|
||||
|
||||
A bucket can be deleted efficiently by deleting the whole collection.
|
||||
Each bucket is stored in one collection, and mapped to folder `/buckets/<bucket_name>` by default. A bucket can be deleted efficiently by deleting the whole collection.
|
||||
|
||||
From version 3.51 name of collection is `filerGroup_bucketname` or `bucketname` if filer group is not set.
|
||||
From version 3.51 the collection name is `filerGroup_bucketname` or `bucketname` if filer group is not set.
|
||||
|
||||
## Support Many Buckets
|
||||
|
||||
@@ -28,467 +27,217 @@ This will add 1 physical volume when existing volumes are full. If using replica
|
||||
fs.configure -locationPrefix=/buckets/ -replication=001 -volumeGrowthCount=2 -apply
|
||||
```
|
||||
|
||||
See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
|
||||
|
||||
|
||||
# Supported APIs
|
||||
Currently, the following APIs are supported.
|
||||
|
||||
Some additional endpoints might be (partially), supported but are not in this list.
|
||||
To be sure, you can look at the function defined in the files `weed/s3api/s3api_*_handlers_*.go`.
|
||||
|
||||
|
||||
```
|
||||
// Object operations
|
||||
* PutObject
|
||||
* GetObject
|
||||
* HeadObject
|
||||
* CopyObject
|
||||
* DeleteObject
|
||||
* ListObjectsV2
|
||||
* ListObjectsV1
|
||||
* DeleteMultipleObjects
|
||||
* GetObjectAttributes
|
||||
* PostPolicy
|
||||
|
||||
// Object Tagging
|
||||
* GetObjectTagging
|
||||
* PutObjectTagging
|
||||
* DeleteObjectTagging
|
||||
|
||||
// User Metadata
|
||||
* PutObject (with x-amz-meta-* headers)
|
||||
* GetObject / HeadObject (returns x-amz-meta-* headers)
|
||||
* CopyObject (with metadata directive)
|
||||
|
||||
// Server-Side Encryption
|
||||
* PutObject (with SSE-KMS, SSE-C, SSE-S3)
|
||||
* GetObject (with automatic decryption)
|
||||
* HeadObject (with encryption metadata)
|
||||
* CopyObject (with encryption/decryption)
|
||||
* Multipart uploads with encryption
|
||||
* Bucket default encryption
|
||||
|
||||
// Conditional Operations
|
||||
* All object operations support conditional headers:
|
||||
- If-Match
|
||||
- If-None-Match
|
||||
- If-Modified-Since
|
||||
- If-Unmodified-Since
|
||||
|
||||
// Bucket operations
|
||||
* PutBucket
|
||||
* DeleteBucket
|
||||
* HeadBucket
|
||||
* ListBuckets
|
||||
* PutBucketLifecycleConfiguration (partially, only for TTL)
|
||||
* GetBucketLifecycleConfiguration (partially, only for TTL)
|
||||
* DeleteBucketLifecycleConfiguration (partially, only for TTL)
|
||||
* GetBucketCors
|
||||
* PutBucketCors
|
||||
* DeleteBucketCors
|
||||
|
||||
// Multipart upload operations
|
||||
* NewMultipartUpload
|
||||
* CompleteMultipartUpload
|
||||
* AbortMultipartUpload
|
||||
* ListMultipartUploads
|
||||
* PutObjectPart
|
||||
* CopyObjectPart
|
||||
* ListObjectParts
|
||||
|
||||
// Object Versioning
|
||||
* PutBucketVersioning
|
||||
* GetBucketVersioning
|
||||
* ListObjectVersions
|
||||
* GetObject (with version ID)
|
||||
* PutObject (with versioning)
|
||||
* DeleteObject (with version ID)
|
||||
* CopyObject (with version ID)
|
||||
* RestoreObject (partial)
|
||||
|
||||
// Object Lock and Retention
|
||||
* GetObjectLockConfiguration
|
||||
* PutObjectLockConfiguration
|
||||
* GetObjectRetention
|
||||
* PutObjectRetention
|
||||
* GetObjectLegalHold
|
||||
* PutObjectLegalHold
|
||||
* BypassGovernanceRetention (via x-amz-bypass-governance-retention header)
|
||||
|
||||
// Bucket Policies
|
||||
* PutBucketPolicy
|
||||
* GetBucketPolicy
|
||||
* DeleteBucketPolicy
|
||||
* Supported Conditions:
|
||||
- s3:ExistingObjectTag
|
||||
```
|
||||
|
||||
Not included:
|
||||
|
||||
|
||||
# User Metadata (Custom Attributes)
|
||||
|
||||
SeaweedFS supports S3 user-defined metadata via `x-amz-meta-*` headers. This allows you to attach custom key-value pairs to objects.
|
||||
|
||||
## Setting User Metadata
|
||||
|
||||
```bash
|
||||
# Set environment variables
|
||||
export S3_ENDPOINT=http://localhost:8333
|
||||
|
||||
# Using AWS CLI
|
||||
aws --endpoint-url $S3_ENDPOINT s3 cp myfile.txt s3://mybucket/myfile.txt \
|
||||
--metadata "expire=2025-12-01,author=john,project=demo"
|
||||
|
||||
# Using curl
|
||||
curl -X PUT \
|
||||
-H "x-amz-meta-expire: 2025-12-01" \
|
||||
-H "x-amz-meta-author: john" \
|
||||
--data-binary @myfile.txt \
|
||||
"$S3_ENDPOINT/mybucket/myfile.txt"
|
||||
```
|
||||
|
||||
## Reading User Metadata
|
||||
|
||||
User metadata is returned in response headers when you GET or HEAD an object:
|
||||
|
||||
```bash
|
||||
# Using AWS CLI
|
||||
aws --endpoint-url $S3_ENDPOINT s3api head-object --bucket mybucket --key myfile.txt
|
||||
|
||||
# Using curl
|
||||
curl -I "$S3_ENDPOINT/mybucket/myfile.txt"
|
||||
# Response includes:
|
||||
# x-amz-meta-expire: 2025-12-01
|
||||
# x-amz-meta-author: john
|
||||
```
|
||||
|
||||
## Updating User Metadata
|
||||
|
||||
To update metadata, use CopyObject with `x-amz-metadata-directive: REPLACE`:
|
||||
|
||||
```bash
|
||||
aws --endpoint-url $S3_ENDPOINT s3 cp s3://mybucket/myfile.txt s3://mybucket/myfile.txt \
|
||||
--metadata "expire=2026-01-01" \
|
||||
--metadata-directive REPLACE
|
||||
```
|
||||
|
||||
## Limits
|
||||
|
||||
Metadata keys are case-insensitive and stored in canonical format (e.g., `x-amz-meta-My-Key` becomes `X-Amz-Meta-My-Key`).
|
||||
|
||||
# Feature difference
|
||||
|
||||
| Feature | SeaweedFS | Amazon S3 |
|
||||
| -------------------------------------- | --------- | --------- |
|
||||
| Multi byte ranges reads | Yes | No |
|
||||
| DeleteObject deletes a folder | Yes | No |
|
||||
| same path for both a file and a folder | No | Yes |
|
||||
| allows more than "/" as a delimiter | No | Yes |
|
||||
| Object Versioning | Yes | Yes |
|
||||
| MFA Delete for versioning | No | Yes |
|
||||
| Server-Side Encryption (SSE-KMS) | Yes | Yes |
|
||||
| Server-Side Encryption (SSE-C) | Yes | Yes |
|
||||
| Server-Side Encryption (SSE-S3) | Yes | Yes |
|
||||
| KMS Providers (Multi-cloud) | Yes | No |
|
||||
| Conditional Headers (All operations) | Yes | Yes |
|
||||
| Range requests with SSE-KMS | Yes | Yes |
|
||||
| Range requests with SSE-C | Yes | Yes |
|
||||
| Range requests with SSE-S3 | Yes | Yes |
|
||||
|
||||
## Empty folders
|
||||
|
||||
SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also.
|
||||
|
||||
To be consistent with AWS S3, SeaweedFS automatically cleans up empty folders asynchronously after file deletions.
|
||||
|
||||
# Server-Side Encryption
|
||||
|
||||
Need encryption at rest? SeaweedFS speaks the same SSE dialects as Amazon S3, so your existing tools and SDKs just work. You can choose from three options:
|
||||
|
||||
- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: Use an external KMS (AWS KMS, Google Cloud KMS, OpenBao/Vault)
|
||||
- **[SSE-C](Server-Side-Encryption-SSE-C)**: Bring your own keys for maximum control
|
||||
- **SSE-S3**: Let SeaweedFS manage keys (explicit `AES256` header or bucket default encryption)
|
||||
|
||||
All encryption types support:
|
||||
- Automatic encryption/decryption
|
||||
- Bucket default encryption
|
||||
- Multipart upload encryption
|
||||
- Cross-encryption copy operations
|
||||
- AWS S3 compatibility
|
||||
|
||||
For detailed setup guides and examples, see:
|
||||
- **[Server-Side Encryption Overview](Server-Side-Encryption)**
|
||||
- **[SSE-KMS Guide](Server-Side-Encryption-SSE-KMS)**
|
||||
- **[SSE-C Guide](Server-Side-Encryption-SSE-C)**
|
||||
|
||||
## Quick Examples
|
||||
|
||||
```bash
|
||||
# SSE-KMS (Key Management Service)
|
||||
aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/kms-encrypted.txt --server-side-encryption aws:kms --ssekms-key-id alias/my-key
|
||||
|
||||
# SSE-C (Customer-provided keys)
|
||||
aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/customer-encrypted.txt --sse-c AES256 --sse-c-key fileb://my-key.bin
|
||||
|
||||
# SSE-S3 (Server-managed)
|
||||
aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/server-encrypted.txt --server-side-encryption AES256
|
||||
```
|
||||
|
||||
# S3 Conditional Operations
|
||||
|
||||
SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent operations and efficient caching:
|
||||
|
||||
- **If-Match**: Execute only if ETag matches (optimistic locking)
|
||||
- **If-None-Match**: Execute only if ETag doesn't match (prevent overwrites, caching)
|
||||
- **If-Modified-Since**: Execute only if modified after date (conditional downloads)
|
||||
- **If-Unmodified-Since**: Execute only if not modified after date (safe updates)
|
||||
|
||||
Conditional operations enable:
|
||||
- **Optimistic concurrency control**: Prevent lost updates
|
||||
- **Efficient caching**: Reduce bandwidth with 304 Not Modified
|
||||
- **Atomic operations**: Operations only proceed when safe
|
||||
- **Data integrity**: Prevent accidental overwrites
|
||||
|
||||
For detailed usage patterns and examples, see **[S3 Conditional Operations](S3-Conditional-Operations)**.
|
||||
|
||||
## Quick Examples
|
||||
|
||||
```bash
|
||||
# Get current ETag
|
||||
ETAG=$(aws --endpoint-url $S3_ENDPOINT s3api head-object --bucket mybucket --key file.txt --query ETag --output text)
|
||||
|
||||
# Conditional update (optimistic locking)
|
||||
curl -X PUT -H "If-Match: $ETAG" -d "updated content" "$S3_ENDPOINT/mybucket/file.txt"
|
||||
|
||||
# Conditional download (caching)
|
||||
curl -H "If-None-Match: $ETAG" "$S3_ENDPOINT/mybucket/file.txt"
|
||||
# Returns 304 Not Modified if unchanged
|
||||
|
||||
# Prevent overwrite (atomic create)
|
||||
curl -X PUT -H "If-None-Match: *" -d "new content" "$S3_ENDPOINT/mybucket/newfile.txt"
|
||||
```
|
||||
|
||||
# S3 Authentication
|
||||
|
||||
> **For a complete overview of S3 configuration options and priority, see [[S3 Configuration]].**
|
||||
|
||||
By default, the SeaweedFS S3 gateway starts in **"Allow All" mode** if no credentials have been configured. This means all S3 operations are permitted without authentication.
|
||||
|
||||
As soon as **at least one identity** is configured (via config file, env vars, or shell), the gateway **automatically transitions** to "Authentication Required" mode.
|
||||
|
||||
| Method | Option | Documentation |
|
||||
|--------|--------|---------------|
|
||||
| **Allow All (Default)** | (No config) | See [[S3 Credentials]] |
|
||||
| **Static Configuration** | `-s3.config=config.json` | [[S3 Credentials]] |
|
||||
| **Dynamic Configuration** | `s3.configure` in weed shell | See below |
|
||||
| **Admin UI** | Web interface | [[Admin UI]] |
|
||||
| **OIDC/JWT (Web Identity)** | `-s3.iam.config=iam.json` | [[OIDC Integration]] |
|
||||
|
||||
## Dynamic Configuration
|
||||
|
||||
Example command:
|
||||
```bash
|
||||
s3.configure -access_key=any -secret_key=any -buckets=bucket1 -user=me -actions=Read,Write,List,Tagging,Admin -apply
|
||||
```
|
||||
|
||||
Output from above `s3.configure` command:
|
||||
```json
|
||||
{
|
||||
"identities": [
|
||||
{
|
||||
"name": "me",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "any",
|
||||
"secretKey": "any"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Read:bucket1",
|
||||
"Write:bucket1",
|
||||
"List:bucket1",
|
||||
"Tagging:bucket1",
|
||||
"Admin:bucket1"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Static Configuration
|
||||
|
||||
To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUP signal without restarting the main process via `pkill -HUP weed`
|
||||
|
||||
You just need to create a user with all "Admin", "Read", "Write", "List", "Tagging" actions.
|
||||
You can create as many users as needed. Each user can have multiple credentials.
|
||||
|
||||
* The "Admin" action is needed to list, create, and delete buckets.
|
||||
* The "Write" action allows uploading files to all buckets.
|
||||
* The "WriteAcp" action allows writing the access control list (ACL) from all buckets.
|
||||
* The "Read" action allows reading files from all buckets.
|
||||
* The "ReadAcp" action allows reading the access control list (ACL) from all buckets.
|
||||
* The "List" action allows listing files from all buckets.
|
||||
* The "Tagging" action allows tagging files from all buckets.
|
||||
* The "Write:<bucket_name>" action allows uploading files within a bucket, e.g., "Write:bucket1".
|
||||
* The "WriteAcp:<bucket_name>" action allows writing ACL within a bucket, e.g., "Write:bucket1".
|
||||
* The "Read:<bucket_name>" action allows reading files within a bucket, e.g., "Read:bucket2".
|
||||
* The "ReadAcp:<bucket_name>" action allows reading ACL within a bucket, e.g., "Read:bucket2".
|
||||
* The "List:<bucket_name>" action allows listing files within a bucket, e.g., "List:bucket2".
|
||||
* The "Tagging:<bucket_name>" action allows tagging files within a bucket, e.g., "Tagging:bucket2".
|
||||
|
||||
### Public access (with anonymous download)
|
||||
|
||||
For public access, you can configure an identity with name "anonymous", usually with just "Read" action, or access to specific buckets.
|
||||
|
||||
```json
|
||||
{
|
||||
"identities": [
|
||||
{
|
||||
"name": "anonymous",
|
||||
"actions": [
|
||||
"Read"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "some_name",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "some_access_key1",
|
||||
"secretKey": "some_secret_key1"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Admin",
|
||||
"Read",
|
||||
"ReadAcp",
|
||||
"List",
|
||||
"Tagging",
|
||||
"Write",
|
||||
"WriteAcp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "some_read_only_user",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "some_access_key2",
|
||||
"secretKey": "some_secret_key2"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Read",
|
||||
"List"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "some_normal_user",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "some_access_key3",
|
||||
"secretKey": "some_secret_key3"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Read",
|
||||
"List",
|
||||
"Tagging",
|
||||
"Write"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "user_limited_by_bucket",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "some_access_key4",
|
||||
"secretKey": "some_secret_key4"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Read:bucket1",
|
||||
"Read:bucket2",
|
||||
"Read:bucket3",
|
||||
"Write:bucket1"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Actions with wildcard
|
||||
|
||||
Wildcard is partially supported for prefix lookup. The following example actions are allowed:
|
||||
|
||||
```
|
||||
Read
|
||||
Read:bucket
|
||||
Read:bucket_prefix*
|
||||
Read:bucket/*
|
||||
Read:bucket/a/b/*
|
||||
```
|
||||
|
||||
## Presigned URL
|
||||
|
||||
Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for example.
|
||||
|
||||
|
||||
# S3 Object Versioning
|
||||
|
||||
SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification.
|
||||
|
||||
For detailed information about object versioning, see the dedicated [[S3-Object-Versioning]] page.
|
||||
|
||||
|
||||
# S3 Cross-Origin Resource Sharing (CORS)
|
||||
|
||||
SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains.
|
||||
|
||||
For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page.
|
||||
|
||||
|
||||
# Reverse Proxy Support
|
||||
|
||||
SeaweedFS S3 API supports deployment behind reverse proxies with full AWS Signature v4 authentication compatibility. This includes support for:
|
||||
|
||||
- **X-Forwarded-Host**: Preserves original host header for signature verification
|
||||
- **X-Forwarded-Port**: Automatically combines with X-Forwarded-Host for non-standard ports
|
||||
- **X-Forwarded-Prefix**: Handles URL path prefix stripping by reverse proxies
|
||||
- **Standard forwarded headers**: X-Forwarded-For, X-Forwarded-Proto, etc.
|
||||
|
||||
## Path Prefix Handling
|
||||
|
||||
When using reverse proxies that strip URL prefixes (e.g., `/s3/`, `/api/s3/`), SeaweedFS automatically handles signature verification for both the original prefixed path and the stripped path. This ensures seamless operation with:
|
||||
|
||||
- API gateways
|
||||
- Multi-tenant deployments
|
||||
- Subpath hosting scenarios
|
||||
|
||||
For detailed configuration examples and setup instructions, see the dedicated [[S3-Nginx-Proxy]] page.
|
||||
|
||||
See [[Path Specific Configuration]]
|
||||
|
||||
## Multiple S3 Nodes
|
||||
|
||||
If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer.
|
||||
|
||||
Usually you would also want to have multiple filers. The easiest way is to run filer together with a S3.
|
||||
You can start multiple S3 instances pointing to a filer for horizontal scaling. The easiest way is to run filer together with S3:
|
||||
|
||||
```bash
|
||||
weed filer -s3
|
||||
```
|
||||
|
||||
# Authentication with Filer
|
||||
# Supported APIs
|
||||
|
||||
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
|
||||
explained in [[Security-Configuration]] -
|
||||
controlled by the `grpc.*` configuration in `security.toml`.
|
||||
The following tables list all standard AWS S3 API operations and their support status in SeaweedFS, using official AWS API operation names.
|
||||
|
||||
**Starting with version 2.84, it is also possible to authenticate the HTTP
|
||||
operations between the S3-API-Proxy and the Filer (especially
|
||||
uploading new files).** This is configured by setting
|
||||
`jwt.filer_signing.key` and `jwt.filer_signing.read.key` in
|
||||
`security.toml`.
|
||||
For IAM action names used in bucket policies (e.g., `s3:ListBucketMultipartUploads`), see [[S3 Bucket Policies]].
|
||||
|
||||
With both configurations (gRPC and JWT), it is possible to have Filer
|
||||
and S3 communicate in fully authenticated fashion; so Filer will reject
|
||||
any unauthenticated communication.
|
||||
## Bucket Operations
|
||||
|
||||
| API Operation | Supported | Notes |
|
||||
|---|---|---|
|
||||
| CreateBucket | Yes | |
|
||||
| DeleteBucket | Yes | Deletes the entire collection |
|
||||
| HeadBucket | Yes | |
|
||||
| ListBuckets | Yes | |
|
||||
| ListDirectoryBuckets | No | S3 Express One Zone |
|
||||
|
||||
## Bucket Configuration
|
||||
|
||||
| API Operation | Supported | Notes |
|
||||
|---|---|---|
|
||||
| GetBucketAcl | Yes | |
|
||||
| PutBucketAcl | Yes | |
|
||||
| GetBucketCors | Yes | See [[S3 CORS]] |
|
||||
| PutBucketCors | Yes | |
|
||||
| DeleteBucketCors | Yes | |
|
||||
| GetBucketEncryption | Yes | See [[Server-Side Encryption]] |
|
||||
| PutBucketEncryption | Yes | |
|
||||
| DeleteBucketEncryption | Yes | |
|
||||
| GetBucketLifecycleConfiguration | Partial | TTL-based expiration only |
|
||||
| PutBucketLifecycleConfiguration | Partial | TTL-based expiration only |
|
||||
| DeleteBucketLifecycle | Partial | TTL-based expiration only |
|
||||
| GetBucketLocation | Yes | Returns default location |
|
||||
| GetBucketOwnershipControls | Yes | |
|
||||
| PutBucketOwnershipControls | Yes | |
|
||||
| DeleteBucketOwnershipControls | Yes | |
|
||||
| GetBucketPolicy | Yes | See [[S3 Bucket Policies]] |
|
||||
| PutBucketPolicy | Yes | |
|
||||
| DeleteBucketPolicy | Yes | |
|
||||
| GetBucketRequestPayment | Yes | Always returns BucketOwner |
|
||||
| GetBucketTagging | Yes | |
|
||||
| PutBucketTagging | Yes | |
|
||||
| DeleteBucketTagging | Yes | |
|
||||
| GetBucketVersioning | Yes | See [[S3 Object Versioning]] |
|
||||
| PutBucketVersioning | Yes | |
|
||||
| GetObjectLockConfiguration | Yes | See [[S3 Object Lock and Retention]] |
|
||||
| PutObjectLockConfiguration | Yes | |
|
||||
| GetPublicAccessBlock | Yes | |
|
||||
| PutPublicAccessBlock | Yes | |
|
||||
| DeletePublicAccessBlock | Yes | |
|
||||
| GetBucketAccelerateConfiguration | No | |
|
||||
| PutBucketAccelerateConfiguration | No | |
|
||||
| GetBucketAnalyticsConfiguration | No | |
|
||||
| PutBucketAnalyticsConfiguration | No | |
|
||||
| DeleteBucketAnalyticsConfiguration | No | |
|
||||
| ListBucketAnalyticsConfigurations | No | |
|
||||
| GetBucketIntelligentTieringConfiguration | No | |
|
||||
| PutBucketIntelligentTieringConfiguration | No | |
|
||||
| DeleteBucketIntelligentTieringConfiguration | No | |
|
||||
| ListBucketIntelligentTieringConfigurations | No | |
|
||||
| GetBucketInventoryConfiguration | No | |
|
||||
| PutBucketInventoryConfiguration | No | |
|
||||
| DeleteBucketInventoryConfiguration | No | |
|
||||
| ListBucketInventoryConfigurations | No | |
|
||||
| GetBucketLogging | No | |
|
||||
| PutBucketLogging | No | |
|
||||
| GetBucketMetricsConfiguration | No | |
|
||||
| PutBucketMetricsConfiguration | No | |
|
||||
| DeleteBucketMetricsConfiguration | No | |
|
||||
| ListBucketMetricsConfigurations | No | |
|
||||
| GetBucketNotificationConfiguration | No | |
|
||||
| PutBucketNotificationConfiguration | No | |
|
||||
| GetBucketPolicyStatus | No | |
|
||||
| GetBucketReplication | No | |
|
||||
| PutBucketReplication | No | |
|
||||
| DeleteBucketReplication | No | |
|
||||
| PutBucketRequestPayment | No | |
|
||||
| GetBucketWebsite | No | |
|
||||
| PutBucketWebsite | No | |
|
||||
| DeleteBucketWebsite | No | |
|
||||
|
||||
## Object Operations
|
||||
|
||||
| API Operation | Supported | Notes |
|
||||
|---|---|---|
|
||||
| CopyObject | Yes | Supports version ID, encryption, metadata directive |
|
||||
| DeleteObject | Yes | Supports version ID; also deletes folders |
|
||||
| DeleteObjects | Yes | Multi-object delete |
|
||||
| GetObject | Yes | Supports range requests, conditional headers, SSE, user metadata |
|
||||
| GetObjectAcl | Yes | |
|
||||
| GetObjectAttributes | Yes | |
|
||||
| GetObjectLegalHold | Yes | See [[S3 Object Lock and Retention]] |
|
||||
| GetObjectRetention | Yes | |
|
||||
| GetObjectTagging | Yes | |
|
||||
| HeadObject | Yes | Returns user metadata, encryption info |
|
||||
| ListObjects | Yes | V1 (legacy) |
|
||||
| ListObjectsV2 | Yes | |
|
||||
| ListObjectVersions | Yes | See [[S3 Object Versioning]] |
|
||||
| PostObject | Yes | Browser-based form uploads via POST policy |
|
||||
| PutObject | Yes | Supports SSE, user metadata (`x-amz-meta-*`), conditional headers |
|
||||
| PutObjectAcl | Yes | |
|
||||
| PutObjectLegalHold | Yes | |
|
||||
| PutObjectRetention | Yes | |
|
||||
| PutObjectTagging | Yes | |
|
||||
| DeleteObjectTagging | Yes | |
|
||||
| GetObjectTorrent | No | |
|
||||
| RenameObject | No | |
|
||||
| RestoreObject | No | |
|
||||
| SelectObjectContent | No | |
|
||||
| WriteGetObjectResponse | No | S3 Object Lambda |
|
||||
|
||||
## Multipart Upload Operations
|
||||
|
||||
| API Operation | Supported | Notes |
|
||||
|---|---|---|
|
||||
| CreateMultipartUpload | Yes | IAM action: `s3:CreateMultipartUpload` |
|
||||
| UploadPart | Yes | IAM action: `s3:UploadPart` |
|
||||
| UploadPartCopy | Yes | |
|
||||
| CompleteMultipartUpload | Yes | IAM action: `s3:CompleteMultipartUpload` |
|
||||
| AbortMultipartUpload | Yes | IAM action: `s3:AbortMultipartUpload` |
|
||||
| ListMultipartUploads | Yes | IAM action: `s3:ListBucketMultipartUploads` |
|
||||
| ListParts | Yes | IAM action: `s3:ListMultipartUploadParts` |
|
||||
|
||||
All multipart operations are implicitly allowed when `s3:PutObject` is granted. See [[S3 Bucket Policies#multipart-upload-permission-inheritance]].
|
||||
|
||||
## Additional APIs
|
||||
|
||||
SeaweedFS also implements these related AWS service APIs on the same S3 endpoint:
|
||||
|
||||
| Service | Operations | Documentation |
|
||||
|---|---|---|
|
||||
| Security Token Service (STS) | AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity | [[OIDC Integration]] |
|
||||
| IAM API | User, policy, access key, and group management | [[Amazon IAM API]], [[AWS IAM CLI]] |
|
||||
| S3 Tables | Table bucket, namespace, and table management (Apache Iceberg) | [[S3 Table Bucket]] |
|
||||
|
||||
# Feature Differences
|
||||
|
||||
| Feature | SeaweedFS | Amazon S3 |
|
||||
|---|---|---|
|
||||
| Multi byte ranges reads | Yes | No |
|
||||
| DeleteObject deletes a folder | Yes | No |
|
||||
| Same path for both a file and a folder | No | Yes |
|
||||
| Allows more than "/" as a delimiter | No | Yes |
|
||||
| Object Versioning | Yes | Yes |
|
||||
| MFA Delete for versioning | No | Yes |
|
||||
| Server-Side Encryption (SSE-KMS) | Yes | Yes |
|
||||
| Server-Side Encryption (SSE-C) | Yes | Yes |
|
||||
| Server-Side Encryption (SSE-S3) | Yes | Yes |
|
||||
| KMS Providers (Multi-cloud) | Yes | No |
|
||||
| Conditional Headers (All operations) | Yes | Yes |
|
||||
| Range requests with SSE | Yes | Yes |
|
||||
| Presigned URLs | Yes | Yes |
|
||||
|
||||
## Empty Folders
|
||||
|
||||
SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also.
|
||||
|
||||
To be consistent with AWS S3, SeaweedFS automatically cleans up empty folders asynchronously after file deletions.
|
||||
|
||||
# Related Pages
|
||||
|
||||
**Configuration & Authentication**
|
||||
- [[S3 Configuration]] — Overview of all configuration options
|
||||
- [[S3 Credentials]] — Access key and identity management
|
||||
- [[OIDC Integration]] — OIDC, STS, and role-based access
|
||||
- [[Amazon IAM API]] / [[AWS IAM CLI]] — IAM API operations
|
||||
- [[Security Configuration]] — mTLS and JWT between S3 gateway and filer
|
||||
|
||||
**Security & Policies**
|
||||
- [[S3 Bucket Policies]] — IAM-style policies with condition operators
|
||||
- [[S3 Policy Conditions]] — Condition key reference
|
||||
- [[S3 Policy Variables]] — Policy variable substitution
|
||||
- [[S3 Object Lock and Retention]] — Object lock, retention, legal hold
|
||||
- [[S3 Object Versioning]] — Version management
|
||||
|
||||
**Encryption**
|
||||
- [[Server-Side Encryption]] — SSE overview
|
||||
- [[Server-Side Encryption SSE-KMS]] — AWS KMS, Google Cloud KMS, OpenBao/Vault
|
||||
- [[Server-Side Encryption SSE-C]] — Customer-provided keys
|
||||
|
||||
**Features**
|
||||
- [[S3 Conditional Operations]] — If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since
|
||||
- [[S3 CORS]] — Cross-Origin Resource Sharing
|
||||
- [[S3 Bucket Quota]] — Storage quota management
|
||||
- [[S3 Rate Limiting]] — Request rate limiting
|
||||
- [[S3 API Audit log]] — Audit logging
|
||||
|
||||
**S3 Tables & Iceberg**
|
||||
- [[S3 Table Bucket]] — S3 Tables API
|
||||
- [[SeaweedFS Iceberg Catalog]] — Apache Iceberg REST catalog
|
||||
- [[Iceberg Table Maintenance]] — Table maintenance operations
|
||||
- [[S3 Tables Security]] — Table bucket security
|
||||
|
||||
**Deployment & Operations**
|
||||
- [[S3 Nginx Proxy]] — Reverse proxy and X-Forwarded-* header support
|
||||
- [[Docker Compose for S3]] — Docker deployment
|
||||
- [[S3 API Benchmark]] — Performance benchmarking
|
||||
- [[S3 API FAQ]] — Frequently asked questions
|
||||
|
||||
**Client Tools**
|
||||
- [[AWS CLI with SeaweedFS]] — AWS CLI usage, presigned URLs
|
||||
- [[s3cmd with SeaweedFS]]
|
||||
- [[rclone with SeaweedFS]]
|
||||
- [[restic with SeaweedFS]]
|
||||
- [[nodejs with Seaweed S3]]
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
# S3 Policy Conditions
|
||||
|
||||
SeaweedFS supports AWS S3-compatible policy conditions for fine-grained access control based on request context such as source IP, transport security, time, and request headers.
|
||||
|
||||
## Overview
|
||||
|
||||
Conditions allow you to restrict when a policy statement applies. They are used in bucket policies and IAM policies to enforce rules like "allow access only from a specific IP range" or "require HTTPS".
|
||||
|
||||
A condition block has the form:
|
||||
|
||||
```json
|
||||
{
|
||||
"Condition": {
|
||||
"<operator>": {
|
||||
"<condition-key>": ["<value1>", "<value2>"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Multiple operators within a `Condition` block are ANDed together. Multiple values for a single key are ORed.
|
||||
|
||||
## Supported Condition Operators
|
||||
|
||||
### String Operators
|
||||
|
||||
| Operator | Description |
|
||||
|----------|-------------|
|
||||
| `StringEquals` | Exact case-sensitive match |
|
||||
| `StringNotEquals` | Negated exact match |
|
||||
| `StringLike` | Case-sensitive match with `*` and `?` wildcards |
|
||||
| `StringNotLike` | Negated wildcard match |
|
||||
|
||||
### Numeric Operators
|
||||
|
||||
| Operator | Description |
|
||||
|----------|-------------|
|
||||
| `NumericEquals` | Equal |
|
||||
| `NumericNotEquals` | Not equal |
|
||||
| `NumericLessThan` | Less than |
|
||||
| `NumericLessThanEquals` | Less than or equal |
|
||||
| `NumericGreaterThan` | Greater than |
|
||||
| `NumericGreaterThanEquals` | Greater than or equal |
|
||||
|
||||
### Date Operators
|
||||
|
||||
Date values must be in RFC 3339 format (e.g., `2025-01-01T00:00:00Z`).
|
||||
|
||||
| Operator | Description |
|
||||
|----------|-------------|
|
||||
| `DateEquals` | Exact date match |
|
||||
| `DateNotEquals` | Negated date match |
|
||||
| `DateLessThan` | Before date |
|
||||
| `DateLessThanEquals` | Before or at date |
|
||||
| `DateGreaterThan` | After date |
|
||||
| `DateGreaterThanEquals` | After or at date |
|
||||
|
||||
### IP Address Operators
|
||||
|
||||
Values can be individual IPs or CIDR ranges (e.g., `192.168.1.0/24`).
|
||||
|
||||
| Operator | Description |
|
||||
|----------|-------------|
|
||||
| `IpAddress` | Source IP is within the specified range |
|
||||
| `NotIpAddress` | Source IP is not within the specified range |
|
||||
|
||||
### Other Operators
|
||||
|
||||
| Operator | Description |
|
||||
|----------|-------------|
|
||||
| `Bool` | Boolean match (`"true"` or `"false"`) |
|
||||
| `ArnEquals` | Exact ARN match |
|
||||
| `ArnLike` | ARN match with wildcards |
|
||||
| `Null` | Check whether a key is present (`"true"` = key absent, `"false"` = key present) |
|
||||
|
||||
### Set Operators
|
||||
|
||||
Prefix any string operator with `ForAnyValue:` or `ForAllValues:` for multi-valued keys:
|
||||
|
||||
- `ForAnyValue:StringEquals` - at least one value matches
|
||||
- `ForAllValues:StringEquals` - all values match
|
||||
|
||||
## Supported Condition Keys
|
||||
|
||||
### AWS Global Context Keys
|
||||
|
||||
| Key | Type | Description |
|
||||
|-----|------|-------------|
|
||||
| `aws:SourceIp` | IP | Client IP address (supports CIDR) |
|
||||
| `aws:SecureTransport` | Bool | `true` if the request was sent over HTTPS |
|
||||
| `aws:CurrentTime` | Date | Current time in RFC 3339 format |
|
||||
| `aws:UserAgent` | String | Client User-Agent header |
|
||||
| `aws:Referer` | String | HTTP Referer header |
|
||||
| `aws:username` | String | Username from principal ARN |
|
||||
| `aws:userid` | String | User ID from principal ARN |
|
||||
| `aws:PrincipalAccount` | String | Account ID from principal ARN |
|
||||
| `aws:principaltype` | String | Principal type (`IAMUser`, `IAMRole`, `AssumedRole`) |
|
||||
| `aws:PrincipalArn` | String | Full principal ARN |
|
||||
| `aws:FederatedProvider` | String | Federated identity provider |
|
||||
| `aws:PrincipalServiceName` | String | Service principal name |
|
||||
|
||||
### S3-Specific Keys
|
||||
|
||||
| Key | Type | Description |
|
||||
|-----|------|-------------|
|
||||
| `s3:prefix` | String | Prefix parameter from list operations |
|
||||
| `s3:delimiter` | String | Delimiter parameter from list operations |
|
||||
| `s3:max-keys` | Numeric | Max keys parameter from list operations |
|
||||
| `s3:ExistingObjectTag/<key>` | String | Value of an existing object tag |
|
||||
| `s3:RequestMethod` | String | HTTP method (GET, PUT, POST, DELETE) |
|
||||
| `s3:authType` | String | Auth type (`REST-HEADER` or `REST-QUERY-STRING`) |
|
||||
|
||||
| `s3:x-amz-*` | String | Any `x-amz-*` request header (e.g., `s3:x-amz-server-side-encryption`) |
|
||||
|
||||
### Identity Provider Keys
|
||||
|
||||
| Key | Type | Description |
|
||||
|-----|------|-------------|
|
||||
| `jwt:preferred_username` | String | Username from JWT token |
|
||||
| `jwt:sub` | String | Subject from JWT token |
|
||||
| `jwt:iss` | String | Issuer from JWT token |
|
||||
| `jwt:aud` | String | Audience from JWT token |
|
||||
| `oidc:sub` | String | Subject from OIDC token |
|
||||
| `oidc:aud` | String | Audience from OIDC token |
|
||||
| `oidc:iss` | String | Issuer from OIDC token |
|
||||
| `oidc:<claim>` | String | Any custom OIDC claim (e.g., `oidc:roles`, `oidc:groups`) |
|
||||
| `saml:username` | String | Username from SAML assertion |
|
||||
| `saml:sub` | String | Subject from SAML assertion |
|
||||
| `saml:aud` | String | Audience from SAML assertion |
|
||||
| `saml:iss` | String | Issuer from SAML assertion |
|
||||
| `ldap:username` | String | Username from LDAP |
|
||||
| `ldap:dn` | String | Distinguished name from LDAP |
|
||||
| `ldap:<attribute>` | String | Any LDAP attribute |
|
||||
|
||||
## Source IP Handling
|
||||
|
||||
When SeaweedFS runs behind a reverse proxy, the `aws:SourceIp` condition key is resolved using the following logic:
|
||||
|
||||
1. If the direct connection is from a **private/loopback IP** (RFC 1918, IPv6 ULA), forwarding headers are trusted:
|
||||
- `X-Forwarded-For` is checked for the rightmost non-private IP
|
||||
- `X-Real-IP` is used as a fallback
|
||||
2. If the direct connection is from a **public IP**, forwarding headers are ignored to prevent spoofing
|
||||
3. Final fallback is always `RemoteAddr`
|
||||
|
||||
This ensures correct behavior whether SeaweedFS is accessed directly or through a load balancer.
|
||||
|
||||
## Examples
|
||||
|
||||
### Restrict Access by IP Address
|
||||
|
||||
Allow reads only from your office network:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3:GetObject",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"IpAddress": {
|
||||
"aws:SourceIp": ["10.0.0.0/8", "192.168.1.0/24"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Deny access from a specific IP range:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"NotIpAddress": {
|
||||
"aws:SourceIp": "203.0.113.0/24"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Require HTTPS
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:*",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"Bool": {
|
||||
"aws:SecureTransport": "false"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Time-Based Access
|
||||
|
||||
Allow access only during a specific window:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3:GetObject",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"DateGreaterThan": {
|
||||
"aws:CurrentTime": "2025-01-01T00:00:00Z"
|
||||
},
|
||||
"DateLessThan": {
|
||||
"aws:CurrentTime": "2025-12-31T23:59:59Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Require Encryption Header
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3:PutObject",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"StringNotEquals": {
|
||||
"s3:x-amz-server-side-encryption": "AES256"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Tag-Based Access Control
|
||||
|
||||
Allow deletion only if the object is tagged as deletable:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3:DeleteObject",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"s3:ExistingObjectTag/status": "deletable"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Combining Multiple Conditions
|
||||
|
||||
Multiple operators are ANDed — all must be true:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3:GetObject",
|
||||
"Resource": "arn:aws:s3:::my-bucket/*",
|
||||
"Condition": {
|
||||
"IpAddress": {
|
||||
"aws:SourceIp": "192.168.1.0/24"
|
||||
},
|
||||
"Bool": {
|
||||
"aws:SecureTransport": "true"
|
||||
},
|
||||
"DateGreaterThan": {
|
||||
"aws:CurrentTime": "2025-01-01T00:00:00Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Policy Evaluation Logic
|
||||
|
||||
Conditions follow AWS-compatible evaluation order:
|
||||
|
||||
1. **Explicit Deny** — if any statement with a matching condition denies, access is denied
|
||||
2. **Explicit Allow** — if a statement with a matching condition allows, access is allowed
|
||||
3. **Default Deny** — if no statement matches, access is denied
|
||||
|
||||
## See Also
|
||||
|
||||
- [[S3 Bucket Policies]] — Managing bucket policies
|
||||
- [[S3 Policy Variables]] — Dynamic policy variables
|
||||
- [[S3 Credentials]] — S3 authentication
|
||||
- [[Amazon IAM API]] — IAM API support
|
||||
- [[OIDC Integration]] — JWT-based authentication
|
||||
+1
@@ -97,6 +97,7 @@
|
||||
* [[S3 Credentials]] (`-s3.config`)
|
||||
* [[OIDC Integration]] (`-s3.iam.config`)
|
||||
* [[S3 Policy Variables]]
|
||||
* [[S3 Policy Conditions]]
|
||||
* [[S3 Bucket Policies]]
|
||||
* [[Amazon IAM API]]
|
||||
* [[AWS IAM CLI]]
|
||||
|
||||
Reference in New Issue
Block a user