71

Amazon S3 API

Chris Lu edited this page 2026-05-19 17:58:53 -07:00

Table of Contents

• How It Works
• Support Many Buckets
• Multiple S3 Nodes
• Supported APIs
• Bucket Operations
• Bucket Configuration
• Object Operations
• Multipart Upload Operations
• Additional APIs
• Feature Differences
• Empty Folders
• Related Pages

To be compatible with Amazon S3 API, a separate "weed s3" command is provided. This provides much faster access when reading or writing files, compared to operating files on the cloud.

How It Works

weed s3 starts a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer. For convenience, weed server -s3 starts a master, a volume server, a filer, and the S3 gateway. weed filer -s3 can start a filer and the S3 gateway together also.

Each bucket is stored in one collection, and mapped to folder /buckets/<bucket_name> by default. A bucket can be deleted efficiently by deleting the whole collection.

From version 3.51 the collection name is filerGroup_bucketname or bucketname if filer group is not set.

Support Many Buckets

Each bucket has its own collection. Usually one collection uses 7 volumes, where each volume is 30GB by default. So if you want to create multiple buckets, you may run out of volumes very quickly unless you have a large disk.

Try to keep the volume size low. For example,

weed master -volumeSizeLimitMB=1024

In addition, you can also configure the per bucket storage this way in weed shell:

fs.configure -locationPrefix=/buckets/ -volumeGrowthCount=1 -apply

This will add 1 physical volume when existing volumes are full. If using replication, you will need to add more volumes, so that it is a multiple of the number of replicas:

fs.configure -locationPrefix=/buckets/ -replication=001 -volumeGrowthCount=2 -apply

See Path Specific Configuration

Multiple S3 Nodes

You can start multiple S3 instances pointing to a filer for horizontal scaling. The easiest way is to run filer together with S3:

weed filer -s3

Supported APIs

The following tables list all standard AWS S3 API operations and their support status in SeaweedFS, using official AWS API operation names.

For IAM action names used in bucket policies (e.g., s3:ListBucketMultipartUploads), see S3 Bucket Policies.

Bucket Operations

API Operation	Supported	Notes
CreateBucket	Yes
DeleteBucket	Yes	Deletes the entire collection
HeadBucket	Yes
ListBuckets	Yes
ListDirectoryBuckets	No	S3 Express One Zone

Bucket Configuration

API Operation	Supported	Notes
GetBucketAcl	Yes
PutBucketAcl	Yes
GetBucketCors	Yes	See S3 CORS
PutBucketCors	Yes
DeleteBucketCors	Yes
GetBucketEncryption	Yes	See Server-Side Encryption
PutBucketEncryption	Yes
DeleteBucketEncryption	Yes
GetBucketLifecycleConfiguration	Yes	See S3 Lifecycle
PutBucketLifecycleConfiguration	Yes	Transition rules not supported
DeleteBucketLifecycle	Yes
GetBucketLocation	Yes	Returns default location
GetBucketOwnershipControls	Yes
PutBucketOwnershipControls	Yes
DeleteBucketOwnershipControls	Yes
GetBucketPolicy	Yes	See S3 Bucket Policies
PutBucketPolicy	Yes
DeleteBucketPolicy	Yes
GetBucketRequestPayment	Yes	Always returns BucketOwner
GetBucketTagging	Yes
PutBucketTagging	Yes
DeleteBucketTagging	Yes
GetBucketVersioning	Yes	See S3 Object Versioning
PutBucketVersioning	Yes
GetObjectLockConfiguration	Yes	See S3 Object Lock and Retention
PutObjectLockConfiguration	Yes
GetPublicAccessBlock	Yes
PutPublicAccessBlock	Yes
DeletePublicAccessBlock	Yes
GetBucketAccelerateConfiguration	Yes	Always returns Suspended
GetBucketAnalyticsConfiguration	Yes	Stub: returns NoSuchConfiguration
ListBucketAnalyticsConfigurations	Yes	Stub: returns an empty list
GetBucketIntelligentTieringConfiguration	Yes	Stub: returns NoSuchConfiguration
ListBucketIntelligentTieringConfigurations	Yes	Stub: returns an empty list
GetBucketInventoryConfiguration	Yes	Stub: returns NoSuchConfiguration
ListBucketInventoryConfigurations	Yes	Stub: returns an empty list
GetBucketLogging	Yes	Always returns an empty BucketLoggingStatus
GetBucketMetricsConfiguration	Yes	Stub: returns NoSuchConfiguration
ListBucketMetricsConfigurations	Yes	Stub: returns an empty list
GetBucketPolicyStatus	Yes	Computes IsPublic from the bucket policy
PutBucketRequestPayment	Yes	Accepts only Payer=BucketOwner
PutBucketAccelerateConfiguration	No
PutBucketAnalyticsConfiguration	No
DeleteBucketAnalyticsConfiguration	No
PutBucketIntelligentTieringConfiguration	No
DeleteBucketIntelligentTieringConfiguration	No
PutBucketInventoryConfiguration	No
DeleteBucketInventoryConfiguration	No
PutBucketLogging	No
PutBucketMetricsConfiguration	No
DeleteBucketMetricsConfiguration	No
GetBucketNotificationConfiguration	No
PutBucketNotificationConfiguration	No
GetBucketReplication	No
PutBucketReplication	No
DeleteBucketReplication	No
GetBucketWebsite	No
PutBucketWebsite	No
DeleteBucketWebsite	No

Object Operations

API Operation	Supported	Notes
CopyObject	Yes	Supports version ID, encryption, metadata directive
DeleteObject	Yes	Supports version ID; also deletes folders
DeleteObjects	Yes	Multi-object delete
GetObject	Yes	Supports range requests, conditional headers, SSE, user metadata
GetObjectAcl	Yes
GetObjectAttributes	Yes
GetObjectLegalHold	Yes	See S3 Object Lock and Retention
GetObjectRetention	Yes
GetObjectTagging	Yes
HeadObject	Yes	Returns user metadata, encryption info
ListObjects	Yes	V1 (legacy)
ListObjectsV2	Yes
ListObjectVersions	Yes	See S3 Object Versioning
PostObject	Yes	Browser-based form uploads via POST policy
PutObject	Yes	Supports SSE, user metadata (x-amz-meta-*), conditional headers
PutObjectAcl	Yes
PutObjectLegalHold	Yes
PutObjectRetention	Yes
PutObjectTagging	Yes
DeleteObjectTagging	Yes
GetObjectTorrent	No
RenameObject	No
RestoreObject	No
SelectObjectContent	No
WriteGetObjectResponse	No	S3 Object Lambda

Multipart Upload Operations

API Operation	Supported	Notes
CreateMultipartUpload	Yes	IAM action: s3:CreateMultipartUpload
UploadPart	Yes	IAM action: s3:UploadPart
UploadPartCopy	Yes
CompleteMultipartUpload	Yes	IAM action: s3:CompleteMultipartUpload
AbortMultipartUpload	Yes	IAM action: s3:AbortMultipartUpload
ListMultipartUploads	Yes	IAM action: s3:ListBucketMultipartUploads
ListParts	Yes	IAM action: s3:ListMultipartUploadParts

All multipart operations are implicitly allowed when s3:PutObject is granted. See S3 Bucket Policies#multipart-upload-permission-inheritance.

Additional APIs

SeaweedFS also implements these related AWS service APIs on the same S3 endpoint:

Service	Operations	Documentation
Security Token Service (STS)	AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity, GetFederationToken, GetCallerIdentity	OIDC Integration
IAM API	User, policy, access key, and group management	Amazon IAM API, AWS IAM CLI
S3 Tables	Table bucket, namespace, and table management (Apache Iceberg)	S3 Table Bucket

Feature Differences

Feature	SeaweedFS	Amazon S3
Multi byte ranges reads	Yes	No
DeleteObject deletes a folder	Yes	No
Same path for both a file and a folder	No	Yes
Allows more than "/" as a delimiter	No	Yes
Object Versioning	Yes	Yes
MFA Delete for versioning	No	Yes
Server-Side Encryption (SSE-KMS)	Yes	Yes
Server-Side Encryption (SSE-C)	Yes	Yes
Server-Side Encryption (SSE-S3)	Yes	Yes
KMS Providers (Multi-cloud)	Yes	No
Conditional Headers (All operations)	Yes	Yes
Range requests with SSE	Yes	Yes
Presigned URLs	Yes	Yes

Empty Folders

SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also.

To be consistent with AWS S3, SeaweedFS automatically cleans up empty folders asynchronously after file deletions.

Related Pages

Configuration & Authentication

• S3 Configuration — Overview of all configuration options
• S3 Credentials — Access key and identity management
• OIDC Integration — OIDC, STS, and role-based access
• Amazon IAM API / AWS IAM CLI — IAM API operations
• Security Configuration — mTLS and JWT between S3 gateway and filer

Security & Policies

• S3 Bucket Policies — IAM-style policies with condition operators
• S3 Policy Conditions — Condition key reference
• S3 Policy Variables — Policy variable substitution
• S3 Object Lock and Retention — Object lock, retention, legal hold
• S3 Object Versioning — Version management

Encryption

• Server-Side Encryption — SSE overview
• Server-Side Encryption SSE-KMS — AWS KMS, Google Cloud KMS, OpenBao/Vault
• Server-Side Encryption SSE-C — Customer-provided keys

Features

• S3 Conditional Operations — If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since
• S3 CORS — Cross-Origin Resource Sharing
• S3 Bucket Quota — Storage quota management
• S3 Rate Limiting — Request rate limiting
• S3 API Audit log — Audit logging

S3 Tables & Iceberg

• S3 Table Bucket — S3 Tables API
• SeaweedFS Iceberg Catalog — Apache Iceberg REST catalog
• Iceberg Table Maintenance — Table maintenance operations
• S3 Tables Security — Table bucket security

Deployment & Operations

• S3 Nginx Proxy — Reverse proxy and X-Forwarded-* header support
• Docker Compose for S3 — Docker deployment
• S3 API Benchmark — Performance benchmarking
• S3 API FAQ — Frequently asked questions

Client Tools

• AWS CLI with SeaweedFS — AWS CLI usage, presigned URLs
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg Table Maintenance

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Deployment with seaweed-up
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware