11

AWS CLI with SeaweedFS

Chris Lu edited this page 2026-04-02 14:33:24 -07:00

Table of Contents

• Installation
• Configuration
• Reverse proxy sub-path configuration
• Execute commands
• Presigned URL
• Server-Side Encryption with AWS CLI
• SSE-KMS
• SSE-C
• SSE-S3 (Server-managed)
• OIDC/JWT to S3
• STS (Security Token Service)
• Get Caller Identity
• Get Federation Token

Installation

See https://aws.amazon.com/cli/

On mac or linux: pip install awscli

Configuration

Configure for AWS. The key values do not matter.

$ aws configure
AWS Access Key ID [None]: any
AWS Secret Access Key [None]: any
Default region name [None]: us-east-1
Default output format [None]:

Also, make sure using AWS signature version 4:

$ aws configure set default.s3.signature_version s3v4

And configure the S3 endpoint:

export S3_ENDPOINT=http://localhost:8333

Reverse proxy sub-path configuration

This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths.
Use this only if you can't create (sub-)domain and use other port!

AWS CLI appends sub-path before actual path so need to add X-Forwarded-Prefix header (set to /s3 for example)

Example for Caddy web server

redir /s3 /s3/
handle_path /s3/* {
	reverse_proxy localhost:8333 {
		header_up X-Forwarded-Prefix /s3
	}
}

Execute commands

# list buckets
$ aws --endpoint-url $S3_ENDPOINT s3 ls
2019-01-02 01:59:25 newbucket

# list files inside the bucket
$ aws --endpoint-url $S3_ENDPOINT s3 ls s3://newbucket
2019-01-02 12:52:44       6804 password

# make a bucket
$ aws --endpoint-url $S3_ENDPOINT s3 mb s3://newbucket3
make_bucket: newbucket3

# add an object
$ aws --endpoint-url $S3_ENDPOINT s3 cp /etc/passwd s3://newbucket3
upload: ../../../../../etc/passwd to s3://newbucket3/passwd

# copy an object
$ aws --endpoint-url $S3_ENDPOINT s3 cp s3://newbucket3/passwd s3://newbucket3/passwd.txt
copy: s3://newbucket3/passwd to s3://newbucket3/passwd.txt

# remove an object
$ aws --endpoint-url $S3_ENDPOINT s3 rm s3://newbucket3/passwd
delete: s3://newbucket3/passwd

# remove a bucket
$ aws --endpoint-url $S3_ENDPOINT s3 rb s3://newbucket3
remove_bucket: newbucket3

Presigned URL

If authentication is enabled, the url is not accessible without proper credentials. But you can presign a url and access it.

# presign url, default to 1 hour
$ aws --endpoint-url $S3_ENDPOINT s3 presign s3://newbucket/t.txt
http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5

# access the url
$ curl "http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5"

Server-Side Encryption with AWS CLI

SSE-KMS

aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://bucket/kms.txt \
  --sse aws:kms \
  --sse-kms-key-id "test-key-123"

SSE-C

# Generate a 256-bit key
openssl rand -base64 32 > key.b64
aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://bucket/ssec.txt \
  --sse-c AES256 \
  --sse-c-key fileb://key.b64

SSE-S3 (Server-managed)

aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://bucket/sse-s3.txt \
  --sse AES256

OIDC/JWT to S3

For Keycloak and other OIDC providers, you can obtain a JWT and access S3 directly (or use STS to assume a role). See OIDC Integration.

STS (Security Token Service)

Get Caller Identity

Verify your credentials and see who you are authenticated as:

aws --endpoint-url $S3_ENDPOINT sts get-caller-identity

Get Federation Token

Obtain temporary credentials scoped down with an optional session policy (only long-term IAM users can call this):

aws --endpoint-url $S3_ENDPOINT sts get-federation-token \
  --name my-session \
  --duration-seconds 3600

With an inline session policy to restrict permissions:

aws --endpoint-url $S3_ENDPOINT sts get-federation-token \
  --name my-session \
  --policy '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::my-bucket/*"}]}'

For AssumeRole and AssumeRoleWithWebIdentity, see OIDC Integration.

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg Table Maintenance

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Deployment with seaweed-up
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware