viaprog/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-06-13 23:36:45 +03:00
Code Issues Packages Projects Releases Wiki Activity
Page: Trino Iceberg Integration
Pages
A typical step‐by‐step example AWS CLI with SeaweedFS AWS IAM CLI Actual Users Admin UI OIDC Admin UI Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Cluster Plan Day 2 Operations Cluster Plan Inventory Reference Cluster Plan Workflow Components Configure Remote Storage Cryptography and FIPS Compliance Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Deployment with seaweed up Directories and Files Distributed POSIX Locks Distributing AI Model Files for Multi GPU Loading Docker Compose for S3 Docker Image Registry with SeaweedFS Doris Iceberg Integration Dremio Iceberg Integration DuckDB Iceberg Integration EC Bitrot Detection Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server File Operations Quick Reference Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Notification Webhook Filer Operation Serialization Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Hobbyest Tinkerer scale on premises tutorial Home Iceberg Table Maintenance Independent Benchmarks Kafka to Kafka Gateway to SMQ to SQL Kubernetes Backups and Recovery with K8up Kubernetes ServiceAccount Authentication Lakekeeper Iceberg Integration Large File Handling Load Command Line Options from a file Master Server API Migrate Maintenance Scripts to Admin Script Plugin Migrate to Filer Store Mount Remote Storage OIDC Integration Optimization for Many Small Buckets Optimization P2P reading in weed mount POSIX Compliance Path Specific Configuration Path Specific Filer Store Plugin Worker Scheduling PostgreSQL compatible Server weed db Production Setup Pub Sub to SMQ to SQL Quick Start with weed mini Replication RisingWave Iceberg Integration Run Blob Storage on Public Internet Run Presto on SeaweedFS Rust Volume Server S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Policies S3 Bucket Quota S3 CORS S3 Conditional Operations S3 Configuration S3 Credentials S3 Lifecycle Architecture S3 Lifecycle Monitoring S3 Lifecycle Operator Guide S3 Lifecycle Troubleshooting S3 Lifecycle vs Volume TTL S3 Lifecycle S3 Nginx Proxy S3 Object Lock and Retention S3 Object Versioning S3 Policy Conditions S3 Policy Variables S3 Rate Limiting S3 Table Bucket Commands S3 Table Bucket S3 Tables Security SFTP Server SQL Queries on Message Queue SQL Quick Reference SRV Service Discovery Seaweed Message Queue SeaweedFS Architecture SeaweedFS Iceberg Catalog SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Side Encryption SSE C Server Side Encryption SSE KMS Server Side Encryption Server Startup via Systemd Simplest S3 Bucket and User Setup Spark Iceberg Integration Store file with a Time To Live Structured Data Lake with SMQ and SQL Super Large Directories Supported APIs vs Minio System Metrics TUS Resumable Uploads TensorFlow with SeaweedFS Tiered Storage Trino Iceberg Integration UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users Worker fstab and systemd mount nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
Clone
3
Trino Iceberg Integration
Chris Lu edited this page 2026-05-03 23:48:41 -07:00
Table of Contents

Trino Iceberg Integration

Trino connects to SeaweedFS Iceberg tables using the iceberg connector with the rest catalog type and SigV4 authentication.

Prerequisites

  • Trino 4xx+ with the Iceberg connector
  • SeaweedFS started as shown in Setup below

Setup

Start weed mini with credentials and a pre-created table bucket via environment variables:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export S3_TABLE_BUCKET=my-table-bucket

weed mini -dir ~/data

This brings up the Iceberg REST Catalog on http://localhost:8181, the S3 endpoint on http://localhost:8333, an admin S3 identity using the AWS env vars (used as Trino's SigV4 credentials below), and the table bucket my-table-bucket pre-created.

Configuration

Create a catalog properties file at etc/catalog/iceberg.properties:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://localhost:8181
iceberg.rest-catalog.warehouse=s3://my-table-bucket

# File format
iceberg.file-format=PARQUET
iceberg.unique-table-location=true

# SigV4 authentication for the REST catalog
iceberg.rest-catalog.security=SIGV4

# S3 FileIO configuration
fs.native-s3.enabled=true
s3.endpoint=http://localhost:8333
s3.path-style-access=true
s3.signer-type=AwsS3V4Signer
s3.aws-access-key=AKIAIOSFODNN7EXAMPLE
s3.aws-secret-key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
s3.region=us-east-1

Recommended: iceberg.unique-table-location=true

Keep Trino's default iceberg.unique-table-location=true (already shown in the configuration above). With it enabled, each new table is created at a unique path s3://<bucket>/<namespace>/<tableName>-<UUID>/. The UUID suffix gives you three operational properties worth keeping:

  • Orphan-resistant DROP + CREATE — if a DROP TABLE cleanup partially fails (filer error, crash mid-cleanup), a fresh CREATE TABLE with the same name lands at a new path and is not blocked or polluted by leftover files.
  • Crash-safe aborted creates — a CREATE TABLE that wrote partial state but failed to commit leaves debris under one UUID; the retry uses a new UUID and proceeds cleanly.
  • Defense-in-depth against client-side empty-location pre-flight checks — Trino runs listFiles(<location>).hasNext() before commit, and so do some other engines. With unique paths the check sees a fresh empty directory regardless of any historical state at the deterministic name.

If you prefer the cleaner s3://<bucket>/<namespace>/<tableName>/ layout (one directory per table, no <tableName>-<UUID>/ sibling), setting iceberg.unique-table-location=false is safe against SeaweedFS — you give up the resistance properties above. Pick whichever trade-off fits your operational model.

Multi-Level Namespaces

To use nested namespaces (e.g., db.schema), add:

iceberg.rest-catalog.nested-namespace-enabled=true

Example SQL

Schema Operations

-- Create a schema (maps to an Iceberg namespace)
CREATE SCHEMA IF NOT EXISTS iceberg.my_namespace;

-- List schemas
SHOW SCHEMAS FROM iceberg;

Table Operations

-- Create a table
CREATE TABLE iceberg.my_namespace.events (
    id INTEGER,
    event VARCHAR,
    ts TIMESTAMP(6)
) WITH (
    format = 'PARQUET'
);

-- Insert data
INSERT INTO iceberg.my_namespace.events VALUES
    (1, 'click', TIMESTAMP '2024-01-15 10:30:00'),
    (2, 'view', TIMESTAMP '2024-01-15 11:00:00');

-- Query
SELECT * FROM iceberg.my_namespace.events;

-- Inspect data files
SELECT file_path FROM iceberg.my_namespace."events$files" LIMIT 5;

Multi-Level Namespace Example

CREATE SCHEMA IF NOT EXISTS iceberg."analytics.web";

CREATE TABLE iceberg."analytics.web".pageviews (
    id INTEGER,
    url VARCHAR,
    ts TIMESTAMP(6)
) WITH (
    format = 'PARQUET'
);

Anonymous Access

When SeaweedFS runs without IAM, remove the SigV4 and credential properties:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://localhost:8181
iceberg.rest-catalog.warehouse=s3://my-table-bucket

fs.native-s3.enabled=true
s3.endpoint=http://localhost:8333
s3.path-style-access=true
s3.region=us-east-1

See Also

Introduction

API

Configuration

Filer

Filer Stores

Management

Advanced Filer Configurations

FUSE Mount

WebDAV

SFTP Server

Cloud Drive

AWS S3 API

S3 Table Bucket

Iceberg Integrations

S3 Authentication & IAM

Server-Side Encryption

S3 Client Tools

Machine Learning

HDFS

Replication and Backup

Metadata Change Events

Messaging

Use Cases

Operations

Rust Volume Server

Advanced

Security

Misc Use Case Examples