1

Doris Iceberg Integration

Chris Lu edited this page 2026-05-03 23:31:55 -07:00

Table of Contents

• Apache Doris Iceberg Integration
• Prerequisites
• Setup
• Configuration
• Example SQL
• Browse the catalog
• Read tables
• Anonymous Access
• See Also

Apache Doris Iceberg Integration

Apache Doris connects to SeaweedFS Iceberg tables using an external catalog of type=iceberg and iceberg.catalog.type=rest. Authentication to the REST catalog uses OAuth2 client credentials via the standard Iceberg credential property; data files are read directly from S3 using s3.access_key / s3.secret_key.

This page reflects the integration verified by the Apache Doris all-in-one 2.1.0 catalog test.

Prerequisites

• Apache Doris 2.1.0 or later (the all-in-one Docker image works for local testing)
• A MySQL-protocol client (mysql CLI or any Go/Java/Python MySQL driver) — Doris speaks MySQL on port 9030
• SeaweedFS started as shown in Setup below

Setup

Start weed mini with credentials and a pre-created table bucket via environment variables:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export S3_TABLE_BUCKET=my-table-bucket

weed mini -dir ~/data

This brings up the Iceberg REST Catalog on http://localhost:8181, the S3 endpoint on http://localhost:8333, an admin S3 identity using the AWS env vars (used as Doris's credential and s3.* keys below), and the table bucket my-table-bucket pre-created.

If Doris runs in a container and SeaweedFS runs on the host, use host.docker.internal (with --add-host host.docker.internal:host-gateway on Linux) in the URLs below.

Configuration

Doris external catalogs are registered with CREATE CATALOG. Connect with any MySQL client and run:

CREATE CATALOG iceberg_catalog PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "uri" = "http://localhost:8181",
    "warehouse" = "s3://my-table-bucket",
    "credential" = "AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "s3.endpoint" = "http://localhost:8333",
    "s3.access_key" = "AKIAIOSFODNN7EXAMPLE",
    "s3.secret_key" = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "s3.region" = "us-east-1",
    "use_path_style" = "true"
);

Key settings:

• uri points at the SeaweedFS Iceberg REST Catalog (default :8181).
• warehouse is s3://<table-bucket-name>. SeaweedFS maps this to the bucket of the same name.
• credential is the Iceberg-standard OAuth2 client credentials in client_id:client_secret form. Doris's REST client exchanges this for a bearer token at <uri>/v1/oauth/tokens using the client_credentials grant.
• s3.endpoint, s3.access_key, s3.secret_key, s3.region are used by the BE workers to read parquet files directly from S3.
• use_path_style=true is required for SeaweedFS's path-style S3.

If you change tables outside of Doris (e.g. via Spark, Trino, or PyIceberg), refresh the catalog so the new metadata is picked up:

REFRESH CATALOG iceberg_catalog;

Example SQL

Browse the catalog

-- List all catalogs registered on this Doris cluster
SHOW CATALOGS;

-- Switch into the Iceberg catalog and list namespaces
SWITCH iceberg_catalog;
SHOW DATABASES;

-- List tables in a namespace
SHOW TABLES FROM iceberg_catalog.my_namespace;

Read tables

-- Three-part identifier: catalog.namespace.table
SELECT * FROM iceberg_catalog.my_namespace.events;
SELECT COUNT(*) FROM iceberg_catalog.my_namespace.events;

-- Identifiers with hyphens or special characters need backticks
SELECT * FROM iceberg_catalog.`my-ns`.`my-table`;

The integration test exercises catalog discovery (SHOW CATALOGS / SHOW DATABASES / SHOW TABLES), schema parsing (column-name projection on id, label), SELECT COUNT(*) against an empty table, and reading three rows that were written by a PyIceberg writer before Doris connected. This validates both the metadata path and the parquet read path.

Write paths from Doris (CREATE TABLE, INSERT INTO) against an Iceberg REST catalog are not exercised by the SeaweedFS test suite. Treat Doris primarily as a reader against tables produced by Spark, Trino, or other writers.

Anonymous Access

When SeaweedFS runs without IAM (e.g. weed mini with no -s3.config), the REST catalog accepts unsigned requests. Drop credential from the catalog properties and leave the s3.* keys set — SeaweedFS accepts any value when IAM is disabled:

CREATE CATALOG iceberg_catalog PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "uri" = "http://localhost:8181",
    "warehouse" = "s3://my-table-bucket",
    "s3.endpoint" = "http://localhost:8333",
    "s3.access_key" = "any",
    "s3.secret_key" = "any",
    "s3.region" = "us-east-1",
    "use_path_style" = "true"
);

See Also

• SeaweedFS Iceberg Catalog - Architecture and concepts
• S3 Table Bucket - Managing table buckets
• Doris integration test - End-to-end reference

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg Table Maintenance

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Deployment with seaweed-up
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware