Bulk Ingestion (Arrow)

IngestionArrowFlight SQLCOPY

Opt-in Implemented · off by default — enable with BULK_PULSE_V2=1 · set as an env var when starting aetheriusd; the standard path works without it

What it is

Bulk ingestion is the path you take to push (or pull) gigabytes of pre-shaped columnar data without paying the per-row cost of a SQL driver. AetheriusDB speaks the Apache Arrow wire format natively over Flight SQL, so a client can stream Arrow batches straight into a table — and stream query results straight back out into a Python or Polars memory space — without an intermediate row-by-row conversion step.

The same path serves COPY FROM against files and cloud object stores: workers download Parquet chunks in parallel and load them into columnar storage. Throughput is bounded by your network and storage hardware, not by the database.

Fast bulk path is opt-in

Start aetheriusd with BULK_PULSE_V2=1 to route bulk loads through the high-throughput path. Without the flag, the standard path is used — slower, but identical results. See Configuration & Availability.

Why use it

The Arrow path makes AetheriusDB a peer in the data-engineering ecosystem. Pandas, Polars, DuckDB, and Spark all speak Arrow natively, so bulk ingestion connects to them directly instead of requiring an adapter.

• Network-bound throughput — pipelines are limited by hardware, not the database.
• No serialization tax — the wire format is the same columnar format end to end.
• Parallel COPY — multi-file loads from S3 or GCS download and ingest concurrently.

flowchart LR
  classDef cli fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
  classDef wire fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef store fill:#1f2640,stroke:#4ade80,color:#e8eaf0

  P[Pandas / Polars<br/>Arrow batches]:::cli --> F[Flight SQL<br/>endpoint]:::wire
  F --> T[Table]:::store
  OBJ[S3 / GCS<br/>Parquet files] --> COPY[COPY FROM<br/>workers]:::wire
  COPY --> T

How to use it

From SQL, bulk loading is a COPY FROM against a file or object-store path. From a Python client, it is a single Arrow-aware call over Flight SQL. Both end up in the same table.

-- Bulk-load a directory of Parquet files from S3 in parallel.
COPY sales
  FROM 's3://my-bucket/exports/2026/*.parquet'
  WITH (format = 'parquet', parallelism = 16);

# Stream a SELECT out as Arrow batches over Flight SQL.
import pyarrow.flight as flight

client = flight.connect('grpc://aether:8815')
batches = client.do_get(flight.Ticket(b'SELECT * FROM sales'))
df = batches.read_all().to_pandas()

Round-trip cost between AetheriusDB and a Pandas DataFrame on the same host is dominated by memory bandwidth, not protocol overhead.

Notes & caveats

Verify the schema first

The schema in your Arrow batches must match the target table column by column. Use a Flight SQL GetSchema call before a DoPut to verify before you stream.

COPY FROM uses your configured credentials

COPY FROM against a cloud bucket uses the credentials configured for the database. Make sure the database’s role has read access to the prefix you target, or individual files will fail mid-load with permission errors.

COPY scales linearly, and can be memory-bounded

COPY appends incrementally — each batch is added without re-serializing the whole table, so a large load stays linear rather than degrading quadratically. COPY also enforces PRIMARY KEY, UNIQUE, and FOREIGN KEY constraints on the incoming rows. On a memory-tight host, start the daemon with AETHERIUS_INGEST_SPILL=1 to evict each cube from RAM as it is flushed durably, bounding the ingest working set (see Configuration).

Continuous ingestion →

Micro-batch streaming where rows trickle in over time.

Source connectors →

Watch S3 buckets and auto-load files as they land.

Configuration & Availability →

The BULK_PULSE_V2 flag and other ingestion defaults.