Native Ingestion
What it is
Section titled “What it is”AetheriusDB ingests data natively — external sources connect straight to the database with no separate ETL service, no Kafka Connect deployment, and no glue job in the middle. A Kafka topic, an S3 prefix, a webhook, or a live Postgres database becomes a first-class object in your schema, and the engine runs the listener itself.
This page is a map of the paths so you can pick the right one. Each path feeds the same tables and shares the same backpressure model: when the database starts to fall behind, it slows the upstream rather than dropping data.
The ingestion paths
Section titled “The ingestion paths”| Path | Best for | Page |
|---|---|---|
| Continuous | High-rate streaming INSERTs from your own producers | Continuous ingestion |
| Bulk (Arrow) | Large one-shot loads of Arrow batches or Parquet files | Bulk ingestion (Arrow) |
| Source connectors | Pulling from Kafka, Pulsar, S3, GCS, Azure, or HTTP | Source connectors |
| CDC bridge | Mirroring a live Postgres or MySQL database | CDC bridge |
flowchart LR classDef ext fill:#1f2640,stroke:#00d4ff,color:#e8eaf0 classDef fab fill:#1f2640,stroke:#7c5cff,color:#e8eaf0 classDef ing fill:#1f2640,stroke:#4ade80,color:#e8eaf0 K[Kafka / Pulsar]:::ext --> AF[Native ingestion]:::fab S3[S3 bucket events]:::ext --> AF WH[HTTP webhook]:::ext --> AF PG[Postgres / MySQL]:::ext --> AF AF --> T[Your tables]:::ing
Why it matters
Section titled “Why it matters”Owning the ingest layer inside the database removes a whole operational tier. You stop managing connector deployments, schema-registry adapters, and dead-letter queues as separate services to monitor and page on.
- Single deployment — the database is the ingest service.
- Shared backpressure — every path slows its upstream under load instead of dropping data.
- One monitoring surface — every path reports through the same producer stats.
How to use it
Section titled “How to use it”Connectors and bridges are declared with CREATE SOURCE / CREATE BRIDGE and
started with START. Continuous tables use WITH (ingest = 'continuous'). Bulk
loads use COPY FROM or Arrow Flight SQL. Follow the linked pages for the full
grammar of each.
-- A Kafka topic feeding a target table (see Source connectors).CREATE SOURCE orders_stream TYPE 'kafka' OPTIONS (brokers = 'broker1:9092', topic = 'orders.v1', format = 'avro') INTO orders;
START SOURCE orders_stream;