Source Connectors
What it is
Section titled “What it is”Source connectors turn a CREATE SOURCE declaration into a running consumer. Each
supported protocol — Kafka, Pulsar, S3, GCS, Azure Blob, HTTP — has a driver that
knows how to talk to its upstream, decode payloads, track its position, and feed
rows into the database. Every driver sits behind one connector contract, so
configuration, startup, monitoring, and shutdown look identical no matter where
your data comes from.
Cursors and offsets are persisted durably. When the database restarts, every connector resumes from exactly the position it last acknowledged — you never reprocess data already ingested and never lose data that was in flight. Credentials come from a pluggable source: environment variables, an IAM role, a secret manager, or a custom provider.
Why use it
Section titled “Why use it”Every external data source eventually becomes its own miniature service — connector pods, dead-letter queues, config repos, dashboards. Folding all of that into the database means one fewer system to deploy, one fewer credential trust boundary, and one fewer thing to page on.
- One config surface — Kafka, S3, GCS, Azure, and HTTP all use the same
CREATE SOURCEgrammar. - Restart-safe — cursors persist; connectors resume cleanly from the last acknowledged position.
- Pluggable credentials — env vars, IAM roles, secret managers, or your own provider.
stateDiagram-v2 [*] --> Created : CREATE SOURCE Created --> Running : START SOURCE Running --> Paused : STOP SOURCE Paused --> Running : START SOURCE Running --> Backpressured : ingest saturated Backpressured --> Running : pressure released Running --> Failed : credential expired Failed --> Running : credentials refreshed Paused --> [*] : DROP SOURCE
How to use it
Section titled “How to use it”Declare connectors with CREATE SOURCE, optionally bind a credentials provider,
then start them. Stopping and restarting is safe — cursors are durable.
-- An S3 bucket watched via its event bus.CREATE SOURCE ledger_files TYPE 'cloud' OPTIONS ( provider = 's3', bucket = 'finance-ledger', prefix = 'daily/', format = 'parquet', events = 'sqs://us-east-1/aether-ledger-events', credentials = 'iam-role' ) INTO ledger;
START SOURCE ledger_files;
-- See every running connector and its current state.SELECT name, kind, running, rows_ingested, cursor FROM aether.sources ORDER BY name;
-- Pause cleanly for maintenance.STOP SOURCE ledger_files;The cursor for ledger_files is checkpointed inside the database; when you
restart the source, it resumes from the last file event it successfully ingested.