Reactive Pipelines

FilesStreamingReactive

Beta Works · surface still evolving · Cascade syntax still evolving

What it is

Reactive Pipelines turn continuous ingest into continuous insight without any glue code. Two concepts work together:

• A pulse table is a table whose rows are atomically replaced each time a new wave of data arrives — the canonical use case is “latest sensor reading” or “current price.”
• A cascade is a directed pipeline of transformations that runs every time a source table gains new data, propagating summaries forward through intermediate tables to a final sink.

A summary that a traditional materialised view would take seconds to refresh stays continuously fresh, recomputed only over the rows that changed.

Why it matters

Traditional databases force a choice: triggers (slow, per-row), materialised views (stale or write-blocking), or external ETL (latency, complexity, separate failure modes). Reactive Pipelines replace all three — logic runs on whole batches at once, triggered automatically when new data lands.

• Low end-to-end latency — raw ingest to summary in the time most systems take just to schedule a job.
• No glue services — the database itself is the streaming runtime.
• Incremental by default — cascades compute deltas, not full re-aggregations, so cost scales with what changed rather than table size.
• Pulse tables for “latest state” workloads.

flowchart LR
  classDef a fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef b fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
  classDef g fill:#1f2640,stroke:#4ade80,color:#4ade80

  S[Source table<br/>new batch arrives]:::a --> N1[Transform 1<br/>group + sum]:::b
  N1 --> N2[Transform 2<br/>filter + join]:::b
  N2 --> SK[Sink table<br/>live summary]:::g

How you use it

Declare a pulse table for the latest-state slot, then a cascade that forwards summarised deltas to a sink. Once defined, the pipeline runs itself.

-- A "latest value per symbol" pulse table. (Illustrative DDL.)
CREATE PULSE TABLE current_price (
  symbol TEXT,
  price  DOUBLE,
  ts     TIMESTAMP
)
FLUSH ON new_writer;

-- A cascade: raw trades → per-minute volume → live dashboard sink.
CREATE CASCADE trade_pipeline AS
  SOURCE      trades
  TRANSFORM   group_by(symbol, minute(ts), sum(qty))
  PULSE STATE per_minute_volume
  TRANSFORM   top(100, by: volume)
  SINK        top_movers;

-- Reads of the sink return the live summary; no refresh needed.
SELECT * FROM top_movers;

Inserts into trades automatically advance the cascade. The sink table stays close to the latest source data, and the intermediate pulse table holds the current minute’s state for fast point lookups.

Key concepts

Concept	What it means
Pulse table	A table whose rows are atomically replaced on each new write wave.
Cascade	A pipeline of transformations from source to sink, triggered by ingest.
Transformation node	A step that summarises or reshapes data as it flows through.
Incremental	Cascades apply deltas, not full recomputation.

Tip

Use a pulse table at the head of a cascade when only the latest state matters (current price, last sensor reading). Use a regular table when downstream nodes need full history.

Heads up

Cascades are deliberately one-way. If a sink needs to feed back into an earlier source, build that as a separate cascade rather than introducing a cycle.

File-Relational Duality →

The dropped-file ingestion path that feeds most cascades.

Git-Native Source Fabric →

Version cascade logic alongside the data it operates on.

In-Database Compute →

How transformation logic runs inside the engine.

Database Branching →

Branches preserve cascade definitions, so experiments include live processing.