File-Relational Duality

FilesIngestionStreamingCSV / Parquet

Beta Works · surface still evolving · Source-table surface still evolving

What it is

File-Relational Duality lets you treat a directory on disk as a streaming table. Instead of running a nightly ETL job, you point AetheriusDB at a directory and it reacts the instant a file is closed: the file’s contents are parsed, validated, and committed as queryable rows. The file you wrote and the rows you can SELECT are two views of the same thing.

Once parsed, the rows are durable and visible to every open query — no manual COPY, no staging table, no cron job.

Why it matters

Most real-world data arrives as files: sensor exports, log dumps, third-party feeds, scheduled extracts from upstream systems. The usual answer is “set up a pipeline,” and that pipeline becomes its own operational burden. File-Relational Duality removes it. The producer writes a file; the database has it as rows.

• No batch lag — rows appear in the table almost immediately after the writer closes the file.
• No pipeline — no Airflow, no Airbyte, no cron job. The database is the ingestion engine.
• Validated at the gate — schema mismatches and anomalies are rejected before bad data lands.
• Self-cleaning — an optional arrival policy archives or deletes the source file once it is durably stored.

flowchart LR
  classDef a fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef b fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
  classDef g fill:#1f2640,stroke:#4ade80,color:#4ade80

  W[External writer<br/>finishes file]:::a --> K[File-close event]:::b
  K --> S[Parse]:::b
  S --> V[Schema +<br/>anomaly check]:::b
  V --> R[Committed rows]:::g
  R --> Q[Queryable table]:::g

How you use it

Declare a source table that points at a directory. Whatever lands there, in the formats you allow, becomes rows in the table.

-- Register a directory as a streaming source. (Illustrative DDL.)
CREATE SOURCE TABLE sensor_readings (
  ts        TIMESTAMP,
  device_id BIGINT,
  value     DOUBLE
)
FROM DIRECTORY '/var/feeds/sensors'
FORMAT csv
ON ARRIVAL archive;

An external job then writes a file into that directory:

cat > /var/feeds/sensors/2026-05-18.csv <<EOF
ts,device_id,value
2026-05-18T10:00:00,42,18.3
2026-05-18T10:00:01,42,18.4
EOF

…and it is immediately queryable:

SELECT avg(value)
FROM   sensor_readings
WHERE  device_id = 42
  AND  ts > now() - INTERVAL '1 minute';

The source table behaves like any other table for read queries. Joins, aggregates, and indexes all work — and the producer never had to do anything except write a file.

Key concepts

Concept	What it means
Source table	A table whose rows are the contents of a watched directory.
Close-triggered	Ingestion fires when the writer closes the file, not on a polling schedule.
Arrival policy	The rule that decides what happens to the source file after ingest — keep, archive, or delete.
Validation gate	Schema and anomaly checks that reject bad data before it becomes rows.

Tip

If the producer writes large files atomically — for example, writing to a temp name then renaming — use that pattern. Partial files trigger anomalies and are rejected rather than half-loaded.

Heads up

File-close events depend on the operating system’s filesystem hooks. Watched directories must be on a supported local mount; remote NFS shares emit events less reliably.

Reactive Pipelines →

Chain ingested files into downstream summaries automatically.

Git-Native Source Fabric →

Version the logic that processes your files alongside the data.

Database Branching →

Branch a live database for safe experimentation.

Loading Data →

Other ways to get data into AetheriusDB.