Data Persistence Engine
What it is
Section titled “What it is”The persistence engine is the layer that makes AetheriusDB a production-grade, crash-recoverable database rather than a fast in-memory cache. It owns the on-disk layout, the durable writes, the periodic checkpoints, and the recovery that brings a server back to its exact committed state after any failure short of physical media loss.
It also lets you run several independent databases inside one server. Each database is fully self-contained — its own directory, its own checkpoint history, its own failure domain — so an issue in one cannot harm another.
Why it matters
Section titled “Why it matters”You get strict durability without sacrificing throughput, clean multi-database isolation, and recovery times that stay bounded even on very large datasets.
- Zero data loss on committed transactions — the commit acknowledgement is only sent after data is durable on disk.
- Bounded recovery time — checkpoints keep restart work proportional to the last few minutes of activity, not the entire history.
- Per-database isolation — each database has its own catalog and storage, so problems stay contained.
How you use it
Section titled “How you use it”Databases and schemas are first-class SQL objects. You manage them with familiar DDL, and the engine handles the file layout, the durability, and the recovery automatically.
-- Create a database (materializes a self-contained directory on disk).CREATE DATABASE analytics_prod WITH data_directory = '/nvme-fast/aetherius/', checkpoint_interval = '2 minutes', compression = 'zstd';
-- Use it; create a schema and a table.USE analytics_prod;CREATE SCHEMA reporting;CREATE TABLE reporting.events ( id BIGINT PRIMARY KEY, occurred TIMESTAMP, payload JSON);
-- Force an immediate checkpoint before a maintenance window.CHECKPOINT;
-- Inspect durable state.SHOW CHECKPOINT STATUS;SHOW RECOVERY STATUS;A clean shutdown (SIGTERM) always takes a final checkpoint, so the next startup
is an instant load with no replay. A forced kill is safe too — recovery resumes
from the last checkpoint and the committed writes after it. See
Crash Recovery & Durability for what a restart looks
like from the user’s side.
Tuning checkpoints
Section titled “Tuning checkpoints”checkpoint_interval trades recovery time against background I/O. A shorter
interval shrinks worst-case recovery; a longer interval reduces steady-state I/O.
Most workloads do well with the default.
Storage format & compression
Section titled “Storage format & compression”By default AetheriusDB stores table data raw and uncompressed, in a columnar layout engineered so large scans run at full memory bandwidth. That is a deliberate speed choice: there is no decompression or decode step on the scan path, which is what keeps large scans fast. So if you are wondering “is my data compressed like Parquet?” — by default, no: it is kept raw so reads run at memory speed.
Raw does not mean “scan everything.” Range- and time-bounded queries automatically skip whole blocks of data that cannot contain a match — the direct equivalent of Parquet’s row-group statistics — so narrow queries never pay for the full table.
At-rest compression (opt-in, off by default)
Section titled “At-rest compression (opt-in, off by default)”For disk- or IO-bound workloads you can compress the on-disk copy of a column while the in-memory copy stays raw. Hot scans keep their full speed — a compressed column is decoded once, when the data is brought into memory, never on every scan. Enable it per server with environment variables:
| Flag | What it does |
|---|---|
AETHERIUS_COMPRESS_DISK=1 | Automatically picks the best lightweight encoding per column — standard codecs including LZ4, frame-of-reference bit-packing, and run-length encoding — and leaves a column raw when compression would not help. |
AETHERIUS_DICT_ENCODE=1 | Dictionary-encodes low-cardinality text columns, so repeated text values take far less space — no change in query results. |
The encoding is chosen per column, so compressed and uncompressed columns coexist safely, and data written before you turned a flag on keeps reading unchanged. Compression is never applied to the hot ingest path or the in-memory working set — only to the cold, at-rest copy.