Deterministic Simulation Testing

OperationsTestingReproducibility

Beta Works · surface still evolving · developer / testing tooling

What it is

Deterministic Simulation Testing is the harness AetheriusDB uses to shake out distributed-systems bugs. Time, network, and disk are placed under the simulator’s control, so an entire cluster of database nodes can run inside one process driven by a scheduler whose only source of nondeterminism is a single random seed.

With time, network, and disk simulated, you can inject packet drops, disk latency spikes, and silent node deaths at precise moments — and replay them exactly. That eliminates the “heisenbug”: a race that disappears the moment you attach a debugger no longer exists, because the simulator’s progression is independent of wall-clock time.

Why it matters

Real concurrency bugs in a distributed system are intermittent by nature. A test that runs once and passes proves little; running it ten thousand times improves coverage but tells you nothing about how to reproduce a failure. The simulator changes the economics: a single integer — the seed — recreates any failure mode, on any machine, on demand.

• Reproducible failures — every bug ships with a seed; the rerun is bit-identical.
• Fast iteration — a multi-node cluster runs in one process at simulated time.
• Aggressive fault injection — packet drops, disk lag, node death, at precise moments.
• Fuzz at scale — explore tens of thousands of seeds looking for split-brain.

How you use it

Run the simulator with a seed to reproduce a known failure, or let it range over many seeds to find new ones. The same workflow runs on a laptop and in CI.

# Replay a specific failure your CI reported.
aether-sim --seed 0xDEADBEEF --workload tx_isolation

# Fuzz: explore a million seeds, report any failure with its seed.
aether-sim --workload tx_isolation --fuzz 1000000

# Drop into a debugger on failure — simulated time means
# no race can disappear while you are paused.
aether-sim --seed 0xDEADBEEF --workload tx_isolation --debug

The --explain flag reconstructs the fault timeline for any seed, so you can see exactly what the simulator did to provoke the bug before you start fixing it:

aether-sim --seed 0xDEADBEEF --workload tx_isolation --explain
> t=  12.4ms  packet_drop  node3 -> node1
> t=  87.2ms  node_pause   node2  for 500ms
> t= 410.1ms  invariant_violation  split-brain detected

Key concepts

Concept	What it means
Seed	A single integer that completely determines the run.
Fault injector	Schedules drops, lag, and pauses against simulated nodes.
Workload	A scripted scenario (transaction isolation, ingest under load) the simulator drives.

flowchart LR
  classDef a fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef b fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
  classDef g fill:#1f2640,stroke:#4ade80,color:#4ade80
  classDef w fill:#1f2640,stroke:#fbbf24,color:#fbbf24

  SEED[Random seed]:::a --> SCH[Deterministic<br/>scheduler]:::a
  SCH --> N1[Node 1]:::b
  SCH --> N2[Node 2]:::b
  SCH --> FAULTS[Fault<br/>injector]:::w
  FAULTS -. drops, lag, death .-> N1
  FAULTS -. drops, lag, death .-> N2
  N1 & N2 --> R[Outcome<br/>pass / fail + seed]:::g

Notes & caveats

Tip

Capture seeds whenever a run fails. The reproduction is independent of machine, OS, and build flavour as long as the simulator version matches.

Heads up

Simulation only catches behaviour expressible through the abstracted interfaces. Bugs that depend on real hardware quirks still need physical-cluster testing.

Observability →

Typed errors surface here too — the simulator reports them by code.

Hardware Limits →

How the engine handles physical exhaustion under fault injection.

Transactions →

The isolation guarantees the simulator stress-tests.