← Back to selected work

Distributed systems / Experimental database

AegisDB

A database built to be interrupted.

Five replicas, crash recovery, and deliberate failure testing.

My contribution
Built the replication runtime, durable storage path, and fault-injection testbed.
Result
The recorded campaign completed 5,000 requests and recovered after a full-cluster restart.

Private repository · source and verification links require access.

FROM THE PROJECT

The recorded failure campaign

Archived verification report · 7 September 2026

  • 5,000 requests completed across five replicas
  • 46 deliberate SIGKILL exits during the campaign
  • Full-cluster restart and damaged-SSTable repair recorded

Summary of retained results, not a live run or execution trace.

Inspect the recorded chaos campaign ↗

Private repository · access required to inspect the report.

5
independent durable replicas
5,000
requests in the recorded chaos run
61
reported regression tests

THE CHALLENGE

What survives when a replica fails?

A successful transaction is only half the problem. A database also has to preserve its meaning when a primary disappears, a connection breaks, or a machine restarts with damaged storage. AegisDB brings these failure cases into the same testbed as the database itself.

ENGINEERING

Ordering writes and recovering durable state

The runtime uses Viewstamped Replication (VSR), with a serialized consensus actor in each daemon. Four native network threads and a separate disk worker handle I/O; C11 single-producer/single-consumer queues connect the work. The compiled daemon embeds CPython rather than running a separate Python service.

Transactions can combine pre-write reads, conditional expectations, and all-or-nothing writes. Reads and analytical queries join the replicated order. The query engine supports SUM, COUNT, MIN, MAX, and AVG using typed column chunks, selection vectors, and null bitmaps.

The storage path fsyncs checksummed WAL blocks before protocol acknowledgement. Checkpoints publish immutable SSTables through an atomically replaced CURRENT manifest. On restart, replicas load the tables and replay the WAL tail; detected corruption quarantines a node until quorum recovery.

Real failures, not only simulation

A supervisor drives five processes through SIGSTOP, SIGCONT, SIGKILL, directed partitions, disconnected sockets, corrupt frames, and a damaged SSTable. A separate deterministic simulator remains available for reproducible traces.

Retry without duplicate transactions

Client identity and increasing sequence numbers support deduplication. Recorded replay checks retry the original 5,000 requests and compare their results without adding duplicate commits.

Separate model proof from implementation evidence

Lean establishes safety of the VSR transition system. Runtime tests independently examine byte-level storage, networking, and recovery. Neither is presented as a substitute for the other.

ARCHITECTURE

Follow a transaction through the cluster

AegisDB architectureTransactions are ordered by the current VSR primary, replicated to peers, persisted before acknowledgement, and answered with a quorum-backed result. Damaged replicas follow quorum recovery.Client transactionReads · expectations · writesCurrent VSR primarySerialized consensus actorFive replica processesReal TCP transportDurable statefsync WAL → SSTablesQuorum recoveryRestart or corrupt storageQuorum-backed responseResult + request deduplication

Transactions are ordered by the current VSR primary, replicated to peers, persisted before acknowledgement, and answered with a quorum-backed result. Damaged replicas follow quorum recovery.

RESULTS AND LIMITS

Recovery with a retained evidence trail

The repository’s 7 September 2026 report records 61 passing regression tests, 5,000 completed requests from 16 clients, and a successful full-cluster restart. It also records 46 deliberate SIGKILL exits and repair of an on-disk SSTable. These are archived results, not tests rerun by this portfolio.

Current boundaries

The Lean proof covers unbounded protocol safety, not a refinement proof of the Python/C daemon, disk parser, or OS calls. Fixed membership and trusted peers remain part of the operating contract.

EXPLORE THE SOURCE

Inspect the recovery and proof records

Repository review: 14 September 2026. Recorded test results retain their original scope; they were not rerun for this site.

← All selected workNEXT PROJECTLingji ↗