Salman Adnan

sqlmill

SQL Database Engine

A database is the filing cabinet a program keeps its information in. This is one built from scratch, and the hard part is making sure that if the power cuts out mid-write, nothing is lost or scrambled on restart. It was killed 300 times at random moments; all 300 times the data came back intact, and every one of 8,504 finished pieces of work was still there.

B+tree Pulse: the index inside the database, drawn as a tree. A lookup walks down from the top to find one row, a range scan sweeps sideways along the bottom to collect many, and the ribbon underneath is the write-ahead log, the safety journal that records every change before it is made. Live and interactive: drag it to orbit, scroll or pinch to zoom. Open full screen
Benchmark charts: sqlmill's insert throughput by durability mode, and point-lookup vs full-scan latency by row count.
From a real single-threaded benchmark run (benchmarks/RESULTS.md).
300 / 300SIGKILL crash rounds
300 / 300queries match SQLite
8,504committed txns verified durable

Overview

A relational database engine written from scratch in Python: a hand-written SQL parser, a cost-based planner, B+tree storage on slotted 4KB pages, a write-ahead log with real crash recovery, and MVCC snapshot-isolation transactions. The core uses only the standard library.

Key features

  • SQL: a hand-written lexer and recursive-descent parser covering DDL, indexes, INNER and LEFT joins, GROUP BY with aggregates and HAVING, transactions, EXPLAIN, and VACUUM. Syntax errors report line, column, and a caret under the offending token.
  • Planner: predicate pushdown, index selection with a row-count cost model, left-deep join-order search, and a hash join vs nested loop choice.
  • Storage: slotted heap pages, a from-scratch B+tree with variable-length keys and split, borrow, and merge rebalancing, an LRU buffer pool, and per-page CRC32 checksums.
  • Durability: write-ahead logging with full-page after-images batched atomically per safe point, fsync on commit, checkpointing, and torn pages detected by checksum and repaired from the log.
  • Transactions: MVCC snapshot isolation with first-committer-wins conflict detection and VACUUM garbage collection. Readers never block writers.

Verification

  • 85 tests covering the parser, planner, B+tree property tests, buffer pool eviction, WAL recovery including torn pages, MVCC, and the differential suite.
  • Crash campaign: 300 rounds passed (seed 2026, 105s); each round SIGKILLs a workload subprocess at a random moment, and 135 rounds also corrupted half of a written page before recovery. 8,504 committed transactions were verified durable; uncommitted work never survived.
  • Differential suite: 300 seeded generated queries run against both sqlmill and stdlib sqlite3, compared modulo row order. 300 of 300 match.
  • Benchmarks: about 1,000 inserts/s with an fsync per commit vs about 2,800/s with fsync off; indexed point lookups stay around 0.2 to 0.3 ms while full scans grow linearly, about 390x slower at 50k rows.

Tech stack

  • Python 3.12
  • Standard library only
  • B+tree storage
  • Write-ahead log
  • MVCC
  • pytest

Book a call

Let's talk about what you're building.

Pick a slot below.