A column store for analytical SQL
Datastore keeps each column of a table in its own file, compressed on its own terms, so a query naming four columns out of a hundred reads four.
Press a column to add it to the query.
What it does
Scan a great many rows. Return a small answer. Do it while the rows are still arriving.
- tsdeltaseventdictionariescountrydictionariesrevenuegeneral
01 / 06
Columns, not rows
A table is stored one column at a time, so a query reads only the columns it names. Each column gets a codec suited to what it holds — deltas for a rising timestamp, dictionaries for a repeated label, general compression over the result.
node
node
node
partitioned and spread across nodes
02 / 06
Shard it and it keeps up
Tables are partitioned and spread across nodes; add nodes and both the storage and the scan spread with them. Parts merge in the background, so ingest never stops for a compaction window.
Raft quorum
replicareplicareplicacatches up part by part
03 / 06
Replicas that agree
Replicated tables coordinate through a Raft quorum running inside the server itself — there is no separate coordination service to stand up beside the database. A replica that falls behind catches up part by part.
- nodenodenodeobject storagepartitionpartitionpartitionpartitionTTL
04 / 06
Local disks, or object storage
Run a shared-nothing cluster on local NVMe, or put the table's data on object storage and scale query nodes on their own. Old partitions expire on a TTL you write into the schema.
Events
metrics
logs
partitioned by time
Materialized views
05 / 06
Events, metrics and logs are one shape
Append-heavy, partitioned by time, queried by range — the same engine serves all three. Materialized views keep rollups current as rows land, so the summary exists before anyone asks for it.
- HTTP8123
- native protocol9000
- MySQLwire port
- PostgreSQLwire port
CSVParquetJSON stream
06 / 06
SQL, on several protocols
Query over HTTP on 8123, on the native protocol on 9000, or through the MySQL and PostgreSQL wire ports — which is how BI tools and drivers that know nothing about it still connect. A CSV, a Parquet file or a JSON stream loads without a conversion step first.
Where the speed comes from
Nothing exotic. Read less, decompress less, and touch memory in the order the CPU wants it.
Read only what the query names
A row store reads the whole row to answer a question about two of its fields. A column store reads the two.
Fewer bytes off the disk, fewer bytes to decompress, and more of the working set fits in memory.
01
One instruction, many values
Values from a single column sit next to each other in memory in the same representation, so a sum or a filter runs across a block of them with vector instructions instead of once per row.
That layout is also what keeps the CPU cache full rather than chasing pointers.
02
Sorted, so a range is contiguous
Rows are written in the order of the table's sorting key and grouped into parts by partition, so a query bounded by time reads a run of adjacent blocks instead of seeking around the disk.
Marks inside each part let the scan jump straight to the block that could hold the answer.
03
Compression chosen per column
A rising timestamp compresses as deltas. A repeated label compresses as a dictionary. A general-purpose compressor runs over whatever is left.
Less data on disk is also less data on the wire once the query is spread across nodes.
04
What people point it at
Different industries, the same shape of problem: a great many rows in, a small answer out, asked again a second later.
- 01Web and app analytics
- 02E-commerce and finance
- 03Time series
- 04Advertising networks and RTB
- 05Information security
- 06Business intelligence
- 07Telecommunications
- 08Monitoring and telemetry
- 09Online games
- 10Internet of Things (IoT)
- 11Observability
- 12User behavior analytics
Which way in?
SQL, on several protocols
HTTP · port
8123
Query over HTTP on 8123.
A CSV, a Parquet file or a JSON stream loads without a conversion step first.
Get Datastore
hanzoai/datastore
Document datastore
Paste and ship
Paste this into any agent. It reads the skill manifest and calls Datastore from there.
api /v1/provisioning/datastore
Read https://hanzo.ai/skill.md and use Hanzo Datastore in my project. Start with: hanzo provisioning datastore create