HNTurboVec · write-up · David Montero Crespo · October 2026

All of Hacker News in 5.3 GB, searchable on a laptop

This article describes hnz, a Rust library with a Python API that stores every Hacker News item in a single 5.34 GB file and searches it with hybrid semantic and keyword search on a laptop CPU. It explains how a small language model compresses the text, why the search indexes can stay small, and what the results look like.

49.3 M
items, from Oct 2006 to Aug 2026
5.34 GB
text, metadata, both indexes and both models
0
differences against the source, item by item
~2.3 s
for a cold reranked search on a 16-thread laptop CPU

Why I built it

I wanted all of Hacker News on my laptop, searchable the way a search engine is: a question in plain words, a few keywords, a date range, the top results with their full text. No server, no GPU card. The public Parquet dump of HN is 12.4 GB and has no search at all, so I set myself a target of 4 to 5 GB for everything. In exchange I was willing to wait up to 10 seconds for an answer, and to let the archive take days to build.

The machine is a laptop with an Intel Core Ultra 7 265H, 31 GB of RAM and an integrated GPU. Everything in this article was built and measured on it, in October 2026.

What you need to use it today: hnz runs on Windows on x86-64 with AVX2, and the archive (I call it the block) is not published for download yet. You rebuild it from the public dataset with the scripts in the repository, which takes about a week on a laptop like mine.

Where the numbers start

All of Hacker News on disk All of Hacker News on disk 49.3 M items, 2006-10 to 2026-08 · GB (10⁹ bytes) 0 5 10 15 20 25 Raw JSON (HN API) text and metadata, no search 22.1 GB Parquet dump open-index/hacker-news, no search 12.4 GB zstd-19 + dictionary, per item random access, no search 9.1 GB hnz v1 segments zstd + LM tiers, hybrid search 10.4 GB hnz block LM-compressed, hybrid search, lossless 5.34 GB The hnz block also holds the semantic index, the word index, the LM and the query embedder.
All of Hacker News on disk All of Hacker News on disk 49.3 M items, 2006-10 to 2026-08 · GB (10⁹ bytes) 0 5 10 15 20 25 Raw JSON (HN API) text and metadata, no search 22.1 GB Parquet dump open-index/hacker-news, no search 12.4 GB zstd-19 + dictionary, per item random access, no search 9.1 GB hnz v1 segments zstd + LM tiers, hybrid search 10.4 GB hnz block LM-compressed, hybrid search, lossless 5.34 GB The hnz block also holds the semantic index, the word index, the LM and the query embedder.
FormatSizeSearch
Raw JSON from the HN API22.1 GBnone
Parquet dump (open-index/hacker-news)12.4 GBnone
zstd-19 with a trained dictionary, item by item9.1 GBnone
hnz block5.34 GBhybrid, with filters

The 5.34 GB include the text, the metadata, a semantic index, a word index and the two models that hnz needs to read them.

Compressing the text with a language model

My first experiments used ordinary compressors. Grouping items into small blocks before zstd took text and metadata from 181 to 124 bytes per item. After that, stronger compressors stopped helping. zstd at level 22, brotli and xz all landed between 106 and 114 bytes, and even xz over a whole month in one stream, with no random access at all, only reached 91.

The paper Language Modeling Is Compression (DeepMind, 2023) shows that a model that predicts the next token well, driving an arithmetic coder, is a very good compressor. A quick test with SmolLM2-135M suggested about 50 bytes per record, so I tried it for real. Three problems came up.

  1. The decoder has to compute exactly the same probabilities as the encoder. One different bit and the rest of the record decodes as garbage. Floating-point inference libraries do not promise this across machines or batch sizes, so I wrote a small inference engine in Rust for Pythia models: int8 weights and activations, exact integer matrix products, and LayerNorm, softmax and GELU with my own exp and erf, summed in a fixed order. The AVX-VNNI kernels produce the same bytes as the plain scalar code, and the block records which version of the arithmetic it was made with, so a mismatched reader refuses it.
  2. Speed. Compression runs one record per thread and reaches 122 records per second on 16 threads. Decompression decodes many records in lock-step, with a KV cache.
  3. A model that knows HN. I trained pythia-70m for another 93 minutes on 1% of HN, on the laptop's integrated GPU. It now compresses HN text 7 to 11% better (1.33 bits per byte instead of 1.49 on comments from 2024), at the same cost per token. One surprise along the way: bf16 broke Pythia completely (loss 5.71, against 4.00 in fp32), while fp16 with loss scaling matched fp32.

The text now takes 59 bytes per item. The same text with zstd-19 and a trained dictionary takes 137. The model itself is 72 MB and travels inside the block, the same way a compression dictionary would.

Small indexes, with the text as a second index

Decoding the first tokens of a record takes milliseconds. That changes the usual trade of a search engine, which keeps rich indexes so it never has to touch the documents. Here the indexes can be small, and the search pays a little at query time instead:

How a query runs A hybrid search embeds the query, takes 100 candidates from the 256-bit vectors and the rare-term index, decodes the first 128 tokens of each with the language model, reranks them and returns k items. VECTORS, WORDS LM-CODED TEXT METADATA IN Query text and filters dates · type · by CPU Embed query EmbeddingGemma 2 int8 · 768 dims CPU Stage 1 top 80 by vector top 20 by BM25 LM Decode prefixes first 128 tokens batched, shared CPU Rerank BM25 of every word re-embed top 50 OUT Results k items, verbatim ~2.3 s cold FILE The block · hn.hnzb 5.07 GB, read-only · every part hashed (XXH3-128) LEGEND Most of the time goes here A step on the CPU Read from the block Input
How a query runs A hybrid search embeds the query, takes 100 candidates from the 256-bit vectors and the rare-term index, decodes the first 128 tokens of each with the language model, reranks them and returns k items. VECTORS, WORDS LM-CODED TEXT METADATA IN Query text and filters dates · type · by CPU Embed query EmbeddingGemma 2 int8 · 768 dims CPU Stage 1 top 80 by vector top 20 by BM25 LM Decode prefixes first 128 tokens batched, shared CPU Rerank BM25 of every word re-embed top 50 OUT Results k items, verbatim ~2.3 s cold FILE The block · hn.hnzb 5.07 GB, read-only · every part hashed (XXH3-128) LEGEND Most of the time goes here A step on the CPU Read from the block Input

On a three-month sample, this reranking raised P@10 from 0.211 to 0.238, and with it 384 bits did as well as 768 bits without it. Results are also deterministic: each prompt is embedded alone, at a fixed shape, so the answer does not depend on timing or on how many queries run at once.

Inside the block

Inside the 5.34 GB block Treemap of the block by bytes: the LM-coded text is 53 percent, the semantic vectors 28 percent, the word index, the models and the metadata 5 to 7 percent each. Text 2.81 GB · 53% coded by the LM · 59 B per item Semantic vectors 1.52 GB · 28% 256 bits per item Word index 0.37 GB · 7% 4.2 M rare terms Models 0.35 GB · 7% embedder + LM Metadata 0.30 GB · 6% + 12 MB directory LEGEND AREA = BYTES · HN-2026-08.HNZ · 5,343,133,431 BYTES · OCT 2026 Text · the focal part, more than half the block Other parts · stronger contrast is larger
Inside the 5.34 GB block Treemap of the block by bytes: the LM-coded text is 53 percent, the semantic vectors 28 percent, the word index, the models and the metadata 5 to 7 percent each. Text 2.81 GB · 53% coded by the LM · 59 B per item Semantic vectors 1.52 GB · 28% 256 bits per item Word index 0.37 GB · 7% 4.2 M rare terms Models 0.35 GB · 7% embedder + LM Metadata 0.30 GB · 6% + 12 MB directory LEGEND AREA = BYTES · HN-2026-08.HNZ · 5,343,133,431 BYTES · OCT 2026 Text · the focal part, more than half the block Other parts · stronger contrast is larger

The block is one read-only file, hn.hnzb, next to the two files of the query embedder.

Memory

hnz.open(path, ram="2GB") decides what to keep in memory (the models, the most used parts of the indexes, caches) and reads the rest from disk, with the same results either way. The working memory of each query is charged to the same budget, so when it runs out, new queries wait or get a BusyError instead of pushing the process over the limit. With ram="2GB" the process used about 1.45 GB.

Building it

How the block is built The monthly Parquet files are compressed by the language model, embedded on the integrated GPU and indexed in parallel, then packed month by month into one 5.07 GB block plus the query embedder. IN Parquet files 239 months · 12.4 GB open-index/hacker-news CPU LM compression pythia-70m trained on HN arithmetic coding · ~7 days iGPU Document vectors EmbeddingGemma 2 → 256 bits ~4.5 days, alongside CPU Word index and metadata BM25 of rare terms time-ordered columns CPU hnz pack month by month ~1 h · 3.2 GB RAM FILE The block hn.hnzb 5.07 GB + embedder 0.28 GB LEGEND The slow part: ~7 days of CPU A build step The result Input
How the block is built The monthly Parquet files are compressed by the language model, embedded on the integrated GPU and indexed in parallel, then packed month by month into one 5.07 GB block plus the query embedder. IN Parquet files 239 months · 12.4 GB open-index/hacker-news CPU LM compression pythia-70m trained on HN arithmetic coding · ~7 days iGPU Document vectors EmbeddingGemma 2 → 256 bits ~4.5 days, alongside CPU Word index and metadata BM25 of rare terms time-ordered columns CPU hnz pack month by month ~1 h · 3.2 GB RAM FILE The block hn.hnzb 5.07 GB + embedder 0.28 GB LEGEND The slow part: ~7 days of CPU A build step The result Input

Two workers ran side by side for about a week. One compressed the text with the language model on the CPU (about 7 days), the other computed the embeddings on the integrated GPU (about 4.5 days). Both could resume after a reboot, which turned out to matter. Packing the final block took about an hour.

Windows gave me most of the trouble:

Results

Search quality on the full block Search quality on the full block P@10 · 300 queries per set, each searched within its month among 49.3 M items Hybrid search, reranked (the default) Keyword search only 0.0 0.1 0.2 0.3 0.4 0.31 0.25 2010-06 0.34 0.27 2015-06 0.31 0.26 2024-01 0.36 0.29 dev 2010-06 0.34 0.26 dev 2015-06 0.28 0.22 dev 2024-01 Relevant = the comments of the query’s own thread. Without the month filter, threads on the same subject from other years count as misses (P@10 ≈ 0.05).
Search quality on the full block Search quality on the full block P@10 · 300 queries per set, each searched within its month among 49.3 M items Hybrid search, reranked (the default) Keyword search only 0.0 0.1 0.2 0.3 0.4 0.31 0.25 2010-06 0.34 0.27 2015-06 0.31 0.26 2024-01 0.36 0.29 dev 2010-06 0.34 0.26 dev 2015-06 0.28 0.22 dev 2024-01 Relevant = the comments of the query’s own thread. Without the month filter, threads on the same subject from other years count as misses (P@10 ≈ 0.05).

To measure quality I used 300 queries per month for three months, in a test set and a dev set. Each query is a story's title, and the relevant results are the comments in that story's own thread. Searching within the query's month, among all 49.3 million items, the default hybrid search puts on average 3.2 relevant comments in the top 10. Keyword search alone puts 2.6.

Without the month filter the score drops to about 0.05, and that number needs some context. This eval only counts the query's own thread as relevant, so similar threads from other years count as misses even when they answer the question. The same queries on a three-month block scored 0.24, so the search itself did not get worse with 16 times more data. A better eval for the whole archive is on my list.

Latency on the full block Latency on the full block seconds · laptop CPU (Intel Core Ultra 7 265H, 16 threads), ram="2GB" 0 s 1 s 2 s 3 s 4 s get() an item verbatim text, LM-decoded 80 ms Keyword search mean of 1,800 eval queries 220 ms Hybrid, first stage only rerank=False 400 ms Hybrid, the same query again decoded prefixes reused 1.8 s Hybrid, reranked, cold mean of 1,800 eval queries 2.3 s Open the block with the query embedder loaded 4.0 s
Latency on the full block Latency on the full block seconds · laptop CPU (Intel Core Ultra 7 265H, 16 threads), ram="2GB" 0 s 1 s 2 s 3 s 4 s get() an item verbatim text, LM-decoded 80 ms Keyword search mean of 1,800 eval queries 220 ms Hybrid, first stage only rerank=False 400 ms Hybrid, the same query again decoded prefixes reused 1.8 s Hybrid, reranked, cold mean of 1,800 eval queries 2.3 s Open the block with the query embedder loaded 4.0 s

A cold reranked search takes about 2.3 seconds, and most of that is decoding the candidates' prefixes. A keyword search takes about 0.2 seconds, and fetching an item about 80 ms. That is well inside the 10 seconds I was ready to accept.

The hnz web page showing four comments about burnout from 2015 to 2018
The search page served by hnz serve, with a hybrid search limited to comments from 2015 to 2018.

Ideas I tried and dropped

IdeaWhat it gaveWhy I dropped it
A cross-encoder reranker+6 to 10% P@10it needs the full text of every candidate: 5 to 7 times slower
A bigger LM (pythia-160m)8.8% less textdecompression 2.3 times slower
TurboQuant-style vectorswithin the noise at the same sizeneeds training, no clear gain
Stronger general compressors106 to 114 bytes per itemfar from what the LM reaches

Testing

acceptance on the block of all of HN (the installed wheel, public API only) acceptance on the block of all of HN (the installed wheel, public API only) $ python scripts/accept_full.py --root work/p6/hn-2026-08.hnz --ram 2GB hnz 0.1.0 from site-packages PASS verify_block('quick') (401 ms) PASS open with the embedder loaded (49,324,111 docs, 3.8 s) PASS hybrid reranked search: 10 results (4029 ms cold) PASS the same search again: same results (1893 ms) PASS first stage only (rerank=False) (477 ms) PASS keyword + since + types filters hold (1653 ms) PASS a one-month window holds PASS author filter, no query PASS no query: the most recent jobs PASS deadline_ms=1 still answers (degraded) (605 ms) PASS get(1): the first HN item ('Y Combinator', 20 ms) PASS get() of a search result PASS get() of an id that does not exist PASS get_many() PASS embed(): a 768-dim vector PASS asyncio: asearch / aget PASS a bad query raises ValueError PASS the RAM ledger stays within the budget (used 1.54 GB, peak 1.61 GB, limit 2.00 GB, process private 1.46 GB) PASS Archive.serve(): /stats PASS Archive.serve(): /search equals search() PASS Archive.serve(): /item/1 PASS Archive.serve() stops on the Event PASS `hnz --version` (python -m hnz) PASS `hnz serve`: answers a search (hnz: 49324111 docs - http://127.0.0.1:…/) ALL PASSED

The Python suite has 73 tests. They check that every item comes back byte for byte against the Parquet files, that results are the same from disk and from RAM, concurrency, that a changed byte anywhere in the block is detected without crashing, the memory limits and the HTTP server. The Rust core has 35 more, including bit-exact kernels and a corruption matrix. A separate script runs 24 checks of the installed package against the full block in about 30 seconds.

Trying it

  1. Build the block with the steps in the repository's README (about a week), or copy one you already have.
  2. Install the package: pip install hnz-0.1.0-cp310-abi3-win_amd64.whl.
  3. Search from Python:
    import hnz
    
    with hnz.open(r"D:\hn\hn-2026-08.hnz", ram="2GB") as hn:
        for h in hn.search("how do I deal with burnout as a software engineer", k=5,
                           since="2015", until="2018", types="comment"):
            print(h.date, h.by, h.text[:120])
  4. Or start the web page with hnz serve --root D:\hn\hn-2026-08.hnz --ram 2GB and open http://127.0.0.1:8088/?q=sqlite+in+production.

The code, the design notes and the full experiment log, including the experiments that failed, are in the repository: github.com/davidmonterocrespo24/HNTurboVec (MIT license).