Retrieval without embeddings

ID/EN

August 3, 2026

Why I built a 256-dimension vector by hand, deterministic and free and deliberately dumb, and the four-stage gate it feeds.

The default way to build retrieval now is embeddings: run text through an embedding model, store the vectors in a vector database, rank by cosine similarity. It works and it's quick to set up. I didn't use it.

The retrieval vectors here are built by hand: 256 dimensions, no model inference, pure Go, hashed with FNV-1a. No API call, no GPU, no per-item cost. This post is about why that was the right call for this system, and how the vector feeds a four-stage filter that decides what to keep.

It started with a conversation with Pak Aria Ghora

Part of this post traces back to a June exchange with Aria Ghora, around antirez's line that raw vector search is a fundamental data structure. The idea of his that stuck: a vector is a general representation, it doesn't have to come out of an LLM — classic ML feature extraction, even a document's layout structure, all of it can be encoded. As he put it (translated from Indonesian):

As long as something can be encoded as a vector, whether via manual extraction or some pretrained model, it can technically be framed as a vector search problem.

My reply at the time was "I'll go dig into it, Pak." The 256-dimension vector below is what came out of the digging.

Why not embeddings

Three reasons, roughly in order of how much they mattered.

Cost. At this volume, an embedding API call per page is a real line item.

Reproducibility. Embedding models change under you. A vendor updates the model, the vectors shift, and a similarity threshold you tuned last month quietly means something different. A hand-built encoder is a pure function. Same input, same 256 numbers, every time, so when a match looks wrong I can reproduce it exactly.

The main reason is structural. An embedding blends everything (topic, category, region, phrasing) into one vector, and cosine similarity mixes all of it together. I needed some of that information to be a hard filter and the rest to be soft ranking: two items must share a category and a region, and only then get ranked by content. There's no clean way to say "these dimensions are a constraint, those are a score" inside a single learned embedding. So I built the vector to keep them separate.

The vector is two blocks

dim = 256

// SHAPE block, dims 0..63: a hard equality filter, never similarity math
catBase   = 0,   catSpan  = 16   // one-hot category
provBase  = 16,  provSpan = 38   // one-hot region
flagBase  = 58,  flagSpan = 6    // structural flags
shapeEnd  = 64

// CONTENT block, dims 64..255: this is what cosine actually measures
contentBase = 64, contentSpan = 192

The first 64 dimensions are the shape block: a one-hot category, a one-hot region, and a few structural flags. It's never used in a distance calculation. It's a filter key: "only compare me to items with the same category and region."

The remaining 192 dimensions are the content block: a feature-hashed fingerprint of the item's text, and the only part cosine similarity touches.

The rule I held to is that shape filters and content ranks, and the two never collapse into one score. Fusing them would put me right back to the embedding's single blended number, which is the thing I was trying to avoid.

Feature-hashing the content

The content block uses the hashing trick, which predates transformers by years. There's no learned vocabulary and no embedding table. You take features (here, character 3-grams of the normalized name, region tokens, and a category-specific bag of attributes), hash each into a bucket, and increment it.

The one detail worth knowing is the signed variant. If every feature only increments, colliding features always reinforce each other and collisions turn into fake signal. Using a second hash bit to pick a sign makes collisions tend to cancel instead:

// Feature hashing with the signed trick (Weinberger et al.).
// A second hash bit decides +1 or -1, so colliding features tend to
// cancel instead of always reinforcing. Collisions become noise, not bias.
func addFeature(vec []float32, token string, base, span int) {
    h := fnv32(token)
    bucket := base + int(h%uint32(span))
    if h&(1<<31) != 0 {
        vec[bucket] -= 1
    } else {
        vec[bucket] += 1
    }
}

A few limits keep it sane (at most 48 tokens per item, deduplicated, minimum 3 letters, stopwords removed), and then the whole vector is L2-normalized so pgvector's cosine operator behaves. That's the entire encoder: a synchronous function call per item, no batching, no network. The vector doesn't know what any word means; it only knows which buckets are set. For this job that's enough, and being deterministic and free is worth more than understanding the text.

Storage: pgvector, partitioned by category

The vectors live in Postgres with pgvector and HNSW. The one non-obvious decision is partitioning the table by category_id (16 partitions plus a default) with a separate local HNSW index on each.

This matters because of how HNSW interacts with filtering. HNSW is approximate: it walks a graph and returns roughly the nearest ef_search neighbors. If you then filter those by category_id = 3, and category 3 is 1% of a million rows, the approximate neighborhood can come back with zero matching rows. Empty results from a table that clearly contains matches, recall collapsing silently. Partitioning by the same key you filter on avoids it: a query for category 3 only touches category 3's index, a small graph where every node already passes the filter.

-- Shape block is a hard equality pre-filter (the "blocking" step).
-- Content block cosine distance (<=>) does the ranking.
-- Because the table is partitioned by category_id, this hits one small
-- per-partition HNSW graph, so recall stays high.
SELECT candidate_id, title, 1 - (vec <=> $1::vector) AS similarity
FROM vector_index
WHERE category_id = $2 AND province_id = $3
ORDER BY vec <=> $1::vector
LIMIT 8;

One bit of hygiene: when the pipeline rejects an item as a duplicate or as irrelevant, its vector is deleted in the same transaction as that decision, so the index only holds vectors worth matching against.

The gate is a funnel

The system decides, per page, whether to keep it and whether it's already covered. That's not one decision but four, arranged so each stage is more precise and more expensive than the last, and runs on only what the previous stage passed.

1. A lexical term list, at crawl time, no AI cost. A hand-seeded domain vocabulary, scored (a title hit is 2 points, a body hit 1) with a threshold below which the page never enters the expensive part of the pipeline. This runs on everything, so it has to be free.

2. A cheap, permissive LLM pre-gate. Extraction (turning a page into structured data) is about 95% of total AI spend, and the precise gate downstream rejects roughly 85% of what extraction produces. So a small, fast model runs first, capped at 150 output tokens, just to drop obvious junk before paying for full extraction. It fails open: on any error or unparseable reply it lets the page through, because a pre-gate that wrongly rejects loses data permanently, while one that wrongly accepts only wastes some money downstream.

3. The precise LLM gate. A deterministic category-range check first (still free), then one call to a stronger model with a hardened classification prompt. If the reply is ambiguous it isn't coerced into pass/fail; the job retries. A silent wrong verdict costs more than a second call.

4. The vector gate, where retrieval happens.

A reject at any stage skips everything below it, so the expensive judgment only runs on the small residue the cheap filters couldn't resolve.

Retrieval is a cascade

The vector gate isn't just "cosine search, top-k." It's a four-step cascade in priority order, and notably none of it is BM25 or full-text search:

1. Exact normalized-title lookup against the reference corpus. The cheapest match; if the title is identical, that's the answer.
2. Vector ANN search, the query above. Shape block filters, content block ranks, top 8.
3. Fuzzy title fallback, when 1 and 2 both come up empty: a trigram-Jaccard similarity written by hand in Go (I chose not to install pg_trgm), bounded by a SQL prefix-scan to at most 500 candidates so it never scans the whole table.
4. Net-new, nothing matched.

When there is a match, the decision combines three independent signals rather than one:

Vector cosine band (deterministic, no AI):
    similarity >= 0.90  ->  near-duplicate, skip
    0.60 .. 0.90        ->  probable match, needs a judgment call
    similarity <  0.60  ->  net-new

+ title trigram similarity     (deterministic, Go)
+ LLM-judged content coverage  (how much of this does the match already cover?)

The cosine band is a fast deterministic first cut. Only the 0.60–0.90 middle, "probably the same thing, not sure," spends an LLM call, to judge how much of the new item the existing one already covers. A final cross-validation call acts as a trust gate, and publishing requires everything to agree:

func shouldPublish(d Decision) bool {
    if d.CulturalGate != "pass" || d.CrossValidation != "accept" {
        return false
    }
    return d.NoveltyVerdict == "net-new" ||
        d.NoveltyVerdict == "enrich" ||
        d.NoveltyVerdict == "supplement"
}

A near-duplicate at 0.90 or above never costs a token. The deterministic band settles it, and the LLM is spent only on the ambiguous middle.

Re-deciding when the classifier changes

There's a failure mode specific to LLM gates. When you fix the classification prompt (you found a class of false positives and hardened against them) everything already published under the old prompt is now suspect, and the fix doesn't reach backward.

So there's a batch re-gate that re-runs the current gate against every published item. One constraint shaped its design: raw page bodies are only retained for 7 days, so for older items the source HTML is gone and the re-gate runs against the stored extracted candidate instead. It's dry-run by default and needs an explicit -execute -confirm=<token> to change anything, since it can un-publish. And because the downstream store has no delete API, it doesn't pretend to; it prints a purge list for a human. Building a filter isn't only deciding well now, it's being able to re-decide what you got wrong earlier.

Why the dumb vector was the right tool

The vector at the center of this has no model behind it and no notion of meaning. That's the point. It's free, so it runs on every item; deterministic, so a threshold means the same thing over time; and fast enough that its only job, narrowing a million rows to eight, leaves the expensive LLM looking at just those eight. An embedding would have been smarter, and also slower, costlier, non-reproducible, and unable to keep the shape/content split the whole system depends on. For this problem the dumb vector was the better tool.