Choosing a Vector Database for Self-Hosted RAG: What Actually Decides It

The decision that gets over-thought

Every RAG project reaches a week where someone builds a comparison spreadsheet of vector databases. It is a comfortable task — vendors publish numbers, the columns line up, and it feels like progress. It is also, in my experience, rarely the decision that determines whether the system works.

I have watched teams migrate between vector databases twice and land at the same mediocre answer quality both times, because the actual problem was that they chunked documents at a fixed 512 tokens with no regard for structure, and never reranked. The engine was retrieving faithfully. It was retrieving the wrong things faithfully.

So this article is deliberately not a benchmark. It is the three questions that genuinely constrain the choice — index size, filtering, and who operates it — and what the realistic options look like against each.

Start with the memory math

Before comparing features, work out how much index you are actually going to hold. The arithmetic is simple and it eliminates most of the debate.

A float32 vector costs dimensions × 4 bytes. An HNSW graph adds edges per node, roughly M × 2 × 4 bytes per vector at the graph’s base layer plus a smaller amount for upper layers, where M is the connectivity parameter (commonly 16). Call the graph overhead 30-50% of the vector bytes at typical settings.

1,000,000 chunks × 768 dims × 4 bytes  ≈  3.0 GB  (vectors)
                       + HNSW graph     ≈  1.0 GB
                       + payload/metadata ≈ varies, often 0.5-2 GB
                                          ─────────
                                          ~5 GB working set

Then double it, because you will want a replica, and add headroom for indexing and query buffers.

Two conclusions fall out immediately. First, a corpus of a few hundred thousand chunks fits comfortably in the RAM of a machine you are probably already paying for, which means distributed anything is premature. Second, the dimension count of your embedding model is a cost decision, not just a quality one — moving from 1536 to 768 dimensions halves your index, and on many corpora the retrieval difference is small enough to disappear behind a reranker.

Count chunks, not documents. Twelve thousand documents chunked into six passages each is seventy-two thousand vectors, which is nothing. That is roughly the shape of the RAG assistant over 12,000 internal docs, and at that size the interesting engineering was never the store.

Metadata filtering is what eliminates options

The second question kills more candidates than performance does: how selective are your filters, and how often do you use them?

Real RAG systems almost never search the whole corpus. They search “documents this user’s team can see, from the last two years, excluding drafts”. That is a filtered approximate nearest-neighbour search, and it is where naive implementations fall apart.

The failure mode is worth understanding because it explains most “our RAG returns nothing” tickets. If the engine runs the vector search first and applies the filter afterwards, a highly selective filter can eliminate every one of the top candidates, and you get an empty or terrible result set even though matching documents exist. If it applies the filter first and then scans exhaustively, you get correct results and unacceptable latency.

Engines handle this differently, and it is the single most important axis to evaluate:

• Qdrant integrates payload filters into HNSW traversal itself, with payload indexes on the filtered fields. This is its strongest argument.
• pgvector leans on Postgres: your filter is a normal WHERE clause with normal B-tree indexes, and recent versions added iterative index scans so a selective filter degrades gracefully rather than returning too few rows. It is genuinely good, and it is regular SQL, which matters more than it sounds.
• Weaviate and Milvus both support filtered search with their own pre-filtering strategies and index-level tuning.

Write your three nastiest real filters down before you evaluate anything, and test with those. Not with an unfiltered query over a public dataset.

Hybrid search fixes more than a bigger model does

Pure vector search is bad at exact tokens. Product SKUs, error codes, version numbers, internal acronyms, surnames — the things enterprise users actually search for — are precisely where embeddings blur and BM25 is exact.

Every serious RAG system I have worked on ended up running both and fusing the results, usually with reciprocal rank fusion:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Reciprocal rank fusion over several ranked ID lists."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

candidates = rrf([bm25_search(query, limit=50), vector_search(query, limit=50)])
final = cross_encoder_rerank(query, candidates[:30])[:5]

So the relevant question about an engine is not whether it does hybrid search, but whether it does it for you or whether you are wiring two systems together. Weaviate and Elasticsearch/OpenSearch have it built in. Qdrant supports sparse vectors alongside dense ones, which gets you there. With pgvector you combine tsvector full-text search and vector search in one SQL query, which is either elegant or fiddly depending on your tolerance for SQL.

That reranking line at the end matters more than the engine choice. A cross-encoder over the top 30 candidates is the cheapest large accuracy win available in RAG, and it is the same lever that took the support bot in the hallucination and evals engagement from 72% to 96% answer accuracy — alongside an eval harness that proved which changes helped.

The realistic shortlist

Engine	Best when	Operational weight	Watch out for
pgvector	You already run Postgres, under a few million chunks, SQL-expressible filters	Lowest — it is a Postgres extension	Sharing an instance with heavy OLTP; very large indexes; dimension limits on indexed types
Qdrant	Filtering is central, you want quantization and a purpose-built store	Moderate — one Rust service, simple to run	One more stateful service to back up and upgrade
Weaviate	You want hybrid search and multi-tenancy out of the box	Moderate	Module surface is large; upgrade paths need attention
Milvus	Hundreds of millions of vectors, genuinely distributed	Highest — etcd, object storage, message queue in cluster mode	Do not adopt cluster mode for a corpus that fits on one node
Elasticsearch / OpenSearch	You already run it for logs or search, hybrid matters most	Moderate to high (JVM)	Memory appetite; kNN tuning is its own discipline
Embedded (LanceDB, local index files)	Tens of thousands of chunks, single writer, batch rebuilds	Near zero	Concurrency and horizontal scaling are your problem

Notice that the column doing the most work is “operational weight”. For a self-hosted system, that is the cost that recurs.

When “just use Postgres” is right, and when it is not

The strongest argument for pgvector has nothing to do with vectors. It is that your embeddings live in the same transaction as your documents. When a document is deleted, its chunks go with it, atomically. When you need to join retrieval results against permissions, ownership, or a tenant table, it is a join. Backups, point-in-time recovery, monitoring, connection pooling, and the person who knows how to operate it all already exist.

Reach for a dedicated engine when one of these is true:

• Your index no longer fits comfortably in RAM alongside your transactional working set, and you are not willing to run a separate Postgres instance for it.
• Query concurrency is high enough that vector search starts competing with your application’s normal load. (Running a dedicated read replica for vector queries buys you a lot of runway here before you migrate.)
• You need aggressive quantization to fit the index in a sane amount of memory.
• Your filtering is graph-like or multi-tenant in a way SQL makes ugly.

Notice that “we might grow” is not on that list. Migrating a vector index is a re-embedding job and a backfill, not a rewrite. It is one of the more reversible decisions in the stack, which is a good reason to make the cheap choice first.

What quantization actually buys

Quantization is the lever that changes the memory math, and it is worth understanding before you size hardware.

Scalar quantization stores each dimension as an int8 instead of a float32 — a 4× reduction in vector memory, with recall loss that is usually small and measurable. This is the default I reach for on anything memory-constrained.

Binary quantization reduces each dimension to a single bit, which is a dramatic reduction, but only works well when paired with a rescoring pass: retrieve a wider candidate set using the binary vectors, then re-score those candidates against full-precision vectors held on disk. Works well for higher-dimensional embeddings, less well for compact ones.

Product quantization sits in between with more tuning knobs and more ways to get it wrong.

pgvector’s halfvec type is the low-effort version of this idea — float16 storage, half the memory, and it also raises the dimension ceiling for indexed vectors. It is often the first thing to try before adopting a new engine for memory reasons.

Whatever you choose, measure recall against a held-out query set before and after. Quantization is one of the few RAG changes where the quality cost is real, bounded, and easy to quantify, so quantify it.

The costs that are not the database

Teams optimise the store and then discover their spend is somewhere else entirely. In self-hosted RAG the recurring costs usually rank like this:

1. Inference. GPU hours for the generation model, and for a self-hosted embedding or reranking model. This dominates almost everything else, which is why self-hosted RAG in production spends more time on serving than on storage.
2. Re-embedding. Every time you change embedding models or chunking, you re-embed the whole corpus. Budget for this happening more than once — it always does — and keep the raw text so you never have to re-extract from source systems.
3. Ingestion. Parsing PDFs, OCR, deduplication, and the pipeline that keeps the index fresh. Index staleness is the quiet killer of RAG trust: users stop believing a system that confidently cites last quarter’s policy.
4. The vector store itself. Memory and disk, which the arithmetic above already told you.

If cost is the pressure you are under, the order of that list is the order to attack it in. Storage is last.

A decision procedure

1. Count chunks, not documents, and multiply by your embedding dimensions to get the raw index size. Under a few hundred thousand chunks, stop over-thinking.
2. Write down your three hardest real filters. If they are SQL, that is a strong pull toward pgvector. If they are selective and constant, that is a pull toward Qdrant.
3. Decide whether you need hybrid search. If your users search for codes, IDs, or acronyms, you do, and it should influence the choice.
4. Ask who operates this at 3am. If the answer is “the same two people who run everything else”, weight operational simplicity heavily.
5. Build the eval set before you pick. Fifty real questions with known-good answers, so that “did this help?” has an answer that is not a vibe.
6. Pick the simplest option that clears 1-4, ship it, and let measured pain drive any migration. Re-embedding into a different store is a weekend, not a quarter.

The teams that end up happy with their RAG systems are not the ones that picked the best vector database. They are the ones that had an evaluation harness early enough to know which changes were working.

If you are building this and want the architecture reviewed before you commit to it, that is RAG systems and AI chatbot work I do, and the serving and GPU side of it sits under MLOps and model deployment.