Vector Databases for RAG 
Choosing and Scaling Your Embedding Store 

Most teams building retrieval-augmented generation discover the hard way that the language model is rarely the thing that's failing — the retrieval is. If the wrong chunks come back, even the best model answers confidently from bad context, and no amount of prompt engineering fixes it. The vector database is the component that decides which chunks come back, which makes it one of the highest-leverage and most underestimated choices in the whole stack. Getting it right is less about picking a trendy product and more about understanding what vector search can and can't do, and matching the store and the retrieval pipeline to the accuracy and scale you actually need.

What a vector database actually does

An embedding model turns a chunk of text into a vector — a list of numbers that positions that text in a high-dimensional space where semantically similar things sit close together. A vector database stores those vectors and answers one core question fast: given a query vector, which stored vectors are nearest? At small scale you could compare against every stored vector exactly, but that doesn't hold up, so vector databases use approximate nearest neighbor (ANN) indexes — most commonly HNSW — that trade a sliver of recall for enormous speed. The practical consequence is that vector search is fuzzy by nature: it finds what's semantically close, not what's exactly right, and it has no idea whether the closest match is actually relevant. Understanding that limitation is what separates a RAG system that works from one that confidently retrieves plausible-but-wrong context.

pgvector or a dedicated vector database?

The first real decision is whether to add vectors to the database you already run or adopt a dedicated one. For most teams starting out, `pgvector` on your existing PostgreSQL is the right answer and a genuinely under-appreciated one: it keeps your vectors next to your relational data, lets you filter by tenant, permissions, and metadata in the same query, and spares you an entire extra system to operate and secure. You outgrow it when you're into the tens of millions of vectors, need very high query throughput, or want vector-native features like sophisticated hybrid search and horizontal sharding — that's when a dedicated store like Qdrant, Weaviate, or a managed service like Pinecone earns its keep. The honest guidance is to start with pgvector and move only when a concrete limit forces you, because the dedicated systems add real operational and cost weight that a smaller workload doesn't need. Whichever you choose, the ability to filter by metadata at query time — especially tenant isolation in a multi-tenant product — matters more than raw benchmark numbers.

Chunking is where retrieval quality is won or lost

The single biggest lever on RAG accuracy isn't the database or the model — it's how you split your source documents into chunks before you embed them. Chunk too large and each vector blurs several topics together so nothing matches cleanly; chunk too small and you shred the context the model needs to actually answer. There's no universal setting: the right approach respects the structure of your content, splitting on semantic boundaries like sections and paragraphs rather than a blind fixed character count, often with a little overlap so ideas that straddle a boundary aren't cut in half. Enriching each chunk with metadata — the document title, section heading, source, and date — pays off twice, once for filtering and once because that context can be handed to the model alongside the chunk. Teams that treat chunking as a throwaway preprocessing step and teams that treat it as a tuned part of the system get very different accuracy out of the identical database and model.

Pure vector search isn't enough: hybrid and reranking

Vector search alone reliably misses a specific class of query — exact names, error codes, SKUs, and rare keywords — because semantic similarity isn't the same as lexical match. The fix that most serious RAG systems converge on is hybrid search: run the vector search and a traditional keyword (BM25) search together and combine the results, so you catch both the semantically related and the exactly-worded. On top of that, a reranking step takes the top candidates from retrieval and uses a more expensive cross-encoder model to reorder them by true relevance to the query, which meaningfully lifts the quality of what actually reaches the model's context window. The pattern to internalize is retrieve broadly, then rerank precisely: cast a wide, cheap net first, then spend compute narrowing it to the few chunks that matter. This two-stage shape is usually the difference between a demo that impresses and a system that holds up on real, messy queries.

Scaling, cost, and keeping the index fresh

Once a RAG system is in production, the ongoing concerns are freshness, cost, and evaluation rather than raw search speed. Your source data changes, so you need a real ingestion pipeline that re-embeds and updates vectors as documents are added, edited, and deleted — a stale index quietly degrades answers in a way no one notices until a customer does. Embeddings and storage aren't free, and re-embedding your whole corpus every time you want to try a new embedding model is a genuine cost and migration event, so version your embeddings and plan for that. Most importantly, you can't improve what you don't measure: build an evaluation set of real questions with known-good answers and track retrieval quality as you change chunking, models, and search strategy, because otherwise every 'improvement' is a guess. And keep an eye on the bill — retrieval, embedding, and generation costs all stack, and the same cost discipline that applies to LLM calls applies to the vector layer feeding them.

How Infiniti Tech Partners builds RAG retrieval

We treat retrieval as the part of RAG that actually determines accuracy, so that's where we spend the engineering. We start most teams on pgvector alongside their existing Postgres — metadata filtering, tenant isolation, and one fewer system to run — and move to a dedicated vector store only when a real scale limit demands it. We tune chunking to the structure of your content, add hybrid search and reranking so exact terms and semantic matches both surface, and stand up an evaluation set so retrieval quality is measured, not assumed. The outcome is a RAG system that returns the right context reliably — grounded, fresh, and cost-aware — instead of one that answers confidently from whatever happened to be nearest in vector space.