28 Aug 2023 AI Engineering 5 min read

Vector Databases: When You Actually Need One

Vector databases raised over 160 million dollars in spring 2023 alone. We explain what approximate nearest neighbor search actually does, how the HNSW index works, and why Postgres with the newly released pgvector 0.5.0 is often sufficient — plus the concrete criteria that justify a dedicated system.

A New Database Category in Twelve Months

Since the release of ChatGPT in November 2022, retrieval-augmented generation has become the standard pattern for grounding large language models in private data. Every such pipeline stores embeddings — dense float vectors, typically 1,536 dimensions from OpenAI's text-embedding-ada-002 — and retrieves the nearest neighbors of a query vector at runtime. A market has formed around this single operation. In April 2023 alone, Pinecone raised 100 million dollars, Weaviate 50 million, and Chroma 18 million. Milvus, open-sourced in 2019, is already a graduated project of the LF AI & Data Foundation.

The question clients ask us first is which vector database to pick. In our view that is the second question. The first is whether a dedicated vector database is needed at all. This post explains what approximate nearest neighbor search does, how the dominant index structure works, what the current systems offer, and where plain PostgreSQL is the correct answer.

A document arrives — scan, mail or upload. 1/4

What Approximate Nearest Neighbor Search Does

Embedding models map text, images, or code to points in a high-dimensional space, where semantic similarity becomes geometric proximity. Retrieval then means k-nearest-neighbor search. The exact version compares the query against every stored vector: one million 1,536-dimensional float32 vectors occupy roughly 6 GB, and each query scans all of it. Cost grows linearly with the corpus. Tree structures that work in low dimensions degrade toward this brute-force scan as dimensionality rises — the curse of dimensionality.

ANN search relaxes exactness. It returns most of the true neighbors, measured as recall: the fraction of the true top-k results found. Production systems typically operate at 90 to 99 percent recall with millisecond query times. Equally important is what ANN does not do. It does not guarantee complete results. Recall can drop sharply when combined with restrictive metadata filters. And the distance metric — cosine, Euclidean, or inner product — must match the embedding model.

How HNSW Finds Neighbors Fast

Hierarchical Navigable Small World graphs, published by Malkov and Yashunin on arXiv in March 2016, are the index behind most vector databases today. HNSW builds a multi-layer proximity graph. Upper layers are sparse and span long distances; the bottom layer contains every vector. A query starts at the top, greedily walks toward the target, and descends layer by layer — the same idea as a skip list, applied to graphs. Search complexity scales logarithmically. In the ANN-Benchmarks comparisons HNSW has led the speed-recall frontier for years; implementations exist in hnswlib, Faiss, and Lucene.

The trade-offs are explicit. The graph lives in RAM; memory consumption grows with both vector count and link density. Index construction is slow compared to flat or IVF structures. Two parameter families govern everything: M, the number of links per node, and ef_construction and ef_search, the candidate list sizes — higher values buy recall at the price of memory, build time, and latency. Deletions require workarounds. HNSW is fast, not free.

The Landscape in August 2023

Five systems dominate the current discussion. Four are dedicated vector databases; one is a Postgres extension. Funding volume and GitHub stars measure market expectations, not fit for your workload. All five implement graph-based ANN — HNSW or a variant — so raw query performance differs less than the marketing suggests; the operational model and the ecosystem differ far more.

System	License and model	Status August 2023	Typical fit
Pinecone	Proprietary managed service	$100M Series B at $750M valuation (April 2023)	Zero-ops teams with cloud budgets
Weaviate	Open source (BSD-3) plus cloud	$50M Series B (April 2023); hybrid search built in	Hybrid keyword-vector search
Milvus	Open source (Apache 2.0)	Graduated LF AI & Data project; distributed architecture	Billion-scale distributed deployments
Chroma	Open source (Apache 2.0)	$18M seed (April 2023); embedded-first API	Prototypes and local development
pgvector	Open source (PostgreSQL license)	v0.5.0 with HNSW released 28 August 2023	Teams already running Postgres

When Postgres With pgvector Is Enough

pgvector 0.5.0 was released today, 28 August 2023, and adds an HNSW index type to PostgreSQL alongside the existing IVFFlat. That closes the main performance gap to dedicated systems for many workloads. The architectural argument is simple: embeddings live in a column next to the rows they describe. One system of record, real ACID transactions, joins between vectors and business data, metadata filtering with ordinary WHERE clauses, and your existing backup, replication, and monitoring stack.

The limits deserve equal clarity. Indexes are capped at 2,000 dimensions. Postgres scales vertically, not horizontally, for this workload. HNSW builds are slow and memory-hungry. There is no built-in embedding pipeline and no automatic hybrid ranking. For corpora up to the low millions of vectors at moderate query rates — which describes most enterprise RAG systems we see — none of this matters. If you already run Postgres, vector search arrives as a migration, not a project: one extension, one column, one index.

When a Dedicated Vector Database Pays Off

A second database is never just a library import. It is a deployment to operate and upgrade, a synchronization pipeline between the system of record and the search index, a consistency model to reason about when the two diverge, and a new failure mode during incidents. This cost is fixed and paid immediately; the benefit depends entirely on scale you may never reach.

Concrete thresholds where dedicated systems earn their keep: hundreds of millions of vectors and beyond, horizontal sharding requirements, sustained high query throughput with strict latency budgets, or heavily filtered search at large scale. If none of these apply today and none are on a measured growth curve, adding one is over-engineering. Start boring; specialize on evidence, not anticipation.

Where Vector Search Goes From Here

Our expectation from the vantage point of August 2023: vector search becomes a feature, not a product category. Elasticsearch added approximate kNN search in version 8.0 in February 2022, Redis followed with vector similarity in 2022, and MongoDB announced Atlas Vector Search in June 2023. Postgres joins that list today. General-purpose databases are absorbing the capability faster than dedicated vendors can differentiate.

We therefore expect consolidation among standalone vendors, hybrid keyword-plus-vector retrieval as the default, and embedding models changing faster than storage layers — making re-indexing pipelines a more durable engineering investment than index selection. Some of these predictions will age badly. One we would bet on: in five years, the question of which vector database to choose will sound like the question of which MapReduce framework sounds today.