26 Jul 2024 AI Engineering 5 min read

ColPali: Visual Document Retrieval Without OCR

ColPali (June 2024) replaces the OCR parsing pipeline with direct embedding of document page images. We explain the late-interaction mechanism inherited from ColBERT, the ViDoRe benchmark results — 81.3 versus 66.1 nDCG@5 — and the storage and scaling trade-offs, and assess where visual retrieval fits in production RAG systems as of July 2024.

Where text-only RAG silently fails

A retrieval-augmented generation system is only as good as its retriever, and most production retrievers operate on extracted text. A PDF is parsed, OCR is applied, the text is chunked and embedded. Everything the parser cannot express as text is gone before the first query arrives. Tables lose their alignment, stamps and signatures vanish, technical drawings become empty strings. The failure is silent: the system still returns plausible passages, and the user never learns that the decisive page was never indexed.

This matters most in the corpora that matter most. Invoices, scanned contracts, engineering drawings, regulatory filings and slide decks carry a large share of their information in layout, figures and visual marks. In our project work at Blue IT Systems we regularly see document sets where the pages a domain expert would pick are exactly the pages a text pipeline cannot represent. Recall failures of this kind rarely show up in evaluation sets built from clean text.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What a parsing pipeline actually costs

A modern ingestion pipeline is a chain of models: layout detection, OCR, table reconstruction, chunking, and optionally captioning of figures with a vision-language model. Each stage has its own error modes, and errors compound. The ColPali authors measure a representative pipeline built on the Unstructured tool — layout detection, OCR, captioning, embedding — at about 7.22 seconds per page on an NVIDIA L4 GPU.

The deeper problem is not speed but information loss. In the common text-only configuration, figures, images and tables are treated as noise and filtered out. The pipeline does not fail loudly on a construction drawing or an approval stamp. It simply indexes nothing.

ColPali embeds the page as an image

ColPali (Faysse et al., arXiv:2407.01449, first version 27 June 2024) removes the pipeline. Each document page is rendered as an image and fed to a vision-language model — PaliGemma-3B, released by Google in May 2024. The SigLIP vision encoder splits the page into 1,024 patches, the patch embeddings pass through the Gemma-2B language model, and a projection layer maps every output token to 128 dimensions.

The result is a multi-vector representation: roughly one thousand contextualized 128-dimensional vectors per page. No OCR, no layout detection, no chunking, no captioning. Indexing takes 0.39 seconds per page on the same L4 GPU — about eighteen times faster than the parsing pipeline, in a single forward pass with fixed sequence length.

The enabling observation is subtle: after multimodal fine-tuning, a VLM's image-token embeddings live in the same latent space as its text-token embeddings. A textual query can therefore be matched directly against visual patches. The authors' contrastive fine-tuning makes that alignment sharp enough for retrieval.

Late interaction does the matching

Scoring follows ColBERT (Khattab and Zaharia, 2020). A query is embedded into one vector per token. For each query token, the late-interaction operator takes the maximum dot product over all patch vectors of a page and sums these maxima into the page score. The rich token-to-patch interaction happens at query time; the expensive page encoding happens offline.

The cost is modest. Query encoding with ColPali's language model takes about 30 ms, versus 22 ms for a BGE-M3 text encoder; the late-interaction scoring adds roughly 1 ms per 1,000 pages in the corpus. The operator is fully differentiable, so the whole retriever is trained end to end — the released model was fine-tuned with LoRA adapters on about 100,000 query-page pairs.

ViDoRe puts numbers on the gap

Because existing benchmarks evaluate text embeddings on clean text, the authors built ViDoRe, the Visual Document Retrieval Benchmark: ten page-level retrieval tasks in English and French, spanning academic datasets (DocVQA, InfoVQA, TAT-DQA, arXivQA, TabFQuAD) and practical domains such as energy, government, healthcare and AI. The main metric is nDCG@5. The benchmark scores three practical requirements together — retrieval quality, query latency and indexing throughput — a framing we consider more honest than embedding-only leaderboards.

The gap is widest where text pipelines discard the most: infographics (InfographicVQA), figures (arXivQA) and tables (TabFQuAD). Notably, ColPali also retrieves text-centric documents better than the text baselines across all evaluated domains and languages — the advantage does not rest on visual edge cases alone.

System	ViDoRe average nDCG@5	Indexing per page (NVIDIA L4)
Unstructured + OCR + BM25	65.5	≈7.22 s (full pipeline)
Unstructured + OCR + BGE-M3	66.1	≈7.22 s (full pipeline)
ColPali (late interaction)	81.3	0.39 s

What ColPali does not solve

The multi-vector representation has a price. Storing 1,024 vectors of 128 dimensions at 16-bit precision costs roughly 260 KB per page; a million-page corpus needs on the order of 260 GB, against a few kilobytes per chunk for single-vector embeddings. Naive late-interaction scoring scans every page; scaling to millions of documents requires optimized engines such as PLAID, and most vector databases offer no native multi-vector support today.

ColPali is a retriever, not a reader. It returns page images; answering still requires a vision-language model downstream, and page-level granularity means no sub-page citations out of the box. The 3B-parameter encoder needs a GPU at indexing and query time. Training data is dominated by English, with French covered; other languages are unverified. And the paper is four weeks old — independent replication is still pending.

Outlook from July 2024

We expect the pattern to outlive the checkpoint. The backbone is replaceable, and better small VLMs will lift retrieval quality without changing the architecture. The obvious engineering targets are storage — pooling and compressing patch vectors — and native multi-vector support in vector databases. The public ViDoRe leaderboard will make progress measurable.

The larger bet is the authors' own: end-to-end RAG in vision space, where a VLM answers directly from retrieved page images and extracted text never exists at all. We consider that plausible for archives where parsing already fails today — drawings, stamps, scans. OCR pipelines will not disappear; where the extracted text itself is the deliverable, they remain mandatory. But for retrieval, treating the page as an image has just become a serious option, and we are evaluating it.