What we learned from comparing text RAG, managed multimodal search, and multimodal embeddings on the same corpus

Over the past few weeks, many people in our community have shared the challenges they are running into while building multimodal search for their products.

Their search systems work reasonably well for text, but struggle when important information appears inside tables, charts, diagrams, scanned pages, images, or multi-format data.

So, how should we approach search when the information is not only text?

We wanted to explore that question practically.

Rather than looking at multimodal search only at the model level, we built three complete retrieval pipelines and tested them on the same corpus and the same set of queries:

1. Text-only RAG using OpenAI embeddings and LanceDB
2. Managed multimodal search using Mixedbread
3. Multimodal retrieval using Voyage embeddings, metadata, and LanceDB

Our goal was to understand what each approach makes easier, where it struggles, and what tradeoffs a team should consider before choosing one. We were not trying to find one universally best solution. There probably is not one.

What we will cover

We evaluated the three pipelines using the same small healthcare corpus, manually verified relevant results, and retrieval metrics.

In this article, we will cover:

• some details about the data we used
• how we created the ground-truth evaluation set
• which metrics we chose and why
• three different retrieval pipeline approaches
• how each retrieval pipeline worked
• what the results revealed