How to Use NeoMME-Retriever for Visual Document RAG

01

Choose a Retriever checkpoint

Use a task-specific NeoMME-*-Retriever checkpoint for document search rather than the raw NeoMME backbone. H Company currently publishes 260M and 800M Retriever variants.

The default Transformers Retriever returns dense and multi-vector embeddings together. The 260M model is a practical efficiency-first starting point, but the correct choice should come from benchmarking both checkpoints on your own corpus.

02

Prepare document pages and metadata

Visual document retrieval normally turns each PDF page into an image before indexing. Keep a stable mapping from each page image to its document ID, page number, source URL or filename, permissions and any useful section metadata.

That metadata remains your application's responsibility. The retriever ranks relevance; it does not automatically enforce access rules, preserve citations or recover the original source document.

03

Load NeoMME-Retriever with the current interface

The current H Company model cards load the Retriever with NeoMMEForRetrieval and a NeoMME/Auto processor. The 800M card, for example, shows AutoProcessor.from_pretrained() and NeoMMEForRetrieval.from_pretrained(..., device_map="auto").

H Company's full 260M example uses the processor's chat-template helper with an explicit query or document task and then runs the same retrieval model over the resulting batches. Recheck the live model card before production because library interfaces can evolve.

04

Use dense and multi-vector representations correctly

NeoMME-Retriever exposes a dense embedding and a multi-vector embedding in one forward pass. H Company documents cosine similarity for the dense vectors and MeanMaxSim for late-interaction scoring.

A scalable architecture can use dense vectors for first-stage approximate-nearest-neighbor retrieval, then rerank a smaller candidate set with late interaction. That pattern is an implementation strategy, not a mandatory NeoMME requirement.

05

Index page images

• render each source page to an image at a consistent policy
• encode the document with the Retriever checkpoint
• store the dense vector and, when needed, the multi-vector representation
• attach document and page metadata to the stored record
• retain the original page image or a durable reference to it

NeoMME does not prescribe a single vector database. Choose the index based on corpus size, filtering needs, latency, cost and whether your stack can support the representation used for reranking.

06

Encode the query and retrieve candidates

At search time, encode the user's text query with the corresponding query task of the same compatible Retriever checkpoint. Do not mix document embeddings from one NeoMME variant with query embeddings from another unless you have explicitly validated compatibility.

Retrieve the nearest dense vectors, optionally rerank the shortlist with MeanMaxSim over the multi-vector embeddings, and return the top pages with their stored source metadata.

07

Add a separate answering model

NeoMME-Retriever is not the generator. Pass the selected page images, OCR text or extracted evidence to a separate LLM or VLM and instruct that model to stay grounded in the retrieved sources.

For visually complex pages, a VLM that can inspect the original page image may preserve chart and layout evidence better than flattening everything back into plain text before generation.

08

Evaluate on your own document corpus

• recall@k and ranking quality on known-answer queries
• indexing throughput and query latency
• dense and multi-vector index size
• cost at the target document volume
• final answer accuracy after the generator step

H Company's ViDoRe and L40S numbers describe its benchmark setup. They do not establish a universal GPU minimum, batch size, shortlist depth or latency target for your application.

09

When visual retrieval is worth the complexity

NeoMME-Retriever is most compelling when page layout, tables, charts, forms, diagrams or typography carry meaning that OCR-first chunking can lose.

For clean digital text, a conventional text-extraction and embedding pipeline may remain cheaper and easier to inspect. Hybrid systems can also use visual retrieval for page relevance and OCR/text extraction for downstream evidence handling.