NeoMME: H Company's Efficient Multimodal Encoder for Visual Document Retrieval

01

What NeoMME is

NeoMME is a family of multilingual, multimodal-native bidirectional encoders from H Company. Unlike many visual-language systems, it does not pair a separately pretrained vision tower with a causal language model. Text tokens and raw image patches travel through one shared Transformer encoder.

That makes NeoMME an encoder rather than a conversational generator. It produces contextual representations for downstream tasks such as retrieval, classification and extraction instead of autoregressively generating answers.

02

NeoMME 260M and 800M

H Company released 260M and 800M model families under Apache 2.0. The current 260M model card lists 263M parameters, a 131,072-token vocabulary, 16,384-token context, 32×32 image patches and default image support up to 2,048 pixels on the longest side.

The base checkpoints are foundation encoders, not finished retrieval applications. H Company's 260M model card explicitly says the raw hidden states require task-specific fine-tuning and directs document-search users to the NeoMME-Retriever checkpoints.

03

Why the single-tower architecture matters

NeoMME is pretrained from scratch with a masked discrete-diffusion text objective while image patches remain visible for multimodal examples. The goal is to learn shared visual and textual representations without carrying the parameter and compute overhead of a generative decoder that retrieval does not need.

For document systems, that architecture is most relevant when the task is ranking or representing pages rather than producing prose. It can preserve visual signals from tables, charts, forms, typography and layout that may be flattened by text-only extraction.

04

NeoMME-Retriever for visual document search

H Company fine-tuned both model sizes into NeoMME-Retriever checkpoints for multimodal document retrieval. The current Transformers variant returns dense and multi-vector embeddings in a single forward pass.

The model cards document cosine similarity for dense embeddings and MeanMaxSim scoring for multi-vector late interaction. That gives teams a compact first-stage representation and a more detailed matching representation from the same checkpoint.

05

Benchmarks and throughput

H Company reports ViDoRe v3 nDCG@10 of 0.5226 for NeoMME-Retriever 260M and 0.5560 for NeoMME-Retriever 800M using late interaction. These are vendor-authored benchmark results, not independent replication.

In H Company's matched throughput test at 2048×2048 input on one NVIDIA L40S, the 260M Retriever encodes about 51 pages per second versus about 26 pages per second for ColModernVBERT. Treat that as a benchmark configuration rather than a universal production speed or hardware requirement.

06

How NeoMME fits into RAG

NeoMME-Retriever can serve as the retrieval layer of a visual RAG pipeline: index page images, encode the user's query, retrieve the most relevant pages, then send those pages or extracted evidence to a separate LLM or VLM for the final answer.

07

Weights, Transformers support and licensing

The pretrained backbones and retrieval checkpoints are available on Hugging Face and integrated into Transformers. Current Retriever model cards use NeoMMEForRetrieval with the NeoMME processor interface for inference.

The released weights are Apache-2.0 licensed. That is permissive for many development and commercial scenarios, but teams should still verify licenses and terms for any datasets, fine-tunes, vector stores and downstream components they combine with the models.

08

Important limitations

• independent replication of the launch benchmarks is still limited
• the base checkpoints are not plug-and-play retrieval systems
• memory and throughput depend on checkpoint, resolution, batch size, precision and indexing strategy
• visual retrieval may add unnecessary complexity for clean text-heavy corpora

Benchmark both NeoMME sizes on the actual document collection. A compact model can be attractive for indexing economics, but production relevance depends on recall, ranking quality, latency, index size and final answer quality on your own workload.