NeoMME: Unpacking the Multimodal-Native and Multilingual Encoder

In the rapidly evolving world of artificial intelligence, models that can seamlessly understand and process information from various sources – like text and images – are becoming increasingly vital. These are known as multimodal models. While many existing solutions often stitch together separate components for different data types, a new innovation named NeoMME (pronounced "nee-oh-me") is changing the game. Released by H Company, NeoMME stands out as an efficient, multimodal-native, and multilingual encoder designed to simplify and optimize how AI handles complex, diverse data. This article dives deep into what NeoMME is, why its approach matters, how it works under the hood, and what its emergence means for AI practitioners and developers.

What is NeoMME?

NeoMME is a family of powerful AI encoders built to understand and represent both text and image data. What makes it "multimodal-native" is its core design: unlike many other systems that use separate modules for vision and language, NeoMME processes both types of input directly within a single, unified bidirectional Transformer. This means it doesn't rely on a "pretrained vision tower" or a "causal language model" that adds extra complexity and computational cost. Think of it like this: instead of having one specialist for reading text and another for seeing images, and then trying to get them to talk to each other, NeoMME is a single, highly capable entity that can do both simultaneously and inherently. It’s also "multilingual," meaning it's designed to handle text in many different languages, making it incredibly versatile for global applications. The H Company researchers, including Aurélien Lac and Tony Wu, officially released NeoMME on September 3, 2026, making it available as open-source under the Apache 2.0 license on Hugging Face. It comes in two sizes: a 260 million parameter version and an 800 million parameter version.

Why NeoMME Matters: Addressing Key Challenges in Multimodal AI

The development of NeoMME is a significant step forward because it tackles several limitations found in conventional multimodal AI architectures.

Overhead Reduction

Many current multimodal models, especially those adapted from generative vision-language models (VLMs), often carry a lot of "overhead." These models are typically designed for generative tasks, like creating text descriptions from images, and thus include components like causal decoders. However, for non-generative tasks such as retrieval, classification, or token labeling, these components are unnecessary. NeoMME's design specifically drops the causal decoder, which translates to a substantial reduction in computational resources and parameters. This makes it much more efficient for tasks where you just need to understand and encode information, not generate new content.

Unified Processing for Better Understanding

By processing images and text within a single Transformer encoder, NeoMME allows for a truly integrated understanding of multimodal information. This "single-tower" approach means that the model learns relationships between visual elements and text tokens more directly, without the potential for information loss or misalignment that can occur when features are passed between separate vision and language encoders. This inherent integration can lead to a richer, more nuanced comprehension of complex documents or media.

Efficiency for Real-World Applications

For AI practitioners and developers, efficiency directly translates to lower operational costs and faster performance. NeoMME’s optimized architecture means quicker processing times and reduced storage requirements, especially critical for large-scale applications like visual document retrieval. Imagine sifting through millions of documents that contain both text and images; NeoMME can do this faster and with less memory.

How NeoMME Works: A High-Level Look at its Architecture

NeoMME’s innovative approach stems from its unique architectural choices and training methodology.

One Transformer for All Modalities

At its heart, NeoMME uses a single, bidirectional Transformer encoder. This is a departure from the common practice of using separate vision and text encoders that then feed into a combined system. For text, it uses factorized token embeddings. For images, it divides them into a grid of non-overlapping 32x32 pixel patches, which are then projected using a small Multi-Layer Perceptron (MLP). Both these text embeddings and image patch projections are fed into the same Transformer encoder. This shared computational path means that images and text are treated as native inputs, fostering a deeper, unified learning process.

Training from Scratch with a Masked Discrete-Diffusion Objective

Instead of relying on existing pretrained vision or text models, NeoMME is trained from scratch. It uses a "masked discrete-diffusion objective," which is a sophisticated way for the model to learn by trying to reconstruct masked (hidden) parts of the input, conditioned on the visible parts. For multimodal examples, the image patches are left visible while the model reconstructs masked text, forcing it to learn image-grounded descriptions rather than just relying on language-only cues. This rigorous pretraining process, involving about 524 billion packed input tokens across various data types (multilingual text, code, mathematics, natural images, and document images), is key to its robust performance.

Dynamic Resolution and Long Context

NeoMME is designed to handle images with dynamic resolutions, meaning images keep their original aspect ratio and size. This is important because it allows the model to dedicate more processing "tokens" to high-resolution, information-dense documents, and fewer to simpler images. Furthermore, both NeoMME models (260M and 800M) boast a substantial context length of 16,384 tokens, which is enough to process up to two standard 4K UHD images simultaneously. This long context window is crucial for understanding large, complex documents or scenarios where a broad view of information is needed.

Modern Encoder Enhancements

The architecture also incorporates several modern advancements in encoder design, such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. These technical improvements contribute to the model's overall efficiency and performance.

NeoMME-Retriever: Specialized for Visual Document Retrieval

To showcase its practical capabilities, H Company fine-tuned NeoMME for visual document retrieval, creating "NeoMME-Retriever." This specialized version uses a "page-image" approach, which means it ranks screenshots of document pages rather than just extracted text chunks. This method is powerful because it bypasses the need for Optical Character Recognition (OCR) preprocessing and, crucially, preserves the original layout, charts, tables, and typography of the document. NeoMME-Retriever is equipped with two jointly trained "heads" that produce different types of embeddings in a single forward pass:

• Dense Head: This generates a compact, normalized vector by mean-pooling hidden states. These "dense embeddings" are excellent for approximate nearest neighbor (ANN) search, which is fast and efficient for initial retrieval.
• Late-Interaction Head: This projects each text token or image patch into a 128-dimensional normalized vector. These "late-interaction embeddings" are more granular and preserve fine-grained matches between query tokens and specific regions of an image, leading to more precise results.

This dual-embedding capability offers immense flexibility for various retrieval scenarios and infrastructure needs.

Performance and Efficiency Benchmarks

NeoMME has demonstrated impressive performance, particularly for visual document retrieval:

• ViDoRe v3 Benchmark: On the ViDoRe v3 benchmark, NeoMME-Retriever-260M achieved an nDCG@10 score of 0.523, making it the highest-scoring model among those strictly below 800 million parameters. The 800M model reached 0.556 nDCG@10.
• Throughput: The 260M model encodes approximately 51 pages per second on an NVIDIA L40S GPU with 2048x2048 image input. This is nearly twice the throughput of ColModernVBERT, meaning it can process documents much faster.
• Storage Efficiency: One of NeoMME's standout features is its incredible storage efficiency for late-interaction embeddings. Using hierarchical token pooling and asymmetric quantization, it reduces index storage from roughly 1.5 MB to a mere 6 kB per page. This is a 255x reduction in storage while still maintaining over 95% of the baseline retrieval performance.
• Text Retrieval: For text-only retrieval on the BEIR-15 benchmark, the 260M model scored 0.4881 nDCG@10 with late interaction, and the 800M model achieved 0.5126.

These benchmarks highlight NeoMME's ability to offer competitive retrieval performance while being significantly more efficient in terms of speed and storage than many existing models.

What NeoMME Means for AI Practitioners and Freelancers

The arrival of NeoMME has several implications for anyone working with AI, from researchers and developers to freelancers leveraging AI tools:

• Democratization of Multimodal AI: By providing an efficient, open-source foundation encoder, NeoMME makes advanced multimodal capabilities more accessible. Developers can integrate these powerful models into their applications without needing to build complex, multi-component systems from scratch.
• Cost-Effective Solutions: The improved throughput and drastic reduction in storage requirements mean lower operational costs. For businesses or freelancers running AI services, this translates to more economical solutions for tasks like document search, content moderation, or visual knowledge bases.
• Enhanced Document Intelligence: For tasks involving document processing, NeoMME's ability to natively handle page images and preserve layout is a game-changer. This is particularly useful for legal tech, medical records, research archives, or any field dealing with visually rich documents where text alone doesn't tell the whole story. Freelancers in data analysis or content management can build more robust and accurate solutions.
• Faster Development Cycles: With a unified architecture and ready-to-use checkpoints on Hugging Face, developers can fine-tune NeoMME for specific tasks more quickly, accelerating the development and deployment of multimodal AI applications.
• New Possibilities for Multilingual Applications: Its multilingual capabilities open doors for global applications, allowing businesses to cater to a diverse user base without needing separate models for each language.

Accessing NeoMME

NeoMME is openly available, making it easy for developers and researchers to experiment with and integrate into their projects.

• Hugging Face Transformers: The models and fine-tuned retrieval variants are available on the Hugging Face Transformers library. This provides a standard and accessible platform for using NeoMME.
• Official Collection: You can find the NeoMME models and related resources in the official Hugging Face collection: Hugging Face NeoMME Collection.
• License: All model checkpoints are released under the Apache 2.0 license, allowing for broad use and modification.
• Visual RAG Demo: A visual Retrieval-Augmented Generation (RAG) demo is also available as a Hugging Face Space, providing a practical example of NeoMME's capabilities.

Conclusion

NeoMME represents a significant leap in efficient multimodal and multilingual AI encoding. By adopting a "multimodal-native" approach with a single bidirectional Transformer, H Company has delivered a powerful, open-source solution that avoids the computational overhead of traditional VLM-based systems. Its impressive performance in visual document retrieval, coupled with its remarkable efficiency in throughput and storage, positions NeoMME as a critical tool for AI practitioners. As AI continues to integrate more deeply into our daily lives and work, tools like NeoMME will be instrumental in building more intelligent, cost-effective, and versatile applications across various industries.

Frequently Asked Questions

What is the main advantage of NeoMME's "multimodal-native" design?

The main advantage is that NeoMME processes both text and raw image patches within a single, unified bidirectional Transformer. This eliminates the need for separate, pretrained vision and language components, reducing computational overhead and leading to a more integrated understanding of multimodal information.

Who developed NeoMME and when was it released?

NeoMME was developed by H Company researchers, including Aurélien Lac and Tony Wu. It was officially released on September 3, 2026.

How does NeoMME improve efficiency for tasks like visual document retrieval?

NeoMME improves efficiency in several ways: it offers significantly faster document encoding throughput (up to 2x faster than comparable models) and drastically reduces the storage required for late-interaction embeddings (by 255x) while maintaining high retrieval accuracy. This translates to lower computational costs and quicker processing for large datasets.

Is NeoMME an open-source project?

Yes, NeoMME is an open-source project. All model checkpoints are released under the Apache 2.0 license and are available on Hugging Face Transformers.