Hot French startup ZML releases free product to speed inference across lots of AI chips

A new player has emerged in the artificial intelligence landscape, promising to make AI inference more accessible and cost-effective. French startup ZML has officially released its ZML/LLMD software, a free product designed to accelerate AI inference across a wide array of AI chips. This development is particularly noteworthy, earning an endorsement from Turing Award winner Yann LeCun, and signals a significant step towards democratizing access to high-performance AI deployment.

What is ZML/LLMD and Why Does It Matter?

ZML is a Paris-based AI startup, founded in 2023 by Steeve Morin, with a clear mission: to build a high-performance AI inference stack for production environments. Their latest offering, ZML/LLMD, is a specialized inference server for Large Language Models (LLMs). The core problem ZML/LLMD addresses is the often prohibitive cost and hardware dependency associated with running complex AI models, especially at scale.

Traditionally, AI inference has been heavily reliant on specific hardware, primarily NVIDIA GPUs, and Python-based frameworks. This creates a vendor lock-in situation and can lead to significant operational expenses. ZML/LLMD aims to break this cycle by providing a hardware-agnostic solution that can compile and run AI models efficiently on various accelerators, thereby reducing the need for specialized, expensive infrastructure and lowering overall operational costs.

The "Model to Metal" Philosophy

ZML's approach can be summarized by its tagline: "Model to Metal." This philosophy emphasizes explicit control, composability, and predictability over the more abstract and often opaque mechanisms found in many existing AI frameworks. Instead of relying on Python-heavy runtime layers that can introduce overhead, ZML uses the Zig programming language, combined with MLIR (Multi-Level Intermediate Representation) and OpenXLA, to compile model computation graphs directly into standalone native binaries.

This deep integration with hardware allows ZML/LLMD to achieve minimal memory overhead and direct hardware access. The result is a more efficient runtime with lower latency between kernel operations, which translates to faster inference and better resource utilization. This is a fundamental departure from the common practice of wrapping Python around CUDA kernels, as seen in many other inference servers like vLLM or Ollama.

Key Features and Technical Innovations in ZML/v2

The recent ZML/v2 release, which arrived on March 24, 2026, represents a complete rewrite of the framework, focusing on developer experience, performance, and composability. This update has brought several significant technical innovations:

• Hardware Agnosticism: ZML/LLMD can compile models directly for a wide range of hardware platforms, including NVIDIA CUDA, AMD ROCm, Intel OneAPI, Google TPUs, and AWS Trainium/Inferentia 2. This means developers can write their AI code once and deploy it across diverse hardware environments without needing to rewrite or port code for each specific chip.
• Zig-based Development: With 92.7% of its codebase written in Zig, ZML leverages this modern systems programming language for its performance predictability and direct hardware control. This eliminates many of the inefficiencies associated with Python-based runtimes.
• Optimized Memory Management: ZML/v2 introduces explicit allocation and use of pinned memory via zml.mem.DmaAllocator. This, along with zero and overlapped copy primitives, significantly speeds up data transfers between the host and accelerators. For example, ZML has demonstrated loading 14.96 GiB of model weights in just 1.165 seconds, achieving a throughput of 12.83 GiB/s.
• Pluggable Attention Backend: The framework now includes a pluggable attention backend system that automatically selects optimized attention implementations based on the target platform and accelerator. This includes support for highly efficient kernels like FlashAttention 2 and 3 on NVIDIA CUDA (for sm80 up to sm121 architectures) and AITER kernels on AMD ROCm.
• Userland Virtual File System (VFS): ZML implements a full userland VFS, allowing models to be loaded directly from various sources—local files, HTTP endpoints, S3 buckets, and Hugging Face repositories—without requiring them to be downloaded to local disk first. This streamlines model deployment and management.
• Complete Build Sandboxing: The build process for ZML/v2 runs entirely on a 100% hermetic LLVM toolchain. This ensures full reproducibility, better cross-compilation support, and a consistent development experience across different operating systems. It also enables remote execution on services like BuildBuddy or NativeLink.
• Powerful Sharding Primitives: ZML/v2 has made sharding a first-class citizen within its programming model, introducing a new public zml.sharding API. This allows for explicit handling of physical and logical meshes, strategies, and placements, leveraging technologies like Shardy or GSPMD transparently based on the target platform.

LLM Support and Current Status

The LLMD inference server, currently in a technical preview or alpha stage, supports popular Large Language Models such as Llama 3.1/3.2, Qwen 3.5, and LFM 2.5. It also offers an OpenAI-compatible API, making it easier for developers to integrate with existing tools and workflows. While powerful, it's important to note some current limitations due to its preview status:

• Single-GPU operation only (multi-GPU sharding is not yet supported).
• A maximum batch size of 16.
• No prefix caching.
• Support primarily limited to Llama and Qwen model architectures.

ZML describes LLMD as a "build-your-own-stack" tool, positioning it for ML systems engineers who are comfortable with low-level infrastructure and looking for highly optimized, hardware-agnostic solutions.

Endorsement and Industry Implications

The feed item highlights an endorsement from Turing Award winner Yann LeCun. While specific details of this endorsement were not found in the public search results, LeCun is a well-known figure in the AI community, recognized for his foundational contributions to deep learning and his consistent advocacy for alternative AI architectures that move beyond the limitations of current large language models. His general stance aligns with ZML's goal of developing more efficient, robust, and hardware-agnostic AI systems.

ZML's work with LLMD represents a broader trend in AI infrastructure: a move away from tightly coupled, Python-centric, and often NVIDIA-exclusive toolchains towards compiled, hardware-portable runtimes. By building on Zig and MLIR, ZML prioritizes performance predictability and true hardware agnosticism over ecosystem maturity. This could have significant implications for the industry:

• Reduced Costs: By making AI inference efficient across diverse hardware, ZML can help companies reduce their infrastructure costs. Community benchmarks, for instance, suggest that AMD's RX 7900 XTX can achieve 80-90% of an RTX 4090's throughput for LLM inference using ZML, at a fraction of the cost.
• Increased Competition: Offering a viable, open-source alternative to proprietary solutions like NVIDIA's CUDA ecosystem fosters competition and innovation in the AI hardware and software markets.
• Broader Accessibility: Hardware agnosticism means AI can be deployed more easily on a wider range of devices, from data centers to edge devices, without extensive code modifications.
• European Innovation: As a French startup, ZML contributes to the growing European AI ecosystem, which is seeing increasing investment and a focus on building sovereign AI capabilities. The Île-de-France Region, Scaleway, VSORA, and ZML have even committed to laying the foundations for the next generation of AI chips in Europe.

Availability and Future Outlook

ZML/LLMD is an open-source project, freely available to developers. The source code and documentation can be found on their official ZML GitHub Repository and their official website.

While still in its early stages with some limitations, ZML's rapid development and innovative approach position it as a project to watch. As the demand for efficient and cost-effective AI inference continues to grow, solutions like ZML/LLMD could play a crucial role in shaping the future of AI deployment, making advanced models accessible to a broader range of users and applications.

Frequently Asked Questions

What is ZML/LLMD?

ZML/LLMD is a free, open-source software released by the French startup ZML. It's an inference engine designed to speed up the execution of AI models, particularly Large Language Models (LLMs), across a variety of hardware chips, aiming to reduce costs and hardware dependency.

What makes ZML/LLMD unique for AI inference?

ZML/LLMD stands out due to its hardware-agnostic design, allowing AI models to run efficiently on NVIDIA, AMD, Intel, Google TPU, and AWS Trainium/Inferentia 2 hardware from a single codebase. It's built with the Zig programming language, MLIR, and Bazel, which helps minimize overhead and provides direct hardware access for peak performance.

Is ZML/LLMD free to use?

Yes, ZML/LLMD is an open-source project and is available for free. It is licensed under the Apache-2.0 license, making it accessible for developers and organizations to use and contribute to.

What are the current limitations of ZML/LLMD?

As of its technical preview stage, ZML/LLMD has some limitations. These include support for single-GPU operation only (no multi-GPU sharding), a maximum batch size of 16, no prefix caching, and current support primarily for Llama and Qwen model architectures.