Perplexity has open-sourced Lily, a specialized local inference engine designed specifically for running the Qwen3.6-35B-A3B model on Apple Silicon. Written in Rust with hand-crafted Metal kernels, Lily bypasses traditional frameworks like PyTorch and MLX to drive generation loops directly through an OpenAI-compatible API.
The engine implements groupwise affine 4-bit quantization, shrinking the 35-billion parameter model down to a 19.4 GB footprint that requires at least 24 GB to 32 GB of unified memory. By bypassing CPU synchronization and fusing custom Metal kernels inside a single GPU command buffer, Lily achieved a 77.4% end-to-end prefill speedup at a 512-token prompt length.
Lily optimizes mixture-of-experts (MoE) execution and long-context processing by grouping operations, utilizing threadgroup memory, and minimizing register-to-memory round trips. The runtime is available for deployment today via Perplexity’s public GitHub repository.
Why it matters
Delivers significant prefill and decode performance gains for open MoE models on consumer Apple hardware.
Demonstrates custom low-level Rust/Metal runtimes outperforming general-purpose frameworks like MLX.
Provides developers with an open-source template for ultra-efficient local LLM serving.
Source: marktechpost.com
