NVIDIA has detailed three distinct optimization techniques to accelerate mixture-of-experts (MoE) inference on its desktop DGX Spark system. Using Ant Group inclusionAI’s 124B-parameter Ling-3.0-flash model as a benchmark, NVIDIA combined a new MoE kernel backend, an online FP8 output layer, and built-in speculative decoding to improve decode throughput on GSM8K tests without sacrificing accuracy.

The optimizations target memory bandwidth bottlenecks typical during single-user desktop serving. Replacing FlashInfer with the Humming kernel library delivered a 9.3% speedup, while quantizing the language model head layer to FP8 increased throughput by 21.7% over baseline. Additionally, leveraging built-in multi-token prediction (NEXTN) for speculative decoding drove test throughput up to 102.88 tokens per second.

NVIDIA cautioned developers about integration traps when combining these techniques, noting that failing to properly share the FP8 LM-head module with draft prediction causes frameworks to fall back and drop overall performance below unoptimized baselines.

Why it matters

  • Desktop systems like DGX Spark can efficiently serve 100B+ MoE models for single users using targeted kernel and quantization optimizations.

  • Stacking FP8 LM-head quantization with built-in speculative decoding requires exact framework configuration to prevent silent performance rollbacks.

  • Humming’s dynamic NVRTC compilation reduces kernel compile times from tens of minutes to seconds compared to traditional CUTLASS approaches.

Source: marktechpost.com