Liquid AI, in partnership with independent validator Artificial Analysis, has launched Pipette, an open-source benchmarking platform for evaluating foundation models on edge devices. Rather than rating standalone models in isolation, Pipette measures performance across complete deployment configurations, combining specific model weights, quantization methods, runtimes, hardware, and context lengths up to 8,192 tokens.
The initial launch dataset includes over 1,000 configurations spanning 30+ models and llama.cpp runtimes evaluated on hardware such as Apple’s M5 Max MacBook Pro, iPhone 17 Pro, and Samsung Galaxy S26 Ultra. Pipette tracks latency, throughput, and memory consumption under strict thermal and load gating, pairing those metrics with off-device quality evaluations on datasets like MATH-500 and GPQA Diamond.
The release addresses a critical bottleneck for mobile software engineers, revealing that identical quantization formats can yield drastically different decode throughputs on identical target hardware.
Why it matters
Provides engineers empirical data showing how runtime and quantization choices alter real-world edge performance before deployment.
Establishes a standardized, hardware-aware testing framework to replace deceptive server-class model cards for mobile AI.
Open-source Apache 2.0 codebase lets teams run continuous integration benchmarking directly on physical target devices.
Source: marktechpost.com
