I’ve been building VeloxQuant, an open-source KV-cache optimization toolkit for MLX focused specifically on LLM inference on Apple Silicon.
The project currently implements 43 KV-cache compression methods, along with custom Metal kernels for several performance-critical paths. The goal is to explore how far we can reduce KV-cache memory and improve inference performance while understanding the trade-offs in latency, throughput, and model quality.
Some of the areas I’ve been working on include KIVI INT4/INT2, RaBitQ, RVQ, fused attention kernels, GQA decode optimization, prefill kernels, and cross-request decode batching.
A recent GQA experiment was particularly interesting: instead of expanding shared KV heads before attention, a two-pass decode path operates on the grouped structure directly. In our tests this reduced GQA attention latency substantially at longer context lengths.
I’m now expanding the benchmarks to newer and larger models and, more importantly, across different Apple Silicon hardware. I currently do most of the development and benchmarking myself, so I’d really like to get more MLX developers involved.
If you have an M1/M2/M3/M4/M5 Mac — MacBook, Mac mini, Mac Studio, etc., I’d especially appreciate help running benchmarks, validating kernels, testing models, or contributing new compression/attention implementations.
I’m also very interested in feedback from people who understand MLX internals, Metal kernels, quantization, attention optimization, or LLM inference systems. If you see something in the architecture or benchmarks that could be improved, please tell me.
Project: https://veloxquant.dev/
GitHub: https://github.com/rajveer43/veloxquant-mlx
Contributions, benchmark results, criticism, and ideas are all welcome. I’d like this to become a genuinely useful MLX performance project rather than something optimized around one Mac or one benchmark.