6
Ran the same eval suite across three quantizations of Llama-3.1-8B on M2 Pro.
q4 is the sweet spot.
Quantize
Q2 vs Q4 vs Q8 on Llama-3.1-8B — actual numbers
0 reply(ies)
sign in to reply.
Ran the same eval suite across three quantizations of Llama-3.1-8B on M2 Pro.
q4 is the sweet spot.