Quantization
All articles
6 total
Small Language Models vs Large Language Models Updated
How to choose between a small on-device model and a large general model …
Comparisons
Added 23 Jun
·
Upd 25 Sep
·9 min
LoRA and QLoRA
Low-Rank Adaptation freezes a model and trains a small pair of low-rank …
Glossary
Added 1 Jul
·
Upd 1 Jul
·7 min
Microscaling (MX) Formats
A family of block-scaled low-precision numeric formats standardized by …
Glossary
Added 6 Jul
·
Upd 6 Jul
·2 min
Quantization
How INT8 and INT4 quantization compress neural network models for faster …
Glossary
Added 28 Mar
·
Upd 30 May
·3 min
Running Distilled Models
What actually changes when you serve a distilled model instead of its …
Guides
Added 3 Sep
·
Upd 3 Sep
·16 min
Self-Hosting LLMs: Hardware and Economics
The full hardware ladder for running your own models, worked VRAM and …
Guides
Added 3 Sep
·
Upd 3 Sep
·20 min
Work with Linda Mohamed, AWS Community Hero