Open-Weight AI · Lesson 1 of the journey

Weights, Precision & the VRAM Equation

What you are actually downloading when you “run an open model” — and how to know if it fits.

1 · The shift in mental model

When you call Claude or GPT, you send text and get text back. Someone else holds the model, the hardware, and the runtime. You rent a behaviour.

An open-weight model is the opposite deal. What gets published is a folder of files — mostly the trained weights: billions of numbers that define the model’s behaviour.A model card on Hugging Face typically lists files like model-00001-of-00004.safetensors. Those are the weights. The rest is config and tokenizer. To turn that folder into something you can talk to, you now supply the two things the API used to hide from you:

That third term is where beginners get burned. The weights have to physically fit in memory before anything can run. So the first skill on this whole journey is arithmetic, not coding.

2 · Precision: how big is one number?

A model with “8 billion parameters” has 8 billion numbers. How much memory that costs depends entirely on how many bytes you spend per number — its precision. Models are trained in 16-bit, but you can store them more cheaply by quantizing: keeping the same count of numbers at lower precision.

Those 4-bit GGUF formats actually cost a touch more than 0.5 — about 0.57 bytes/param — because each block of weights carries a little metadata. Close enough for back-of-envelope sizing; we’ll get precise in the quantization lesson. [ref]

3 · The equation

Weights memory is just a multiplication:

An 8B model at 4-bit: 8 × 0.5 ≈ 4 GB. The same model at FP16: 8 × 2 = 16 GB. Same model — a 4× swing in what hardware you need, decided purely by precision.

But weights aren’t the whole bill. Two overheads ride along:

• Runtime overhead (~15–20%): the engine, activations, CUDA buffers.
• KV cache — memory that grows with how much text is “in context.” It’s the hidden killer people forget; long prompts or many simultaneous users can rival the weights themselves.We size the KV cache properly in the serving lesson, when concurrency starts to matter. [ref]

4 · Feel it — the calculator

This is the tool, not the table. Drag the model size and flip the precision; watch what GPU you’d need. Try to predict the number before you let go of the slider — that prediction is the skill we’re building.

5 · What this unlocks for your mission

You’re cloud-GPU-first and still deciding what hardware you’d need. The equation is exactly how that decision gets made — for real, before spending a cent:

• Want to run a 70B model? At 4-bit that’s 70 × 0.57 × 1.2 ≈ 48 GB — one A6000/L40S, or two 24 GB cards. At FP16 it’s ~168 GB — multiple A100s. The precision choice changed the rental bill by ~4×.
• On your Mac, “VRAM” is unified memory shared with the system, so usable budget is roughly (total RAM − ~6 GB). A 32 GB Mac comfortably runs an 8B–14B model at 4-bit — your first hands-on lesson.
• When we reach fine-tuning, the same equation explains why QLoRA (4-bit frozen base + small trainable adapters) lets a 70B model train on a single 24 GB card — it shrinks the dominant term.

6 · Retrieval practice

Close the lesson above in your mind first. Answering from memory is what builds durable retention — re-reading only feels like learning.

▸ Primary source for this lesson

Read once, slowly: The Math Behind Local LLMs — VRAM Requirements. It walks the same formula with the KV-cache term spelled out. For the quantization side, skim LLM Quantization Explained (Q4/Q8/FP16). Both are tracked in RESOURCES.md.