The nouns of the open-LLM world. Skim it once; return whenever a lesson uses a term you want to pin down. Lessons link here rather than re-defining terms.
Models & openness
- Open-weight model
- A model whose trained weights (the numbers) are downloadable, so anyone can run or fine-tune it. Most "open source" LLMs are really open-weight: you get the weights, but not the training data or full recipe. Qwen, Llama, DeepSeek, GLM, Mistral, gpt-oss are open-weight. The practical thing you care about — can I download and run it? — is "yes."
- Open-source (truly)
- Weights plus training code and (ideally) data, under an OSI-style license. Rarer. For your mission the distinction matters mostly for licensing: check the model card. Apache-2.0 (Qwen) and MIT (DeepSeek, GLM) are the safest for commercial use; some "open" licenses add use restrictions.
- Parameters (e.g. "27B", "70B")
- How many learned weights the model has, in billions. Bigger ≈ smarter but heavier. Parameter count drives how much VRAM you need to run it.
- MoE & "active parameters" (e.g. "753B total / 40B active")
- Mixture-of-Experts: the model is huge in total but only routes each token through a small active slice. You get big-model quality at small-model speed — but you must still fit the total in memory. Most 2026 frontier open models (DeepSeek V4, GLM-5.2, Qwen3) are MoE.
- Quantization (Q4_K_M, Q8, FP16…)
- Compressing the weights to fewer bits so the model fits in less memory and runs faster, at a small quality cost. FP16 = full-ish precision (2 bytes/param);
Q4 ≈ 4 bits/param ≈ ~¼ the memory. Q4_K_M is the common sweet spot: near-original quality, big memory savings. Rule of thumb: a model's VRAM need ≈ (params) × (bytes per param) + overhead.
Hardware & running
- VRAM
- The memory on a GPU. The model's weights must fit here to run fast. The single most important spec when choosing a GPU or a model. Rough 2026 fits at Q4: 8GB→7B, 12GB→14B, 16–24GB→24–32B, 40GB+→70B.
- Unified memory (Apple Silicon)
- On Macs, the CPU and GPU share one big pool of fast memory. So a Mac with 128GB unified memory can run a 70B model that would need two NVIDIA cards — which is why Macs punch above their weight for local LLMs (slower, but huge models fit).
- Inference vs training
- Inference = running a finished model to get answers (what you'll do 99% of the time). Training = building/teaching the model (expensive, out of scope for now). Almost everything in this course is about inference.
- Token & context window
- A token is a chunk of text (~¾ of a word). Models read and bill per token. The context window is how many tokens the model can hold at once (2026 open models reach 200K–1M). Hosted APIs price as $/million input tokens + $/million output tokens.
Software that serves models
- Inference engine / runtime
- The program that actually executes the model. Two that matter: llama.cpp (portable, runs anywhere, powers most local tools) and vLLM (production-grade, ~16–20× the concurrent throughput for many users at once).
- Local runner (Ollama, LM Studio, Jan)
- Friendly "experience layers" that wrap an engine so you can download and chat with a model in minutes. Ollama = CLI + local server; LM Studio / Jan = GUI apps. Great for one user; not built for serving a crowd.
- Serving / concurrency / batching
- Handling many simultaneous requests efficiently. vLLM's tricks (PagedAttention, continuous batching) are why it serves a production app while Ollama is happiest with one user. Concurrency is the dividing line between "local toy" and "production server."
- OpenAI-compatible endpoint
- The universal adapter. Nearly every host and engine exposes the same API shape as OpenAI's, so your code switches providers by changing a base URL + API key — no rewrite. This is why "swap the model" is cheap and why you're not locked in.
Where models run (the cloud ladder)
- Hosted inference API
- Someone else runs the model; you call it and pay per token. Zero ops. OpenRouter, DeepInfra, Together, Fireworks, Groq. The default starting point.
- Serverless GPU / scale-to-zero
- You deploy a container or model; the platform spins a GPU up on demand and bills per second of active compute, down to zero when idle. Modal, RunPod Serverless, HF Inference Endpoints. Middle ground: more control than an API, no idle cost.
- GPU rental (on-demand vs spot)
- Rent a whole GPU machine by the hour and run whatever you want on it. RunPod, Vast.ai, Lambda. On-demand = stable price, won't be reclaimed; spot = cheaper but can be interrupted. You manage the server.
- TCO & break-even
- Total Cost of Ownership — not just the GPU, but power, cooling, networking, and the engineer-hours to run it (often $750–3,000/mo in labor). Break-even is the usage level where owning/renting beats paying per token. In 2026 that's roughly 50–100M tokens/month, or when privacy forces on-prem.
- Pod / instance
- One rented GPU machine (RunPod calls it a "pod," Vast.ai an "instance"). You pick the GPU, a Docker image, and storage; it boots, you use it, you terminate it to stop paying. The clock runs the whole time it's running, whether or not you're using it.
- Network volume / persistent storage
- A disk that survives after a pod is destroyed. Mount it at the model cache path (e.g.
/root/.cache/huggingface) so a 20GB model download persists — otherwise every fresh pod re-downloads (re-paying time) the weights.
- On-demand vs spot/interruptible
- On-demand: fixed price, the machine is yours until you stop it. Spot/interruptible: cheaper, but the provider can reclaim it with little warning. Use on-demand while learning; spot for fault-tolerant batch jobs.
- Cold start
- The delay before a freshly-started server can answer — downloading weights, loading them into VRAM. Why your first request 502s for a minute or two. Serverless rungs (Modal, RunPod Serverless) trade some cold-start latency for scale-to-zero billing.
Hugging Face nouns
- The Hub
- "GitHub for ML." Where models, datasets, and demos live (2.4M+ models). You download weights from here.
- Model card
- The README on a model's Hub page: license, intended use, size, benchmarks, quant variants. Read it before you commit — especially the license.
- Spaces
- Host a live demo web app (Gradio/Streamlit) for a model, with a public URL and optional free ZeroGPU quota. Good for a quick playground or sharing a prototype.
- Inference Endpoints
- HF's managed "deploy one model on a dedicated GPU behind an API" product, with autoscaling and scale-to-zero. Sits between hosted API and rolling your own.