Lesson 02 · Your hands on a real GPU
You have no strong computer — so you'll borrow one. By the end you'll have rented a GPU in the cloud, served an open model on it behind a standard API, called it from your own code, and shut it down. That single loop is the skill your whole mission rests on.
Tick each box as you go. Each step has a checkpoint — a way to know it worked before moving on. That's your feedback loop; don't advance on faith.
Why: 24GB is the budget sweet spot — it runs everything up to ~32B at quant.2 The vLLM image gives you a real OpenAI-compatible server, not a toy. Checkpoint: pod status shows Running.
Why: the model is a ~5–20GB download. Put it on a persistent volume so next session you don't re-download (and re-pay for) it.1 Checkpoint: the volume appears mounted in the pod's storage.
Why: vLLM listens on 8000; RunPod gives you a public proxy URL to it. The --api-key is not optional — without it, anyone who finds the URL runs inference billed to you.3 Command is on the cheat sheet. Checkpoint: logs show Application startup complete.
Why: if memory use is >0%, the model is really on the GPU; if it's 0%, you're accidentally on CPU and everything will crawl.1 Checkpoint: nvidia-smi shows VRAM used and a python process.
Why: this is the whole point — your normal OpenAI client, repointed by base URL + key, talking to a model you're hosting.4 First call may 502 for a minute while weights load — wait and retry. Checkpoint: you get a completion back.
Why: a running pod bills every second, used or not. $0.50/hr left on overnight for a week is ~$84 — your whole budget gone to nothing.2 Checkpoint: the pod is gone from your dashboard and billing has stopped.
The 8B model proved the loop. To feel an actually strong open model, redeploy with Qwen/Qwen3.6-27B-Instruct on the same 24GB card — in mid-2026 it ties a frontier closed model (Sonnet-class) on agentic coding benchmarks and still fits a single rented GPU.5 Same steps, one bigger model name. That's the moment the thesis stops being abstract: a near-frontier model, running on a $0.50/hr machine you rented in five minutes.
You now know, in your hands and not just your head: an open model is a file, a rented GPU is a few clicks and cents-per-hour, and the access layer is the same OpenAI call you already write. The "select few gatekeepers" future has an exit, and you've used it. The remaining question is purely economic — when is this worth it versus just paying a hosted API per token? That's the comparison Lesson 03 will make concrete.
From memory — these are the gotchas that bite people for real money.
1. You finish experimenting for the night. To stop being billed you must:
2. Why is the --api-key on your serve command genuinely important?
3. The point of the network volume mounted at the cache path is so that:
4. Switching this call from your rented pod to OpenRouter later would mainly mean:
Keep beside you: RunPod's official "Deploy vLLM" docs3 (and Vast.ai's Ollama template guide1 if you went the cheaper route). These are the canonical, current step-by-step references; the lesson gives you the judgement around them.
And when a real error stops you, r/LocalLLaMA has almost certainly seen it this month. Search before you stall.
I'm your teacher — if a pod won't boot, a call 502s, or the bill looks wrong, paste it to me. Debugging your live setup is exactly the kind of help I'm here for.
Sources (verified June 2026; prices/models drift — re-check before quoting):