Training a model and using a model are almost nothing alike operationally — one happens once (or occasionally), costs enormous compute, and adjusts weights; the other happens on every single request, is comparatively cheap, and never changes the model at all.
3 min read
Every deployed ML system has two distinct phases, and conflating them is a common source of confused expectations about cost, speed, and what's even possible to change:
"Can we make the model learn from this user's correction right now?" is a question with a real, often disappointing answer: no, not directly — a deployed model's weights are fixed until an actual retraining run happens, which is expensive and deliberate, not automatic per-request. What products that feel like they're learning in real time are usually doing instead is one of: storing the correction somewhere and feeding it back into a future, separate retraining run, or (far more common with today's LLMs) putting the correction into the context sent with each request — the model itself never actually changes, but what it's shown at inference time does. This context-based approach is exactly what makes the "fine-tuning vs. prompting vs. RAG" lesson later in this domain worth understanding precisely — they're three different answers to "how do I make a model behave differently without retraining it from scratch."
Training a large model can genuinely cost millions of dollars in compute, spent once. Inference costs much less per call — but multiplied by potentially billions of calls across a product's lifetime, aggregate inference cost commonly exceeds the original training cost many times over. This is exactly why techniques that make inference faster or cheaper (quantization — using lower-precision numbers for the weights; smaller "distilled" models trained to mimic a larger one) are such an active, commercially important area — a percentage saved on inference cost multiplies across every single request forever, in a way a training-cost saving never does.
A user typing a message to a chatbot is waiting, in real time, for inference to complete — this is a genuine latency-sensitive path, not a background batch job. This is a big part of why an LLM API response streams back token by token rather than waiting for the entire response to finish generating: it lets the user start reading immediately rather than staring at a blank screen for the full generation time, directly analogous to how HTTP responses can begin rendering before a full page finishes downloading.
Every time you call an LLM API, you are doing inference on a model somebody else already trained — you are not training or modifying that model in any way, regardless of how the conversation goes. This single fact is the correct mental model for nearly every practical question about building LLM-powered features: your own product code can shape what gets sent into each inference call (the prompt, retrieved context, conversation history) but never changes the underlying model's weights — exactly the distinction the AI Agents & LLM Integration domain assumes throughout its own material.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. What is the key operational difference between training and inference?
2. Why can't an LLM API 'permanently learn' from a correction a user gives it during a conversation?
3. Why does an LLM response often stream back token by token instead of arriving all at once?
AI & ML Fundamentals