AI Inference

AI inference is the process of running a trained AI model on new input to produce an output — for example, generating a ChatGPT response. It is distinct from training, which is the one-time process of building the model.

How it works

Once a model is trained, every individual prediction it makes — every answer, image, or line of code — is an inference. Inference runs on specialized hardware (GPUs or AI accelerators) and is usually priced per token (a chunk of text). For high-traffic products, inference is the dominant ongoing AI cost.

Thank you for reading this post, don't forget to subscribe!

Why it matters

The cost of inference fell roughly 280-fold between late 2022 and late 2024 — from about $20 to around $0.07 per million tokens — which is a primary reason AI adoption accelerated so quickly. See AI Adoption Statistics 2026.