Learn
What is local inference?
Last reviewed July 16, 2026
Local inference means running an AI model on your own hardware instead of calling a provider's servers. Prompts and outputs never leave the machine, requests have no per-token cost, and it works offline. The constraint is capability per hardware: consumer machines run small and mid-size open-weight models, not closed frontier ones.
What it takes
A runtime (Ollama is the common one), an open-weight model sized to your memory, and enough GPU VRAM or unified memory: roughly 8 GB for small models, 16 GB for capable mid-sizes, 24 GB and up for the 27B class. Token speed scales with the hardware; a workstation GPU feels interactive, a thin laptop does not.
The hybrid pattern
Local-only is a constraint; local-first is a strategy. The pattern that works: route what your machine can serve locally (private, free), and fall back to hosted models for jobs that outgrow it, inside the same interface, so privacy and capability stop being an either-or.
Pair a computer running Ollama and idapt routes to it as a free provider: a hardware-fit filter shows what your machine can serve, and one toggle prefers local with cloud fallback in the same chat.
Frequently asked
See it in practice
The fastest way to understand it is to use it. Start free, no credit card.
Try idapt free