Run AI Models Locally with Ollama: A Practical Setup Guide

By
ollama ai local llm homelab privacy

Ollama runs large language models on your own machine. No API keys, no usage limits, no data leaving your network. A 7B model on a machine with 16GB RAM answers questions in seconds.

Install Ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download the installer from ollama.com. It runs as a background service.

Docker:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Pull and Run a Model

ollama pull llama3.1:8b
ollama run llama3.1:8b

The first pull downloads the model weights (~4.7GB for the 8B quantized). After that, ollama run loads it in seconds.

Models Worth Running

Model Size Best For
Llama 3.1 8B 4.7GB General chat, quick answers
Qwen2.5-Coder 14B 8.9GB Code tasks, scripting
Mistral 7B 4.1GB Fast text tasks
DeepSeek-R1 14B 9GB Reasoning, analysis
Phi-4 14B 8.9GB Smaller footprint, strong reasoning

Add a Web UI

Ollama's CLI is fine for quick tests. For real use, run Open WebUI — a ChatGPT-style interface that connects to Ollama:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000. Create an account (local only, no registration). Select your Ollama model from the dropdown.

GPU Acceleration

CPU inference works but runs slowly on large models. A GPU cuts inference time by 5–10x.

NVIDIA: Install the NVIDIA Container Toolkit, then:

docker run -d --gpus=all -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama

AMD (ROCm): Use the ollama/ollama:rocm image.

Hardware worth buying for local AI:

What Ollama Won't Replace

Cloud models from Anthropic and OpenAI still win on multi-step reasoning and complex tasks. Local models are for: private data, offline use, high-volume tasks where API costs add up, and scripting workflows that call the model hundreds of times a day.

As an Amazon Associate, IT Lad earns from qualifying purchases made through links on this page. Read the full affiliate disclosure.

Previous Post Next Post