---
title: 'Run AI Models Locally with Ollama: A Practical Setup Guide'
url: 'http://itlad.uk/blog/ollama-local-ai-models-guide'
markdown: 'http://itlad.uk/blog/ollama-local-ai-models-guide.md'
date: '2026-06-01'
description: 'Set up Ollama to run AI models locally on your own hardware — model selection, GPU acceleration, Open WebUI frontend and hardware recommendations.'
taxonomy:
  category:
    - ai
    - homelab
  tag:
    - ollama
    - ai
    - 'local llm'
    - homelab
    - privacy
---

[ Home ](http://itlad.uk/)     [ Blog ](http://itlad.uk/blog)      Run AI Models Locally with Ollama: A Practical Setup Guide    

# [Run AI Models Locally with Ollama: A Practical Setup Guide](http://itlad.uk/blog/ollama-local-ai-models-guide)

 By [IT Lad](http://itlad.uk/about) 1 Jun 2026 

  [ollama](http://itlad.uk/blog/tag:ollama#body-wrapper) [ai](http://itlad.uk/blog/tag:ai#body-wrapper) [local llm](http://itlad.uk/blog/tag:local%20llm#body-wrapper) [homelab](http://itlad.uk/blog/tag:homelab#body-wrapper) [privacy](http://itlad.uk/blog/tag:privacy#body-wrapper)  

Ollama runs large language models on your own machine. No API keys, no usage limits, no data leaving your network. A 7B model on a machine with 16GB RAM answers questions in seconds.

## Install Ollama

**Linux:**

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

**Windows:** Download the installer from ollama.com. It runs as a background service.

**Docker:**

```bash
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

## Pull and Run a Model

```bash
ollama pull llama3.1:8b
ollama run llama3.1:8b
```

The first pull downloads the model weights (~4.7GB for the 8B quantized). After that, `ollama run` loads it in seconds.

## Models Worth Running

| Model | Size | Best For |
|---|---|---|
| Llama 3.1 8B | 4.7GB | General chat, quick answers |
| Qwen2.5-Coder 14B | 8.9GB | Code tasks, scripting |
| Mistral 7B | 4.1GB | Fast text tasks |
| DeepSeek-R1 14B | 9GB | Reasoning, analysis |
| Phi-4 14B | 8.9GB | Smaller footprint, strong reasoning |

## Add a Web UI

Ollama's CLI is fine for quick tests. For real use, run Open WebUI — a ChatGPT-style interface that connects to Ollama:

```bash
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main
```

Open `http://localhost:3000`. Create an account (local only, no registration). Select your Ollama model from the dropdown.

## GPU Acceleration

CPU inference works but runs slowly on large models. A GPU cuts inference time by 5–10x.

**NVIDIA:** Install the NVIDIA Container Toolkit, then:

```bash
docker run -d --gpus=all -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama
```

**AMD (ROCm):** Use the `ollama/ollama:rocm` image.

Hardware worth buying for local AI:

- [NVIDIA RTX 4060 Ti 16GB](https://www.amazon.com/s?k=nvidia%2Brtx%2B4060%2Bti%2B16gb&tag=itlad-21) — 16GB VRAM handles 13B models comfortably
- [NVIDIA RTX 3060 12GB](https://www.amazon.com/s?k=nvidia%2Brtx%2B3060%2B12gb%2Bgraphics%2Bcard&tag=itlad-21) — cheaper, runs 7–13B models, good starting point
- [32GB DDR4 RAM kit](https://www.amazon.com/s?k=32gb%2Bddr4%2B3200mhz%2Bram%2Bkit&tag=itlad-21) — minimum for comfortable CPU inference on 13B models

## What Ollama Won't Replace

Cloud models from Anthropic and OpenAI still win on multi-step reasoning and complex tasks. Local models are for: private data, offline use, high-volume tasks where API costs add up, and scripting workflows that call the model hundreds of times a day.

---

*As an Amazon Associate, IT Lad earns from qualifying purchases made through links on this page. [Read the full affiliate disclosure](http://itlad.uk/affiliate-disclosure).*

 [ Previous Post](http://itlad.uk/blog/linux-commands-every-it-pro-needs) [Next Post ](http://itlad.uk/blog/microsoft-power-apps-guide)

#### Syndicate

 [ Atom 1.0](http://itlad.uk/blog.atom) [ RSS](http://itlad.uk/blog.rss)

---

## Navigation

- Parent: [Blog](http://itlad.uk/blog.md)
- Previous: [Microsoft Power Apps: What IT Pros Actually Need to Know](http://itlad.uk/blog/microsoft-power-apps-guide.md)
- Next: [Linux Commands Every IT Professional Should Know Cold](http://itlad.uk/blog/linux-commands-every-it-pro-needs.md)