Run a private ChatGPT-style assistant at home with Ollama and Open WebUI

Quick comparison

RAM figures are catalog estimates from our tests.

What we measured

• gpt-oss:20b (a 13 GB download) ran 100% on the GPU and used about 16.5 GB of the RTX 3090's 24 GB of VRAM at Ollama's default 32k context.
• Generation speed was about 122 to 125 tokens per second on a 300-token answer. The very first request took about 47 seconds because the model had to load; after that the answer started right away.
• llama3.2:1b, a tiny test model, generated about 285 tokens per second and used about 3 GB of VRAM in total.
• The LXC (6 vCPU, 6 GB RAM) sat at about 0.7 GB of RAM idle and about 2.9 GB with the 20B model loaded, because the weights live in VRAM. The Open WebUI container itself used 0.5 to 0.7 GB.
• Disk: the Open WebUI image is about 6.5 GB and our two models take 15 GB, so a 40 GB disk is the realistic minimum.
• These are short single-user tests, not a full benchmark. Long prompts, long conversations and several users at once will be slower and use more VRAM.

Before you start

• A GPU makes the difference. As a rough rule, the model file should fit in VRAM with a few GB to spare. Without a GPU, small models still run, but slowly.
• For an NVIDIA GPU you need the NVIDIA driver on the Proxmox host. If you run this in an LXC container, the GPU has to be passed through to it, which is a separate guide of its own.
• Open WebUI runs in Docker. Inside an LXC container, Docker needs the nesting feature enabled.
• Pick a model that fits your VRAM. We use gpt-oss:20b on 24 GB. Smaller cards should start with a model of 7 to 8 billion parameters.

Setup steps

1. Install Ollama by following the official instructions at ollama.com/download. In our fresh Debian LXC the installer needed the zstd and pciutils packages first.
2. Check that it works: run ollama --version, then ollama pull for the model you picked, for example ollama pull gpt-oss:20b.
3. Test the model from the API before adding any interface: send a request to http://127.0.0.1:11434/api/generate with curl. We found this more reliable than ollama run when the shell is not a real terminal.
4. Install Docker, then start Open WebUI with the compose file on our Open WebUI page. It uses host networking, so the container reaches Ollama on 127.0.0.1:11434.
5. Open http://your-server-ip:8080. The first account you create becomes the administrator, so create it right away, with a strong password.
6. In the admin settings turn off new sign-ups, or set them to require approval, unless you want other people to register.
7. Choose your model from the drop-down and start chatting. Check ollama ps and nvidia-smi once to confirm the model really runs 100% on the GPU.
8. For access from outside your home, use a VPN such as Tailscale rather than opening the port to the internet.

Running it day to day

• Ollama unloads a model after a few minutes of idle time and frees the VRAM. That is useful if the GPU is shared with other jobs, but the next request pays the load time again.
• If several GPU tools share one card, run one heavy job at a time. Two big models together can run out of VRAM or RAM.
• Update both parts from time to time: pull the newer Open WebUI image and re-create the container, and update Ollama with its installer.
• Back up the Open WebUI data volume. It holds users, chats and settings. The models themselves can always be downloaded again.

Honest caveats

• Open models are good, but a 20B model is not the same as the newest paid cloud models. Expect weaker results on hard reasoning, long documents and very recent facts. Judge by your own tasks.
• Ollama listens on all interfaces in our setup, and its API has no login. Keep it on a trusted network or behind a VPN, never on the open internet.
• Open WebUI's licence is not a standard open-source one: it is BSD-3-Clause based with a clause that forbids removing the Open WebUI branding once you have more than 50 users. For a home or family setup this does not matter, but read it for anything bigger.
• Models have their own licences, and some limit commercial use. Check the licence of each model you download.
• A big model keeps a GPU busy and draws power. In our short test the card drew about 162 W while generating.
• Our figures come from one machine and short prompts. Menu names and defaults change between versions; if something does not match, follow the current official documentation.