Local AI Document Search on CachyOS with Ollama + Open WebUI

Run a private AI on your own machine that can read your PDFs, notes, and files and answer questions about them — no cloud, no subscriptions, nothing leaving your computer. This guide walks through the whole setup from a fresh CachyOS install using Ollama (the model runtime) and Open WebUI (the browser interface), wired together for document search (RAG).

The technique is called RAG — Retrieval-Augmented Generation. Instead of hoping the model already knows your documents, the system searches your files for the relevant passages and hands them to the model as context, so answers are grounded in your data.

How the pieces fit together

A local RAG setup has three layers, and it helps to know which does what before installing anything:

The embedding model turns your documents into vectors so they can be searched by meaning. This is the layer that actually finds the right passages.
The vector store holds those vectors. Open WebUI includes one, so there’s nothing extra to install.
The chat model (LLM) reads the passages it’s handed and writes the answer.

The counter-intuitive part: for document search, retrieval quality matters more than raw model size. A small, fast chat model paired with a good embedding model beats a huge model with poor retrieval. That’s why this guide uses a small chat model — it’s the right choice, not a compromise.

Prerequisites

A working CachyOS (or any Arch-based) system.
Around 8 GB of free RAM for the small models below; more is better.
Comfort with a terminal and sudo.
An AUR helper — CachyOS ships paru by default (used below). yay works identically.

Step 1 — Install Ollama

Pick the package that matches your hardware. Check whether you have an NVIDIA GPU, an AMD GPU, or neither, then install one of these:

sudo pacman -S ollama-cuda    # NVIDIA GPU
sudo pacman -S ollama-rocm    # AMD GPU (ROCm)
sudo pacman -S ollama         # CPU only, no GPU

On modern AMD cards (RDNA 3 and newer), ollama-vulkan is often faster than ROCm and avoids ROCm’s unsupported-GPU headaches — worth trying instead of ollama-rocm if you’re on recent AMD hardware. A GPU is recommended but not required; the small models here run fine on CPU, just slower.

Enable the service so it starts on boot and runs in the background:

sudo systemctl enable --now ollama

Verify it’s up. Ollama listens on port 11434:

curl http://localhost:11434
# should print: Ollama is running

Ollama runs as its own ollama system user and stores models in /usr/share/ollama.

Step 2 — Pull your two models

You need one chat model and one embedding model:

ollama pull qwen3:4b            # the chat model (fast, small, good at grounded answers)
ollama pull nomic-embed-text    # the embedding model (does the actual document search)

Together these are only about 2.8 GB. nomic-embed-text is the community-standard local embedder in 2026 — small, fast, and purpose-built for search.

If you want something even lighter for the chat model, qwen2.5:3b or llama3.2:3b are excellent and a touch faster. Going below the ~3B class starts to hurt answer quality on document questions, which is exactly what you don’t want here.

Confirm both landed:

ollama list

Step 3 — Install Open WebUI

On Arch/CachyOS, Open WebUI is packaged in the AUR and set up as a proper systemd service:

paru -S open-webui

Enable and start it:

sudo systemctl enable --now open-webui.service

Important: the first start is slow. The service fetches a Python runtime and downloads a default embedding model from Hugging Face before it will answer — this can take a few minutes, during which the browser will say “connection refused.” That’s normal on first boot. Watch it come up:

journalctl -u open-webui -f

Wait for a line like Application startup complete or Uvicorn running on http://0.0.0.0:8080, then press Ctrl-C to stop watching the log (the service keeps running). Confirm it’s listening:

ss -tlnp | grep 8080

Open WebUI serves on port 8080 (this AUR package hard-codes 8080; it can’t easily be changed because it’s wrapped with uv). Its data — settings, accounts, chats, knowledge bases, and the vector store — lives in /var/lib/open-webui.

Step 4 — Create your account

Open http://localhost:8080/ in your browser. You’ll get a first-run signup screen. The first account you create becomes the admin — make it yours. There’s no email verification; it’s all local.

Once in, Open WebUI should already see your Ollama models in the model dropdown, since both run on the same machine.

Step 5 — Point Open WebUI at the good embedder

Open WebUI defaults to a small built-in embedder. Switch it to the nomic-embed-text you pulled:

Go to Settings → Admin → Documents.
Set Embedding Model Engine to Ollama.
Set Embedding Model to nomic-embed-text.
Click Save.

While you’re on this page, two settings worth knowing:

Chunk Size / Overlap (defaults around 1000 / 100) control how documents are split before embedding. The defaults are a fine starting point.
Hybrid Search (toggle it on if your files are technical) combines semantic search with keyword matching, which greatly helps with exact terms — part numbers, error codes, function names — that pure semantic search can miss.

One-way door: if you ever change the embedding model later, you have to re-upload your documents. Old vectors were made by the old embedder and aren’t comparable to new ones, so retrieval quietly degrades until you re-index. Set your embedder before loading files and leave it alone.

Step 6 — Raise the context window (the #1 RAG mistake)

This step is easy to skip and it’s the most common reason local RAG gives bad answers.

Ollama defaults to a small context window (historically 2048 tokens). RAG stuffs your retrieved passages plus your question plus the system prompt into that window — so with a small window, the retrieved text gets silently truncated and the model answers from almost nothing.

Fix it by baking a bigger context into a custom model. Create a file called Modelfile:

FROM qwen3:4b
PARAMETER num_ctx 8192

fish shell users: fish doesn’t support bash’s <<'EOF' heredoc. Just create the file in a text editor, or use printf 'FROM qwen3:4b\nPARAMETER num_ctx 8192\n' > Modelfile.

Then build the tuned model (this copies no weights — it just adds a small manifest, so it’s instant and uses no extra disk):

ollama create qwen3-4b-rag -f Modelfile

Use qwen3-4b-rag as your model from now on. 8192 is a good balance for a small model on a laptop.

Step 7 — Build your knowledge base

In the sidebar, go to Workspace → Knowledge → Create. Give it a name (e.g. “My Documents”).
Inside it, use the + to Upload File or Upload Directory. Open WebUI extracts the text, chunks it, and embeds each chunk with nomic-embed-text.
Start with just a couple of files to confirm the pipeline works before loading everything — it’s much faster to spot a problem that way.

Scanned PDFs: if a file is an image-only scan (photographed pages, no selectable text), the default extractor gets nothing from it. For those, turn on PDF Extract Images (OCR) in Settings → Documents, or switch the Content Extraction Engine to an OCR-capable one.

Step 8 — Ask your documents a question

Two ways to use the knowledge base:

Per chat: in any new chat, type #, pick your collection, then ask.
Bound to a model: in Workspace → Models, edit qwen3-4b-rag and attach the knowledge base under Knowledge Source so it’s always available.

Test it with a question that can only be answered from your files — a specific number, a name, a line from a document. If the model could answer from general knowledge, the test proves nothing. A correct, specific answer means retrieval is working end to end.

Troubleshooting

“Connection refused” at localhost:8080 — Almost always the service is still booting (especially the slow first start) or isn’t running. Check systemctl status open-webui, start it if needed, and watch journalctl -u open-webui -f for the startup-complete line before reloading.

It’s on 8080, not 3000 — Docker installs of Open WebUI commonly use port 3000; this AUR/systemd install uses 8080. If you’re following API examples from elsewhere, change the port to match.

The model gives vague or made-up answers — Confirm you’re using the num_ctx-bumped model (qwen3-4b-rag), not the stock one. The stock model truncates retrieved passages. Also check the knowledge base is actually attached to the chat.

“The content provided is empty” when adding files via the API — Uploads process asynchronously; the file needs a few seconds to finish extracting and embedding before it can be added to a knowledge base. Wait, then retry.

A note on security

This setup is private by design — everything runs on localhost. Two things to keep it that way:

Ollama’s API has no authentication. By default it only listens on 127.0.0.1, which is what you want. Do not set OLLAMA_HOST=0.0.0.0 unless you deliberately want other machines to use it — and if you do, put it behind a firewall, because anyone who can reach the port can use your models.
Keep Open WebUI on localhost too, or put a reverse proxy with real authentication in front of it before exposing it beyond your machine.

Optional — Store models somewhere else

Ollama defaults to /usr/share/ollama. To keep models on a different drive or in your home directory, point OLLAMA_MODELS at a new path via a service override:

sudo systemctl edit ollama

Add (adjust the path and, on a single-user machine, the user):

[Service]
Environment="OLLAMA_MODELS=/your/new/path/models"
User=youruser
Group=youruser

Move any existing models to the new path, make sure the directory is owned by the user the service runs as, then sudo systemctl daemon-reload && sudo systemctl restart ollama. Note that pointing into your home directory means the service user must be able to read it — running the service as your own user (as above) avoids cross-user permission problems.

Setup: CachyOS · Ollama + Open WebUI (AUR) · qwen3:4b + nomic-embed-text. Everything local, everything private.

Install Local AI On CachyOS