Private AI Chat for Company Documents: A RAG Deployment

• RAG
• FastAPI
• ChromaDB
• Ollama
• Redis
• ARQ
• Docker Compose
• Nginx
• Let's Encrypt
• Google Cloud
• OpenRouter
• DeepSeek
• SQLite
• Tailwind CSS

The goal

A real estate developer wanted people to ask questions in plain English and get answers from the company's own data: sales leads, customer queries, finance records, inventory and engineering documents. Every answer had to show where it came from, and the data had to stay on a server the company controls.

The system is built on the EuroRAG starter kit (I compare it with RAGFlow in Part 1 of my Production RAG series). My job was to turn that starter kit into a working, branded deployment: server, data sources, tuning and the problems that only show up once real data goes in.

demo documents indexed (PDF, DOCX, TXT)

33

demo database rows across 14 tables

748

Docker services behind one reverse proxy

5

fixes to the starter kit's connectors

3

How it answers

The AI does not answer from memory. A question is turned into numbers, the vector database finds the most similar pieces of the company's documents, and only those pieces go to the language model with the question. The answer streams back with the source documents listed.

Browser → Nginx (HTTPS) → App (FastAPI)
                            │  question → embedding (Ollama) → search (ChromaDB)
                            │  top matching chunks + question → chat model
                            ▼
                         answer + sources

Worker (ARQ) ← Redis queue: reads documents, splits them, creates embeddings, stores them

The search combines meaning-based and keyword search, and the chat runs in a strict mode: it answers only from the retrieved text, and says it has no information when the documents do not cover the question. That is the same idea I explain in Part 4 of the series.

The stack, and why

Everything runs as Docker containers on one Google Cloud server, with only the reverse proxy open to the internet. The app, the background worker, the vector database, the embedding model and the queue each have their own container, and they restart on their own after a reboot.

• Embeddings run locally. The embedding model runs on the server through Ollama, so the documents are not sent anywhere to be indexed.
• The chat model is a hosted API. The server has no GPU. A 7B model on the CPU took minutes to answer and timed out, while a hosted model through OpenRouter answers in a few seconds. Changing the model is one setting.
• HTTPS with automatic renewal through Nginx and Let's Encrypt.

Connecting the data

Three kinds of source, all from the admin screen:

• A folder of files. PDF, Word, Excel, PowerPoint, CSV and text files, synced on a schedule. The demo uses 33 documents across three folders: company reports and procedures, market research and department FAQs.
• A database. SQLite, PostgreSQL and MySQL are supported. The demo uses a SQLite file with 14 tables (leads, customer queries, bank reconciliation, inventory, drawings and more). Each row becomes its own searchable document, written out as plain field: value text.
• A website crawler. It follows links from a start page to a set depth, only on domains the admin has approved.

Making it the client's own

I did not edit the starter kit's templates. It loads a custom folder first, so the login page, the navigation, the logo, fonts and colours all live in one folder outside the core code. The core pages use a default blue for every button and focus ring, so I remapped that one colour scale in the base template instead of changing eight core files. Every page extends that template, so the whole app took on the brand colours at once, and future updates of the starter kit will pick them up too.

Problems I hit and fixed

Database rows were rejected as unsupported files

Symptom

Every database row showed as Failed with an 'unsupported file type' error.

Root cause

Each row was saved as a document with no file extension, and the document processor refuses a blank one.

Fix

Gave each row document a .txt name. Old failed rows stayed in the list, so I told the client they were safe to delete.

A local database was blocked by the EU-only check

Symptom

The SQLite source was refused when the strict EU setting was on.

Root cause

The check looked for an EU domain in the connection string, and a local file has no domain at all.

Fix

Treated a local SQLite file as compliant, since its data never leaves the server.

The crawler crashed on its own depth setting

Symptom

A crawl failed with a type error as soon as it started.

Root cause

The admin screen saved the maximum depth as text, and the crawler compared it with a number.

Fix

Converted the depth and page limit to numbers when the crawler reads them.

The worker showed as unhealthy while working fine

Symptom

The worker container was marked unhealthy in the status list even though it was processing jobs.

Root cause

The default health check calls a web address, and the worker is a background job runner with no web server.

Fix

Replaced the check with a simple test that the main process is alive, in an override file that has to be recreated on a fresh server.

Per-user document privacy had no effect

Symptom

After setting documents to be visible per user, a new account's chat still returned another account's documents as sources.

Root cause

A value in the settings file only reaches a container if the compose file lists it, and this one was missing for both the app and the worker. The app silently used its default, which searches everyone's documents.

Fix

Added the variable to both services, then checked it from inside the running container.

Tuning the answers

• How many chunks to retrieve. With the starting value of 5, a question about customer queries missed relevant rows, because 70 similar rows compete for 5 places. Raising it to 15, and later tuning it further, fixed those misses.
• Chunk size. Chunks are capped at 512 tokens, with a mix of splitting strategies.
• Strict mode. The model answers only from what was retrieved.

Two limits I documented for the client instead of hiding them:

• Counting does not work. Search returns the closest chunks, not every row, so a question like "how many leads are there?" cannot be answered reliably. Totals and sums belong in the database.
• Questions outside the data return nothing. The demo data ends in June 2026, so "this month" finds nothing. Asking for a named month works.

What I did

• Set up the server, containers, reverse proxy and HTTPS, and wrote the setup and operations guide the client's team uses.
• Loaded the demo documents and database, and tuned retrieval.
• Customised the look of the app without touching the starter kit's core templates.
• Found and fixed the problems above.
• Made the demo video: the script, the cloned voice, the animation and the thumbnail.

Related work

The ideas behind this project are in my free Production RAG video series, and I build assistants on real data in Milo, a real estate assistant and self-hosted AI agents for a team chat.

Want your documents to answer questions?

If your company's knowledge is spread across folders, files, databases and websites, I can set up a private chat like this, with the sources shown for every answer. Tell me about your documents on the contact page.