Local AI Homelab System
Project Overview
This project is a personal homelab initiative designed to transform consumer-grade hardware into a secure, manageable, and sustainable local artificial intelligence infrastructure.
My goal was not simply to run a few AI models. I wanted to manage devices with different operating systems within a unified architecture, test GPU-based workloads, provide secure remote access, and build the entire system in a way that is documented, reversible, and maintainable over the long term.
System Architecture
The system consists of several devices, each with a specific role:
- MacBook: Primary management and development device
- Raspberry Pi 400: Central control and coordination node
- PC1: Secondary GPU worker and test system
- PC2: Primary local AI and GPU worker
- Ubuntu laptop: Lightweight auxiliary node
Secure connectivity between devices is provided through Tailscale. No router ports are forwarded, and SSH, RDP, and web services are not exposed directly to the public internet.
Using the custom lab and homelab commands I developed, I can manage device status, active operating systems, GPUs, Docker services, network services, and power operations from a central interface.
MacBook → Tailscale → Raspberry Pi 400 → PC1 / PC2 / Laptop
Primary AI Worker: PC2
The main local AI environment runs on PC2.
Hardware
- Intel Core i5-10600K
- NVIDIA GeForce RTX 3080 with 10 GB VRAM
- 16 GB RAM
- 1 TB SSD for Ubuntu
- 500 GB NVMe drive for Windows
- ASUS PRIME B460-PLUS motherboard
PC2 runs Ubuntu 24.04 with Docker, NVIDIA Container Toolkit, and Ollama. A software-enforced 240 W GPU power limit is applied to support stable and sustainable long-running workloads. I also developed fan control, temperature monitoring, and periodic telemetry services.
Services are not left running continuously. AI containers are stopped when they are not needed, and unnecessary network ports remain closed.
Technologies Used
The main technologies used in the project include:
- Linux and macOS
- Docker and Docker Compose
- NVIDIA Container Toolkit
- Ollama and Open WebUI
- Tailscale, SSH, and RDP
- Bash and Python
- Git and GitHub
- systemd services and timers
- Beszel and custom telemetry scripts
Local Model Evaluation
Approximately 30 local models were downloaded and tested on PC2 for different use cases. These included models from the Gemma, Qwen, DeepSeek, Granite, Llama, Mistral, Phi, and DeepCoder families, as well as embedding and vision models.
I did not evaluate the models solely on speed. I developed a custom Python-based evaluation system to compare them across areas such as:
- Code generation and code correction
- JSON and structured output
- RAG and source-grounded answering
- Language and reasoning tasks
- Agent planning
- Compliance with safety policies
- Response consistency
- Latency and resource usage
A campaign consisting of 12 models and 40 tasks produced a total of 480 model outputs.
Initial results showed that Granite models performed strongly in general stability and JSON output, while Qwen Coder models were particularly effective in coding tasks. Rather than identifying a single “best” model, the tests demonstrated that different models are better suited to different types of work.
System Issues and Troubleshooting Approach
The project involved not only AI model testing but also real infrastructure and hardware-related issues.
For example, PC2 occasionally experienced a rare physical power-off problem. To investigate it, I developed logging and evidence-collection tools covering systemd, ACPI, PCIe, GPU, network adapter, and Wake-on-LAN layers.
Problems were not addressed through random configuration changes. They were investigated through controlled testing, single-variable experiments, log analysis, and reversible modifications.
The core working principle of the project is:
Observe first, make a small change, verify the result, and document it.
Outcome
This project gave me practical experience not only in running local AI models, but also in designing and operating an end-to-end technical system.
It combines skills in:
- Linux system administration
- GPU and hardware optimization
- Docker-based service architecture
- Secure networking and remote access
- Python and Bash automation
- AI model evaluation
- Monitoring and telemetry
- Incident debugging
- Git-based documentation and change management
The system can now be managed securely from remote locations. PC1 and PC2 can be used as GPU workers, while PC2 includes a local model cache, evaluation tools, telemetry, fan control, and GPU power protection.
The next goals are to improve task-based model selection, build a RAG-based personal knowledge system, and develop secure local agents that initially operate with read-only and recommendation-level permissions.
Project type: Personal Homelab / Local AI Infrastructure
Focus areas: Linux, Docker, NVIDIA GPU, Local LLMs, Automation, Monitoring, Secure Remote Access