01
Guided first run
Create or join from one launcher. The saved role starts automatically on later launches.
Low-power distributed inference
Use the computers you already have.
Distributed LLM Universal connects compatible Windows and Linux machines into one llama.cpp inference cluster. Run the source directly or use the admin-free portable worker, approve machines, and launch a GGUF model from the dashboard.
One command on every machine
$ python run.py
Choose:
1. Create a cluster
2. Join a cluster
Coordinator dashboard:
http://YOUR-IP:7000
What is included
A practical control layer around native inference.
The management app stays lightweight. Model execution is delegated to official, hardware-optimized llama.cpp binaries, while portable workers use an authenticated outbound TLS tunnel.
02
Mixed hardware
Detects operating system, architecture, CPU features, RAM, GPUs, and network address.
03
LAN discovery
Finds coordinators automatically, then protects enrollment with a join code and approval.
04
Native GGUF runtime
Downloads a suitable CPU, Vulkan, CUDA, or ROCm backend and runs quantized GGUF models.
05
Smart placement
Benchmarks workers and creates a RAM- and performance-weighted tensor split.
06
Admin-free workers
Extract and run without Python, sudo, services, inbound worker ports, or an installer.
Quick start
From extraction to chat.
Start the coordinator
Run python run.py and choose Create a cluster.
Join the workers
Run the normal app or extract the portable worker, enter the coordinator address and six-digit code, then approve it.
Run a model
Approve workers, install the backend, choose a GGUF model, then select Download & run.
Network safety: portable workers encrypt RPC through an authenticated outbound tunnel and do not expose port 50052. Direct workers and the initial HTTP enrollment flow remain intended for a trusted LAN or private VPN.
Default catalogue
Start small, then use the cluster's measured capacity.
| Model | Quantization | Approx. weights | Use |
|---|---|---|---|
| Qwen3 0.6B | Q8 | 0.64 GB | Backend test |
| Qwen3 1.7B | Q8 | 1.83 GB | Very weak devices |
| Qwen3 4B | Q4_K_M | 2.50 GB | Low-power default |
| Qwen3 8B | Q4_K_M | 5.03 GB | Medium cluster |
| Qwen3 14B | Q4_K_M | 9.00 GB | Larger mixed cluster |
The dashboard adds runtime, context, and operating-system headroom before marking a model as suitable. More machines mainly add usable memory and throughput; they do not multiply the speed of one response.
Project history
The earlier research materials are still included.
The restored prototype site, architecture drawing, dashboard preview, and whitepaper document the original custom-transformer design. The current Universal release instead uses llama.cpp RPC for the model runtime; those older materials are preserved as project history, not as current setup instructions.
