Hardware Detection

Octomil detects your GPU, CPU, and memory automatically on every serve and pull. This drives the model optimizer to pick the right quantization and offload strategy.

Supported Hardware

Backend	GPUs
NVIDIA CUDA	RTX 5090-2060, GTX 1080 Ti-1650, H200/H100/A100/L40S/T4/V100, Jetson (AGX Orin, Xavier, TX2, Nano)
AMD ROCm	RX 9070 XT/9070, 7900 XTX-7600, 6950 XT-6600, MI300X/MI250X/MI210/MI100
Apple Silicon	M1-M4 (all variants). Unified memory treated as VRAM minus 4 GB OS reserve
CPU	x86_64 (AVX-512, AVX2), ARM64 (NEON)

Multi-GPU setups sum VRAM across all GPUs.

Troubleshooting

If your GPU isn't detected, Octomil logs the reason. Common issues:

• NVIDIA: nvidia-smi not in PATH, or driver not installed
• AMD: ROCm not installed. Octomil falls back to lspci and sysfs, but VRAM may be estimated
• Jetson: Detected via /etc/nv_tegra_release or device tree. Ensure Tegra drivers are installed