What it is
llama.cpp is an open-source inference engine, written in plain C/C++, that runs large language and vision models with minimal setup. It uses the GGUF model format and 1.5-bit to 8-bit quantization to fit models into less memory, and supports Apple Metal, CUDA, AMD HIP, Vulkan, SYCL and many other backends, including splitting work between CPU and GPU. It ships llama-cli for the terminal and llama-server, an OpenAI-compatible API server. Many local AI apps are built on it.
It is free under the MIT license. The team behind it, ggml.ai, joined Hugging Face in February 2026, and the project remains open source. It is a developer tool: you compile or download builds and work from the command line.
Best for
- Running models on CPUs, laptops and consumer GPUs
- Developers who want full control over local inference
- Serving GGUF models through an OpenAI-compatible API