Ollama, vLLM or llama.cpp: which should a company use to run local AI?

To test a model on your own computer, use Ollama; to run on a modest machine, on a Mac or with a model larger than the graphics card's memory, use llama.cpp; to serve dozens of people at the same time on a GPU server, use vLLM. All three are free and open source, and the choice depends less on which one is "best" and more on how many people will use the AI at the same time.

Have you ever installed an AI model on your laptop, been impressed, and then found out it choked when three colleagues used it together? This text explains what each tool does, where each one shines and which one we chose in a real technical assessment of local AI for a company.

All three do the same basic thing: they load an open language model and answer requests through an API, the door other systems use to talk to the model. The difference is who each one was built for.

Ollama. It is MIT licensed, installs on macOS, Windows and Linux or runs in Docker, and exposes a REST API on the machine itself. It is the fastest way to go from zero to a model answering. The official repository shows the model running with a few commands.

llama.cpp. It is also MIT licensed and written in C/C++. The project README says its goal is to run models with minimal setup and strong performance on a wide range of hardware. It treats Apple Silicon as a first-class citizen, supports 1.5-bit to 8-bit quantization, which shrinks the model to fit in less memory, and can split the model between CPU and GPU when it does not fit entirely on the card. Its server, llama-server, has an OpenAI-compatible API, parallel decoding for multiple users and continuous batching.

vLLM. It is Apache 2.0 licensed and describes itself as a high-throughput, memory-efficient inference engine. The README highlights PagedAttention, which organizes conversation memory in blocks, plus continuous batching of requests, prefix caching and multiple LoRA adapters on the same model. The technique was published in a 2023 paper by Woosuk Kwon and colleagues.

There is a line attributed to William Gibson, the author of Neuromancer: "the future is already here, it's just not very evenly distributed". Quote Investigator traced the line and concluded there is strong evidence that Gibson used it and coined it, although the earliest citations are not in his books and the wording varies.

Local AI is exactly like that. The model that impresses on one person's laptop is the future arriving. The trouble starts when it has to arrive equally for legal, finance and customer service at the same time. That is when the choice of model server stops being a technical detail.

A language model generates text token by token. When only one person is using it, almost any tool does fine. When many people ask at the same time, what decides is whether the server can group the requests into one batch without wasting memory.

A test published in the witcheer/rtx-5090-benchmarks dataset, with the Qwen3-14B model quantized to 4-bit AWQ on vLLM, on an RTX 5090, recorded 150 tokens per second with one request at a time, 1,188 with 8 requests together and 3,937 with 32. The same card delivered more than 26 times more text in total when it served 32 requests in a batch. That gain is what vLLM was built to exploit.

Ollama also serves requests in parallel, but with care. According to the official FAQ, the default is to process 1 request at a time per model (the OLLAMA_NUM_PARALLEL variable). When you raise that number, the memory needed grows with the context size, and requests that do not fit go to a queue of up to 512.

Ollama, to start and to develop. It is for a developer testing models, building a proof of concept or having an assistant on their own computer. It is the lowest-friction choice.

llama.cpp, for limited or mixed hardware. It makes sense on a Mac, on a machine without a strong graphics card or when the model is larger than the GPU memory and needs to use RAM. Together with llama-swap, an MIT tool that swaps models according to the request, you can have several models on the same machine without keeping them all loaded.

vLLM, to serve the company. It is the choice when the server has a dedicated GPU and needs to serve many people with fast answers. It also has a sleep mode that moves the model weights from the GPU to RAM, freeing the card for another task without restarting the server.

In October 2026 we carried out a technical assessment of local AI infrastructure for a server with a single RTX 5090 that had to serve employees across several departments. The recommendation was vLLM as the main server. The alternatives on record were SGLang and llama.cpp with llama-swap, which is simpler but less efficient with many users. Ollama was kept for development only.

The reason was the count of people at the same time. For the chosen model, a quantized Gemma 4 26B-A4B, our estimate was 16 to 32 answers being generated in parallel, which would depend precisely on vLLM's continuous batching and memory management. That estimate has to be measured with real questions before any purchase decision, and the plan included a load test going from 1 to 48 simultaneous users.

In front of vLLM sat a model router, LiteLLM, which gives each department its own key and spending limit and sends the request to a cloud service when the GPU is busy. Since all three servers speak the same OpenAI API language, swapping one for another later does not force you to rebuild the systems that use the AI.

Can I start with Ollama and migrate later? Yes. Since Ollama (OpenAI compatibility), llama.cpp and vLLM offer OpenAI-style APIs, the systems that talk to the model change little. What changes is the server configuration.

Does vLLM run on any computer? It was designed for servers with GPUs. For a Mac or a machine without a strong card, llama.cpp is usually the most direct path.

Which one is the fastest? It depends on the scenario. With one person using it, the difference is usually small. With dozens of people at the same time, continuous batching makes the biggest difference, which is why the test has to imitate real use.

DOUVRAS Labs chooses the model server based on measured use, not on hype: we compare models with real questions in Portuguese, run the load test, set up routing with spending limits and connect the AI to the company's documents and systems. It is the work we offer in local AI for legal and accounting firms, AI agents and assistants and automation and integrations.

And in your company, has the AI that works on one person's computer been tested with the whole team using it at once?

Douglas Henrique, DOUVRAS Labs.

Ollama. Official repository and FAQ (How does Ollama handle concurrent requests?). github.com and docs.ollama.com

ggml-org. llama.cpp: LLM inference in C/C++. github.com

vLLM Project. vLLM: a high-throughput and memory-efficient inference and serving engine for LLMs. github.com

Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023. arxiv.org

mostlygeek. llama-swap. github.com

witcheer. RTX 5090 benchmarks, report FP4 on consumer Blackwell. Hugging Face, 2026. huggingface.co

vLLM. Sleep Mode. docs.vllm.ai

LiteLLM. Proxy: users, budgets and rate limits. docs.litellm.ai

Quote Investigator. The Future Has Arrived, It's Just Not Evenly Distributed Yet. 2012. quoteinvestigator.com

Gibson, William. Neuromancer. Ace Books, 1984.