What determines the cost of running a local LLM on an RTX 5090?

The cost of running a local LLM on an RTX 5090 is not the card: it is the combination of video memory, RAM, energy, the number of people using it at the same time and the operating hours the server demands. A 5090 has 32 GB of memory and serves a mid-size model to several users well, but it does not run heavy text, image and video models at the same time, and that limit is what weighs most in the math.

Have you ever seen someone buy the most powerful machine on the market and find out later that it does not do what they needed? This article shows the factors that define the real cost, with measured numbers and estimates marked as estimates, based on a technical survey of local AI infrastructure we carried out in October 2026 for a server with a single RTX 5090.

According to NVIDIA's spec sheet, the RTX 5090 has 32 GB of GDDR7 memory, a total graphics power of 575 W and requires a 1,000 W system power supply. Those three numbers already say a lot: what fits in memory, how much energy goes into the bill and what kind of machine has to surround the card.

In practice, a mid-size text model takes up a good part of that memory. Gemma 4 26B-A4B, an open model under the Apache 2.0 license, takes about 17.2 GB in the 4-bit quantized version published on Hugging Face. Adding the memory the server uses to hold the context of conversations, our estimate in the survey was about 25 GB in use. A good-quality image model needs 20 GB or more. The two do not fit together on the same card.

In Neuromancer, by William Gibson, the hacker Case lives for his deck, an Ono-Sendai Cyberspace 7. The book spends little time admiring the device and a lot of time showing what it demands: access, people, time and risk. The deck is the visible part. The rest is operations.

Local AI works the same way. The card is the purchase that shows up on the invoice. The real cost shows up later, when you have to decide what stays on during business hours, what goes into a queue at night, what happens when the server restarts and who takes care of it.

GPU memory. It defines which model fits and how many conversations run at the same time. With a single card, the solution we recommended was to split the day into modes: a large text model during business hours, a smaller model and light image generation in the evening, and the whole card for image, video and training overnight.

Server RAM. A less obvious bottleneck. In the survey, 64 GB of RAM proved insufficient for heavy video and for the sleep mode of the model server. vLLM, the model server we use, has a mode that moves the weights off the GPU into RAM to free the card in seconds, but that only works with RAM to spare. That is why our recommendation was to move to 128 GB before thinking about a second card.

Energy and infrastructure. 575 W on the card alone, a compatible power supply, cooling and a server that stays on all day.

Training. Adapting the model to the company's tone does not always fit on the machine. According to Unsloth's documentation, training a LoRA adapter on Gemma 4 26B-A4B needs more than 40 GB of memory, while Gemma 4 31B with QLoRA fits in 22 GB. When it does not fit, the way out is to rent a larger GPU for a few hours.

Licenses. Not every open model can be used commercially. Some well-known image models have non-commercial licenses, and others are only free below a certain revenue. This has to be checked before going into production.

Operations. Someone has to update, monitor, measure and respond when something breaks. This is usually the most forgotten line in the math.

It depends on the model, the size of the questions and how many people ask for an answer at the same time. A test published by Patrick Gawron with Qwen3.6-35B-A3B on vLLM, on an RTX 5090, recorded about 105 tokens per second with one conversation and 160 in total with two.

For Gemma 4 26B-A4B, our estimate was 16 to 32 answers being generated at the same time. If at peak only a small share of logged-in users is waiting for an answer, that serves somewhere between 100 and 250 active employees. It is an estimate with a stated premise, and the premise has to be measured against real usage. The comfort criterion we adopted was the first word of the answer arriving in under 2 seconds for 95% of requests and generation staying above reading speed.

The rule we proposed was not to buy on intuition. The second card comes in when one of these signals holds for two to four weeks: the wait for the first word frequently exceeds 3 seconds at peak; marketing needs to generate images and video during the day and the overnight queue does not solve it; the monthly spend on cloud services used as a fallback exceeds the monthly amount of amortizing a card over 24 to 36 months, including energy; or the data cannot leave the company and the large model has to be available all the time.

Donald Knuth wrote in 1974 that we should forget about small efficiencies about 97% of the time, because "premature optimization is the root of all evil" (Structured Programming with go to Statements). Buying hardware before measuring usage is the expensive version of that.

The first step we recommend is a proof phase of about two weeks: install the model server, compare two or three models with more than 100 real questions in Portuguese and run a load test increasing the number of simultaneous users. Alongside that, a model router such as LiteLLM lets you set a spending limit per department and send the request to the cloud when the GPU is busy or down, with personal data masked first.

At the end of this phase, the company knows which model to choose, how many people the card can handle and whether more RAM or a second GPU is needed. The purchase decision is then made with numbers.

Is one RTX 5090 enough for a company? For a mid-size text model with a few dozen people using it at the same time, our estimate says yes. For heavy text, image and video together, no: you need to schedule usage or have another card.

Is it worth training your own model? Rarely as a first step. What changes often, such as prices, stock and policy, should live in documents the AI consults, not in training. Fine-tuning is more useful for tone of voice and format.

Can you just use the cloud and forget the server? Yes, and sometimes that is the best path. A local server makes sense when the data cannot leave the company or when the volume of use makes the cloud more expensive than the machine.

DOUVRAS Labs does this sizing before the purchase and builds the operation afterwards: model server, GPU time scheduling, routing to the cloud with spending limits, document base and agents for each department. It is the same work we offer in local AI for law and accounting firms, AI agents and assistants and automation and integrations.

And in your company, has anyone measured how many people would use AI at the same time before choosing the machine?

Douglas Henrique, DOUVRAS Labs.

NVIDIA. GeForce RTX 5090, specifications. nvidia.com

cyankiwi. gemma-4-26B-A4B-it-AWQ-4bit. Hugging Face. huggingface.co

vLLM. Sleep Mode. Documentation. docs.vllm.ai

Unsloth. Gemma 4: how to train. Documentation. unsloth.ai

Gawron, Patrick. Qwen3.6-35B NVFP4 on vLLM on an RTX 5090, 2026. patrickgawron.com

Knuth, Donald E. Structured Programming with go to Statements. ACM Computing Surveys, 1974. dl.acm.org

Gibson, William. Neuromancer. Ace Books, 1984.

LiteLLM. Users, teams and budgets. Documentation. docs.litellm.ai