THE TRUTH ABOUT: Planning gpu memory for a private rag service - Caught on Camera

A private retrieval augmented generation service often begins with a model that runs comfortably on a developer workstation. The difficulty appears when several employees submit long questions, the retrieval layer adds supporting documents, and the application must keep responding while an indexing job is running. Loading the model is only the first capacity test.

For a production RAG service, GPU memory planning should account for model weights, the attention cache, runtime overhead, and concurrent requests. System RAM, storage, and CPU capacity need their own budgets. Treating these resources separately makes it easier to identify whether a proposed server fits the application or merely fits the model file.

When comparing dedicated AI server infrastructure, begin with a workload description that a provider can evaluate: model version, serving framework, expected prompt lengths, target concurrency, and acceptable response time. A useful configuration discussion starts with those requirements, before anyone chooses a GPU from a product list.

Map the request before sizing the GPU

A RAG request can involve authentication, document retrieval, permission filtering, reranking, prompt assembly, inference, and output validation. Not every step belongs on the same accelerator. A vector search service may primarily need RAM and storage, while a reranker can introduce a separate CPU or GPU requirement.

Draw the request path and record which components remain active simultaneously. If the embedding model runs on the inference GPU, its memory use must be included even when document ingestion is an occasional task. If embedding runs elsewhere, network transfer and queueing still affect the time a user waits.

Also distinguish ingestion from retrieval. Importing a large document collection may involve parsing, chunking, and embedding thousands of files. Allowing this work to compete freely with interactive questions can create a slowdown that looks like an inference problem. Separate schedules or resource limits may solve it without changing the language model.

Build a memory budget with explicit assumptions

The raw weight estimate is parameter count multiplied by bytes per stored parameter. For illustration, eight billion parameters stored at two bytes each represent approximately 16 billion bytes, or 14.9 GiB. This calculation is not the full memory requirement and does not describe every model architecture or loading format.

Quantization can reduce weight storage, but metadata, unsupported operations, and runtime allocations complicate the final footprint. Measure the deployed model with the actual inference engine. Do not approve a server solely because a downloaded checkpoint is smaller than its advertised GPU memory.

The key value cache stores information used during attention computation. Its footprint depends on model architecture, token counts, precision, and the serving implementation. Longer active sequences and more simultaneous requests can consume memory that looked spare during a single short demonstration.

The vLLM serving documentation describes controls for GPU memory utilization and cache allocation. These settings help manage an available memory budget; raising a utilization limit does not create physical capacity. Review the documentation for the version deployed, because defaults and supported features can change.

Measure useful context rather than maximum context

An application does not need to send every retrieved passage to the model. Excess context can increase memory use and response time while adding irrelevant information. Establish how many passages are normally required, how large they are, and how permission filtering affects the result set.

Build a test collection with short questions, long conversations, and difficult searches that retrieve unusually large documents. Include the system instructions and conversation history when measuring prompt length. Looking only at the latest user message understates the input that actually reaches the model.

A context policy should define truncation behavior. For example, the application may preserve instructions, keep the most recent conversation turns, and include the highest ranked authorized passages within a fixed token allowance. Validate that the policy still produces acceptable answers. A capacity saving that removes essential evidence is not an improvement.

Output limits matter as well. Generating a short answer and generating a lengthy report create different service demands. Give each product feature a sensible output budget rather than exposing the model maximum everywhere. Explain limits in the interface so users understand when a long request should become a background job.

Test concurrency as a traffic pattern

Ten requests arriving together are different from ten requests spread across a minute. A realistic benchmark should reproduce bursts, pauses, and a mixture of prompt sizes. Otherwise, a configuration can pass an average throughput test while delivering poor response times during normal working hours.

Track time to first token, total completion time, successful requests per minute, and the proportion of requests that exceed the application deadline. For streaming interfaces, a quick first token is useful, but the full answer still needs to arrive within a reasonable time.

Use a small progression of tests:

• Establish the single request baseline with the final model and prompt format.
• Increase simultaneous requests while keeping the test set unchanged.
• Repeat with long contexts and the expected output lengths.
• Add retrieval, reranking, and authentication to the measured path.
• Run a sustained test long enough to expose memory growth and queue buildup.

Record rejected requests and retries rather than dropping them from the results. A benchmark that reports only completed requests can hide overload. The system also needs a policy for a full queue: reject promptly, apply a deadline, or offer an asynchronous workflow.

Give the rest of the server a separate budget

System RAM supports the operating system, application processes, search indexes, and data preparation. It is not interchangeable with GPU memory. Some serving approaches can offload work or weights to system memory, but the performance consequences must be tested with the intended hardware and framework.

Storage capacity should include model versions, document collections, indexes, logs, and temporary files. Keep enough room to stage a replacement model while the previous version remains available for rollback. A disk sized only for the active checkpoint can make routine updates unexpectedly difficult.

Storage speed also affects recovery. A service may perform well after warming up but take too long to load after a restart. Measure cold start time, index initialization, and the interval until the health check represents genuine readiness. A process listening on a port is not necessarily ready to answer questions.

CPU capacity deserves the same attention. Tokenization, parsing, TLS, and orchestration can become limiting factors even while the GPU has spare capacity. Look at utilization alongside queue lengths and latency to avoid assigning every slowdown to the accelerator.

Keep private data boundaries visible

Dedicated hardware can provide control over where a workload runs, but privacy depends on the complete application. Document sources, temporary files, telemetry, support access, and backups may all contain sensitive information. Map them before deciding that a deployment is private.

For retrieval, access controls must apply before passages reach the model. Test with two users who have different document permissions and confirm that neither retrieved text nor generated answers expose restricted material. Capacity testing should use sanitized data or an approved test dataset.

Logging needs restraint. Full prompts are convenient during development, but retaining them indefinitely can create an additional sensitive dataset. Decide which diagnostic fields are necessary, who can access them, and when they are deleted. Preserve enough information to investigate failures without casually duplicating customer documents.

Approve the configuration against a repeatable test

Test the failure behavior of the document layer as well. If retrieval returns nothing, the application should follow an agreed response policy rather than generate an apparently supported answer from an empty evidence set. If one document source is temporarily unavailable, record whether the service can continue with reduced coverage. These cases affect both the usefulness of the product and the infrastructure signals operators need. A healthy inference process can coexist with a broken retrieval pipeline, so readiness checks should represent the full service rather than only the model process.

Before committing to a server, write a short acceptance sheet containing the model identifier, framework version, test dataset, traffic pattern, latency targets, and observed peak memory. Include the date, because later model or application changes can invalidate the result.

Ask the provider to confirm the specific GPU model, usable memory, CPU and RAM configuration, storage, network terms, and available support scope. Multi GPU capacity also requires a serving design that can use it; separate GPU memory pools should not be treated as one automatically available pool.

The final decision should explain what the configuration can support today and what will trigger an upgrade. That trigger might be sustained queue growth, insufficient memory for a required model, or missed response deadlines. With those boundaries documented, a private RAG service can grow through measured changes instead of repeated guesses about a larger GPU.