Local LLMs are two different infrastructure decisions

“Run the model locally” sounds like one architecture. It is at least two.

The first is on-device inference: a model runs on a laptop, workstation, or edge system close to one user and their data. The second is self-hosted serving: an organisation runs models on its own accelerator infrastructure and exposes them as a shared service.

llama.cpp and vLLM map naturally to those different boundaries. They overlap, and neither is limited to one topology, but they optimise for different problems.

That distinction matters more than the shared “local LLM” label.

What local inference can improve

Local or self-hosted inference can reduce dependence on a commercial model API. It can keep prompts and retrieved documents inside a controlled environment, remove an external provider’s rate limit from the request path, and make a chosen model available when a third-party service is unavailable.

Those are capabilities, not automatic outcomes.

Privacy depends on the whole data path. Running model weights locally does not guarantee that an application is offline. Model downloaders, telemetry, retrieval connectors, agent tools, web search, and logging destinations may still cross the boundary. A useful privacy claim must describe where prompts, retrieved context, outputs, traces, and tool arguments travel.

Cost moves rather than disappears. A self-hosted stack avoids a provider’s per-token charge, but it introduces hardware, electricity, idle capacity, operations, upgrades, observability, and incident response. A laptop already owned by the user has a different cost model from a continuously provisioned GPU fleet.

Availability becomes your responsibility. Removing an external outage dependency also removes the provider’s managed redundancy. Local inference still has hardware limits, software failures, capacity queues, and model-loading time.

The benefit is control. Whether that control is cheaper or more resilient depends on demand and operating discipline.

llama.cpp: make the model fit the device

The original LLaMA research release was published by Meta in February 2023.¹ llama.cpp appeared shortly afterwards as a C/C++ inference implementation and has since expanded far beyond that initial model family.

Its core strength is broad, portable inference. The project documents optimised paths for Apple silicon, x86 instruction sets, and backends including CUDA, HIP, Vulkan, SYCL, and others.² This makes it suitable for laptops, workstations, heterogeneous environments, and edge systems where a large serving cluster would be disproportionate.

Quantization is central to that fit.

A model commonly stores each parameter in 16-bit floating point form. Quantization represents weights with fewer bits—such as 8, 6, 5, 4, or lower—reducing the memory and bandwidth required for inference.² Lower precision can affect output quality, and runtime memory includes more than the weights, so the smallest file is not automatically the best deployment.

The order of magnitude is still useful:

• An 8-billion-parameter model needs roughly 16 GB for raw FP16 weights.
• At a theoretical 4 bits per weight, the raw weights are roughly 4 GB, before format and runtime overhead.
• A 70-billion-parameter model is roughly 140 GB at FP16 and 35 GB at 4 bits, again before overhead.

This corrects a common misconception: ordinary 4-bit quantization can bring an 8B-class model near 4 GB, but it does not bring a 70B model to 4 GB.

Why GGUF matters

llama.cpp commonly uses GGUF, a file format designed for GGML-based inference. The GGUF specification describes single-file deployment, extensible metadata, and memory-mapped loading.³

In practical terms, one .gguf file can carry the tensors and the metadata needed by the runtime, including tokenizer and model information. That makes a quantized model easier to move between compatible machines and tools than a loose set of framework-specific artifacts.

The file is only the deployable unit. You still have to choose:

• A model whose licence permits the intended use.
• A quantization level that fits memory without unacceptable quality loss.
• A context length whose Key-Value cache fits alongside the weights.
• CPU, GPU, or hybrid offload settings that meet latency expectations.
• A prompt template and sampling configuration compatible with the model.

For one developer, a private document assistant, an offline code helper, or an edge application, those are often the right constraints to optimise.

vLLM: make shared inference efficient

vLLM starts from a different question: how should an accelerator serve many variable-length requests efficiently?

Its documentation supports online serving through OpenAI-compatible endpoints, including completions, chat completions, responses, and embeddings for applicable models.⁴ The hardware documentation covers NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, and other backends with feature-specific limitations.⁵

The shared-service problem is not only loading the weights. Each active sequence also creates a Key-Value cache, or KV cache: the attention keys and values retained from earlier tokens so the model does not recompute the entire sequence for every new token.

KV-cache memory grows with concurrent requests and sequence length. Poor allocation creates fragmentation and reduces the number of requests that can fit on an accelerator.

The vLLM research introduced PagedAttention, which stores KV-cache blocks in non-contiguous memory using an approach inspired by virtual-memory paging.⁶ This enables more flexible allocation and sharing of KV-cache blocks. The published evaluation reported higher throughput than the compared systems under its tested models and workloads; that benchmark should not be treated as a universal production guarantee.

vLLM also schedules work across active sequences. Often described as continuous batching, the engine can form batches from requests at different stages rather than waiting for every sequence in a static batch to finish. The goal is accelerator utilisation and service throughput, not making one isolated laptop request simpler.

Advanced serving optimisations need workload evidence

vLLM supports several optimisations beyond basic serving:

• Speculative decoding uses a proposal method—often a smaller draft model—to predict candidate tokens that the target model verifies. Current vLLM documentation supports multiple speculation methods and describes their different latency and workload characteristics.⁷
• Disaggregated prefilling places prompt processing and token decoding in different instances so operators can tune time-to-first-token and inter-token latency separately. The documentation labels this feature experimental and explicitly notes that it does not improve throughput.⁸
• Distributed deployment can spread model execution or serving capacity across accelerator infrastructure. The vLLM production-stack project provides a Kubernetes-oriented reference with routing and observability components.⁹

These features are not a maturity ladder every team must climb. Each adds configuration, failure modes, and observability requirements. Use them only when traces and load tests show which latency or capacity constraint needs to change.

The APIs overlap; the operations do not

Both projects expose OpenAI-style HTTP interfaces.¹⁰⁴ That can reduce application changes when moving a chat, Retrieval-Augmented Generation (RAG), agent, or code-assistant workflow between backends.

“Drop-in replacement” still needs qualification.

Endpoint coverage, request parameters, chat templates, tool-calling behaviour, structured output, model names, token accounting, and error semantics can differ. llama.cpp itself says it does not make a strong claim of complete OpenAI API compatibility, even though its chat-completions endpoint supports many applications.¹⁰

The model matters too. Moving the same client code to a different open-weight model can change answer quality, context handling, tool use, safety behaviour, and output format.

Treat the shared API as an integration bridge, not proof of behavioural equivalence.

RAG, agents, and code assistants sit above the engine

Neither inference engine creates a complete advanced application by itself.

RAG retrieves relevant information and inserts it into the model context. The application still needs document ingestion, chunking, embeddings, retrieval, access control, and evaluation.

Agents allow a model to select tools or steps. The surrounding system still needs tool permissions, input validation, timeouts, audit trails, and human approval for consequential actions.

Code assistants need repository context, language-aware retrieval, secret filtering, and a safe boundary between generated suggestions and executable changes.

Local inference can keep more of those flows under your control. It does not remove the need to govern them.

Choose the boundary before the engine

Use llama.cpp as the default candidate when:

• The model must run on a personal computer, workstation, or edge device.
• Memory fit and hardware portability matter more than multi-user throughput.
• The application should keep working with limited or no network connectivity.
• One user or a small local service owns the workload.

Use vLLM as the default candidate when:

• Multiple users or applications share the model.
• Throughput, time-to-first-token, and inter-token latency have explicit objectives.
• The team operates accelerators, containers, routing, monitoring, and capacity management.
• Distributed execution or specialised serving optimisations are justified by measured demand.

Then validate five things with the actual model and workload:

1. Quality: Does quantization or model choice meet an agreed evaluation threshold?
2. Memory: Do weights, KV cache, runtime buffers, and concurrency fit?
3. Performance: Are latency and throughput acceptable at realistic prompt and output lengths?
4. Boundary: Do prompts, retrieved data, logs, and tools stay where policy requires?
5. Total cost: Is the measured cost of hardware and operations better for this demand profile?

The engine decision comes after the operating boundary. A laptop runtime and a shared inference platform solve different problems—even when the client sends the same JSON.

Scope note: This is an operating-model comparison based on the cited project documentation and research. Hardware support and experimental features change frequently. Benchmark the exact model, quantization, backend, prompt distribution, and concurrency before making a production decision.

References

Footnotes

1. Meta AI, LLaMA: Open and Efficient Foundation Language Models, published 24 February 2023. Accessed 28 July 2026. ↩

2. ggml-org, llama.cpp, including supported backends and quantization levels. Accessed 28 July 2026. ↩ ↩²

3. ggml-org, GGUF specification, including single-file deployment, metadata, and memory mapping. Accessed 28 July 2026. ↩

4. vLLM, OpenAI-compatible server, including supported APIs. Accessed 28 July 2026. ↩ ↩²

5. vLLM, GPU installation and supported hardware, including backend-specific limitations. Accessed 28 July 2026. ↩

6. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. Accessed 28 July 2026. ↩

7. vLLM, Speculative Decoding, including supported speculation methods. Accessed 28 July 2026. ↩

8. vLLM, Disaggregated Prefilling, marked experimental and documented as a latency-control technique rather than a throughput improvement. Accessed 28 July 2026. ↩

9. vLLM Project, vLLM Production Stack, a Kubernetes-oriented reference stack. Accessed 28 July 2026. ↩

10. ggml-org, llama.cpp HTTP server documentation, including OpenAI-style endpoints and compatibility limitations. Accessed 28 July 2026. ↩ ↩²