Local large language models

What it is and why it exists

A large language model (LLM) predicts subsequent tokens from existing context. With a local LLM, the model weights, runner, and inference state live on a device you control instead of sending each request to a hosted API. That device may be a laptop, workstation, or private server. “Local” describes the deployment boundary; it neither says that the model is small nor guarantees a completely offline system.

This arrangement first addresses data boundaries and availability. Input, output, and runtime logs can remain inside your infrastructure, and inference can continue without an internet connection. Model downloads, telemetry, update checks, log forwarding, and backups may still create external data flows, though. A genuinely offline deployment requires checking and disabling every channel you do not need.

Local inference also returns operational responsibility to your team. Capacity planning, patches, model updates, rate limits, monitoring, and recovery no longer belong to a cloud provider. Owned hardware does not make inference free: power, machine time, engineering work, and idle capacity still cost money, and a local service does not gain more accelerator memory when traffic spikes.

Good local workloads tend to have a clear data boundary, an acceptable model, and a fairly steady load. Internal document classification, offline drafting, and repeatable development tests can fit. If a task depends on the strongest hosted model, multi-region elasticity, or a formal service-level agreement, local deployment is unlikely to be the default. Compare quality, latency, throughput, and cost on the same evaluation inputs.

Ollama and llama.cpp solve model-running and serving problems; they do not prove that a model fits your business. Ollama provides model management, a command line, and a local HTTP API. llama.cpp runs GGUF artifacts directly and offers command-line and compatible HTTP servers. Neither replaces model-card and license review, domain evaluation, or application-level input and output validation.

How it works

A local inference stack needs four mutually compatible parts. The model artifact contains weights and metadata, the tokenizer maps text to tokens, the chat template arranges messages in the form the model saw during training, and the runner executes computation on a CPU, GPU, or another backend. Similar-looking labels do not establish compatibility. A mismatched template or tokenizer can run successfully while producing consistently poor results.

During loading, the runner maps weights into system or accelerator memory and allocates runtime buffers. After a prompt arrives, it first performs prefill over all input tokens. Decode follows and produces one token, or a small group of tokens, at each step. Streaming returns completed fragments to the caller sooner, but it does not remove the computation needed to finish the generation.

A context window limits the tokens a model can access in one request. Input, system instructions, conversation history, and reserved output all consume that budget. During inference, the key-value cache (KV cache) retains attention state already computed for prior tokens, so each decode step need not recompute the entire history. It grows with context, batch size, and concurrent slots.

Quantization uses lower-precision representations for some model data, most often its weights. This usually reduces artifact size and weight memory, but it does not shrink the tokenizer, runtime buffers, or KV cache by the same proportion. Lower precision can also change output quality. The effect depends on the model, quantization method, and task, so one universal percentage would be misleading.

GGUF is the model file format used by llama.cpp. It packages tensors with standardized metadata for loading and distribution. A GGUF file is still not a complete release record: preserve the license, provenance, evaluations, runner version, and artifact digest separately. Ollama can hide some file-management detail, but a team still needs to know the exact model and quantization it loaded.

The diagram ends at application policy, not at model output. Text generated on your own machine can still be wrong, out of bounds, or influenced by malicious instructions in the prompt. The application must inspect completion state, parse the result, and enforce domain constraints. Permission decisions and external side effects stay outside the model.

Choosing a runner and artifact

Start with requirements, not a leaderboard. Record the allowed licenses, deployment operating system, available memory, maximum context, concurrency target, acceptable latency, and domain cases that must pass. Only then shortlist models and quantizations and measure them on the target machine. Parameter count roughly describes weight scale; it does not establish quality, speed, or peak memory.

With Ollama, the local API defaults to http://localhost:11434/api. /api/generate accepts fields including model, prompt, stream, and runtime options. A streamed response consists of multiple JSON objects; its terminal object contains done plus timing and token counts. The API is not strictly versioned, so pin the Ollama version and keep contract tests for every response field your application needs.

For llama.cpp, the current entry points are llama-cli and llama-server, and source builds use CMake. llama-server can expose an OpenAI-compatible HTTP interface. The old draft’s ./main, ./server, and Make-first build commands are stale. Do not copy them from an old tutorial into a current deployment script.

Before downloading a model, check its publisher, license, model card, file list, and digest. A model tag may move to a different artifact, so recording only a mutable label may not reproduce the same bytes during rollback. Preserve the source, immutable revision or digest, quantization, template, and runner version, then evaluate that combination with the application release.

Examples

The next three executable examples cover a capacity floor, request budgeting, and stream completion in that order. They do not download a model, so their output is deterministic and easy to verify in a normal Python environment. Real model generations are probabilistic evaluation artifacts. They should be saved from a real runner, not passed off as output from an editorial environment without one.

Calculate the weight-memory floor

The theoretical weight floor is the parameter count multiplied by the number of bits per weight. This example uses the decimal 8B parameter count and converts the result to binary GiB. The number answers “about how large are the bare weights,” not “will this model run on the device.”

A real GGUF file can differ from that multiplication because of block proportions, scales, and metadata. At run time, the computation graph, temporary buffers, KV cache, and driver also need space. Use this result only as a floor that eliminates plainly unsuitable candidates, then observe actual peak memory with the target runner, context, and concurrency settings.

Build a bounded request

An application should not give arbitrary input the entire context window. The following builder expects its caller to count input tokens with the target model’s tokenizer and then checks the input and maximum-output total. num_ctx selects the runtime context, while num_predict limits how many tokens Ollama may generate.

import json


def build_request(model, prompt, input_tokens, context_limit, max_output):
    if input_tokens + max_output > context_limit:
        raise ValueError("input and output budgets exceed the context limit")

    return {
        "model": model,
        "prompt": prompt,
        "stream": False,
        "options": {"num_ctx": context_limit, "num_predict": max_output},
    }


request = build_request("gemma4", "Return one short status line.", 180, 4096, 96)
print(json.dumps(request, indent=2))

try:
    build_request("gemma4", "Oversized input", 4050, 4096, 96)
except ValueError as error:
    print(f"rejected: {error}")

{
  "model": "gemma4",
  "prompt": "Return one short status line.",
  "stream": false,
  "options": {
    "num_ctx": 4096,
    "num_predict": 96
  }
}
rejected: input and output budgets exceed the context limit

The 180 in this example is a caller-supplied measurement, not a guess from character count. Different tokenizers split the same text differently, and a chat template adds special tokens. Production code should count the complete request with the tokenizer distributed with the model, then explicitly reject, chunk, or truncate by a tested rule. Silently dropping the earliest messages changes the task’s meaning.

After installing Ollama and reviewing a model, the following current command starts interactive inference. Ollama is absent from this editorial environment, so there is no fabricated model answer; gemma4 comes from the official quick start checked during this review.

# not executed here: Ollama is not installed in the editorial runner
ollama run gemma4

Not executed in this environment.

Collect a stream and its terminal metadata

A streaming client must distinguish content fragments from the final terminal object. This example consumes an intentionally constructed contract-test trace whose fields match Ollama’s /api/generate response; it is not a hardware benchmark. The code accepts a result only after it sees done, then calculates generation rate from the nanosecond duration and token count in that terminal object.

import json


trace = [
    '{"response":"Local", "done":false}',
    '{"response":" inference", "done":false}',
    '{"response":" is ready.", "done":false}',
    '{"response":"", "done":true, "done_reason":"stop", '
    '"eval_count":60, "eval_duration":3000000000}',
]


def collect(events):
    text = []
    final = None
    for line in events:
        event = json.loads(line)
        text.append(event.get("response", ""))
        if event.get("done"):
            final = event

    if final is None:
        raise ValueError("stream ended without a terminal event")
    return "".join(text), final


answer, final = collect(trace)
seconds = final["eval_duration"] / 1_000_000_000
rate = final["eval_count"] / seconds
print(answer)
print(f"done_reason={final['done_reason']}")
print(f"generation_rate={rate:.1f} token/s")

Local inference is ready.
done_reason=stop
generation_rate=20.0 token/s

The displayed rate verifies arithmetic only: the test trace deliberately supplies 60 tokens and three seconds. When evaluating a real service, separately record load time, input-evaluation time, time to first visible fragment, and end-to-end latency. If the connection closes before a terminal object arrives, mark accumulated text incomplete. A retry starts a new generation and cannot resume the old stream.

Pitfalls

Fix: map data flows for downloads, inference, logging, telemetry, and backups. When you need local-only operation, disable cloud features and test an offline start. Log only approved, redacted diagnostic fields, and give model-cache and log directories file permissions and retention rules that match the data’s sensitivity.

Fix: pin the complete artifact identity and runner version, then run representative, boundary, and adversarial cases on the target hardware. Put quality results beside peak memory, cold-start time, time to first token, and sustained generation rate in the same decision record.

Fix: use the weight formula as a floor, then measure the peak at the planned num_ctx, concurrency, and backend. The capacity test should include the longest allowed input and output and retain headroom. If the target remains unstable, reduce context, concurrency, or model size.

Fix: commit a result only after the protocol’s terminal event, preserve the stop reason and counts, and give incomplete output a distinct state. Bound retries. A new request is a new generation, and any downstream side effect also needs idempotency controls outside the model call.

Fix: measure cold starts and warm requests separately. Report time to first token, end-to-end latency, input-evaluation rate, generation rate, and queue time. Use percentiles from the real input-length and concurrency distribution, and run quality evaluation at the same time so speed does not excuse unusable output.

Further reading

• Ollama README and quick start
• Ollama API reference
• Ollama context length
• llama.cpp README and quick start
• Hugging Face GGUF documentation