How to Choose an LLM: Open vs Closed Models, Size, Cost and Latency

How to choose an LLM for a product: hosted APIs vs open-weight models, model size, benchmarks vs your own evals, cost, latency, privacy and lock-in.

Beginner 3 min read · Updated 29 October 2026

Choosing a model is one of the first design decisions in an AI product, and the one most often made on vibes. A good choice weighs quality on your task (not leaderboard rank), cost per task, latency, context length, data privacy and residency, operational burden and lock-in. The usual answer in production isn't one model but a portfolio: a strong model for hard requests, a cheap fast one for easy ones, and a fallback.

Hosted API or open-weight model

	Hosted API (closed model)	Open-weight model, self-hosted
Quality	Usually the frontier for general reasoning	Strong and improving; can be fine-tuned freely
Time to start	Minutes	Days to weeks (GPUs, serving stack, scaling)
Cost model	Per token; no idle cost	Per GPU-hour; cheap at high, steady utilisation
Latency control	Provider's; varies with their load	Yours; can co-locate and tune
Data control	Data leaves your network (check retention and region terms)	Stays in your environment
Customisation	Prompting, some fine-tuning options	Full fine-tuning, quantization, custom decoding
Ops burden	Low	High: capacity, upgrades, on-call

Many teams start on APIs, then self-host a smaller open model for high-volume, narrow tasks once traffic is steady and the task is well understood. Cloud platforms that host open models sit in between.

Size, speed and cost

An illustrative model portfolio: relative cost per million tokens for small, medium and large tiers of the same family. Real prices vary by provider; the spread between tiers is commonly an order of magnitude or more.

Larger models are better at multi-step reasoning, nuanced instructions and rare knowledge; smaller ones are faster and far cheaper.
Most production traffic is easy (classification, extraction, short answers). Routing it to a small model is the biggest cost lever; see LLM gateway.
Reasoning modes that spend extra "thinking" tokens raise quality on hard problems at the cost of latency and tokens; enable them per request type.

Benchmarks vs your own evals

Public benchmarks measure general ability and are increasingly saturated or leaked into training data. They're a filter, not a decision. Decide with a task-specific eval set:

A model selection process. Shortlist from public information, then decide on your own eval set, with cost and latency measured on real prompts.

Build the eval set from real (or realistic) inputs, including hard and adversarial cases, and grade it as described in LLM evaluation.

Other selection criteria

Context window: enough for your largest realistic prompt, with headroom.
Tool calling and structured output support: reliability of function calls and JSON schemas varies a lot between models.
Multimodality: images, audio, documents as input; image or audio output.
Languages: quality in your users' languages, not only English.
Rate limits and capacity: tokens per minute you can actually get at peak.
Compliance: data retention, training-on-your-data terms, regions, certifications.
Model lifecycle: how long a version stays available; deprecations force re-evaluation.

Avoiding lock-in

Call models through one internal interface or gateway, not provider SDKs sprinkled through the code.
Keep prompts and eval sets versioned per model; prompts tuned for one model often need adjustment for another.
Re-run the eval suite on every model upgrade before switching; treat a model change like a code release.

Where it’s used

Customer-facing chat

A strong model for complex questions, a fast one for greetings and simple lookups, behind a router.

Bulk extraction

A small model or a fine-tuned open model, run in batch, where cost per document dominates.

Regulated data

Self-hosted open models or region-pinned hosted endpoints with zero data retention.

In an interview

Compare hosted and self-hosted options on cost, latency, privacy and ops.
Propose a portfolio with routing rather than one model for everything.
Decide with a task-specific eval set, not leaderboards.
Explain how you'd swap models later without rewriting the app.

Frequently asked questions

Should I use a hosted LLM API or self-host an open model?

Start with hosted APIs for speed and quality. Self-host open-weight models when traffic is high and steady, data must stay in your environment, or you need deep customisation, and you can run GPUs efficiently.

How do I compare LLMs for my product?

Shortlist a few models, run them on your own evaluation set of realistic cases, and compare quality per slice against cost per task and latency, rather than relying on public leaderboards.