Self-Hosted LLM Coding Assistants: When They Make Sense

Self-hosting an LLM coding assistant makes sense in a narrow set of cases: a hard regulatory or contractual ban on sending code to third parties, an air-gapped environment, or a genuinely unusual data-sensitivity posture. For most funded startups with 5-25 engineers, it does not — you give up meaningful model quality, take on GPU costs that rarely beat API pricing at your scale, and inherit an inference platform to operate. The middle path most teams should evaluate first is a cloud API under a zero data retention (ZDR) agreement.

When does self-hosting actually make sense?

Self-hosting is the right call when the constraint is absolute, not preferential. If a customer contract, classification requirement, or regulator says code and data cannot transit a third-party inference provider under any terms, no API agreement fixes that — you run the model yourself. The same logic applies to air-gapped or sovereign-cloud environments where outbound traffic simply isn’t available.

Weaker versions of the concern — “we’re worried about our code being used for training” or “our SOC 2 auditor will ask about this” — are usually addressable contractually. Auditors care that you assessed the vendor and controlled the data flow, not that you bought GPUs; we cover what they actually ask in our look at AI coding assistants and SOC 2. If compliance pressure is what’s driving the question, that’s an infrastructure-and-vendor-management problem first — the kind of thing our SOC 2 infrastructure package exists to sort out. (Usual note: this is engineering guidance, not legal advice.)

What quality do you give up with open models?

The open-model ecosystem is real and improving fast. Ollama’s model library lists serious coding-capable options: Qwen3-Coder (30B and 480B variants, built for agentic coding), Qwen2.5-Coder up to 32B, DeepSeek-Coder-V2, Codestral, and OpenAI’s open-weight gpt-oss models at 20B and 120B. For autocomplete, boilerplate, and single-file edits, a well-served 30B-class coder model is genuinely usable.

The gap shows up where the money is: long-horizon agentic work. Multi-step tasks across a large repo — refactoring a Terraform module tree, tracing a bug through services, driving a migration — are exactly where frontier hosted models have pulled ahead, and exactly the work that justifies the tool’s cost in the first place. The models that top agentic coding comparisons are the big hosted ones, which is why they dominate our roundup of the best AI coding tools for Terraform and Kubernetes work. Running the largest open models (a 480B Qwen3-Coder, a 671B DeepSeek) at usable speed puts you in multi-GPU territory, which brings us to cost.

What do the GPUs really cost?

The honest math: at 5-25 engineers, self-hosted inference rarely undercuts API spend. Per Lambda’s published on-demand pricing, an H100 runs roughly $3.99-$4.29 per GPU-hour; AWS’s smallest H100 option, per its P5 instance docs, is a single-GPU p5.4xlarge, with the standard config jumping to 8 GPUs. Serving a strong 30B+ coder model with decent context length and concurrency typically means at least one big GPU running whenever engineers are working — call it roughly $3,000-$7,000/month for a modest always-on setup before you’ve paid anyone to operate it.

Option	Ballpark monthly cost (10-eng team)	Quality tier	Where code goes
Cloud API, default terms	Per-seat or token spend, typically low thousands	Frontier	Vendor, with limited-time logs
Cloud API + ZDR	Same as above	Frontier	Vendor, no prompt/output retention
Self-hosted (Ollama/vLLM on rented GPUs)	~$3-7k GPU + ops time	Open-model	Stays in your VPC

The hidden line item is operations: model serving, GPU capacity, upgrades, evals when a new open model ships. That’s a platform, and someone owns it. It’s the same “cheap until you price the labor” trap we see in cloud bills generally.

What’s the middle path: cloud APIs with zero data retention?

For most teams, a ZDR agreement gets you the security property you wanted without the GPU bill. Per Anthropic’s privacy docs, under a ZDR agreement inputs and outputs are not stored (narrow carve-outs for legal compliance and abuse prevention aside); it’s available to commercial API customers subject to approval, and covers Claude Code used with commercial organization keys or Enterprise plans. OpenAI’s docs describe a similar posture: API abuse-monitoring logs are retained up to 30 days by default, and approved ZDR customers can have customer content excluded, though some endpoints don’t qualify.

That contractual answer — vendor assessed, retention terms documented, data flow diagrammed — is what an auditor actually wants to see, and it slots cleanly into the vendor-management controls in our SOC 2 infrastructure checklist. Pair it with client-side hygiene so secrets never reach any model, hosted or local — see how to keep secrets out of AI coding tools.

What do we recommend clients do?

Decide on the constraint, not the vibe. The sequence we walk clients through:

• Name the actual requirement. A written customer or regulatory prohibition on third-party inference means self-host; anything softer means negotiate terms.
• Get ZDR in writing first. Both major vendors offer it to commercial customers on approval — ask before you architect around its absence.
• If you must self-host, scope it small. One well-chosen coder model behind vLLM or Ollama on rented GPUs, for the restricted repos only — not a blanket replacement for frontier tooling on AWS, GCP, or Azure.
• Re-run the math twice a year. Open-model quality and GPU pricing both move quickly; a decision from last year is stale.

A predictable pattern: teams that self-host everything for comfort quietly drift back to hosted frontier models because the productivity gap is felt daily and the GPU bill is felt monthly. If you want a second pair of eyes on the security review, the vendor terms, and whether the numbers work for your team, that’s a standard part of our audit and packages.