Open Source Models in Production: Licensing and Practical Limits

A client asked us last year to swap their support-triage feature off a commercial API and onto something self-hosted. Cost was the stated reason. Three weeks in, the actual blockers had nothing to do with tokens per second: a clause about naming derivative models, an EU restriction on the multimodal variant they wanted, and a procurement team who wanted to know who indemnifies them if the training data turns out to be someone’s copyrighted book.

That is the shape of most open source LLM projects we have shipped. The inference part is a solved engineering problem with good tooling. Licensing, evaluation and honest accounting of what the model cannot do are where projects quietly die.

This article was written after doing it, not before. Numbers, clauses, and the cases where we tell clients to stay on a hosted API.

Open weights is not open source, and the gap is where projects stall

The Open Source Initiative publishes a definition. Almost none of the headline model releases meet it. What you get instead is a binary blob (the weights) under a bespoke agreement that permits commercial use subject to conditions, with no training data, no training code, and a usage policy you are contractually bound to enforce downstream.

That distinction matters for three concrete reasons. Your legal team cannot rubber-stamp it under an existing OSS policy. Your licence obligations flow to anyone you distribute the model or a fine-tune to, including your own clients. And the acceptable use policy is not a suggestion: if you build a product that your model’s licence prohibits, the licence terminates and you are running unlicensed software in production.

Call them open weights internally. It changes how people treat the review.

The licences that actually ship, read properly

Here is the state of play for the families we see on client projects, as of early 2026. Always read the LICENSE file in the specific repo, because terms change per release and sometimes per model size within a release.

Llama (Meta): permissive with teeth

The Llama Community License allows commercial use, fine-tuning, and (since 3.1) using model outputs to train other models. The conditions people miss:

• 700 million MAU threshold. If your product had more than 700 million monthly active users in the month before the model’s release date, you need a separate licence from Meta. Practically irrelevant for agencies, relevant if you are a platform vendor.
• Attribution. You must display “Built with Llama” on a related website, interface or documentation.
• Derivative naming. If you distribute a fine-tune, its name must begin with “Llama”.
• EU restriction on multimodal. The vision-capable releases exclude individuals domiciled in, and companies with a principal place of business in, the EU from the grant. End users in the EU can use a product built on them, but a Berlin agency cannot take the licence itself. We have had to route around this twice.

<!-- Footer attribution. Required by the Llama Community License,
     and the cheapest compliance item you will ever ship. -->
<p class="small text-muted">
  Built with Llama. Summaries are generated and may contain errors.
</p>

Gemma (Google)

Commercial use is allowed under the Gemma Terms of Use. The clause that gives counsel pause is the restriction mechanism: Google reserves the right to require you to stop using or distributing the model if it is being used in violation of the prohibited use policy. You also have to pass the use restrictions to downstream recipients. Fine for an internal tool. Harder if you are shipping weights inside a product someone else deploys.

Mistral: read the model card, not the brand

Mistral splits cleanly. Mistral 7B, Mixtral, Nemo and the Mistral Small line are Apache 2.0, full stop, no attribution, no usage policy. The larger and specialist models sit under the Mistral Research License, which is non-commercial, with a paid commercial licence available. Teams routinely assume “Mistral is Apache” and deploy something that is not. Check per model.

Qwen, DeepSeek, OLMo

Qwen3 shipped Apache 2.0 across the lineup, which fixed the inconsistency in Qwen2.5 where some sizes carried a bespoke licence with a 100 million MAU clause. DeepSeek-R1 weights are MIT, including permission to distil. OLMo from AI2 is Apache 2.0 and publishes the training data, which is the only family we can honestly call open source in the OSI sense. If your constraint is legal simplicity rather than raw benchmark position, start with these three.

What running an open source LLM in production actually costs

Start with memory, because that is what determines your bill. Weights at bf16 are roughly 2 bytes per parameter, so a 70B model is about 140 GB. FP8 halves it to 70 GB. Four-bit quantisation gets you to roughly 38 to 40 GB.

Then the part people forget. The KV cache scales with context length and concurrency, and on a 70B model with grouped-query attention it is not small:

# Llama 3.3 70B: 80 layers, 8 KV heads, head_dim 128, bf16 cache
def kvbytespertoken(layers, kvheads, headdim, dtypebytes=2):
    # x2 because we store both K and V
    return 2  layers  kvheads  headdim  dtype_bytes

pertoken = kvbytespertoken(80, 8, 128)      # 327,680 bytes = 0.31 MB
print(pertoken * 128000 / 1e9, "GB for one 128k session")   # ~42 GB
print(pertoken  8000  32 / 1e9, "GB for 32 concurrent 8k sessions")  # ~84 GB

One long-context session can cost more memory than the quantised weights. This is why the single highest-leverage flag in a vLLM deployment is --max-model-len. Setting it to the context you actually need instead of the model maximum is usually the difference between two GPUs and four.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --max-model-len 16384 \          # not 131072. be honest about your real inputs
  --max-num-seqs 64 \
  --enable-prefix-caching \        # big win when every request shares a system prompt
  --gpu-memory-utilization 0.92 \
  --tensor-parallel-size 2

On cost: H100 on-demand pricing has fallen hard, and in 2026 you can find it in the low single-digit dollars per hour on the specialist GPU clouds, considerably more on the hyperscalers. Call it $2,000 to $3,000 per month per GPU running continuously, and remember you pay for idle time. A hosted 70B-class endpoint charges per token. Do the division honestly: for most B2B products we have measured, breakeven lands somewhere in the hundreds of millions of tokens per month, and almost nobody hits it with a chat feature.

So the self-hosting decision is rarely about money. It is about data residency, latency floors you control, fine-tuning on proprietary data, and not having a vendor deprecate your model with ninety days’ notice. Those are good reasons. Say them out loud instead of pretending the spreadsheet works.

Where open models still lose, measured

A well-prompted Llama 3.3 70B or Qwen3 handles classification, extraction, summarisation and rewriting at a quality your users will not distinguish from a frontier API. We have shipped all four. The gaps show up elsewhere:

• Multi-step reasoning with tool calls. Chains of four or more tool invocations are where open models drift, hallucinate arguments, or loop. Budget for a hard step cap and a fallback.
• Long-context recall. Advertised context windows are not usable context windows. Accuracy on retrieval-in-the-middle degrades well before the stated maximum on every open model we have tested. Chunk and retrieve rather than stuffing 100k tokens in.
• Quantisation damage is task-dependent. FP8 on Hopper has been effectively free in our evals. INT4 (AWQ, GPTQ, GGUF Q4KM) is fine for extraction and classification but costs measurable accuracy on arithmetic and multi-hop reasoning. Test on your own eval set, not on MMLU.
• Strict schema output. Open models need constrained decoding to be reliable. Use grammar-based structured output rather than prompting and hoping, and keep a repair pass. We wrote up the full approach in structured output from LLMs.

Build the eval set before you pick the model. Two hundred real examples from your own data, scored by something deterministic, beats any leaderboard. The same discipline we described in shipping an AI feature users trust applies here, only more so, because you no longer have a vendor silently improving the model underneath you.

Fine-tuning can make you the provider

The EU AI Act’s general-purpose AI obligations began applying in August 2025, with enforcement powers phasing in through 2026. The part that catches teams building on open weights: if you perform a substantial modification of a general-purpose model, you can become the provider of that modified model, with your own technical documentation, copyright policy and training-content summary obligations for the modification.

There is an exemption for free and open-source models released under a genuinely free licence with public weights and parameters, provided the model does not present systemic risk. Two caveats. The exemption is about how the model is released, not how you use it, so it does not cover you once you embed the model in a commercial product. And several of the licences above, with their usage policies and naming conditions, are arguably not “free and open-source” for this purpose anyway.

Practical advice: keep a one-page model card for every fine-tune you deploy, recording base model, licence, dataset provenance, eval results and date. Takes an hour. Saves a week when procurement asks.

The question your client’s lawyer will ask

“If we get sued over generated output, who covers us?” With the major commercial APIs, there is usually some form of copyright indemnity attached to paid tiers. With open weights, there is none. Every one of these licences disclaims warranties and liability in capital letters, and you have no visibility into the training corpus.

This is the same trade-off we laid out for generated imagery in AI image generation for the web: the self-hosted option gives you control and removes the counterparty you could point at. For internal tools, summarisation of the user’s own content, and classification, the exposure is low and nobody cares. For anything that publishes model output under a client’s brand, put a human in the loop and say so in the contract.

How we actually decide

1. Is the data allowed to leave the building? If no, self-host. The decision is made. Pick an Apache 2.0 or MIT model and move on.
2. Is the task extraction, classification, routing or rewriting? An 8B to 32B open model, FP8, on one GPU will probably do it. Prove it with your eval set in a day.
3. Does it need agentic tool use or strong reasoning? Start on a hosted frontier model. Revisit in six months. We have changed our answer here twice in two years and will again.
4. Under 50 million tokens a month? Use a hosted endpoint for the open model instead of running GPUs. You keep portability and the licence story, without the pager.
5. Shipping weights to someone else’s infrastructure? Apache 2.0 or MIT only. The attribution, naming and downstream-policy obligations of the bespoke licences are a support burden you do not want.