13 Aug 2023 AI Engineering 5 min read

Llama 2: What Open Weights Mean for Enterprises

Meta released Llama 2 on July 18, 2023 under a license that permits commercial use. We examine what open weights change for enterprise architectures: data residency without third-country transfers, fixed-cost inference instead of per-token billing, the measured quality gap to GPT-4, and where the 7B, 13B, and 70B models fit today.

The API Dependency Problem

Most enterprise LLM integrations in mid-2023 follow one pattern: application data is sent to a hosted API, the provider runs the model, and usage is billed per token. This works. It also creates three dependencies that recur in every architecture review we run: prompts leave the company's infrastructure, cost scales linearly with usage, and model versions change on the provider's schedule — not yours.

For prototypes these constraints are acceptable. For production systems that process regulated or contractually protected data, they often are not. Until this summer there was no commercially usable open-weight model of credible quality. On July 18, 2023, Meta changed that baseline with Llama 2.

A request arrives — the router classifies its difficulty. 1/4

What Llama 2 Actually Is

Llama 2 is a family of pretrained and fine-tuned language models released by Meta on July 18, 2023, in three sizes: 7, 13, and 70 billion parameters. All were trained on 2 trillion tokens of publicly available data with a context window of 4,096 tokens. The 70B variant uses grouped-query attention for faster inference. A 34B variant was trained but not released.

The chat variants, Llama-2-Chat, were aligned with supervised fine-tuning and RLHF on over one million human annotations. The weights are downloadable after accepting the license, and the models are available through the Azure AI model catalog, AWS, and Hugging Face. What you receive are the parameters — not the training data, not the training pipeline.

Reading the License Carefully

Meta calls Llama 2 open source. By the Open Source Definition it is not, and the Open Source Initiative said so publicly on July 20, 2023. The Llama 2 Community License grants a royalty-free right to commercial use with two restrictions: services exceeding 700 million monthly active users at the release date require a separate license from Meta, and Llama 2 outputs may not be used to improve other large language models. An acceptable-use policy applies on top.

For most enterprises the 700-million clause is irrelevant. The practical consequence is different: your legal review must read the actual license text, not the label. We use the term open weights. It states precisely what is open — and what is not.

Data Residency Under Your Control

Self-hosted weights mean prompts and completions never leave your infrastructure. The model runs on-premises or in a dedicated EU cloud tenant. No third-country transfer occurs, no subprocessor sees the data, no provider retains it. For workloads under GDPR, professional secrecy obligations, or works-council agreements, this removes the hardest questions from the data protection assessment.

Honest scoping: self-hosting does not make a system GDPR-compliant by itself. Lawful basis, purpose limitation, and governance of model outputs remain your responsibility, exactly as before. What changes is the scope of the assessment: the transfer question and the vendor-access question disappear, because there is no vendor in the request path.

The Economics of Self-Hosting

API pricing scales linearly with token volume. Self-hosting inverts this: fixed GPU cost, near-zero marginal cost at high utilization. The numbers are concrete. Llama 2 70B needs roughly 140 GB for weights alone at 16-bit precision — two 80-GB A100s. With 4-bit quantization (GPTQ, llama.cpp) this drops to roughly 35–40 GB. The 7B and 13B models run on a single workstation GPU. Serving stacks exist: Hugging Face Text Generation Inference, and vLLM (June 2023) with PagedAttention for high-throughput batching.

The trade-off is equally concrete. Break-even against an API requires sustained volume. Operations, monitoring, security patching, and capacity planning move onto your team. For low or spiky traffic, a hosted API remains the cheaper and simpler option. Cost control means predictability — not automatically a lower bill.

The Quality Gap Against GPT-4

Meta's paper is unusually candid about where its models stand. On academic benchmarks — reported in the paper's Table 4 alongside OpenAI's published GPT-4 figures — the 70B model reaches the level of GPT-3.5 on knowledge tasks, but not on code or mathematics:

The reading is unambiguous. Llama 2 70B matches GPT-3.5 on knowledge and grade-school math, trails it clearly on code, and sits far behind GPT-4 everywhere. Any deployment plan that expects GPT-4-level reasoning from Llama 2 will fail. The correct question is not whether Llama 2 beats GPT-4 — it does not — but which of your tasks do not need GPT-4.

Benchmark	Llama 2 70B	GPT-3.5	GPT-4
MMLU (5-shot)	68.9	70.0	86.4
GSM8K (8-shot)	56.8	57.1	92.0
HumanEval (0-shot)	29.9	48.1	67.0

Where Open Weights Fit Today

The fit is narrow, high-volume, data-sensitive tasks: classification, information extraction, summarization of internal documents, retrieval-augmented question answering over company data, first drafts. These tasks rarely require frontier-level reasoning. They do require data residency and predictable volume economics — exactly what open weights provide.

Fine-tuning shifts the balance further. With LoRA and QLoRA (May 2023), a 7B or 13B model can be adapted to one domain task on a single GPU with 24–48 GB of memory. A small model tuned on your task and your data can outperform a prompted general model on that task — and only on that task. The pragmatic architecture in August 2023 is a router: sensitive and high-volume traffic to self-hosted Llama 2, hard reasoning to a frontier API.

Outlook From August 2023

Three expectations guide our planning. First, the quality gap will narrow. Llama 2 70B already sits at GPT-3.5 level on knowledge benchmarks; we expect stronger open weights, including code-specialized variants, within twelve months. Second, inference cost will fall faster than model quality rises — quantization and serving optimizations like PagedAttention are months old and improving weekly.

Third, regulation pushes in the same direction. The EU Parliament adopted its negotiating position on the AI Act in June 2023; European enterprises will increasingly need deployments they can inspect and control. Our working assumption: the end state is not one model but a portfolio — several small, specialized open-weight models beside one frontier API. Weights are becoming infrastructure. Enterprises should start treating them that way.