21 Apr 2024 AI Engineering 5 min read

Llama 3: Open Weights Get Serious

Meta's Llama 3 (April 18, 2024) narrows the gap between open weights and closed APIs. We examine the 8B and 70B benchmarks, what 4-bit quantization does to hardware requirements, the utilization math of self-hosting versus API pricing, and the privacy-bound workloads where an open model is the only viable option.

The Gap Between Open and Closed

Until this week, teams with privacy-bound workloads faced an uncomfortable choice. Frontier model quality lived behind hosted APIs, and data such as patient records, HR files, or client documents may not leave controlled infrastructure. Open models could be self-hosted, but they trailed by roughly a generation: the Llama 2 70B base model scored 69.7 on MMLU (5-shot), while OpenAI reported 86.4 for GPT-4. Capability or control — rarely both.

That gap was the argument. Every architecture discussion about on-premise inference ended with the same caveat: the model you are allowed to run is measurably worse than the model you are not. We at Blue IT Systems see this weekly in projects with health and legal data. Llama 3 changes the size of that caveat. It does not remove it.

A request arrives — the router classifies its difficulty. 1/4

What Meta Released

On April 18, 2024, Meta released Llama 3 in two sizes, 8B and 70B parameters, each as a pretrained base model and an instruction-tuned variant. Both were pretrained on over 15 trillion tokens — seven times the Llama 2 corpus — with four times more code and over 5% non-English data across more than 30 languages. The architecture is a standard decoder-only transformer with a 128K-token vocabulary, grouped query attention on both sizes, and an 8,192-token context window. Knowledge cutoffs: March 2023 (8B) and December 2023 (70B).

The license is the Meta Llama 3 Community License. Commercial use is permitted unless the licensee exceeded 700 million monthly active users at release; products must display "Built with Meta Llama 3"; outputs may not be used to improve other LLMs. This is open weights, not open source: training data and training code remain closed. For self-hosting, the weights are sufficient.

Benchmarks in Context

The instruction-tuned models carry the release. Meta's published comparison places the 70B at or above Claude 3 Sonnet, Anthropic's mid-tier API model. Shot counts differ between vendors — Meta reports GSM-8K with 8-shot chain of thought, Anthropic with 0-shot — so treat the table as indicative, not as a controlled experiment.

Meta also reports a human evaluation on 1,800 prompts across twelve use cases, in which annotators preferred Llama 3 70B over Claude 3 Sonnet, Mistral Medium, and GPT-3.5. Vendor-run human evaluations deserve skepticism, but the direction matches the automated benchmarks.

What the numbers do not say: the context window is 8K tokens, against 128K for GPT-4 Turbo and 200K for Claude 3. Llama 3 is text-only. And GPT-4 (86.4 MMLU) and Claude 3 Opus (86.8) remain clearly ahead. The gap has narrowed to one tier, not to zero.

Benchmark	Llama 3 8B Instruct	Llama 3 70B Instruct	Claude 3 Sonnet
MMLU (5-shot)	68.4	82.0	79.0
HumanEval (0-shot)	62.2	81.7	73.0
GSM-8K (CoT)	79.6	93.0	92.3

Quantization Makes 70B Practical

In FP16, 70 billion parameters occupy roughly 140 GB — a multi-GPU deployment before the first token. Post-training quantization changes the arithmetic: at 4 bits per weight (GPTQ, AWQ, or llama.cpp's GGUF formats), the 70B weights shrink to roughly 35 to 40 GB and fit on a single 80 GB A100 or H100 with room for the KV cache. The 8B model drops from 16 GB to around 5 GB and runs on a consumer GPU or a laptop.

Quantization is not free. Quality degrades measurably below 4 bits, and even 4-bit variants need task-specific evaluation before production use. Day-one community conversions of a new model routinely carry tokenizer and chat-template bugs; Llama 3's first GGUF builds were no exception. The serving stack, however, is ready: vLLM, Hugging Face TGI, llama.cpp, and Ollama supported Llama 3 at launch.

The Economics of Self-Hosting

API pricing in April 2024, per million input/output tokens: GPT-4 Turbo at $10/$30, Claude 3 Sonnet at $3/$15, Claude 3 Haiku at $0.25/$1.25. These are list prices; volume discounts exist on both sides. Self-hosting replaces marginal cost with fixed cost: an 80 GB A100 rents on demand for roughly two to four US dollars per hour depending on provider — regardless of how many tokens it serves.

Break-even is therefore a utilization question. A batched vLLM deployment of a quantized 70B running near capacity around the clock undercuts mid-tier API prices per token; the same GPU at 5% load inverts the calculation. Add the costs the API price already includes: monitoring, patching, capacity planning, on-call. Self-hosting pays off under steady load or hard data constraints — not for spiky, low-volume workloads.

When Open Weights Beat the API

Some requirements end the comparison before it starts: data that may not leave your infrastructure under GDPR processing agreements, professional secrecy obligations, or works-council agreements. Model versions that must be pinned and reproducible for audits. Fine-tuning on confidential data no third party may see. Air-gapped operation. In these cases an API is not the weaker option — it is not an option.

The reverse list is equally real: frontier reasoning, contexts beyond 8K tokens, multimodal input, and highly unpredictable load favor the API. What Llama 3 changes is the price of the constraint: the quality penalty for staying on your own hardware has dropped from a full generation to a measurable but small margin on many tasks. The EU AI Act, adopted by Parliament on March 13, 2024, will raise documentation obligations on both paths.

Outlook

Meta states that the 8B and 70B models are the beginning: models with over 400 billion parameters are still in training, and longer context windows, multimodality, and additional languages are announced for the coming months. If the 70B's trajectory holds, an open-weights model at GPT-4 level within this year is plausible.

Our expectation as of April 2024: a dense ecosystem of fine-tunes and quantized builds within weeks, hosted open-model endpoints pressing per-token prices down, and — for regulated European industries — self-hosted open weights becoming the default starting point, with frontier APIs reserved for tasks that demonstrably need them. We are aligning our reference architectures accordingly. The gap has stopped being the argument.