07 Oct 2024 AI Engineering 5 min read

Small Language Models: When 3 to 9 Billion Parameters Suffice

Phi-3 and Gemma 2 show that models with 3 to 9 billion parameters can carry a large share of production traffic at a fraction of frontier cost. We examine which tasks small models cover, what on-premises and edge deployment requires, how routing directs easy traffic to cheap models, and how to measure quality per euro.

The Cost of Defaulting to Frontier Models

In 2023 the default architecture for language features was simple: send every request to the largest available model. That default is expensive in three currencies. Money: frontier APIs bill per token at premium rates. Latency: large models answer in seconds, not milliseconds. Data: every prompt leaves your infrastructure. In the production systems we operate at Blue IT Systems, a large share of requests are classification, extraction, or short summarization — tasks far below the capability ceiling of a frontier model. Paying frontier prices for that traffic is a habit, not a decision.

2024 changed the calculus. Microsoft released Phi-3 in April, Google released Gemma 2 in June. Both are open-weight model families in the 2-to-27-billion-parameter range that run on a single GPU — the smallest on a phone. The question is no longer whether small models are good. It is which traffic they can carry, and how to route it there.

A request arrives — the router classifies its difficulty. 1/4

What Small Models Deliver in 2024

Phi-3-mini has 3.8 billion parameters and was trained on 3.3 trillion tokens of heavily filtered web data plus synthetic data. Microsoft reports 69% on MMLU and 8.38 on MT-Bench — comparable to Mixtral 8x7B and GPT-3.5, models many times its size. Gemma 2 9B was trained with knowledge distillation from a larger teacher on 8 trillion tokens and reaches 71.3% on MMLU, ahead of Llama 3 8B. The mechanism behind both is the same: better training data and distillation substitute for parameters.

The table shows vendor-reported figures. They come from different evaluation pipelines and are indicative, not directly comparable. GPT-4o mini is included as a hosted reference point: it scores higher, but its weights are closed and every request leaves your infrastructure. Benchmark scores also say little about your specific task; an in-domain evaluation set of a few hundred examples says more.

Model	Parameters	Release	MMLU (5-shot)	Context
Phi-3-mini	3.8B	Apr 2024	69%	4K / 128K
Phi-3-small	7B	May 2024	75%	8K / 128K
Gemma 2 9B	9B	Jun 2024	71.3%	8K
GPT-4o mini (hosted reference)	not disclosed	Jul 2024	82%	128K

Where 3 to 9 Billion Parameters Suffice

Small models suffice where language competence matters more than world knowledge: intent classification, entity and field extraction, short-form summarization, rewriting, structured output such as JSON, and answering questions over context supplied by retrieval. In a RAG pipeline the retriever provides the facts; the model only has to read, select, and formulate. That is precisely the profile of a 3-to-9B model, and it covers a surprising share of enterprise workloads.

They do not suffice for everything. Small parameter budgets store less factual knowledge — the Phi-3 report itself notes weak TriviaQA results and recommends search augmentation. Multi-step reasoning degrades faster than on large models. Quality outside English is thinner. And long agentic chains amplify per-step error rates. State this openly in design reviews: a small model is a component with a narrow contract, not a general substitute.

On-Premises and Edge Deployment

Open weights enable deployments that API models exclude. Under GDPR and sector-specific rules, many of our customers cannot send documents to a US-hosted API. A 4-bit quantized Phi-3-mini occupies roughly 2.3 GB and runs at more than 12 tokens per second on an iPhone 14, per Microsoft's report. Gemma 2 9B fits on a single 24 GB consumer GPU; the 27B variant runs on one H100. Tooling is mature: llama.cpp, Ollama, ONNX Runtime, and vLLM cover everything from laptop to server.

The trade-off is operational. Model updates, security patches, evaluation, and capacity planning are yours. A hosted API improves silently; your on-premises model does not. Budget for this in euros and headcount — self-hosting is a product decision, not merely an infrastructure decision.

Routing Easy Traffic to Small Models

A router places each request: easy traffic goes to the small model, hard traffic to the large one. RouteLLM from LMSYS (July 2024) trained routers on Chatbot Arena preference data and reports 95% of GPT-4 quality on MT-Bench with cost reductions of up to 85% compared to using GPT-4 alone. The framework and datasets are open source.

In production we mostly see simpler variants: rule-based routing by endpoint or task type, and cascades that try the small model first and escalate on low confidence or failed validation. Two caveats. The router is itself a model and misroutes; it needs its own evaluation set and monitoring. And published savings do not transfer — they depend entirely on your traffic mix. Measure the share of easy traffic before promising numbers.

Quality per Euro

Hosted small models set the reference price. GPT-4o mini has cost 15 cents per million input tokens and 60 cents per million output tokens since July 2024 — an order of magnitude below frontier pricing. Any self-hosted deployment must beat that price after including hardware, energy, and engineering time. At low or spiky volume it usually cannot: a rented GPU sitting idle is more expensive than any API call it never makes.

Self-hosting wins under three conditions: steady high utilization, data-residency constraints that rule out external APIs, or hard latency budgets. The deciding metric is quality per euro per task — measured on your own evaluation set, not on MMLU. We have seen tenfold cost reductions where the traffic profile fits, and none where it does not.

Outlook from October 2024

Three developments look robust from today's vantage point. First, distillation pipelines will keep compressing frontier capability into small models with a lag of six to twelve months; the 2024 release cadence of Phi and Gemma suggests this is now routine. Second, consumer hardware is being built for local inference: Copilot+ PCs ship with NPUs, and Apple has announced an on-device model of roughly 3 billion parameters for Apple Intelligence.

Third, we expect heterogeneous model fleets — one frontier model, several small specialists, and a router — to become the default production architecture in 2025. If that holds, engineering teams will differentiate on routing quality and evaluation infrastructure, not on model choice. The frontier model becomes the escalation path, not the default.