22 Dec 2023 AI Engineering 4 min read

Gemini and the New Model Competition

Google announced Gemini 1.0 on 6 December 2023 in three sizes: Ultra, Pro and Nano. We separate delivered capability from marketing — benchmark methodology and the edited demo video — and examine what a second frontier-class vendor means for builders: routing policies, provider-neutral interfaces and task-level evals instead of single-vendor bets.

One Frontier Vendor Was Never a Strategy

For twelve months the top of the model market had one occupant. GPT-4, released in March 2023, set the reference for complex reasoning, and most serious LLM applications were built against a single vendor's API. That is concentration risk: one price list, one deprecation policy, one set of rate limits, one outage story. The OpenAI governance crisis in November 2023 made the risk tangible — for five days it was unclear who would run the company behind most production LLM workloads.

On 6 December 2023 Google announced Gemini 1.0. Whether Gemini Ultra beats GPT-4 is a benchmark debate. That a second vendor now credibly plays at the frontier is a structural fact. It shifts the question from "which model is best" to "how do we build so that the answer is allowed to change".

A request arrives — the router classifies its difficulty. 1/4

What Google Announced on December 6

Gemini 1.0 comes in three sizes: Ultra for complex tasks, Pro for the broad band of workloads, Nano for on-device use. All three are natively multimodal — trained from the start on text, code, audio, image and video — with a context window of 32,768 tokens. Google reports state-of-the-art results for Ultra on 30 of 32 academic benchmarks.

What shipped on day one was less. Bard has run on a fine-tuned Gemini Pro since 6 December (English, more than 170 countries). The Pixel 8 Pro uses Gemini Nano for summaries in the Recorder app and Smart Reply in Gboard. Since 13 December, Gemini Pro is available via API in Google AI Studio (free, 60 requests per minute) and in Vertex AI. Ultra remains in private preview; "Bard Advanced" is slated for early 2024.

Reading the Benchmarks Carefully

The headline: 90.04% on MMLU, above the human-expert mark of 89.8% and ahead of GPT-4's 86.4%. The footnote: the 90.04% is produced with "CoT@32" — 32 sampled chains of thought with uncertainty routing. GPT-4's 86.4% is standard 5-shot. Same method, same metric: Gemini Ultra scores 83.7% 5-shot — behind GPT-4.

This is not fraud; the details are in the technical report. But it shows what vendor benchmarks are: marketing constrained by footnotes. The multimodal result is more robust — 59.4% on MMMU is state of the art. Our conclusion stays the same: benchmark deltas of a few points do not predict performance on your task. Only your own evals do.

The Demo and the Delivered Reality

The launch video "Hands-on with Gemini" showed a model responding fluently, in real time, by voice, to live video. It was viewed millions of times. On 7 December, Bloomberg reported — and Google confirmed — that the interactions were built from still frames and text prompts, then edited; latency was cut and outputs were shortened. Real-time voice-and-video interaction was not demonstrated because it was not delivered.

The actual capability — strong image understanding on still frames — is roughly on the level of GPT-4V, which OpenAI introduced in September 2023. Native multimodality is a real architectural difference. In December 2023 it is not yet a product difference you can call through an API. Plan against the API surface, not against the promotional video.

Three Sizes Make It a Portfolio

The more durable signal is the packaging: one family, three sizes, three target platforms. This is the same tiering logic as GPT-4 next to GPT-3.5 Turbo, or Claude 2.1 next to Claude Instant — plus an explicit on-device tier. Nano-1 has 1.8 billion parameters, Nano-2 has 3.25 billion, 4-bit quantized and distilled from the larger models.

Model selection thus stops being a single decision and becomes a routing policy: which tasks need the frontier tier, which run on a mid tier at a fraction of the cost, which run on the device without data leaving it. That policy has to be re-evaluated whenever a vendor moves.

Model	Parameters	Status on 22 December 2023	Target platform
Gemini Ultra	not disclosed	Private preview for selected partners; Bard Advanced planned for early 2024	Data center (TPU)
Gemini Pro	not disclosed	In Bard since 6 Dec; API in AI Studio and Vertex AI since 13 Dec	Cloud API
Gemini Nano	1.8B (Nano-1) / 3.25B (Nano-2)	Shipping on Pixel 8 Pro (Recorder and Gboard)	On-device via AICore on Android 14

What a Second Strong Vendor Changes

Concretely, four things. First, price pressure: Gemini Pro is free in preview and will be billed per 1,000 characters at general availability; near-frontier capability is being priced against GPT-3.5-class budgets. Second, negotiating position: enterprise buyers now have a second bid. Third, failover becomes realistic: two APIs of comparable capability turn multi-vendor operation into an engineering task instead of a wish.

Fourth, it forces architecture. We treat model access as a replaceable dependency: a thin provider-neutral interface, prompts under version control, task-level evals that run against every candidate model. An abstraction layer does not make prompts portable — prompt behavior differs per model and must be re-evaluated on every switch. And note the third force: Mixtral 8x7B, released on 11 December with open weights under Apache 2.0, reports GPT-3.5-level results. The portfolio is not just two vendors.

Outlook From December 2023

Predictions we are willing to write down: Gemini Ultra ships in early 2024 and lands roughly at GPT-4 level; the frontier becomes a band, not a point. Benchmark headlines lose authority; task-level evals become the procurement instrument. Prices per token keep falling as the mid tier commoditizes.

Real-time multimodality — the thing the video anticipated — arrives later than the marketing suggests, but it arrives; the video describes a target state, perhaps 2024/25. For builders the job is unchanged: architectures that survive a model swap. The best time to remove a hard vendor dependency was before December 6. The second-best time is now.