15 Sep 2024 AI Engineering 4 min read

OpenAI o1: Test-Time Compute and the Price of Thinking

OpenAI's o1-preview and o1-mini, released September 12, 2024, trade latency and cost for accuracy by spending compute on hidden reasoning tokens at inference time. We examine where trained deliberation helps — math, code, planning — what it costs, why the chain of thought stays hidden, and what test-time compute means for engineering.

The Limits of Fast Answers

A large language model spends roughly the same amount of compute on every token it generates. Whether the question is trivial or an olympiad problem, the forward pass is the same. For tasks that require search, decomposition and backtracking — competition mathematics, algorithmic programming, multi-step planning — that fixed budget is the bottleneck. The model commits to an early token and cannot revise it.

Prompt engineering has treated the symptom. Chain-of-thought prompting, published by Wei et al. in January 2022, asks the model to write out intermediate steps and measurably improves accuracy. But the reasoning itself remains untrained: the model imitates step-by-step text; it was never optimized to find correct steps. Since GPT-3, the primary lever has been pretraining compute. That lever now has a counterpart.

A request arrives — the router classifies its difficulty. 1/4

What OpenAI o1 Actually Does

On September 12, 2024, OpenAI released o1-preview and o1-mini, the first models of a new series trained with reinforcement learning to produce a long internal chain of thought before answering. According to OpenAI, the training teaches the model to recognize its own mistakes, break hard steps into smaller ones, and switch approaches when one fails.

In the API, this thinking materializes as reasoning tokens. They occupy space in the 128,000-token context window and are billed as output tokens, but they are not returned. Only their count is visible in the usage object. Output including reasoning is capped at 32,768 tokens for o1-preview and 65,536 for o1-mini.

Test-Time Compute as a Second Axis

OpenAI reports that o1's accuracy improves smoothly along two axes: more reinforcement learning during training, and more time spent thinking at inference. The second axis is the new one. Test-time compute means a model can buy accuracy with inference budget on a per-request basis instead of requiring a larger model.

The AIME 2024 numbers illustrate the scaling. o1 solves 74% of problems with a single sample, 83% with consensus over 64 samples, and 93% when 1,000 samples are re-ranked with a learned scoring function. Same weights, three compute budgets, three results. The relevant constraint shifts from parameter count to inference spend.

Where Deliberate Reasoning Pays Off

The published numbers concentrate the gains where solutions can be checked: competition mathematics, competitive programming, and hard science questions. Single-sample (pass@1) results from OpenAI's evaluation:

o1 is also the first model to surpass recruited PhD experts on GPQA Diamond. For code, o1-mini is the relevant option: it nearly matches o1 on Codeforces at an 80% lower price than o1-preview. Planning tasks profit for the same reason. Wherever a problem rewards trying, checking and revising steps, trained deliberation beats a single pass.

Benchmark (pass@1)	GPT-4o	o1-preview	o1
AIME 2024	9.3%	44.6%	74.4%
Codeforces (Elo / percentile)	808 / 11th	1,258 / 62nd	1,673 / 89th
GPQA Diamond	50.6%	73.3%	77.3%

The Price in Latency and Cost

o1-preview costs $15 per million input tokens and $60 per million output tokens; o1-mini is 80% cheaper at $3 and $12. Reasoning tokens are billed at the output rate even though they are never shown. A single request can consume hundreds to tens of thousands of them; OpenAI recommends reserving around 25,000 tokens for reasoning and output.

Latency follows the same curve: responses take seconds to minutes. The beta is narrow: no system messages, no streaming, no function calling, no image input, API access only for usage tier 5 at 20 requests per minute, and ChatGPT Plus launched with 30 o1-preview messages per week. If the token limit is hit mid-reasoning, you pay without receiving visible output.

Hidden Chains of Thought

OpenAI does not show the raw chain of thought. Users see a model-generated summary. The stated reasons: the unaltered chain must remain free of policy training so it can be monitored for signs of manipulation, plus user experience and competitive advantage. OpenAI acknowledges the disadvantages of this decision openly.

For engineering, this changes the trust model. Intermediate steps cannot be audited, logged or diffed. Costs cannot be predicted precisely before a request runs. Verification has to move to the output: unit tests for generated code, recomputation for numeric results, schema checks for extracted data. We treat o1 as a stronger black box — not a transparent one.

What o1 Does Not Solve

o1 is not the better model for everything. In OpenAI's own human preference evaluations, GPT-4o was preferred for personal writing and editing text; o1-preview won clearly on computer programming, data analysis and mathematical calculation. Deliberate reasoning pays only when a problem actually requires search. For retrieval, formatting or short factual answers, it adds cost without accuracy.

Prompting habits also invert. OpenAI advises simple, direct prompts; few-shot examples and instructions to think step by step can degrade performance because the model already reasons internally. Our working pattern at Blue IT Systems: route the bulk of a workload to a fast default model and escalate isolated hard subproblems to a reasoning model — with output verification on both paths.

Where Test-Time Compute Goes Next

OpenAI researcher Noam Brown wrote on release day that future versions should be able to think for hours, days, even weeks. If that holds, test-time compute becomes a second scaling law next to pretraining — and how long a query may think becomes an engineering parameter, the way a timeout budget is today.

We expect three developments: prices fall, as they did from GPT-4 to GPT-4o mini; routers choose automatically between fast and slow models, which OpenAI already names as a goal for ChatGPT; and reasoning models take the planner role in agent systems while cheaper models execute. The open question is whether hidden reasoning remains acceptable once such systems make consequential decisions.