Engineering notes
Since 2023 we have documented what actually works in AI systems — with numbers, sources and honest limits.
Agentic workflows replace single-pass prompting with iterative loops: reflection, tool use, planning, and multi-agent collaboration. We review the measured gains behind each pattern — from Reflexion's HumanEval results to Devin's SWE-bench score — state what the techniques do not deliver, and argue for starting with the simplest loop that an evaluation can verify.
Anthropic's Claude 3 family — Opus, Sonnet, and Haiku — replaces the single-model default with a tiered portfolio: one API, three price points, vision input, and a 200K context window across all tiers. We outline the verified launch numbers, the trade-offs, and a practical routing pattern that assigns each workload the cheapest model passing its evaluation.
Google's Gemini 1.5 Pro accepts up to one million tokens of context. Does that make retrieval-augmented generation obsolete? We run the numbers: per-query cost, latency, multi-needle recall, freshness, and access control. The conclusion from February 2024: long context changes how retrieval pipelines are built — it does not replace them.
OpenAI's text-embedding-3 models, released on 25 January 2024, support shortening vectors via a dimensions parameter. We explain the technique behind it — Matryoshka Representation Learning — verify the benchmark numbers, calculate storage savings at scale, and show why migrating from ada-002 still means re-embedding the entire corpus.
A field taxonomy of RAG failure modes as of January 2024: retrieval misses, stale indexes, the lost-in-the-middle effect, and citation drift. We define each mode, cite the measurements behind it, and argue for evaluating retrieval and generation as separate systems — recall@k on one side, faithfulness metrics on the other.
JSON mode from OpenAI's DevDay 2023, function calling as a schema contract, grammar-constrained decoding with Outlines and llama.cpp, and validation-first pipelines built on Pydantic v2. A survey of structured output techniques as of January 2024: what each method guarantees, what none of them guarantee, and why structure beats parsing prose.
Google announced Gemini 1.0 on 6 December 2023 in three sizes: Ultra, Pro and Nano. We separate delivered capability from marketing — benchmark methodology and the edited demo video — and examine what a second frontier-class vendor means for builders: routing policies, provider-neutral interfaces and task-level evals instead of single-vendor bets.
Pure vector search fails on part numbers, error codes, and other exact identifiers. We explain why, define BM25 and dense retrieval precisely, show how Reciprocal Rank Fusion combines both without score normalization, and review measured gains from hybrid retrieval and early rerankers such as Cohere Rerank — with numbers from 2023 benchmarks.
On November 6, 2023, OpenAI announced GPT-4 Turbo with a 128k context window, the Assistants API with Retrieval and Code Interpreter, and GPTs. We summarize the verified numbers, the new API primitives, and the concrete limits — and state what remains your responsibility: evaluation, data governance, cost control, and vendor risk.
Why demos and gut feeling are not an evaluation strategy for LLM applications. We cover golden test sets, model-graded evaluation with GPT-4 as judge, the position and verbosity biases quantified by Zheng et al. in 2023, and a layered evaluation stack that runs in CI — including what each technique does not measure.
OpenAI opened GPT-3.5 Turbo fine-tuning on August 22, 2023. We define what fine-tuning actually improves — output format, tone, narrow tasks — and what it does not: adding knowledge. Using the October 2023 price list we calculate training cost, the eightfold inference premium and the break-even, and show why retrieval-augmented generation remains the correct channel for facts.
Spring 2023 made autonomous agents famous: AutoGPT passed 74,000 GitHub stars within weeks, BabyAGI fit into 140 lines of Python. Six months later, few run in production. We trace the ReAct pattern behind the hype, quantify runaway loops and real costs, and describe the constrained tool-use loop that works: budgets, schemas, checkpoints.