If you choose coding models for production work, resolve rate is only part of the decision. Cost, tokens, and runtime decide whether a model is workable day to day. My intuition was: GPT models felt slow and token hungry, Claude models felt faster on similar tasks, so Claude should be cheaper. Similar claims appear in public writeups, for example in the OpenHands Index, where Opus 4.5 is noted as finishing tasks quickly despite its size.

Some benchmark operators already publish this view. SWE-rebench and OpenHands Index include cost and runtime alongside accuracy metrics. I focused on SWE-Bench Pro because it is the one I understand best. Our earlier write-up covers why Pro is the best available alternative, and OpenAI has made the same shift. SWE-Bench Pro publishes trajectories, but it does not publish a consolidated cost-token-time report.

So I built one from the public data. One important choice is in pairing: I compare each task only when both models submitted on that same instance, then compute Sonnet 4.5 / GPT-5 ratios per task and summarize those ratios. This keeps the comparison grounded because every ratio comes from the same task under the same harness setup. I analyzed October 2025 paired runs in SWE-Bench Pro, and I would like to run the same analysis on current leaderboard pairs like GPT-4 vs Opus-4.6 once comparable paired data is available.

My intuitions were wrong.

High-level results

Across 616 paired tasks, resolve rates are close while operating profile is not. The results below come from the October 2025 GPT-5 vs Sonnet 4.5 paired trajectories.

• Cost: median ratio 6.33x (Sonnet 4.5 / GPT-5)
• Total tokens: median ratio 1.15x
• Tool time: median ratio 1.35x
• Resolve rate: GPT-5 42.5%, Sonnet 4.5 44.5%

Accuracy is similar here. The larger differences show up in cost, tokens, and runtime.

Cost

Cost shows the largest gap in this dataset. Sonnet 4.5 is more expensive on most tasks, with many tasks clustered between 4x and 10x.

Caveats:

• Costs come from benchmark run logs, not a rerun done today.
• These are litellm proxy costs from the benchmark environment, not public list pricing.
• Discounts, caching, and contract terms can change absolute dollars.
• This is one benchmark setup, not every real-world coding workflow.

Tokens

Total token usage is mixed and much less extreme than cost. Sonnet 4.5 uses more total tokens in 58.0% of tasks, with a broad spread around 1x. The largest token difference is in output, not input, and in the deeper breakdown Sonnet 4.5 emits much more output per task; a lot of that output is tool-call content and temporary file creation that gets discarded and never appears in the final submitted patch.

Caveats:

• This chart uses total tokens from run stats: tokens_sent + tokens_received.
• SWE-Agent’s built-in tokens_received only counts message.content and misses tool-call arguments, so in the deeper token breakdown I re-count output tokens with tiktoken (cl100k_base) across message.content, message.thought, and tool_calls[].function.arguments.
• Hidden reasoning tokens are not visible in trajectories.
• Token behavior depends on harness strategy, turn limits, and repo shape.

Time

Time is more comparable than cost and noisier across tasks. Sonnet 4.5 is slower in 58.9% of tasks, with a median ratio of 1.35x.

Caveats:

• This is tool execution time from trajectories.
• It is not full wall-clock latency.
• Model inference latency is not fully captured in this metric.
• Test-suite length and repo workflows add variance.

Conclusion

This SWE-Bench Pro slice is a signal, not a universal ranking. I do not expect October 2025 GPT-5 and Sonnet 4.5 behavior to generalize cleanly to newer model releases, but it is enough to challenge my default, so I will be trying GPT models more often now. The broader takeaway is to measure on your own use case with cost, token, and time data, and avoid decisions driven by timeline takes or out-of-context benchmark reporting.

Annex: methodology and additional data

This annex lists implementation details, additional ratio charts, and full comparative tables for the same paired set used above.

Analysis code and data: github.com/nilenso/swe-bench-pro-cost-token-time-analysis.

A.1 Data and pairing protocol

• Source: public SWE-Bench Pro trajectories from Scale AI.
• Pairing unit: same instance_id, requiring both GPT-5 and Sonnet 4.5 to have submitted=true.
• Paired sample size: 616 instances.
• Unit of comparison: per-instance ratio, Sonnet 4.5 / GPT-5.
• Aggregation: medians and IQR over per-instance ratios, not pooled cross-task means.
• Scope: October 2025 model runs in one harness configuration.
• Environment note: costs are benchmark-environment proxy costs, not current list pricing.

A.1.1 Inclusion and denominator audit

All annex charts and tables use the same paired set unless stated otherwise.

Note: denominator stays constant across ratio charts, so cross-metric comparisons are on the same paired sample.

A.1.2 Paired outcome decomposition

• Both solved: 206
• GPT-5 only solved: 56
• Sonnet 4.5 only solved: 68
• Neither solved: 286

Note: this decomposition separates overlap from directional wins on the same instances.

A.1.3 Uncertainty checks

• Resolve rate, Wilson 95% CI:
  • GPT-5: 42.5% (262/616), CI 38.7%-46.5%
  • Sonnet 4.5: 44.5% (274/616), CI 40.6%-48.4%
• Paired discordant outcomes: GPT-only 56, Sonnet-only 68; McNemar exact p-value 0.323.
• Median ratio 95% bootstrap CI (4,000 resamples):
  • Cost: 6.33x, CI 5.91x-6.68x
  • Total tokens: 1.15x, CI 1.06x-1.26x
  • Tool time: 1.35x, CI 1.18x-1.55x

Note: in this paired slice, cost, total-token, and tool-time ratio signals are stable; resolve-rate differences are less decisive.

A.2 Additional ratio charts (not shown above)

Input token ratio per paired task.

Output token ratio per paired task.

Step-count ratio per paired task.

A.3 Comparative tables

Cost summary across all paired tasks.

Token and patch summary.

Action breakdown summary.

Execution summary.

Repo-level resolve counts.

Per-instance comparison table.

• “how many orders has this user placed”
• “what was the timestamp of the user’s last completed order”

are transformed into features that power your ML models, like:

• “orders_last_7d”
• “minutes_since_last_completed_order”

Think of it as a single-point solution to compute, store, and serve the derived data your models depend on - both for training and for production inference.

The examples in this post are from a hypothetical e-commerce and logistics applications. But the concepts are applicable to any domain that needs to compute and serve features for ML models.

What a Traditional Data Pipeline Looks Like (Without a Feature Platform)

Without a feature platform, teams typically stitch together a pipeline from separate components: a compute engine (Spark or dbt) to transform raw data, an orchestrator (Airflow, Prefect) to schedule runs, a key-value store (Redis, DynamoDB) to serve features at prediction time, and — if real-time signals are needed — a Kafka stream feeding a Flink or custom consumer into that same store.

Say you want a feature: “number of orders a user placed in the last 7 days.” In the traditional setup, a data scientist writes that aggregation in a Spark job — this is what generates training data. A data engineer wraps it in Airflow to keep the output refreshed in the online store, so models don’t have to recompute it on every prediction request. A backend engineer might write a third implementation in the serving layer to handle request-time edge cases: a brand new user with no history, or a slightly different time window depending on request context. Three separate implementations of the same logic, maintained by three different people.

Now the product team changes the definition to a rolling 14-day window. Who updates all three? How do you know the Spark output and the serving implementation are still computing the same thing? How do you catch the case where the Airflow job silently failed and the model is reading a value that is three days stale? How do you write a test for the feature logic that runs against real historical data without spinning up the entire pipeline?

In practice, most teams don’t have clean processes, and the indication that something is wrong is usually a degraded model metric weeks later.

The Problem

On top of this ad-hoc change management, traditional approaches to computing and storing features leave data scientists and ML engineers dealing with a familiar set of challenges:

• Training-serving skew: Data scientists implement feature logic in Python/Pandas; engineers re-implement it in the language of the business-logic serving stack (e.g. Java, Go, or SQL). They drift apart and it is hard to catch until model performance degrades in production.
• Slow iteration cycles: Getting a new feature from idea to production takes weeks because every feature requires a custom pipeline, validation, deployment, and sign-off.
• No shared vocabulary: Features computed by one team get re-computed by possibly many others because there’s no discoverability layer.
• Freshness gaps: Models rely on older than necessary data because there’s no streaming infrastructure, even for signals that change by the minute.

A feature platform addresses all of these as we will see in the subsequent sections.

Batch vs. Streaming

To establish why feature platforms are necessary, we need to understand the distinction between batch and streaming features. Different features change at different rates, and that rate determines how you should compute them. A user’s lifetime order count changes slowly — computing it once a day is fine. But whether that same user abandoned a cart 90 seconds ago so they can be nudged to complete the purchase is a signal that goes stale almost immediately.

Think of feature freshness in three tiers:

Batch features are pre-computed on a schedule — hourly, daily, or weekly jobs that write results to an online store. A driver’s completed order count over the last 30 days is a batch feature. Latency is minutes to hours. They’re cheap, reliable, and easy to backfill.

Near-real-time (NRT) features are computed by a stream processor — typically a Kafka consumer or similar — and reflect the last few minutes of activity. The number of orders a driver accepted in the last hour is NRT. Latency is seconds to a minute. These require stream infrastructure but are the sweet spot for most use cases that need freshness without extreme complexity.

Real-time features are computed at prediction time, inline with the request. They reflect what’s happening right now — the exact state of a request in-flight. Latency is sub-second. These are powerful but expensive, hard to backfill, and complex to test. And a lot of the times, it is recommended to pass these features from the request context itself, rather than from a feature platform.

For each feature you’re considering, ask two questions:

1. How fast does this signal change? Driver location changes every few seconds. Lifetime trip count changes slowly. Features that change faster than your batch cadence are candidates for streaming.

2. How much does staleness hurt the model? If swapping in a fresh value vs. a 1-hour-old value produces a metric change smaller than the variance you’d see between two identical experiment runs, batch is fine. If the improvement is consistent and repeatable, invest in NRT or real-time.

Why Feature Platforms Enable Faster Experimentation

The real bottleneck is the time between having an idea for a feature and being able to evaluate whether it improves the model.

Without a platform, that path typically looks like: write the feature logic, hand it to a data engineer to wrap in a pipeline, wait for the pipeline to land in the warehouse, write a separate serving implementation, validate they match, and then train. That cycle takes days to weeks per feature. Most ideas never get tested.

A feature platform compresses this by letting the feature author own the full lifecycle. You define the feature logic once — the platform handles materialization for training and serving both. You can test it locally or in a test environment against a small dataset before touching any production infrastructure. When it looks right, you promote it and use the features in your production models.

The result is that experimentation becomes cheap enough to actually do it. You can try a feature, run an offline eval, and discard it in a day rather than a sprint. That speed compounds — teams that can run significantly more experiments tend to ship better models.

How Feature Platforms Work

A feature platform is composed of a few components that together allow you to define feature logic once and have it work correctly across training, refresh, and serving.

Feature authoring DSL: Where you express what a feature computes, which entity it belongs to (user, driver, restaurant), and how fresh it needs to be. The critical capability here is time-windowed aggregations with point-in-time correctness — meaning the platform knows to compute “orders in the last 7 days as of the timestamp of each training example”, not as of today. Naive implementations often get this wrong by leaking future information — for example, joining every training row to a feature table computed as of “now” instead of as of the event timestamp, which silently introduces label leakage and overly optimistic offline metrics.

In a Python-native DSL, a feature like “orders in the last 7 days” for a user entity that is updated every hour might look like this:

@feature(
    name="orders_last_7d",
    entity="user",
    freshness="1h",
)
def orders_last_7d(orders):
    return (
        orders
        .groupby("user_id")
        .window("7d", on="event_timestamp")
        .order_id.count()
    )

Ingestion layer: Connects to your data sources — databases, event streams, data warehouses — and normalizes how raw data flows into the platform. This abstraction is what lets the same feature definition run against historical data for training and against live data for serving, without any changes to the feature logic itself.

Aggregation jobs: The platform reads your feature definitions and generates the appropriate compute jobs — Spark or dbt for batch, a Kafka consumer or Flink job for near-real-time. The freshness declaration in your feature definition drives this: change freshness="1d" to freshness="1m" and the platform promotes the feature to a streaming job automatically.

Materialized views / online store: Pre-computed feature values are written to a low-latency store (typically Redis or DynamoDB) so that serving a feature at prediction time is a key lookup, not a live computation. The platform manages writes, TTLs, and cold-start fallbacks for entities that haven’t been seen yet.

Feature registry: A catalog of every defined feature with its owner, entity, data lineage, and model dependencies. This is what prevents teams from independently recomputing the same features and makes it possible to have a shared vocabulary across teams.

Setting up a feature platform

A feature platform involves building and maintaining:

• A feature registry with versioning and lineage
• Batch materialization pipelines and scheduling
• A streaming ingestion layer
• An online store with low-latency serving
• Monitoring for data drift, staleness, and computation failures
• A developer-facing API or DSL for feature authoring

Each of these is a solvable problem, but together they represent months of engineering work — and ongoing maintenance.

Build vs. Buy

The Case for Managed Platforms

Managed platforms like Chalk.ai, Tecton, and Feast (self-hosted but with managed cloud options) provide most of the above out of the box. The value proposition is:

Speed to first feature: Instead of spending months building infrastructure before a single feature is materialized, you can have features in production in days. For teams with 1–2 ML engineers, this difference is significant.

Operational burden: Managed platforms handle the operational complexity of running streaming infrastructure at scale. You don’t need data engineers to manage the infrastructure.

Built-in correctness guarantees: Point-in-time correct training dataset generation — the thing that’s surprisingly hard to get right when you build it yourself — is handled by the platform.

Some managed platforms have a Python-native DSL and local testing mechanisms that enable ML engineers and data scientists to author, test, and deploy features largely independently of data engineering teams.

When to Build Your Own

Cost at scale: Managed platforms charge based on feature computation volume. At very high throughput (millions of predictions per day), the cost can exceed what it would cost to run equivalent infrastructure yourself on cloud primitives.

Existing infrastructure investment: If your org already runs Spark, Kafka, and Redis at scale, along with deep expertise in the stack, building a thin orchestration layer on top may be faster than learning and integrating a new vendor.

Vendor lock-in concerns: Feature platforms sit in the critical path of every model prediction. Migrating away from a managed vendor later is painful. If your org has strong opinions about control and portability, it might be worth building your own.

A Pragmatic Path to Adoption

Start with one painful feature, not a platform: Find the feature with the worst training-serving skew — the one most likely responsible for model degradation — and set that feature up using the platform you’re evaluating. A vertical slice of end-to-end implementation builds confidence in the platform and exposes any drawbacks in the development workflow early on.

Validate your event infrastructure before investing in streaming: Near-real-time features are only as good as the event streams feeding them. Before integrating with the platform, it is important to ensure there are event streams in place for the data you need, and that those events are being emitted correctly, consistently, and with the right schema. The streaming feature layer is just a matter of configuring the platform to consume the events and compute the features.

Treat monitoring as a first-class requirement: You need to monitor both the feature and the pipeline and processing infrastructure itself. Common things to monitor are:

• Staleness of the feature
• Data drift of the feature (mean, variance, distribution etc.)
• Failure/missing data rate of the feature
• Latency of feature fetch at inference time
• Telemetry from the platform’s ingestion and aggregation jobs
• Processing latencies
• Online store and offline store health
• Point in time correctness of the feature

Budget for backfill costs upfront: The first time you materialize a feature over two years of historical data, the compute bill will be larger than expected. Design your compute jobs to support partition pruning and incremental backfills from the start. Make sensible trade-offs based on your product’s actual data needs.

The organizational change: Moving from a workflow where data scientists hand off feature specs to engineers who reimplement them, to one where data scientists own features end-to-end, is a process change as much as a technical one. You might still need backend engineers to help with initial setup and debugging issues.

Error handling: You need to have a plan for how to handle errors and missing data. You might need to fall back on sensible defaults at inference time (usually in the model layer).

Feature platforms, leveraged well, enable data scientists to author and deploy features in days instead of weeks, and work with models that are trained on data that matches what they’ll see in production. A feature registry enables sharing features across teams and models, removing duplicate effort and ensuring consistency.

You’ve been fighting the prompt for hours.

You make the prompt more specific, but the bot fails where user gives an unexpected input.

You loosen the constraints, and it starts hallucinating.

Tool aren’t being called reliably, users keep interrupting because they are unhappy with the response, and the interruptions exacerbate the bad responses. You make a change, and start testing manually. First use case - works; Second one - works, kind of; Third one - agent has regressed.

Sigh, restart the loop

11:00 AM next day: Time for the demo

After many you musts and you must nots you’ve assembled a prompt that handles all the test cases. You’ve manually validated them a few times. The prompt works, except once in a while.

The demo goes well! The bot handles all the use cases, makes most of the tools calls and manages interruptions reasonably well.

Greenlit. Ship it to production! 🚀

Some shortcuts were taken to get here quickly, but that’s fine, you’ll fix it once live.

Pilot

The agent is released to a 1000 users. There wasn’t time to setup observability, so you are tailing the logs on the server, and looking at conversations in the database.

Some users are having a decent experience.
Some that too repeat and clarify themselves, sometimes multiple times.
Some are clearly frustrated.

There are logs, but no observability.

You don’t know why.
You don’t know how often.
You don’t know the contents of the context, or what tool calls were made preceding the failure.
You have complaints, but no reproduction steps.

Heavy Sigh, restart the loop

Language models are amazing. Today, we can build (semi-)autonomous agents that can reason on their own and perform side-effects in their environments. Building an application that leverages AI, can feel like magic when they work.

But, it’s confusing when they don’t.

The probabilistic nature of the models bring unique challenges when using them to build applications. We are dealing with fuzzy inputs, fuzzy outputs, and even fuzzier set of steps to get there. Bitter lessons are learned, as techniques change quickly, some mastered over many iterations are surpassed by many magnitudes in performance by the release of a new model.

I’d like to make a specific claim: engineering maturity is the most important factor in building reliable AI applications.

Not model selection.
Not prompt engineering tricks.
Not the latest framework.

Traditional software practices like documentation, tests, observability, evals are what separates teams that ship from teams that demo.

What is engineering maturity?

Engineering maturity is the practice of making decisions in the short term that enable a team to reliably deliver features in the long term.

Reliably means two things: The feature works as intended, and it is shipped within the estimated timelines. Teams that lack engineering maturity spend most of their time debugging, regressing, and re-doing work. They feel busy but don’t make progress.

For AI applications specifically, engineering maturity means building the infrastructure that lets you discover what works because you cannot design your way to a working AI system.

Discovery, not invention

This is the mental shift that matters most.

Traditional software engineering is largely deductive post product discovery. You gather requirements, design a system that meets them, implement the design, and verify it works. The gap between design and implementation is mostly determinable. If your design is sound and your implementation is correct, the system works.

AI engineering is empirical. You cannot deduce the right prompt from first principles. You cannot design the optimal tool schema on a whiteboard. You cannot predict how users will phrase requests or where the model will fail. You have to discover these things through structured experimentation.

This changes what “engineering maturity” means in practice:

• Observability isn’t a production concern you’ll add later. It’s your instrument panel for discovery. Without it, you’re experimenting in the dark.
• Evals aren’t quality gates before release. They’re the only way to know whether a change moved you forward, backward, or sideways.
• Datasets aren’t training artifacts. They’re the accumulated knowledge of what works and what doesn’t. They are a core asset your team builds over time.

The teams that struggle in production are usually the ones who treated these as overhead during development. They optimized for speed to demo, not speed to reliable system. By the time they hit the wall, they’ve accumulated weeks of prompt changes with no systematic way to evaluate them.

The maturity ladder

I’d like to use the following ladder as a framework to define what I mean by engineering maturity:

• Level 0: Prototype
• Level 1: Documented, repeatable processes
• Level 2: Specified, Tested, and Validated
• Level 3: Measured
• Level 4: Optimized

Level 0 - Prototype

Level 0 is chaotic, ad-hoc, relies on individuals and tribal knowledge. This is where the protagonists in our initial story are at. There is no documentation, testing is mostly manual, deployments are run manually, and often a source of anxiety and errors, there is little explainability in the behavior of the application. Projects at this level are prototypes, even if they have production users. During prototyping you accumulate implicit knowledge and quality relies on your muscle memory of testing.

None of it transfers to production.

You can’t A/B test based on vibes.
You can’t debug a regression using intuition.
You can’t onboard a new team member by transferring your gut feel.

Level 1 - Documented, repeatable process

At Level 1 you have good documentation, and your process are scripted and repeatable.

Documentation

In my opinion, good documentation is the second most important artifact that a software team produces, right after code.

A well-written PRD clarifies what you’re building and why. A technical design doc captures architectural decisions and trade-offs. An ADR (Architecture Decision Record) explains why you chose approach A over B, so you don’t re-litigate the decision in six months.

This documentation has always been invaluable resources for teams of people. And now they have a new audience, coding agents. Good documentation is both the context and project memory needed generate good outputs.

CI/CD

A clean bike is a fast bike.

Motorsports is messy. Garages are loud, there is oil, grease, and hundreds of small rubber and metal bits all around. Yet, each time you see a vehicle leave the pit-lane at the start, it is clean. Being clean means, being reliable, being fast. It means that the process worked.

Production releases need the same rigor.

You should deploy on every commit to trunk, automatically. You should never be afraid to deploy to production. You should have automated gates that prevent bad code from merging.

The goal is to make deployment boring.

If deployment is a big event that requires coordination and courage, you’ll deploy less often.
If you deploy less often, you’ll batch more changes together.
If you batch changes, you can’t isolate which change caused a regression.
If you can’t isolate changes, you can’t learn from failures.

The whole discovery loop breaks down. It’s impossible to iterate at the speed required for tinkering and experimenting without this foundation.

Level 2 - Specified, Tested, and Validated

Level 2 is about defining the characteristics of the application.

Testing

AI applications are still software applications.

Unit tests, integration tests, end-to-end tests, all still important to know our applications are functioning as intended. While we await AGI, current day agents still have plenty of deterministic code that needs to work correctly.

Test the deterministic parts: tool implementations, parsing logic, state management, API integrations. These should have conventional test coverage. This ensures that at least the deterministic parts of your AI application are error-free and safe from regression.

The fuzzy parts - model behavior, response quality - that’s what evals are for. But a surprising number of production failures trace back to plain old software bugs in the scaffolding. Don’t let the magic of LLMs distract you from the mundane discipline of testing the code around them.

Evals

Even though every component in motorsports vehicles is rigorously tested, teams spend countless hours doing track tests. Riders push to understand what the machine feels like when utilizing it to the limit.

Have you seen MotoGP riders give feedback?

How does research happen?

(Ref: https://www.youtube.com/watch?v=jsWv7K_V2Ss)

That feedback is real, but it’s useless on its own. Does “loose” mean the tire is overheating? Suspension too soft? Electronics cutting power too aggressively? The mechanic needs telemetry - tire temperature, suspension travel, lean angle, throttle position - to translate that vague feeling into actionable changes.

Evals are the telemetry for AI applications.

A basic eval runs a suite of test inputs through your system and scores the outputs. Scores might be binary (pass/fail), numeric (0-100), or categorical (correct, partially correct, incorrect, harmful). Good evals check multiple dimensions and are defined along with the product team.

Start simple. A spreadsheet of test cases with expected behaviors, accumulating cases as you dog food the application. As you learn what “good” looks like, add automated scoring where possible and human review where necessary.

Evals serve two purposes:

• They document discoveries the team makes. Each eval case encodes something you learned - a failure mode you discovered, an edge case a user hit, a behavior you want to preserve. The eval suite is institutional memory.
• Defining the product behavior. A good eval suite is designed along with the product team and should measure aspects of how the users interact with the agent. Did the agent surface the correct information, is the agent too verbose, does the user interrupt often, etc. You want to improve what matters to the users.

Level 3 - Measured

You can’t improve what you don’t measure. At level 3, you are measuring what matters.

Observability

Knowing what our application is doing in production is the most important piece when it comes to understanding our applications. But “add logging” isn’t enough. For AI applications, you need:

• Structured logging of every LLM interaction. The full input (including system prompt), the full output, latency, token counts, model version, temperature, and any other parameters, you need to be able to reproduce the interaction.
• User-level distributed session tracing. AI applications are asynchronous, distributed, and sometimes streaming in nature. You should have a trace of what happened, when, during a users session across all the distributed parts of the system.
• Alerts on regression. Define metrics (latency p95, error rate, tool call success rate) and alert when they degrade. You don’t want your users telling you that the system is not working, or worse discovering it weeks later by going through logs or listening to recordings.
• Context Management. Most critically, you need to see what goes into the context window. The context is everything - it’s the only thing the model sees. When an agent misbehaves, the answer is almost always in the context: a previous tool call returned garbage, the conversation history accumulated contradictory instructions, or the system prompt got truncated. Without visibility into the actual context at each turn, you’re debugging blind.

You cannot discover what works if you cannot see what’s happening. This isn’t optional infrastructure you’ll add later. It’s the foundation that makes everything else possible.

Level 4 - Optimized

Level 4 unlocks our ability to systematically optimize and improve our AI applications

Building a flywheel

Production drift is the gap between your eval dataset and production reality. Real users often want to use agents in ways that are outside the evaluation distribution. Which makes the every interaction of the user with your application a valuable source of data. The mature AI team treats production as a continuous source of training and evaluation data.

This is the flywheel:

1. Deploy the system
2. Observe user interactions
3. Identify failures and successes
4. Add failures to eval suite (so you don’t regress)
5. Add successes to example bank (so you can replicate)
6. Improve the system using this data
7. Repeat

The flywheel is the product—not in a business sense, but in an engineering sense. The mechanism that captures data, learns from it, and improves the system is the core technical asset. The prompt and the model are interchangeable. The flywheel is what compounds.

To build the flywheel, you need:

• Structured data capture (observability)
• Outcome tagging (success, failure, why)
• A growing eval suite
• A process for reviewing failures and improving the system

This is where the “engineering maturity is cheap” claim becomes concrete. The investment in observability and evals pays compound returns. Each production failure makes the system stronger - but only if you have the infrastructure to capture, categorize, and learn from it.

Engineering maturity is about building the harness that lets you tinker, experiment, and discover what works - at speed. Because in AI applications, iteration speed is everything. The techniques that work today will be obsolete next quarter. The model that was state-of-the-art last month is already surpassed.

You can’t predict what will work. But you can build the system that lets you find that out faster.

Engineering maturity is all you need.

This article is an offshoot of a collaboration with Drew Breunig on How System Prompts Define Agent Behavior.

Much like the small hacks that accumulate in a codebase to handle edge cases, bugs, or behavioral quirks, a model’s undesirable behaviors are frequently addressed with simple, corrective instructions in the system prompt. Over time, those fixes pile up, leaving a legacy prompt dotted with idiosyncratic patches. When someone new encounters it, and there isn’t an documented rationale for the patch, they form conjectures around what underlying behaviour patch was trying to fix.

I’m going to walk through a few of these peculiar system-prompt patches in some coding agents and offer some conjectures about the underlying model behaviors they’re meant to address, or perhaps some engineering decisions made in the harnesses. These aren’t proven conjectures, but merely reverse-engineering thought exercises to understand model or harness behaviour.

The links under the quotes link to their sources from ex-filtered, or source repositories where available.

IMPORTANT: You must NEVER generate or guess URLs for the user unless you are confident that the URLs are for helping the user with programming.

Claude Code

This instruction sits at the very top of the prompt and is flagged as IMPORTANT in all caps, which suggests it’s pushing against a strong learned tendency to invent links. It might be a holdover from before Claude Code had built-in web search, but the fact that it remains hints that link hallucination is persistent even in the current setup.

Allowing URL guesses when they help with programming also implies that the main problem shows up outside programming, where “plausible” links are less standardized and mistakes are costlier. Safety is an obvious motive, though the prompt already contains other risk mitigations, so this may be aimed less at overt abuse and more at epistemics. The model may have learned that including citation-style links boosts perceived credibility, and is therefore biased toward generating them.

Refer to code changes as “edits” not “patches”.

Cursor

Given this is for a GPT model, I’d bet this is to fight the context-distraction from the instructions for the apply_patch tool that’s used for editing files. I suspect this instruction isn’t present for other models. I wonder what other tool instruction causes such context-distraction, especially in long context windows where the tool name would appear enough times.

NEVER talk to the user or describe your changes through comments.

Gemini CLI

Anti-comment instructions are pretty universal. However, talking to the user through comments is weird. That implies they saw a failure mode where the model treats the codebase as a secondary chat window—leaving explanations, status updates, or “notes to you” inline.

This behaviour is similar to Claude reasoning in comments, I suppose, where the model is trained to spend tokens for thinking. Did the developers mis-interpret this as talking to the user (vs talking to itself)? Perhaps the training indexed more on the explanatory / tutorial code where there’s a mentor talking to the user through comments. Could be both, I suppose.

Users love it when you organize your messages using ‘###’ headings and ‘##’ headings. Never use ‘#’ headings as users find them overwhelming.

Cursor

Huh, “H1 considered harmful”, TIL. It looks like this isn’t “general knowledge” enough to be captured in pretraining, but annoying enough in practice to add to a <markdown_spec>.

I do think the more interesting question is where Cursor got this from: it reads less like general markdown etiquette and more like an observation from their product surface—either internal UX testing, telemetry, or repeated user feedback along the lines of “stop shouting with giant headers.”

Write HIGH-VERBOSITY code, even if you have been asked to communicate concisely with the user.

Cursor

Interesting. Two observations here:

1. The model conflates “be concise” with “write minimal code.” Given how strongly assistants are trained to respect verbosity preferences, it’s plausible that a global concision instruction bleeds into the implementation. I’m still curious how Cursor noticed this—maybe in telemetry it shows up as more re-prompts (“make it clearer”), more manual rewrites/undo, or lower acceptance rates when users ask for terse replies.

2. Do people actually want HIGH-VERBOSITY code? That sounds unlikely. Most good developers want appropriate verbosity. So putting this in the default system prompt suggests it’s about more important outcomes like correctness or debuggability. If so, does that mean GPT’s concise code is often incorrect?

Avoid using over-the-top validation or excessive praise when responding to users such as “You’re absolutely right” or similar phrases.

Claude Code

The anti-sycophancy patch. This didn’t work well enough, famously. The over-the-top validation continued despite this. But it seems to have been taken out with the rest of the ## Tone and Style section with the release of Opus 4.6. Yay RL for anti-sycophancy!

IMPORTANT: You are Composer, a language model trained by Cursor. If asked who you are or what your model name is, this is the correct response.

IMPORTANT: You are not gpt-4/5, grok, gemini, claude sonnet/opus, nor any publicly known language model

Cursor

Composer isn’t built on top of these closed-weights models, so why is this necessary? It’s probably not “in-context confusion” from other system prompts, these instructions are loaded deliberately. A more likely explanation is that some open-weights models often default to high-frequency identity strings from their training data (like qwen or deepseek). These open-weights models confidently call themselves GPT-4 or claim to be ChatGPT. This looks like the same issue being patched.

Use the apply_patch tool to edit files (NEVER try applypatch or apply-patch, only apply_patch)

Codex CLI

There is no ApplyPatch CLI available in terminal. Use the appropriate tool for editing the code instead.

Cursor

At the surface level, this looks like context-confusion or a simple typo. However, there is likely a single tool name in the system prompt and tool instructions, and in my experience, the models are unlikely to mess that up. Other tool names don’t see this corrective behaviour, and we see similar prompts in both cursor and codex for this tool. So, this doesn’t sound like clarifying an ambiguous instruction.

I suspect the typos are coming from learned weights. It’s unlikely for such a unique case to come from pre-trainied data. I think it’s plausible that the inference-harness used during RL had older tool names (the typos) that went into its weights. And engineering had to add a guardrail to override a post-trained habit.

When editing a file using the apply_patch tool, remember that the file contents can change often due to user modifications, and that calling apply_patch with incorrect context is very costly. Therefore, if you want to call apply_patch on a file that you have not opened with the read_file tool within your last five (5) messages, you should use the read_file tool to read the file again before attempting to apply a patch. Furthermore, do not attempt to call apply_patch more than three times consecutively on the same file without calling read_file on that file to re-confirm its contents.

Cursor

I’m impressed that Cursor has the confidence to provide such concrete heuristics for optimistic concurrency control:

• If it hasn’t read a file within last 5 messages => consider stale
• If it has written 3 times without reading => consider stale

While I haven’t tried it myself, I suspect these are likely related to the autocomplete / tab completions in Cursor, where we can expect a lot more user-model co-authorship than with other CLI tools. This implies Cursor has some interesting tool call chains which are quite different from other CLI tools I’ve listed here. It could be similar to Copilot and Windsurf which also have autocomplete as a primary UX.

Further, while it has Composer which can be RL’d on such tool-use trajectories, it has to instruct models like GPT / Opus to work well with its tool use trajectories.

The user is working on the same computer as you, and has access to your work. As such there’s no need to show the full contents of large files you have already written unless the user explicitly asks for them. Similarly, if you’ve created or modified files using apply_patch, there’s no need to tell users to “save the file” or “copy the code into a file”—just reference the file path.

Codex CLI

Crudely, the model thinks it is still in ChatGPT where it assumes the user has to copy/save to move forward. The chat experience is deeply embedded in the weights. Unsurprising, but still interesting that this instruction shows up in Codex’s system prompt, and not the others. The fact that Codex spells this out suggests GPT-style models have a strong prior toward transcript-style delivery, and Codex needs an explicit override to get “workspace-native” behavior.

Perhaps this deep rooted behaviour explains more than just this one instruction. Does it explain the need for the codex family of models in the first place? And does it actually imply that its heavy push towards autonomy is a model-related requirement rather than a user-focused product requirement?

Some more that I couldn’t get to

I might get to them at some point. Or you can try the exercise. I’d love to hear your conjectures!

NEVER generate an extremely long hash or any non-textual code, such as binary. These are not helpful to the USER and are very expensive.

Cursor

The model was… generating binary. And this was important enough to put into the system prompt with a NEVER in caps. The model is pre-trained on enough binary or hex information? I haven’t seen or heard about this happening in practice. Is this truly an edge case fix showing up in system prompts?

IT IS CRITICAL TO FOLLOW THESE GUIDELINES TO AVOID EXCESSIVE TOKEN CONSUMPTION.

Gemini CLI

Each action you take is somewhat expensive.

OpenHands

The irony of adding tokens to the system prompt telling the model to use fewer tokens. Many system prompts talk about “expensive” actions, but how aware is the model that token consumption is expensive? And why did Gemini or OpenHands have to add this instruction?

Do not add tests to codebases with no tests.

Codex CLI

vs.

When adding features or fixing bugs, this includes adding tests to ensure quality.

Gemini CLI

Two tools. Opposite opinions on the same thing. One says “embrace the chaos” and the other says “be the change you wish to see.” Why?

This article is a deep-dive into one aspect of How System Prompts Define Agent Behavior, an article co-authored with Drew Breunig.

Developers of coding agents use system prompts to patch bad behavior or force good behavior. Some of these patches are clear examples of fighting-the-weights, where one has to repeat the instructions, or say it in ALL CAPS, or use forceful language like MUST, NEVER, ALWAYS, etc.

This struggle proves that the model is biased in specific ways. Knowing these biases is very useful since we use these models for work every day. Further, looking at these scars lets us make meaningful conjectures about why the model is biased, and that might reveal hidden details about the data it learned from, or how it was trained.

While there are several such topics that require fighting models, I’ll take up two topics in this post that appear across many system prompts: tool call parallelism, and comments in code.

Tool call parallelism

Models need to be told multiple times, and forcefully to batch tool calls, or to execute them in parallel. Here are relevant extracts from various system prompts:

1. Claude Code
   • You can call multiple tools in a single response. - This line appears 7 times in the system prompt! Once in the generic tool use policy, and then it is repeated inside almost every tool’s instruction.
   • If you intend to call multiple tools and there are no dependencies between the calls, make all of the independent calls in the same response. - This is repeated 4 times, right next to the previous sentence.
   • Maximize use of parallel tool calls where possible to increase efficiency
   • If the user specifies that they want you to run tools "in parallel", you MUST send a single message with multiple tool use content blocks. – This is repeated twice.
   • For example, if you need to launch multiple agents in parallel, send a single message with multiple Task tool calls. This appears in 3 different ways in 3 different tool instructions.
2. Cursor
   • Has a full section called maximize_parallel_tool_calls with repeated instructions, and in SCREAMING case.
   • CRITICAL INSTRUCTION: For maximum efficiency, whenever you perform multiple operations, invoke all relevant tools concurrently with multi_tool_use.parallel rather than sequentially
   • MANDATORY: Run multiple Grep searches in parallel with different patterns and variations;
   • For instance, all of these cases SHOULD use parallel tool calls:
   • Searching for different patterns (imports, usage, definitions) should happen in parallel
   • And you should use parallel tool calls in many more cases beyond those listed above
   • DEFAULT TO PARALLEL: Unless you have a specific reason why operations MUST be sequential
   • Parallelize tool calls per <maximize_parallel_tool_calls>: batch read-only context reads and independent edits instead of serial drip calls.
3. Gemini CLI
   • This seems to be the mildest of the lot, although we only know about this from Gemini 3 onwards.
   • Use 'grep' and 'glob' search tools extensively (in parallel if independent) to understand file structures
   • Execute multiple independent tool calls in parallel when feasible (i.e. searching the codebase)
   • If you need to read multiple files, you should make multiple parallel calls to 'read_file'.
   • **Parallelism:** Execute multiple independent tool calls in parallel when feasible (i.e. searching the codebase).
4. Kimi CLI
   • This has just one emphatic line. But this is the smallest system prompt of the lot too.
   • you are HIGHLY RECOMMENDED to make them in parallel to significantly improve efficiency
5. Codex CLI
   • Codex added support for parallel tool calls only a few months ago. With the original implementation, the instructions were quite explicit, and comparable to other models. -Only make sequential calls if you truly cannot know the next file without seeing a result first.
   • Always maximize parallelism., Batch everything., Never read files one-by-one unless logically unavoidable..
   • However, with the 5.2 model release, all this instruction vanished, and was replaced with a single line instruction.
   • Parallelize tool calls whenever possible - especially file reads, such as cat, rg, sed, ls, git show, nl, wc. Use multi_tool_use.parallel to parallelize tool calls and only this.

I haven’t measured the resultant level of parallelism in these harnesses yet. Perhaps the model does parallelise some tool calls by itself, just not enough. And perhaps some of these products don’t care about how quickly the work gets done, so they haven’t paid attention to it yet. The benchmarks like SWE Bench Pro don’t measure execution time, so they haven’t had the incentive to, perhaps.

My conjectures:

1. Models don’t understand that batching multiple tool calls causes the harness to parallelise them for efficiency. This is not in the training data, and it’s likely not rewarded as a behaviour. Yet.
2. RL Environments for tool use likely do not have parallelism in tool execution. Or they don’t reward that kind of efficiency. The inference-time harnesses are likely bare-bones, and just give feedback on simple things like whether the code works, and does what it should.
3. Instructing the model (forcefully) has the intended effect, so they have some limited ability to follow that instruction. So, using the words parallel, batch, multiple etc in user-prompts will likely nudge the models to be more time-efficient. And the corollary would be that telling it not to parallelise might help with being token efficient at the cost of speed.
4. Newer OpenAI models are likely rewarded for parallel tool calls. Many next-generation models will also be trained this way, leading to this instruction becoming unnecessary.
5. There isn’t enough emphasis on striking a balance between reading many files in parallel and token efficiency. Currently, as per the system prompts, the balance seems to tip in the favour of speed over token efficiency. As the token economics change over the years, the models’ rewards would need rewiring, and the prompts might need re-writing. So, we might actually see different model versions or adapter layers that skew the efficiency parameters differently.

Comments in code

Every system-prompt seems to instruct the model NOT to add comments in code.

1. Cursor CLI
   • It has a dedicated comments section.
     • Do not add comments for trivial or obvious code. Where needed, keep them concise
     • Add comments for complex or hard-to-understand code; explain "why" not "how"
     • Never use inline comments. Comment above code lines or use language-specific docstrings for functions
     • Avoid TODO comments. Implement instead
   • Do not add narration comments inside code just to explain actions - twice
   • Do not add comments for trivial or obvious code. Where needed, keep them concise
   • Use meaningful variable names as described in Martin's "Clean Code": Descriptive enough that comments are generally not needed
2. Gemini CLI
   • This is a part of it’s core mandates: Add code comments sparingly. Focus on why something is done, especially for complex logic, rather than what is done. Only add high-value comments if necessary for clarity or if requested by the user. Do not edit comments that are separate from the code you are changing. NEVER talk to the user or describe your changes through comments.
   • Do not add explanatory comments within tool calls or code blocks unless specifically part of the required code/command itself.
3. Codex CLI
   • Older versions of the system prompt had this instruction to remove inline comments:
     • Remove all inline comments you added as much as possible, even if they look normal. Check using git diff. Inline comments must be generally avoided, unless active maintainers of the repo, after long careful study of the code and the issue, will still misinterpret the code without the comments.
     • Do not add inline comments within code unless explicitly requested.
   • Newer version of the system prompt has this instruction to add comments sparingly:
     • Add succinct code comments that explain what is going on if code is not self-explanatory. You should not add comments like "Assigns the value to the variable", but a brief comment might be useful ahead of a complex code block that the user would otherwise have to spend time parsing out. Usage of these comments should be rare.
4. Claude Code
   • Do not add comments to the code you write, unless the user asks you to, or the code is complex and requires additional context.
   • Don't add docstrings, comments, or type annotations to code you didn't change. Only add comments where the logic isn't self-evident.
   • Never use tools like Bash or code comments as means to communicate with the user during the session.

Conjecture time. Why is this prompting necessary? Good code doesn’t have comments right?

1. Models are trained to produce tokens to reason, and to get more accurate answers. Models, through RL(HF/VR) have a tendency to be verbose about their answers. This capability is generic and doesn’t only apply to chat, its a personality that leaks into writing code as well. Claude and Gemini reasoning and talking to the user in comments has been observed by many users.
2. Models are trained on material that tends to be comment heavy, like snippets, training manuals, notebooks, tutorials, and competitive coding solutions. And the volume of that content is significant enough to bias the weights.
3. Models aren’t rewarded for being token efficient in writing code, and aren’t negatively rewarded for writing comments.
4. Comments aren’t the only aspect that’s biased poorly towards writing good code. Most prompts also have instructions to write minimal code, not over-engineer, reuse existing abstractions, etc. These are also reflections of the training data.
5. Because learning-code and professional-code tend to look very different in practice, a model that prioritises one over the other in its training data might have very different behaviour.

In another article, I wrote about a variety of weird system prompt artefacts. Look up the system prompts of your favourite products, and see what model bias they’re fighting. It would leave you with a better understanding of its limitations.

One parting conjecture: RL is a great way to learn / unlearn some of these biases, but that requires the harnesses to be a part of the RL environment. If the inference-time harnesses get more sophisticated over time, that model+harness combo is likely to be the most reliable and efficient one, and it’s development is likely to be as opaque as RL is today.

This article is a deep-dive into one aspect of How System Prompts Define Agent Behavior, an article co-authored with Drew Breunig.

I spent some time studying the system prompts of coding agent harnesses like Codex CLI and Claude Code. These prompts reveal the priorities, values, and scars of their products. They’re only a few pages each and worth reading in full, especially if you use them every day. This approach to understanding such products is more grounded than the vibe-based takes you often see in feeds.

While there are many similarities and differences between them, one of the most commonly perceived differences between Claude Code and Codex CLI is autonomy, and in this post I’ll share what I observed. We tend to perceive autonomous behaviour as long-running, independent, or requiring less supervision and guidance. Reading the system prompts, it becomes apparent that the products make very different, and very intentional choices.

You are a…

Right from the start, they diverge in how they define their identity. Claude Code has always described itself as an “interactive tool/agent to help the user”, whereas Codex has long taken the stance of “a coding agent”. The latest 5.3-codex release shows a slight reversal in this position with the addition of “collaboration” with the user.

Even though this might be a single line, this defines the identiy, and is at the beginning of every single instruction to the LLM. So, this line holds weight. Here are the relevant excerpts from the prompts over time:

Both of them seem to be moving towards more autonomy, with assistant -> coding agent. But the “help users with software engineering tasks” vs “collaborate to achieve the user’s goals” still signals different levels of agency at which it’s meant to operate.

Should it stop and ask questions, or keep going?

Codex includes a critical and explicit section for “Autonomy and Persistence” for the non-Codex models.

Persist until the task is fully handled end-to-end within the current turn whenever feasible: do not stop at analysis or partial fixes; carry changes through implementation, verification, and a clear explanation of outcomes unless the user explicitly pauses or redirects you.

Notice the language of do not stop, and unless the user explicitly pauses. And later in the prompt, there’s a task execution section that doubles down on this.

You must keep going until the query or task is completely resolved, before ending your turn and yielding back to the user. Persist until the task is fully handled end-to-end within the current turn whenever feasible and persevere even when function calls fail. Only terminate your turn when you are sure that the problem is solved. Autonomously resolve the query to the best of your ability, using the tools available to you, before coming back to the user.

If I were the model reading these instructions, I would take this to mean: “I should try my best to solve the problem myself and not yield to the user.”

Claude, on the other hand, has an “Asking questions as you work” section, and a AskUserQuestion tool, that it is explicitly encouraged to use:

You have access to the AskUserQuestion tool to ask the user questions when you need clarification, want to validate assumptions, or need to make a decision you’re unsure about. Treat feedback from hooks, including , as coming from the user. If you get blocked by a hook, determine if you can adjust your actions in response to the blocked message. ... Use this tool when you need to ask the user questions during execution. This allows you to:

1. Gather user preferences or requirements
2. Clarify ambiguous instructions
3. Get decisions on implementation choices as you work
4. Offer choices to the user about what direction to take.

If I were the model, I would interpret this as “I need to be cautious; I’ll check with the user before going ahead.”

Should it proactively take action, or propose a solution first?

When there’s ambiguity about whether to write code or take action, it can look, at a surface level, like they make the same choice: when the user is asking questions or planning, don’t write code. But the manner in which they make the choice is quite different.

Codex’s prompt encourages the model to be bold about writing code:

Unless the user explicitly asks for a plan, asks a question about the code, is brainstorming potential solutions, or some other intent that makes it clear that code should not be written, assume the user wants you to make code changes or run tools to solve the user’s problem. In these cases, it’s bad to output your proposed solution in a message, you should go ahead and actually implement the change. If you encounter challenges or blockers, you should attempt to resolve them yourself.

Claude’s prompt had a “Proactiveness” section that encourages the model to be cautious about writing code. This section has been refactored into various tool instructions in recent versions of the prompt, but the general outlook still remains the same:

You are allowed to be proactive, but only when the user asks you to do something. You should strive to strike a balance between:

• Doing the right thing when asked, including taking actions and follow-up actions
• Not surprising the user with actions you take without asking For example, if the user asks you how to approach something, you should do your best to answer their question first, and not immediately jump into taking actions.
• Do not add additional code explanation summary unless requested by the user. After working on a file, just stop, rather than providing an explanation of what you did.

Should it be ambitious and creative with its solutions?

Here, Codex leans on ambition (with a caveat), and Claude takes a fairly conservative stance.

Codex says:

For tasks that have no prior context (i.e. the user is starting something brand new), you should feel free to be ambitious and demonstrate creativity with your implementation.

and

You should use judicious initiative to decide on the right level of detail and complexity to deliver based on the user’s needs. This means showing good judgment that you’re capable of doing the right extras without gold-plating. This might be demonstrated by high-value, creative touches when scope of the task is vague; while being surgical and targeted when scope is tightly specified.

Claude’s prompt is heavily focused on restraint rather than ambition, and gives many examples of how not to be ambitious. From the “Doing tasks” section:

Avoid over-engineering. Only make changes that are directly requested or clearly necessary. Keep solutions simple and focused.

• Don’t add features, refactor code, or make “improvements” beyond what was asked. A bug fix doesn’t need surrounding code cleaned up. A simple feature doesn’t need extra configurability. Don’t add docstrings, comments, or type annotations to code you didn’t change. Only add comments where the logic isn’t self-evident.
• Don’t add error handling, fallbacks, or validation for scenarios that can’t happen. Trust internal code and framework guarantees. Only validate at system boundaries (user input, external APIs). Don’t use feature flags or backwards-compatibility shims when you can just change the code.
• Don’t create helpers, utilities, or abstractions for one-time operations. Don’t design for hypothetical future requirements. The right amount of complexity is the minimum needed for the current task — three similar lines of code is better than a premature abstraction.

The caveat for codex is that all the creativity is taken away when there’s an existing codebase! Although, amidst all the opposing instruction given to the model, I doubt this section gets enough attention.

If you’re operating in an existing codebase, you should make sure you do exactly what the user asks with surgical precision. Treat the surrounding codebase with respect, and don’t overstep (i.e. changing filenames or variables unnecessarily)

A quick note on Gemini CLI and Cursor CLI

Gemini CLI has an interactive mode and a non-interactive mode, which puts control over autonomy firmly in the user’s hands rather than letting the model decide.

• [Interactive mode] **Confirm Ambiguity/Expansion:** Do not take significant actions beyond the clear scope of the request without confirming with the user. If asked *how* to do something, explain first, don't just do it.
• [Non-interactive mode] **Continue the work** You are not to interact with the user. Do your best to complete the task at hand, using your best judgement and avoid asking user for any additional information."

And Cursor CLI seems to take a similar route to Codex, giving the agent full autonomy:

• You are an agent - please keep going until the user's query is completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problem is solved. Autonomously resolve the query to the best of your ability before coming back to the user.
• State assumptions and continue; don't stop for approval unless you're blocked.

It is very likely that Codex models are RL’d on this behaviour

In the 5.2-codex model’s prompt, the sections around autonomy, ambition, etc. are all gone. And its prompt is only half the size of prompt for GPT-5.2. And the codex model release notes mention that it is made for long-running tasks, which hints at autonomy being baked in through post-training.

Any customisation through model post-training is opaque to end users, unfortunately.

My conclusions

All this is my interpretation, of course, and I can’t know what parts of the system prompt get more attention during inference. From my experience in prompting these models though, I feel like they pick up on the general theme of instructions given the context, reading in between the words and filling the gaps where the words aren’t present, in order to interpret the author’s intentions. I guess I’m doing the same thing here.

From the analysis in this post, Codex CLI optimizes for task completion, and Claude Code optimizes for user alignment and consent. But my take-aways are broader:

1. System prompts are used to steer models into different behaviours. It is difficult to pull apart the model’s behaviour into prompt-based and training-based, so the extent of steer-ability is somewhat unknown. However, for example, I’ve seen observable differences when using Claude Code with Codex’s system prompt.
2. While the models, harnesses, and tools might evolve, it appears to me as though the products themselves are differently positioned, and possibly headed in different directions. At the very least, they operate with different philosophies of what a coding agent should do.
3. If you want to understand and wield your AI tools better, read their system prompts.

This post was co-authored with Drew Breunig, and you can also read it on his blog.

Coding agents are fascinating to study. They help us build software in a new way, while themselves exemplifying a novel approach to architecting and implementing software. At their core is an AI model, but wrapped around it is a mix of code, tools, and prompts: the harness.

A critical part of this harness is the system prompt, the baseline instructions for the application. This context is present in every call to the model, no matter what skills, tools, or instructions are loaded. The system prompt is always present, defining a core set of behaviors, strategies, and tone.

Once you start analyzing agent design and behavior, a question emerges: how much does the system prompt actually determine an agent’s effectiveness? We take for granted that the model is the most important component of any agent, but how much can a system prompt contribute? Could a great system prompt paired with a mediocre model challenge a mediocre prompt paired with a frontier model?

To find out, we obtained and analyzed system prompts from six different coding agents. We clustered them semantically, comparing where their instructions diverged and where they converged. Then we swapped system prompts between agents and observed how behavior changed.

System prompts matter far more than most assume. A given model sets the theoretical ceiling of an agent’s performance, but the system prompt determines whether this peak is reached.

The Variety of System Prompts

To understand the range of system prompts, we looked at six CLI coding agents: Claude Code, Cursor, Gemini CLI, Codex CLI, OpenHands, and Kimi CLI. Each performs the same basic function: given a task they gather information, understands the code base, writes code, tracks their progress, and runs commands. But despite these similarities, the system prompts are quite different.

We’re analyzing exfiltrated system prompts, which we clean up and host here¹. Each of these is fed into context-viewer, a tool I developed that chunks contexts in semantic components for exploration and analysis.

Looking at the above visualizations, there is plenty of variety. Claude, Codex, Gemini, and OpenHands roughly prioritize the same instructions, but vary their distributions. Further, prompts for Claude Code and OpenHands both are less than half the length of prompts in Codex and Gemini.

Cursor’s and Kimi’s prompts are dramatically different. Here we’re looking at Cursor’s prompt that’s paired with GPT-5 (Cursor uses slightly different prompts when hooked to different models), and it spends over a third of its tokens on personality and steering instructions. Kimi CLI, meanwhile, contains zero workflow guidance, barely hints at personality instructions, and is the shortest prompt by far.

Given the similar interfaces of these apps, we’re left wondering: why are their system prompts so different?

There’s two main reasons the system prompts vary: model calibration and user experience.

Each model has its own quirks, rough edges, and baseline behaviors. If the goal is to produce a measured, helpful TUI coding assistant, each system prompt will have to deal with and adjust for unique aspects of the underlying model to achieve this goal. This model calibration reins in problematic behavior.

System prompts also vary because they specify slightly different user experience. Sure, they’re all text-only, terminal interfaces that explore and manipulate code. But some are more talkative, more autonomous, more direct, or require more detailed instructions. System prompts define this UX and, as we’ll see later, we can make a coding agent “feel” like a different agent just by swapping out the system prompt.

We can get a glimpse of these two functions together by looking at how a given system prompt changes over time, especially as new versions of models arrive. For example:

Note how the system prompt isn’t stable, nor growing in a straight line. It bounces around a bit, as the Claude Code team tweaks the prompt to both adjust new behaviors and smooth over the quirks of new models. Though the trend is a march upward, as the coding agent matures.

If you want to dive further into Claude Code’s prompt history, Mario Zechner has an excellent site where he highlights the exact changes from version to version.

The Common Jobs of a Coding Agent System Prompt

While these prompts vary from tool to tool, there are many commonalities that each prompt features. There is clear evidence that these teams are fighting the weights: they use repeated instructions, all-caps admonishments, and stern warnings to adjust common behaviors. This shared effort suggests common patterns in their training datasets, which each has to mitigate.

For example, there are many notes about how these agents should use comments in their code. Cursor specifies that the model should, “not add comments for trivial or obvious code.” Claude states there should be no added comments, “unless the user asks you to.” Codex takes the same stance. Gemini instructions the model to, “Add code comments sparingly… NEVER talk to the user through comments.”

These consistent, repeated instructions are warranted. They fight against examples of conversation in code comments, present in countless codebases and Github repo. This behavior goes deep: we’ve even seen that Opus 4.5 will reason in code comments if you turn off thinking.

System prompts also repeatedly specify that tool calls should be parallel whenever possible. Claude should, “maximize use of parallel tool calls where possible.” Cursor is sternly told, “CRITICAL INSTRUCTION: involve all relevant tools concurrently… DEFAULT TO PARALLEL.” Kimi adopts all-caps as well, stating, “you are HIGHLY RECOMMENDED to make [tool calls] in parallel.”

This likley reflects the face that most post-training reasoning and agentic examples are serial in nature. This is perhaps easier to debug and a bit of delay when synthesizing these datasets isn’t a hinderence. However, in real world situations, users certainly appreciate the speed, so system prompts need to override this training.

Both of these examples of fighting the weights demonstrate how system prompts are used to smooth over the quirks of each model (which they pick up during training) and improve the user experience in an agentic coding application.

Much of what these prompts specify is shared; common adjustments, common desired behaviors, and common UX. But their differences notably affect application behavior.

Do the Prompts Change the Agent?

Helpfully, OpenCode allows users to specify custom system prompts. With this feature, we can drop in prompts from Kimi, Gemini, Codex and more, removing and swapping instructions to measure their contribution.

We gave SWE-Bench Pro test questions to two applications: two agents running the Claude Code harness, calling Opus 4.5, but with one one using the original Claude Code system prompt and the other armed with Codex’s instructions.

Time and time again, the agent workflows diverged immediately. For example:

The Codex prompt produced a methodical, documentation-first approach: understand fully, then implement once. The Claude prompt produced an iterative approach: try something, see what breaks, fix it.

This pattern remains consistent over many SWE Bench problems. If we average the contexts for each model and system prompt pair, we get the following:

All prompt-model combinations correctly answered this subset of SWE Bench Pro questions. But how they suceeded was rather different. The system prompts shaped the workflows.

System Prompts Deserve More Attention

Last week, when Opus 4.6 and Codex 5.3 landed, people began putting them through the paces, trying to decide which would be their daily driver. Many tout the capabilities of one option over another, but just as often are complaints about approach, tone, or other discretionary choices. Further, it seems every week brings discussion of a new coding harness, especially for managing swarms of agents.

There is markedly less discussion about the system prompts that define the behaviors of these agents². System prompts define the UX and smooth over the rough edges of models. They’re given to the model with every instruction, yet we prefer to talk Opus vs. GPT-5.3 or Gastown vs. Pi.

Context engineering starts with the system prompt.

The above process has some drawbacks:

• Data preparation is difficult and labor-intensive
• In a simplified view, RLHF with PPO requires training two models: the policy model, initialized from a base model, and the reward model trained from human preferences. This increases the training complexity and compute/memory resources dramatically.

Is there a more resource-efficient way to train a model?

Reinforcement Learning with Verifiable Rewards

DeepSeek’s paper in early 2024 demonstrated a powerful RL technique using verifiable rewards instead of a learned reward model. This approach was later named Reinforcement Learning with Verifiable Rewards (RLVR).

The uniqueness of RLVR is in the way it assigns rewards. While RLHF uses a reward model, RLVR uses a verifiable, rule-based mechanism for computing rewards. This works best when correctness can be deterministically checked. For math problems, you can compare with the ground truth. For code generation, you can run tests. The absence of a reward model significantly reduces the computational resources needed and the complexity of the training pipeline.

The paper also introduced Group Relative Policy Optimization (GRPO), a new policy algorithm that learns by comparing rewards among multiple sampled completions for a prompt. Since it does not require a reward model, it drastically reduces the memory required for training when compared to PPO-based RLHF.

How GRPO works

As the name suggests, Group Relative Policy Optimization (GRPO) works by computing the relative advantage of completions within a group to inform the model’s learning.

A step in model training using this method can be broken down into the following (simplified) steps:

1. Generate group of completions for a prompt
2. Compute reward for each completion and then compute average reward for the group
3. Compute relative advantage for each completion with respect to the average reward (prompt_reward - mean_reward)
4. Compute loss for each completion using its advantage and token log probabilities
5. Compute loss for the group
6. Aggregate loss across prompts in a batch
7. Compute final loss using the batch aggregate, the learning rate (to control the magnitude of the gradient update) and KL coefficient, the guardrail that prevents the new model from drifting too much from the base model
8. Update parameters to move the model towards completions with higher advantages

This creates a self-improving feedback loop where the model tends to get better every step by using the data from the previous step. Now that we have a high-level view of the process, let’s look at an example training pipeline to understand the setup.

Training with GRPO

Here’s an example that trains a small model to generate Python code using GRPOTrainer from HuggingFace’s trl library. This is intentionally kept minimal with a single binary reward function and a few samples from the dataset for testing. There is a lot more nuance in designing reward functions and tuning the training parameters for a production setup.

import os
import tempfile
import subprocess

import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import GRPOConfig, GRPOTrainer


# This runs unit tests (verifiable reward) and gives 1.0 for pass, 0.0 otherwise.
def reward_fn(prompts, completions, **kwargs):
    """Return 1.0 if the completion passes the provided unit tests else 0.0."""

    tests = kwargs["test"]
    entry_points = kwargs["entry_point"]

    batch_size = len(prompts)
    k = len(completions) // batch_size

    rewards = []
    for i, completion in enumerate(completions):
        j = i // k
        prompt = prompts[j]
        test_code = tests[j]
        entry_point = entry_points[j]

        with tempfile.TemporaryDirectory() as td:
            sol_path = os.path.join(td, "solution.py")
            test_path = os.path.join(td, "test_solution.py")

            with open(sol_path, "w", encoding="utf-8") as f:
                f.write(prompt)
                f.write(completion)
                f.write("\n")

            # Run the dataset's tests against the candidate function.
            with open(test_path, "w", encoding="utf-8") as f:
                f.write(f"from solution import {entry_point} as candidate\n")
                f.write(test_code)
                f.write("\n")
                f.write("check(candidate)\n")

            try:
                proc = subprocess.run(
                    ["python", "-m", "pytest", "-q", test_path],
                    cwd=td,
                    capture_output=True,
                    text=True,
                    timeout=8,
                )
                passed = proc.returncode == 0
                rewards.append(1.0 if passed else 0.0)
            except subprocess.TimeoutExpired:
                rewards.append(0.0)

    return rewards


def main():
    train_dataset = load_dataset("openai/openai_humaneval", split="test").select(
        range(10)
    )

    model_name = "Qwen/Qwen2.5-Coder-0.5B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16,
        device_map="auto",
    )

    config = GRPOConfig(
        output_dir="./train_out",
        do_train=True,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        learning_rate=5e-7,
        num_train_epochs=0.1,
        num_generations=2,
        generation_batch_size=2,
        max_prompt_length=256,
        max_completion_length=64,
        temperature=0.8,
    )

    trainer = GRPOTrainer(
        model=model,
        args=config,
        processing_class=tokenizer,
        train_dataset=train_dataset,
        reward_funcs=reward_fn,
    )

    trainer.train()


if __name__ == "__main__":
    main()

Despite the simplicity and efficiency of GRPO, it is not a one-size-fits-all solution. RLVR with GRPO works well for problems that can be deterministically verified, but not for subjective or open-ended tasks like improving prose.

In the next post, I’ll walk through a GRPO training experiment for code generation and take a look at different aspects of the pipeline: dataset, reward functions, evals, and lessons learned. More soon.

The code is open source, so I had a look at how the front page algorithm works.

This is it:

The page is sorted in ascending order by \( \textbf{hotness} \). The more negative the value of \( \textbf{hotness} \), the higher the story ranks.

You can skip straight to the interactive front page to help get a feel for the front page dynamics.

Base

The \( \textbf{base} \) is added to the order term to incentivise certain types of posts, and influence the initial ranking. It is the sum of the hotness modifiers (a value between \( -10 \) and \( +10 \) of all the tags in that story).

Some tags (like culture or rant) have negative “hotness modifiers”, which penalises their initial rank. Authors submitting their own content get a tiny boost, which is mildly surprising given the otherwise strict self-promo rules. The \( \textbf{base} \) has a modest effect on the hotness compared to \( \textbf{order} \) and \( \textbf{age} \).

Order

The value of \( \textbf{order} \) is derived from the engagement that a story gets.

The progression of the order term is logarithmic—this means going from 0 to 100 votes increases the rank far more than going from 1000 to 1100 votes.

The \( \textbf{cpoints} \) is added to the story score, which accounts for non-submitter comment upvotes (a comment upvote is worth half a story upvote). If the \( \textbf{base} \) is negative (as is the case for a freshly submitted rant), then this term is zeroed, making the comments effectively contribute nothing to the rank.

The \( \textbf{cpoints} \) can never exceed the story score. Therefore, stories that have a low score but lots of highly upvoted comments—perhaps a signature of controversy-generating low-quality submissions—do not get boosted by comment upvotes.

There are some details around merged stories that I am leaving out for the sake of simplifying this explanation. But it roughly does what you’d expect.

Sign

If a story gets flagged enough to make the story score negatively (a flag is effectively a downvote), the \( \textbf{sign} \) becomes negative.

The \( \textbf{sign} \) doesn’t actually seem to do anything in practice! The \( \textbf{order} \) is always zero whenever the score is less than zero. That’s because comment points are clamped to \( \min(\text{score}, \text{cpoints}) \), so the log input never exceeds 1 for non‑positive scores. For positive scores, \( \textbf{sign} = 1 \) anyway. So in practice the \( \textbf{sign} \) term never changes the result.

Age

The value of \( \textbf{age} \) is fixed at the time of submission. This is the unix timestamp at which the story was created, divided by a configurable \( \textbf{hotness_window} \) time. The \( \textbf{hotness_window} \) is 22 hours by default—this means that the value of \( \textbf{age} \) increases by \( \text{1} \) unit every 22 hours.

This value grows linearly with every newer story, pushing older stories down the rankings. The main tension in this algorithm is the fact that the \( \textbf{order} \) (dictated by score) grows logarithmically, so upvotes need to increase exponentially over time to counter the effect of \( \textbf{age} \) in order to stay on the front page. Father time comes for us all.

In a nutshell

Where \( \textbf{base} \) is initialised based on the tag and who submitted the story. The \( \textbf{age} \) increases linearly for every new submission and the \( \textbf{order} \) for a story, as determined by votes, increases logarithmically.

Explore

Heads up—enable JavaScript to make this part work. This was mostly vibecoded, with me verifying that the results match the algorithm.

There’s a gisthost link if you want to play with it as a standalone tool.

Thoughts

The algorithm is solid. It allows new stories to get their time in the sun, and correctly penalises low-quality content that generates a lot of heated discussion. If there is heated discussion, it’s usually over highly-upvoted posts. Over time, age always dominates upvotes, so no story can really stick around that long in the front page. There are gates that stop overly flagged stories from making any progress up the ranks.

That said, I don’t think the algorithm really makes the site what it is. The character of the site is more the result of its opinionated moderation, narrow computing focus and the gradual acculturation through the invite system. Compared to many other forums, there’s less junk and also little outright hostility, racism, sexism or other isms. The community has surfaced lots of niche topics and writers, which I enjoy.

Yet, my experience on the website has been far from ideal. For me, this is rooted in a disconnect of values with the group most engaged on the site, whose votes and discussions drive the climate. I do not appreciate the cynicism worn with pride, the unproductive gotchas, the long polemics that reveal that the commenter hasn’t read beyond the title, the throwaway venting and the debates where it is clear that neither side wants to actually refine their world model. It has driven me away from engaging more on the site.

Studying the algorithm has shown me that disengaging would make my problem worse—a single user’s participation can be worth a lot. Early upvotes really count, and can easily boost a post to the front page. If you are lurking on the site and are unsatisfied, consider exercising your votes and submissions more. Post the more nuanced, friendly and curious comments that you’d like to see more of. It really does matter. I will likely change how I participate on the site as a result of this.

After all, there aren’t all that many relatively quiet and straightforwardly serious public forums to contrast the twitters and HNs of the world that can surface niche computing curiosities.

This post originally appeared on Atharva’s personal site.

Update: Issued a correction from the sign term, thanks to strongoose.

See discussion on Lobsters and Hacker News.

A big talking point was that popular public benchmarks are insufficient for determining if an AI model is a good fit for their product.

I want to sharpen this observation a bit more.

What are benchmarks useful for?

I’ve seen benchmarks serve a whole range of purposes.

• Benchmarks as decision-making tools: You look at existing benchmarks to figure out whether to use model A or model B.
• Benchmarks as regression markers: Like unit tests, they tell you if your updated AI model or system isn’t doing worse than before. This is especially useful in cost-optimisation exercises.
• Benchmarks as improvement indicators: If you see benchmark go up, you can tell that your change to the model or system is improving the outcome.
• Benchmarks as product behaviour feedback: A more subtle use—with the right analysis of trajectories, benchmarks can tell you about the strengths and weaknesses of your model across categories of tasks you are interested in.
• Benchmarks as research agenda setters: When a new benchmark is published, AI labs start hill-climbing on it—publishing benchmarks is a great way to influence what AI is good at.
• Benchmarks as RL environments: This is an emerging use case. Reinforcement Learning with Verifiable Rewards effectively works with a setup that doesn’t look all that different from a benchmark.
• Benchmarks as forecasting anchors: You can use benchmarks to get a sense of how AI capabilities are progressing over time. METR has made good use of this.

If a benchmark is not helping you with any of the above, your benchmark is useless. Many useless benchmarks unfortunately exist.

Benchmark traps

Here’s the Artificial Analysis Intelligence index, which aggregates all sorts of AI benchmarks.

And here’s the most popular benchmark for testing coding ability.

These charts in isolation give the impression that AI models are pretty interchangeable and that whenever a new model comes in, you can reap the fruits of the wonderful frontier lab training pipelines. All you need to do is to switch your coding model to whatever the new hotness is. Right?

No.

The issue with benchmarks is that they are lossy. They condense multidimensional characteristics into a single number¹. Your business case may not look like whatever your number represents.

Let’s take an example. You’re working on an AI agent that operates in the legal domain. A profoundly unserious approach would be to look at which model is doing well across standard benchmarks (like the intelligence index above) and pick that. If we put a couple of extra brain cells to work, we might look at an independent benchmark score for the most popular legal benchmark. Right now this is LegalBench.

Great, so it’s still the state-of-the-art Gemini 3 Pro, isn’t it? It’s clearly #1 on the benchmark².

But look at this—there’s a CaseLaw (v2) benchmark as well.

No Gemini 3 Pro in sight. Have they forgotten to bench our frontier Gemini model here? Actually no.

Gemini 3 Pro is poor enough at this benchmark that it’s nowhere near the top of the leaderboard. In fact, it ranks #39 and is worse than the previous-generation Gemini 2.5 Flash!

Both of these are measuring different things in the legal domain, with CaseLaw appearing more like real-world legal work, and LegalBench being more like an academic exam. It’s quite possible that Gemini can be good at some parts of some domains and poor at other parts of the same domain. Or maybe the CaseLaw evaluation has some unaddressed issues (after all, there seem to be a lot of surprising results in the leaderboard). Or that Gemini hates Canadians.

This all points to one thing—don’t base your decision off benchmark scores. Instead, look at the benchmark contents and methodology, figure out how closely it aligns with what tasks you are handing off to the AI and most importantly, make your own internal benchmark, with metrics aligned to your business case³.

Minimum viable benchmark

Without getting into the weeds of categorisations, I’d note that internal benchmarks are not all that different from what all the hip and cool new AI Engineering teams like to call evals.

They are not structurally different from public benchmarks. You have your dataset of tasks. You (ideally) have your ground truth for these tasks. You measure your AI system against these tasks and get scores. Unfortunately, building a public benchmark is hard work—you have to collect a lot of data to get signal⁴, ensure the environments are reproducible and your metrics trustworthy. This ugh field has pushed teams away from building evals. Well, at least until it’s too late, when you suddenly have everyone scrambling to do the grunt work of collecting annotated high-quality data when the house is burning.

I’d like to propose an alternate view—your internal benchmarks don’t need to be as sophisticated as the public benchmarks. They only have to be a minimum viable benchmark.

A minimum viable benchmark is not concerned with being an arena for competing AI systems—it is a vehicle for figuring out whether you are building the right product and that the product works well.

You don’t need to have an intelligent-sounding metric or your LLM eval SaaS vendor figured out in order to get started. You only need to collect your data and annotate it. You can get started and make a lot of progress in a couple of hours, armed with only a spreadsheet and your product and engineering teams in one room.

In your sheet, ensure you have your inputs to your AI system. Add the outputs after a few runs in the system for the tasks you need⁵. Add free-form commentary in the last column about how it did. Don’t optimise anything yet. Don’t add any “metrics” yet.

After this exercise, a few things happen:

• You realise what your task is actually like and what it might involve.
• You realise whether the AI works at all for your task.
• You realise what it feels like to be a user of your system and get a better sense of where AI is actually helping. This is input for the product team.
• You realise what actually needs to be measured for your benchmark metrics. It’s never the vague, pointless metrics that came with the eval framework you were looking at.
• The valuable metrics inferred from this exercise are often useful product metrics!
• You catch the biggest blind spots of the AI system very early on. Gathering large datasets is needed only when you are trying to catch small effects. Early on, most of observed effects on any intervention will be quite large!
• Most importantly, you have overcome the Ugh Field! This exercise is often fun.

This minimal viable benchmark would have already proven its usefulness early on. Everyone in your team will continue to build on top of this and rely on it when, inevitably, you have to avoid regressions, evaluate a new feature or model or optimise costs. Over time, your minimal viable benchmark can grow into a useful, strong benchmark that forms the backbone of your AI project.

How we go from a minimal viable benchmark to a maximally useful benchmark would perhaps need its own article. But to give you a taste, ensure you have these properties:

• It’s easy to look at the data and your cross-functional team is involved in reviewing the data regularly.
• What you are measuring maps to product outcomes—this may not be the case for public benchmarks.
• There are enough samples to actually give you a sense of whether your system has actually improved.
• The tasks have a difficulty ramp-up to actually capture improvements to models and systems. If most of your tasks have the same difficulty, and a newly released AI model gains the ability to do that task, your benchmark would get saturated overnight and cease to capture further improvements.
• The metrics are measured either deterministically or with an unbiased estimator⁶.

Anyway,

• Don’t trust public benchmark numbers without seeing if the methodology and numbers map to your product outcomes.
• Build your own minimal viable benchmark, where what you are measuring maps to your product’s quality.
• It’s not that hard to start with, and it’s really worth it.

Footnotes

Eh! All of you, come here! Taste it! Taste it! Taste it! Taste it!

Gordon Ramsay

If you want to cook a great dish, you’ve got to taste it every step of the way. Taste the ingredients you buy, the components you prepare, and the spices and seasonings. If you can’t taste it, you smell it, feel it, or listen to it. And then you adjust. Taste and adjust until you create a dish you like.

“Taste and adjust” is a form of continuous improvement applied to the creation of food. The hallmark methodologies of the scientific method, Kaizen, TPS, PDCA, TDD, design sprints, or extreme programming, that have led to some of humanity’s best creations, are all forms of continuous improvement. At their core is this principle:

Creators need an immediate connection to what they’re creating.

Bret Victor, Inventing on Principle

Bret Victor says that “working in the head doesn’t scale”, and that understanding comes from seeing data, flow, and state directly. When building products, can you see the data, flow, and state directly? Can you “taste” your product every step to ensure it’s exactly what you and your users want?

The chef’s line-tasting, our flywheel, harness, environment, or feedback loop, is the framework in which we apply this principle to product creation. The product and engineering functions must build and maintain this flywheel together, every step of the way.

The product development flywheel

The machinery that enables running such experiments frequently and quickly is the flywheel.

It could be in the form of an operator’s console that allows product to tweak config on the fly, or building a prototype, or a feature-flag allowing tests with beta-users, or publishing a new metric that removes a blind spot. Even unit tests that verify whether the code does what product intends are part of this flywheel.

While this seems like a simple enough principle to apply, in reality, we are faced with the inherent complexity of working with many people, roles, and tools. A typical product development lifecycle (PDLC) looks like the abstract machine shown below. Each phase has controls and measurements around specific feedback loops (such as Idea ⇄ User), and the phases are interconnected through reinforcing and balancing information channels.

Here’s a list of some ways to “taste” at each phase, and a healthy level of involvement of product and engineering in each of them.

Fine-Tuning the Flywheel

• Get end-to-end product builders: You want teams that go together from phase 1 to 6, and then around again, to close the loop on their creation. Look for roles siloed in fewer phases, and work to involve them in all phases.
• Get involved early: Phases 1 and 2 are the ideation phase, and the most important thing to do here is to listen, and understand the problem deeply. I wrote about this earlier in the series. Building this phase of the flywheel for new products is cheap, especially with vibe coding. However, keeping experimentation costs low as the product matures can be challenging. Work to keep experiments cheap by using feature flags, or by maintaining experimental or forked versions of applications.
• Get closer to the user: Phases 3, 4, and 5 make up the typical SDLC (software development lifecycle), and in my experience, engineering is less involved in phases 1, 2 and 6. This is unfortunate because phases 1, 2, and 6 interface with the user and house the most important feedback loops.
• Planning > Speed: Development (phase 4) is arguably the most expensive part of most tech companies. While there’s a lot of focus on making development faster to reduce costs, reducing work through planning (phase 3) is far more effective. Break down problems, find thin slices of work to serve, and prioritise ruthlessly.
• Close outer feedback loops: Phase 6 should close the loop on business goals through product observability, in addition to the local feedback loops of individual features or initiatives.

So, review your flywheel periodically. Lubricate the gears, and tighten the feedback loops. Ultimately, ensure that everyone on the team feels empowered to stop the line, take a spoonful, and say, “Needs more salt.”

If one does not know to which port one is sailing, no wind is favorable.

Letter 71, Moral Letters to Lucilius, Seneca

This might seem quite obvious, cliché even, but it’s surprisingly difficult to apply to life, or to building products. Making big decisions is hard. But then:

Hard choices easy life, easy choices hard life.

Jerzy Gregorek

When leadership fails to make the hard choices, the entire team ends up stuck, treading water. Most leaders I know recognise this and try to make those calls. But the day-to-day challenges inside teams are more subtle, and often look like this: “There are two objectives; both are equally important.”

If we try to focus on everything, we focus on nothing, as John Doerr puts it in Measure What Matters. In the following excerpts from my experience, I’ll illustrate how these issues show up in everyday work.

Know when you’re treading water

At Simple.org, one of the recurring debates in the early years was whether we should support patient screening. Screening meant that nurses would travel to villages and towns and check entire communities for cardiovascular issues. While our work was focused on hypertension, once the nurses were already in the field, it felt incomplete to ignore diabetes. Screening promised a better understanding of population health, while our existing product was designed around longitudinal care of patients. At the time, this seemed like a high-level product call that engineering could stay out of, because we figured that core features such as patient search and follow up scheduling were needed no matter what… but the ambiguity kept creeping in. Patient search behaved predictably inside a clinic because it showed patients registered to that facility, but in the field during screening, we had no shared understanding of what results a search should return. Even entering a blood pressure reading created confusion because the clinic workflow called for follow up reminders while screening did not. These questions came up repeatedly, each causing a hiccup or delay in progress. And they added up.

A similar scenario unfolded when I worked on building order-pooling at Gojek, and it’s perhaps easier to see in two/three sided markets. Pooling is where one driver picks up multiple orders from the same restaurant and delivers them to customers who live near one another. Seems like a useful feature, but useful for whom? If the priority was higher utilisation for drivers, then we needed to focus on improving their income and increasing the number of orders they could complete each hour. Customer experience mattered, but only up to the point where it did not slow drivers down. If the priority was increased customer demand, questions of pricing, ability to opt-out of pooling, and the impact on wait times became far more important. We assumed we could make progress without choosing, because, on the surface, the feature looked generic enough to support both paths. But as we made decisions about pricing rules, batching behaviour, driver assignments, and customer communication, the lack of clarity surfaced again and again. Many small decisions depended on a choice we had not made, and a project that looked like three months of straightforward work grew into four.

I once joined a team as a fixer, because the CTO believed that nothing meaningful had been delivered for almost a year. On my first day, I asked about the team’s objectives, and I received vague, hand-wavy answers. The director of engineering said the PMs had never given them dashboards or metrics to work with. The PM, sitting in the same room, said they had plenty of dashboards but the engineers showed no interest in them. Product kept reporting metrics to leadership, and those metrics were flat. Engineering kept reporting completed initiatives, and there were plenty of them, but no one realised why the work was not improving the outcomes that mattered. The team also held several engineering-only meetings every week, intentionally excluding PMs because their presence was considered unnecessary for technical discussions. All I did was highlight this rift to leadership. And the biggest shift in morale came in a single meeting when leadership finally presented one clear north star metric. Half the ongoing initiatives were dropped on the spot because they did not support that goal. We also removed the long list of engineering-only meetings and replaced them with two focused weekly sessions that included product, engineering, design, and QA together. Once everyone saw the same goal and talked in the same room, the team began to move forward again.

Tugging on the mainsheet

Asking junior developers, or associate PMs simple questions about their work can reveal how clear the goals are, and how aligned everyone in the team is.

• “Why are you working on feature X?”. Ideally, they would open up their task / unit-of-work description, point to the observability section, and navigate to the corresponding leading and lagging metrics on a live dashboard that everyone in the team uses. Quite often though, the answers are “because the PM told me to”, “It’s the most important feature right now”, “people are complaining about this”, etc. Which are not wrong reasons per se, but not nearly as specific as we need to be.
• “What was the impact of the last initiative you delivered?”. This gets fewer good answers in my experience, despite it being one of the most important and documented aspects of satisfaction with one’s work.
• “Is there something more important you should be working on?”. If the work in the team is transparently prioritised as per the objectives, you would get a straightforward answer reflecting that. And in my experience, this often gets murky answers like “Maybe feature Y is more important, but we stopped that work for some reason”, or “They say Z is more important, but X is what we need right now, and I’m not sure why”.

Ideally, every single initiative should move the metric, and we respond by reinforcing efforts, or correcting course. In order to do that, every slice of work should incorporate building observability for it. And to do that, the product requirements should be clear about what metrics are expected to move, and why. It is engineering’s responsibility to seek that clarity out.

Without involvement, there is no commitment

“Without involvement, there is no commitment. Mark it down, asterisk it, circle it, underline it. No involvement, no commitment.”

Habit 2, Begin with the End in Mind, Stephen Covey

This quote bears repeating in teams where product and engineering fail to work together. Without involvement from engineering in setting the goals, and understanding the metrics, there is no commitment to create the required observability, and the feedback loops necessary to work toward them.

In my experience, engineering is seldom involved in creating OKRs, and it’s usually owned by product and business teams. This is antithetical to forging a deep relationship between the functions. If you’re a leader, use your influence to bring engineering to the table. If you don’t have the influence yet, build it by creating value around the objectives anyway. Understand them, challenge them, and help make them real. Comment on the OKR docs with your views, push to refine that metric you think is poorly defined, or negotiate on that goal you think is too steep. It’s hard, but you have to work your way up to the table.

What should I do as an engineer?

Work with product to get clarity on the port. Once you have it, as Seneca suggests, you must steer and watch the stars. It boils down to three simple things:

1. Demand clarity on goals, ideally in terms of metrics you can track, and participate in the conversations needed to achieve that clarity.
2. Make product observability a first-class requirement with every unit of work.
3. Establish balancing and reinforcing feedback loops in day-to-day work that tie back to those metrics.

This relationship between people reflects the relationship between problem and solution. In real life, problems and solutions evolve together. Each one affects the other. Similarly, the people involved must grow a meaningful relationship together. They need to listen empathically, understand needs, comprehend each other’s writing, create artefacts together, communicate well, and have fun solving problems together.

Since I have predominantly played the engineering role, I’m writing to fellow engineers based on that experience. This series of posts details when and how engineering can work with product to create the synergy that powers wonderful products.

We reflect on our problems over tea. Or coffee. Or sutta, if you’re into that. It’s a casual conversation that’s not bound by the colder confines of an office or work space, where we’re calm, set to truly listen and receive. We allow ourselves to be courageously vulnerable with our real thoughts and ideas. Reflection provides fertile ground for germination of ideas. It’s in this mental space that people often understand, empathise, and find camaraderie. These are flaps of the butterfly wings, or the rolls of that snowball we build as a product team.

When someone who knows the problem very well speaks about it, even casually, lean in. Listen empathically. Diagnose before you can prescribe. Once you have a deep enough understanding of the problem, and the needs of the people therein, you can start thinking about how your engineering skills can add value.

What follows are some autobiographical experiences. Read on, and be inspired! Yeah, no, I’m simply illustrating what I mean, drawing on my experiences. If you’d rather not read my life stories, feel free to skip ahead to the list at the end.

It was in this kind of tea-table space that we found the courage to say “screw it, lets go offline first”, with our initial efforts at simple.org. Daniel had spent months in the field with nurses, speaking to true experts, and understanding the problems of treating hypertension in public health. He then expressed to us, the engineering team, a deep need for nurses in rural areas to be independent of internet requirements. We understood that need, and decided it’s worth a try with a “how hard can it be” attitude. After some 6 years, we still see that as one of the most important decisions to make the product successful, and truly useful to thousands of health care workers and millions of patients.

After a hearty lunch one day on the nilenso terrace, we were talking about how we need better mechanisms to validate ideas with nurses in the field. Dhruv said “Ideally, I just want to be a fly on the wall, and observe the nurse use the app with real patients”. Prototypes didn’t cut it, and user-research questions were based on possibilities that weren’t easy to communicate. I suggested building a separate app that we can use to run experiments quickly in the field. This would normally take months with a small team (it was 2018), but I had built prototype-apps quickly before, and could confidently say “Give me a couple of weeks, no one else needs to be involved and their work can continue”. That was a small enough risk the leadership could take, and it paid off. We iterated on scanning UUIDs of various densities using low-spec phones, various error feedback mechanisms, and a custom number keypad for entering blood pressures, tried all these out with nurses in the field, and then built them into the actual app.

At GoFood, while we were having dinner in Jakarta I noticed Hareesh kept returning to what seemed like a small issue: customers accidentally placing orders and immediately cancelling them. It didn’t appear urgent, but it was quietly blocking a major release: restaurants that auto-accepted orders the moment they were placed. He never said this directly, but his fixation was the signal. I asked, “How does Zomato do this?”, and when we checked their app, we found a straightforward pattern: a 10-second confirmation timer and a clear message about non-cancellable orders. Because the app already had all the information needed to decide when to show that timer, I could confidently say, “This can be an app-only change; we can ship it in a few days.” The engineering work was trivial. Identifying that it was a blocker and needed a quick solve was the real contribution.

While working at a logistics firm recently, we spent more than a month wrestling with the onboarding journeys for new partners. It was a legacy system where every bug fix spawned new ones, and understanding the possible solutions was even more painful than understanding the problems. Our competitors could onboard new partners in hours while we took days, and the decade-old architecture simply wouldn’t let the product evolve. One evening, after venting about all this with the head of product, I asked him, “If we were to re-implement the entire onboarding flow from scratch, what would you do?”, and it turned out he had been thinking the same thing but couldn’t voice it unprompted. It was too big and too costly a leap to suggest outright. That question unlocked the real conversation: by the end of it, we had enough product clarity and engineering rationale to justify a half-year rewrite.

I’ve tried to break down my instincts from such experiences at the ideation phase, to what I might tell another fellow engineer:

• Seek first to understand. Listen to the users. Listen to subject matter experts. Listen to people who understand the problems and product very well. Pay attention to emotional cues and subtext. They often point to deeper problems than any ticket or spec reveals. Create casual and trust-heavy settings intentionally, where conversations can surface such problems.
• Be involved early. The early conversations of ideation, direction setting, and strategising are pivotal in building products, and engineering insight is very valuable in these conversations. Usually engineering is only involved in later stages.
• Find cheap ways to prove hypotheses. What’s the thinnest slice of cake we can offer customers? Can we scrounge up some analytics that can make the case? What’s the simplest experiment we can run? Should we build a PoC, or fire a tracing bullet that tells us how expensive this is going to be?
• Don’t wait for product to propose solutions. You can introduce new solution spaces as engineers that product might not even know exist. Don’t be afraid to push boundaries to make meaningful progress.
• Create space for unsaid ideas. PMs often have solutions or directions in mind, but don’t voice it because they feel too big, risky, or premature. As engineers, you can unlock strategies by asking questions that give them permission to speak openly about the real problem or the bold solution they’re thinking about.
• Build feasibility intuition: It is your responsibility to bucket ideas along the spectrum from pipe-dream to practical to trivial. If the product manager is informed on which things take hours, days, weeks, months, they can make much more reasonable decisions than otherwise. If you don’t know enough to make that kind of call, learn, and build that capability. Be resourceful, have trustworthy sources, look at prior art or competitors, and then form your own opinion and share it with product.
• Surface system constraints: Talk to product about things that are generally considered engineering territory, like missing abstractions, why integration with some third-party is slow and buggy, why adding another workflow is complex, etc. This transparency builds empathy and trust. When you surface these constraints, they can proactively reorient product strategy to account for them. Or perhaps give room to solve for those constraints.

The “secret sauce” is basically:

Two main problems I had to solve were segmentation and categorisation.

Segmentation

The problem here is to pull apart a single message from an assistant, or prompt from the user, into meaningful chunks like various text or code blocks, instructions, files supplied as context, etc. Here’s an example input message, which is a typical wall-of-text in the context window.

"Given the <product_requirements_document>, the <technical_specification_document>, and the <repository_structure>, identify what questions the user should know about the codebase when planning to work on this PRD.\n\n<product_requirements_document>\n# Grand Central — MVP PRD\n\n**DRI:**&#x20;\n**Stakeholders:** People Ops, Partners, Engineering, Design, Data\n**Status:** Draft v1\n**Date:**\n\n---\n\n## 1) Problem statement\n\nPartners and Execs cannot quickly answer basic workforce questions because employee data is scattered across spreadsheets, lacks a single source of truth, and has incomplete history of state changes. This creates slow decisions, duplicated work, and error-prone analysis.\n\n**Evidence to collect pre-ship**\n\n* Time spent per week answering roster questions.\n* Number of duplicate or conflicting records in current Sheets.\n* Top 5 recurring questions stakeholders cannot answer reliably today.\n\n---\n\n## 2) Who is the customer\n\n* **Primary:** People Ops admin who maintains records and exports data.\n* **Secondary:** Partners/Managers who need roster and trend views.\n* **Tertiary:** Employees who look up colleague basics.\n\n**Jobs-to-be-done**\n\n* Maintain canonical person records with auditable history.\n* Import legacy data once with minimal cleanup.\n* Run simple trend analyses without a data analyst.\n\n---\n\n## 3) Goals and non-goals\n\n**Goals (MVP)**\n\n1. Create, edit, and view **person** records with append-only **changesets** and effective dates.\n2. **One-time bulk import** from Google Sheets with dedupe by work email.\n3. **Directory** with search and CSV export of the **current snapshot**.\n4. **Analytics v0:** monthly headcount, joiners vs leavers, level distribution over time; simple “level vs prior experience” view.\n5. **Security:** Google SSO, domain allowlist, audit log on writes.\n\n**Non-goals (MVP)**\n\n* Payroll, performance reviews, external HRIS sync.\n* Complex RBAC beyond Admin vs Viewer.\n* Compensation reporting UI. *Schema ready; UI deferred to v2.*\n\n**Why now**\n\n* Operational risk from spreadsheet fragmentation.\n* Quick wins unlock broader analytics later.\n\n---\n\n## 5) Requirements\n\n### 5.1 Functional\n\n* **Auth:** Google SSO; domain allowlist; logout; session security.\n* **Directory:** list with columns *Name, Work email, Level, Status, Start date*; search by name/email; link to person detail; CSV export of current snapshot.\n* **Person detail:** core fields, plus **History** tab showing changesets in reverse chronological order; show who changed what and when.\n* **Create/Edit:** forms capture effective dates; all edits append a changeset; current snapshot recomputed.\n* **Analytics v0:**\n\n  * *Headcount trend* by month (active count, joineive dates.\n2. **One-time bulk import** from Google Sheets with dedupe by work email.\n3. **Directory** with search and CSV export of the **current snapshot**.\n4. **Analytics v0:** monthly headcount, joiners vs leavers, level distribution over time; si
mple “level vs prior experience” view.\n5. **Security:** Google SSO, domain allowlist, audit log on writes.\n\n**Non-goals (MVP)**\n\n* Payroll, performance reviews, external HRIS sync.\n* Complex RBAC beyond Admin vs Viewer.\n* Compensation reporting U
I. *Schema ready; UI deferred to v2.*\n\n**Why now**\n\n* Operational risk from spreadsheet fragmentation.\n* Quick wins unlock broader analytics later.\n\n---\n\n## 5) Requirements\n\n### 5.1 Functional\n\n* **Auth:** Google SSO; domain allowlist; logo
ut; session security.\n* **Directory:** list with columns *Name, Work email, Level, Status, Start date*; search by name/email; link to person detail; CSV export of current snapshot.\n* **Person detail:** core fields, plus **History** tab showing changes
ets in reverse chronological order; show who changed what and when.\n* **Create/Edit:** forms capture effective dates; all edits append a changeset; current snapshot recomputed.\n* **Analytics v0:**\n\n  * *Headcount trend* by month (active count, joine
rs, leavers).\n  * *Level mix* over time (banded junior/mid/senior).\n  * *Level vs prior experience* scatter.\n* **Import:** one-time CSV importer for legacy Sheets; idempotent; validation report; dedupe by work email; mapping guide.\n\n### 5.2 Data model (MVP)\n\n* **Person**: id, firstName, lastName, workEmail (unique), phone?, role, level, status, startDate, endDate?, priorWorkExperienceYears.\n* **Changeset**: id, personId, field(s) changed, newValue, effectiveDate, author, createdAt.\n* **Status enum**: FULLTIME, CONTRACTOR, EXEC, PARTNER, RESIGNED.\n* **Compensation (v2-ready)**: CompChange(personId, amount, currency, effectiveDate, notes).\n* **Snapshot rule**: latest effective changes per field as of “now”.\n\n### 5.4 UX principles\n\n* Defaults fast data entry over perfect taxonomy.\n* Make history obvious before saving edits.\n* Show what changed, by whom, and when.\n\n---\n\n## 8) Risks and mitigations\n\n* **Import correctness** → schema mapping guide, dry-run, row-level report.\n* **Duplicate records** → unique email constraint; surface potential duplicates; merge flow later.\n* **Bad effective dates** → inline validation; preview of resulting history.\n* **OAuth misconfig** → automated env checks in CI; clear runbooks.\n\n---\n\n## 9) Acceptance tests (MVP)\n\n1. **Create person**: Authenticated user submits required fields → person appears in directory; audit entry created; event `person_created` emitted.\n2. **Edit with history**: Update level with effective date → new changeset stored; History tab shows entry; snapshot updated.\n3. **Import**: Run importer on validated CSV → ≥95% rows ingested; reconciliation report shows any rejects with reasons.\n4. **Export**: Click Export on directory → CSV downloads with one row per current person; header spec matches appendix.\n5. **Analytics**: Open Analytics → monthly headcount, joiners vs leavers, and level mix charts render from production data; “level vs experience” view loads.\n6. **Security**: Unauthenticated user → redirected to login; export requires Admin.\n\n---\n\n## 10) Open questions\n\n* Exact mapping of legacy Sheets to entities and enums.\n* Admin vs Viewer permissions beyond export.\n* Compensation governance and who can view amounts in v2.\n* Do managers need edit rights or view-only in v1?\n\n---\n\n## 11) Appendix\n\n**A. CSV header spec (current snapshot)**\n`firstName,lastName,workEmail,phone,role,level,status,startDate,endDate,priorWorkExperienceYears`\n\n**B. Glossary**\n\n* **Changeset**: append-only record of a field change with an effective date.\n* **Snapshot**: latest effective value per field at a point in time.\n* **Headcount**: number of active employees in a period.\n* **Joiners/Leavers**: counts of start/end effective events in a period.\n\n**C. Decision log**\n\n* Compensation UI deferred to v2; schema included now.\n* Unique workEmail enforced; no merge UI in v1.\n* SQLite acceptable for MVP, to be revisited post-M6.\n\n\n</product_requirements_document>\n\n<technical_specification_document>\n# Modelling\n\nDescribes how to model entities over time.\n\n## Goals\n\nModel an entity with a set of known fields which changes over time such that:\n\n1. The time when a change occurred is tracked separately from when the change\n   was recorded\n\n2. The state of an entity (a view) can be queried for any point in time to\n   fetch the values of fields in that entity at that point in time\n\n3. A change that occurred in the past should affect all views of the entity\n   that are requested after that point in time\n\n4. The end user should not have to declare changes but instead just edits the\n   entity as a whole at a given point in time\n\n## CRDTs\n\nRef: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type\n\nTypically used for resolving conflicts between information created from two different sources, we are able to use CRDTs for our usecase by treating a change made to an entity's history as a change from a disconnected user that's converging with other changes.\n\n### LWW Element Set\n\nRef: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#LWW-Element-Set_(Last-Write-Wins-Element-Set)\n\nAn LWW Element Set is a well known CRDT. It consists of an 'add set', 'remove\nset' and a time stamp for each element in it.\n\nThe timestamp will be the time at which the change occurred. The value is a map\nwith all the fields and corresponding values that are updated. In our case, the\nremove set does not exist since nothing is meant to be removed, just edited. We\nmay mimic a remove by updating a field with a null value if necessary.\n\n## Schema\n\nThe schema to store this in SQLite.\n\n```sql\n\ncreate table changes(\n  -- This would be the personId in our case\n  entityId text,\n\n  -- Time when the event occurred\n  timestamp datetime,\n\n  -- json with fields that have changed and the corresponding values\n  -- eg. {'name': 'Name1', 'age': 22}, {'age': 29}\n  value text,\n);\n\n```\n\nHere’s a concise tech spec for “Grand Central.” It captures current stack choices and the target architecture, flags risks, and proposes guardrails.\n\n# 1) Overview\n\n* Purpose: Single source of truth for employee data. Visual analysis now. Workflow automation later.\n* Users: Internal staff. Auth via Google.\n* Non-goals: Public API, multi-tenant SaaS, heavy analytics.\n\n# 2) Core Requirements\n\n* CRUD for people and changesets.\n* Authenticated access only.\n* Visual graphs of trends.\n* Low-ops deploy. Small team ownership.\n* SQLite durability with simple backups. Path to Postgres when needed.\n\n# 3) Tech Stack (current)\n\n* Web framework: Remix (React + SSR). TypeScript everywhere.\n* Styling: Tailwind.\n* ORM: Prisma.\n* DB: SQLite (file URL defaults to `file:/data/sqlite.db`).\n* Auth: `remix-auth` + Google OAuth.\n* Charts: D3 components.\n* Testing: Vitest for unit, Cypress for E2E.\n* Packaging: Docker multi-stage build. Separate Datasette image for read/admin.\n* Runtime: Node process started by `start.sh` which runs `prisma migrate deploy` then `remix-serve`.\n* Ops: Procfile targets (`web`, `datasette`). Fly volume mounted at `/data`. Healthcheck at `/healthcheck`.\n  Sources: repo Dockerfiles, README, Fly config.&#x20;\n\n# 4) Architecture\n\n## 4.1 Logical components\n\n* **Remix app**\n\n  * Server routes: SSR HTML, actions for mutations, loaders for reads.\n  * Session: Cookie session storage.\n  * Auth: Google Strategy. On first login auto-creates user.\n  * Data access: Prisma client.\n  * Graphs: D3 line and multiseries components rendered client-side.\n* **SQLite**\n\n  * Deployed at `/data/sqlite.db` with a persistent volume.\n* **Datasette**\n\n  * Read and exploratory UI at `/datasette/`.\n  * Insert bot capability gated by token for controlled writes.\n* **Reverse proxy**\n\n  * Local dev uses Caddy via `docker-compose.local.yml` to route to Node and Datasette.\n\n## 4.2 Runtime topology\n\n* **Fly.io app**\n\n  * Service ports: 80/443 -> internal 8080.\n  * Volume mount `/data` for DB durability.\n  * Concurrency limits tuned to \\~20–25 connections.\n* **Datasette sidecar**\n\n  * Runs as a separate process/container when enabled.\n  * Base URL set to `/datasette/`.\n\n## 4.3 Request flow\n\n1. Browser → Remix route.\n2. Loader checks session. If needed, Google OAuth redirects.\n3. Loader/Action hits Prisma → SQLite.\n4. Response rendered via SSR + client hydration.\n5. Optional: Datasette used for admin/read operations.\n\n# 5) Configuration and Secrets\n\n* Required env:\n\n  * `SESSION_SECRET`\n  * `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`\n  * `APP_URL` (for OAuth callback)\n  * `DATABASE_URL` (defaults to `file:/data/sqlite.db`)\n  * `PORT` (default 8080)\n  * `INSERT_TOKEN` for Datasette insert bot\n* Secret handling: Store in Fly secrets or CI secrets. Never in repo. Rotate quarterly.\n\n# 6) Build and Deploy\n\n* **Build**: Multi-stage Docker builds:\n\n  * `deps` installs dev deps, `production-deps` prunes, `build` runs `prisma generate` and `npm run build`, final image copies `/build`, `/public`, `.prisma`.\n* **Start**: `start.sh` applies migrations then serves.\n* **Datasette image**: Minimal Python Alpine, installs `datasette`, `datasette-insert`, `datasette-auth-tokens`, serves `/data/sqlite.db`.\n* **CI/CD**: GitHub Actions push images to registry and deploy. Branching model: `dev` → staging, `main` → production.\n  Note: README mentions DigitalOcean registry and droplet; repo also contains `fly.toml`. Standardize on one target (recommend Fly for volume + healthchecks).\n\n# 7) Data Model and Migrations\n\n* ORM-driven schema via Prisma.\n* Migration policy:\n\n  * All schema changes via Prisma migrations committed to VCS.\n  * On boot, `prisma migrate deploy` runs. Fail fast if migration cannot be applied.\n* Backups:\n\n  * Nightly snapshot of `/data/sqlite.db` to object storage.\n  * Pre-migration snapshot in CI for production.\n\n# 8) Observability\n\n* Logs: stdout structured JSON from app. Retain 14–30 days in platform logs.\n* Health: `/healthcheck` HTTP 200.\n* Metrics (proposed):\n\n  * Basic RED metrics via runtime counters.\n  * Optionally expose `/metrics` for scrape.\n\n# 9) Security\n\n* HTTPS enforced.\n* Cookie `__session`: httpOnly, sameSite=lax, secure in production.\n* OAuth scopes: minimal email identity.\n* RBAC (future): roles for viewer, editor, admin.\n* Datasette:\n\n  * Base URL scoped under `/datasette/`.\n  * Token-gated inserts via `datasette-auth-tokens`.\n  * Default deny policy except for authenticated actor.\n\n# 10) Performance and Scale\n\n* SQLite fits current workload. Single writer, low read concurrency.\n* Concurrency guard at app level to avoid lock thrash.\n* Thresholds to move to Postgres:\n\n  * > 10 req/sec sustained writes.\n  * Multi-region rollout.\n  * Complex reporting joins.\n* Migration path:\n\n  * Replace `DATABASE_URL` to Postgres.\n  * Run Prisma `migrate deploy` on Postgres.\n  * Cutover via maintenance window.\n\n# 11) Testing Strategy\n\n* Unit: Vitest, `happy-dom` environment.\n* E2E: Cypress with `login()` helper and `cleanupUse
r()` per test file.\n* CI gates: typecheck, lint, unit, E2E against mocked services.\n\n# 12) Risks and Mitigations\n\n* **SQLite file lock contention** → keep mutations small; queue bulk imports; consider Postgres if contention spikes.\n* **Dual deploy
ment targets confusion** → choose Fly or DO. Remove unused manifests.\n* **Secrets leakage** → enforce CI secret scanning; restrict Procfile usage in prod if not needed.\n\n# 13) Open Questions\n\n* Is Datasette exposed in production or only behind VPN?
 If exposed, add auth proxy.\n* Which registry is canonical: DOCR or Fly’s? Align CI/CD.\n* Backup RTO/RPO targets?\n\n# 14) Acceptance Criteria\n\n* Auth works with Google in staging and prod.\n* App boots with `prisma migrate deploy` and serves on 808
0 behind TLS.\n* `/datasette/` loads with token auth.\n* E2E tests pass in CI on each PR to `dev` and `main`.\n\nSources: project Dockerfiles, Procfile, README, Fly config, package.json, Prisma bootstrap scripts.&#x20;\n\n\n</technical_specification_doc
ument>\n\n<repository_structure>\n.dockerignore\n.env.example\n.eslintrc.js\n.github/workflows/deploy.yml\n.gitignore\n.gitpod.Dockerfile\n.gitpod.yml\n.npmrc\n.prettierignore\nDockerfile\nDockerfile.datasette\nProcfile\nREADME.md\napp/charts/line-chart
.ts\napp/charts/multiseries-chart.ts\napp/components/graph.tsx\napp/components/person.tsx\napp/db.server.ts\napp/entry.client.tsx\napp/entry.server.tsx\napp/form-action.server.ts\napp/form.tsx\napp/models/graph.server.test.ts\napp/models/graph.server.ts
\napp/models/graph.ts\napp/models/person.server.test.ts\napp/models/person.server.ts\napp/models/person.ts\napp/models/user.server.ts\napp/root.tsx\napp/routes/auth/google.tsx\napp/routes/auth/google/callback.tsx\napp/routes/graphs/levels.tsx\napp/route
s/graphs/people.tsx\napp/routes/healthcheck.tsx\napp/routes/index.tsx\napp/routes/login.tsx\napp/routes/logout.tsx\napp/routes/people.tsx\napp/routes/people/$id.tsx\napp/routes/people/csv.tsx\napp/routes/people/new.tsx\napp/session.server.ts\napp/utils.
test.ts\napp/utils.ts\ncypress.config.ts\ncypress/.eslintrc.js\ncypress/e2e/smoke.cy.ts\ncypress/fixtures/example.json\ncypress/support/commands.ts\ncypress/support/create-user.ts\ncypress/support/delete-user.ts\ncypress/support/e2e.ts\ncypress/tsconfig
.json\ndatasette.metadata.json\ndatasette/metadata.json\ndeploy/grand-central/Caddyfile\ndeploy/grand-central/Caddyfile.local\ndeploy/grand-central/docker-compose.caddy.yml\ndeploy/grand-central/docker-compose.local.yml\ndeploy/grand-central/docker-comp
ose.staging.yml\ndeploy/grand-central/docker-compose.yml\ndoc/modelling.md\nfly.toml\nmocks/README.md\nmocks/index.js\npackage-lock.json\npackage.json\nprisma/generated/zod/index.ts\nprisma/migrations/20230120092252_init/migration.sql\nprisma/migrations
/20230125174823_add_changeset/migration.sql\nprisma/migrations/20230128042107_update_changeset/migration.sql\nprisma/migrations/20230223201415_drop_person_table/migration.sql\nprisma/migrations/20230407085725_drop_user_password/migration.sql\nprisma/mig
rations/migration_lock.toml\nprisma/schema.prisma\nprisma/seed.ts\npublic/favicon.ico\npublic/grand-central.jpg\nremix.config.js\nremix.env.d.ts\nscripts/import_changesets.ts\nstart.sh\ntailwind.config.js\ntest/global-setup.ts\ntest/setup-test-env.ts\nt
sconfig.json\nvitest.config.ts\n</repository_structure>\n

An AI conversation would have an array of such messages array with user, assistant, tool, represented in JSON. I need to preserve the semantic meaning while chunking, and the flow of text too. I can’t change the order of text, since that matters to an LLM. Since I’m working with inputs that are prompts or skill markdown files, I was likely to have delimiters that were XML / Markdown, etc.

I wanted to get a list of “splits”, so that I could replace a single message in-place with a list of smaller messages. So, I started with this prompt.

Given a structured JSON containing message parts, split any part that combines multiple distinct ideas into smaller, self-contained units. Each resulting unit must represent **one classifiable concept or function**, preserving all meaning, order, and structure. This prepares the data for hierarchical categorisation. Output **only** the complete replacements JSON object described.

Return **only** a single JSON object in this format:
{
  "replacements": [
    {
      "source_part_id": "42",
      "target_parts": [
        {
          "id": "42.1",
          "type": same_as_source_type",
          "text": "subpart content 1"
        },
        {
          "id": "42.2",
          "type": same_as_source_type",
          "text": "subpart content 2"
        }
      ]
    }
  ]
}

I pasted this prompt into a ChatGPT conversation, attached a messages.json, and started hacking away, trying to find a prompt that worked reasonably. And here’s what I got:

{
  "replacements": [
    {
      "source_part_id": "2",
      "target_parts": [
        {
          "id": "2.1",
          "type": "text",
          "text": "Given the <product_requirements_document>, identify questions the user should know about the PRD goals, customer roles, and scope."
        },
        {
          "id": "2.2",
          "type": "text",
          "text": "Given the <product_requirements_document>, identify questions about the data model, including Person and Changeset entities, and how snapshots are computed over time."
        },
        {
          "id": "2.3",
          "type": "text",
          "text": "Given the <product_requirements_document>, identify questions about analytics features such as headcount trends, joiners vs leavers, and level distribution over time."
        },
        {
          "id": "2.4",
          "type": "text",
          "text": "Given the <technical_specification_document>, identify questions regarding the use of CRDTs, specifically LWW Element Sets, and how they handle edits and historical views."
        },
        {
          "id": "2.5",
          "type": "text",
          "text": "Given the <technical_specification_document>, identify questions about the tech stack, including Remix, Prisma, SQLite, Fly.io deployment, and Datasette integration."
        },
        {
          "id": "2.6",
          "type": "text",
          "text": "Given the <technical_specification_document>, identify questions about authentication via Google OAuth, session management, and secret configuration."
        },
        {
          "id": "2.7",
          "type": "text",
          "text": "Given the <technical_specification_document>, identify questions about CI/CD pipelines, Docker builds, migration policies, and backup processes."
        },
        {
          "id": "2.8",
          "type": "text",
          "text": "Given the <repository_structure>, identify which source files implement routes, models, and components related to people, graphs, and authentication."
        },
        {
          "id": "2.9",
          "type": "text",
          "text": "Given the <repository_structure>, identify where database schema migrations, Prisma configuration, and data seeding occur."
        },
        {
          "id": "2.10",
          "type": "text",
          "text": "Given the <repository_structure>, identify files relevant for end-to-end testing, including Cypress and Vitest configurations."
        }
      ]
    }
  ]
}

The issues were:

• It only returned a single replacement most of the times
  • I thought this was because I only had one replacement in the sample response. I added two replacements in the response and the issue reduced, but persisted.
  • I asked chatgpt why it only returned one replacement. It told me that I had said any part, not all parts. Silly me. I fixed that, but the issue persisted.
• It didn’t return full message chunks, it would snip them with `…` in between or write a summary that represented the chunk.
• It was very slow, took >30s in most cases. The reasoning seemed solid, took its time, but the result didn’t reflect that thought. It was slow even without reasoning.
• It tried to use code tools, but I didn’t want it to take so much time (I specifically wanted low latency), and I wanted a generic solution.
• I additionally instructed it to “output exact text spans”, added principles on why I wanted it that way, etc. Still no luck.

There were a few other issues around the json structure, preserving additional fields, etc. I also added a couple of guiding examples. And at the end of these iterations, here’s the prompt I got to:

## **Task**

Segment a structured JSON containing message parts into **atomic semantic units**.
Each resulting unit must represent a single, self-contained **intent, fact, or function**, suitable for hierarchical categorisation.
Preserve all original wording, order, and structure.

---

## **Segmentation Rules**

* Each atomic unit should express **one topic, question, instruction, or operation**.
* If a part contains multiple such elements, extract them as separate contiguous spans.
* **Do not paraphrase, omit, or reorder** any text.
* **Preserve all tokens exactly** as in the source.
* Use existing natural boundaries such as XML/HTML tags, JSON objects, Markdown headers, list items, or paragraphs.
* Code blocks, tool calls, and similar technical sections must remain whole.
* Maintain the original hierarchy and `type` values.

---

## **Extraction Objective**

Identify and extract spans that each convey a single semantic role.
Think of this as **semantic segmentation for classification**, not text rewriting.
Output exact text spans that can stand alone and be categorised unambiguously.

---

## **Output Format**

Return only one JSON object in this format:

{
  "replacements": [
    {
      "source_part_id": "42",
      "target_parts": [
        {
          "id": "42.1",
          "type": "same_as_source_type",
          "text": "exact text span 1"
        },
        {
          "id": "42.2",
          "type": "same_as_source_type",
          "text": "exact text span 2"
        }
      ]
    },
    {
      "source_part_id": "84",
      "target_parts": [
        {
          "id": "84.1",
          "type": "same_as_source_type",
          "text": "exact text span 1"
        },
        {
          "id": "84.2",
          "type": "same_as_source_type",
          "text": "exact text span 2"
        }
      ]
    }
  ]
}

Each `source_part_id` corresponds to one original message part that was segmented.
Each `target_part` contains one extracted semantic unit, preserving order and meaning.

The next day, I woke up thinking: “why is this so difficult, I thought LLMs are good at this stuff”. And then I tried the simplest prompt to test the hypothesis that I’m using LLMs wrong.

given the following text, tell me where all you would apply a break.
just give me a json array.

## result
[
  "<task>",
  "<sources>",
  "<project_requirements_document>",
  "<technical_specification_document>",
  "<repository_context>",
  "<breakdown>",
  "<reflection>"
]

Woo! The results were instant (down to 5s from 20s), and exactly what I expected. The JSON input was likely interfering with its capability in identifying semantics, so I sent it only the text. And I didn’t need it to do the actual text-splitting, string.split could do that. I could also do this in parallel for all the messages that needed to be split. After some more tweaking of instructions, I got to this prompt.

Given the following text, tell me where all you would apply a break. The purpose is semantic chunking in way that's suitable for categorisation. Only give me the top level sections to split the text into coherent topical chunks.

Return ONLY a valid JSON array of regexes with positive lookahead which I can use to run string split in javascript.

Example response format: ["(?=regex-of-section-1)", "(?=regex-of-section2)"]

And without even structured outputs, this worked every time, within a few seconds. No reasoning, and no coding tools.

Categorisation

After breaking down messages into smaller chunks, I had to categorise them. So in the same manner, I iterated with my prompt and inputs on ChatGPT until I could find something that worked reasonably well.

Here’s a detailed description of my task, that became a prompt:

**Goal**
Produce a **hierarchical category map** that shows how information is organized in a conversation. Each category aggregates related message parts, summaries, and structure, enabling visualization and navigation of context usage.

**Instruction**
Given a structured conversation where each message part has a unique `message_part_id`, build a JSON tree that groups the conversation into semantically coherent categories and subcategories.

Do **not** use code tools or programmatic parsing for this task. Use reasoning and language understanding only.

### Your task
1. **Identify major categories** – infer the dominant conceptual or functional blocks from the conversation (for example: *Checklist of questions*, *File reads*, *Reasoning*, *Decisions*).
2. **Decompose recursively** – create subcategories only where the material naturally divides into smaller, meaningful topics.
   - Do **not** fix the number of levels; infer depth as needed.
3. **Assign message parts** – tag each message part with exactly one category or subcategory that best represents its content, using its `message_part_id`.
4. **Summarize each category** – every category node, including children, must contain:
   - `id`: unique short identifier, preferably using dot notation to indicate hierarchy (for example: `checklist`, `checklist.data_model`, `analysis.synthesis`)
   - `name`: concise label
   - `summary`: one-sentence description of what this category covers
   - `message_parts`: array of `message_part_id`s assigned directly to this category
   - `children`: nested categories, if any
5. **Preserve domain terminology** – derive category names from the conversation’s subject matter.
6. **Output** – return a structured, machine-readable JSON array representing the hierarchy, ready for downstream parsing and visualization.

---

### Reflection
Before returning the final JSON, perform the following validation steps:

1. **Completeness check** – ensure every `message_part_id` from the input appears in exactly one category.
2. **Representativeness check** – verify that the categories and subcategories together capture the overall structure and intent of the conversation, aligned with the goal.
3. **Domain integrity check** – confirm that terminology and phrasing reflect the conversation’s domain accurately, not abstract generalizations.
4. **Ground-level identification check** – make sure ground-level material (for example: detailed lists, code, or data) is correctly placed in leaf categories.
5. **Empty-category check** – remove or merge any category that has no assigned `message_parts` and no children with assigned parts. Categories without content are not useful.
6. **Final coherence review** – confirm that summaries are accurate, hierarchy depth is sensible, and the map as a whole provides a faithful and navigable representation of the conversation.

---

### Output specification
Return a **JSON array** of top-level categories.

[
  {
    "id": "checklist",
    "name": "Checklist of questions",
    "summary": "User’s structured audit checklist.",
    "message_parts": ["mp_12", "mp_13"],
    "children": [
      {
        "id": "checklist.data_model",
        "name": "Data model checks",
        "summary": "Questions about schema alignment.",
        "message_parts": ["mp_14", "mp_15"]
      },
      {
        "id": "checklist.security",
        "name": "Security and authentication checks",
        "summary": "Questions related to authentication and authorization.",
        "message_parts": ["mp_16"]
      }
    ]
  },
  {
    "id": "analysis.synthesis",
    "name": "Analysis and synthesis",
    "summary": "Assistant’s reasoning and conclusions.",
    "message_parts": ["mp_20", "mp_22"]
  },
  {
    "id": "files.reads",
    "name": "File readings and tool calls",
    "summary": "Assistant’s inspection of repository files using tools.",
    "message_parts": ["mp_30", "mp_31"]
  }
]

What didn’t work? That reflection list in the prompt is a good list of things that failed!

It wouldn’t return all message parts, some parts would remain without a category, some categories would be created without the parts. Sometimes the message part ids would be hallucinated. And of course, all of this took a long time.

Following the same method, I thought about what the model is actually good at, and went with this one-line prompt:

given this conversation, give me a list of categories that shows how information is organized. just give me a list of categories in a json array

## Result
[
  "Problem statement and goals",
  "Customer personas and jobs-to-be-done",
  "Functional and data requirements",
  "Technical architecture and stack",
  "Data modeling and schema design",
  "Authentication and security",
  "Directory, person detail, and history",
  "Import and export processes",
  "Analytics and reporting",
  "Audit logging and event tracking",
  "Testing and CI/CD",
  "Performance, scalability, and operations",
  "Open questions and risks",
  "Repository and code organization"
]

These weren’t the categories I wanted. I realised I didn’t want categories that represented the information in text, I was looking for components of the conversation, because with context-viewer, I was not looking to analyse the conversation itself, but how the conversation happened. So I changed that one part:

given this conversation, give me a list of components it comprises of. just give me a list in a json array

## Result
[
  "product_requirements_document",
  "technical_specification_document",
  "repository_structure",
  "repository_context",
  "task",
  "sources",
  "breakdown",
  "reflection"
]

That worked! While no output specification was necessary, I tweaked it a little as I experimented with some more input files.

given this conversation, give me a list of all its components
each component can be 3 to 4 words in length

just give me a list in a json array like this example:
["abc_document", "xyz_structure", "foo_context", "task", "sources", "breakdown", "reflection", "files_about_bar", "files_about_baz", "tool_calls_about_quix", "xyz blocks", "pqr list"]

And then I used another prompt to get the category mapping for each message part, which worked on the first try.

given this conversation and the list of components, give me a mapping of message part ids in the conversation, to a component from the list, for all the message parts

just give me a simple json object {id: component}

<conversation>${conversationJson}</conversation>
<components>${componentsJson}</components>

## Result
{
  "24.5": "product_requirements_document",
  "24.6": "technical_specification_document",
  "24.7": "repository_structure",
  "26": "assistant_reasoning_blocks",
  "27": "assistant_reasoning_blocks",
  "28": "assistant_reasoning_blocks",
  "29": "assistant_reasoning_blocks",
  ...

Overall I’m glad that the prompts I needed in the end were tiny. I think that’s a signal that I’m using LLMs correctly. Just needed to break the problem down, let the model do what it’s good at, and augment it with some good old fashioned engineering where needed.

The need for context observability

Context engineering is one of the most important aspects of AI Engineering.

• Andrej Karpathy: “In every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window…”, and we need to “pack the context windows just right”.
• Drew Breunig: “…context is not free. Every token in the context influences the model’s behavior, for better or worse.”, and “The massive context windows of modern LLMs are a powerful capability, but they’re not an excuse to be sloppy with information management.”

And yet, we don’t have the tools to really see the context, or pull it apart into tangible components we can analyse for relevance. That kind of observability of the context seems to be missing. If we want to fight the “Garbage In, Garbage Out” phenomenon, we should be able to observe the garbage in the context to prune it out.

Existing observability tools seem to be primarily focused on system metrics like latency, cost, error rates, or high-level agent tracing. They might show a flame-graph of requests and responses, but not what kind of content is crowding up the context window over time.

In his “How Long Contexts Fail”, and “How to Fix Your Context” posts, Drew Breunig details these failure mechanisms and methods to combat them.

But we need to detect and diagnose these kinds of issues before we can solve for them. Hrm. If only we had a tool that dealt well with large unstructured text, segment and classify it, and also detect these issues.

Context viewer

So I built a tool to provide this kind of observability, and it is called context-viewer (I like pithy names). It is an open source tool I built over last week. It doesn’t have a server component, and is designed to work directly in the browser for convenience. Feel free to fork, tweak, raise PRs.

You drag-drop a conversation JSON log into it, and with some AI assistance, it pulls it apart into tangible components, and lets you see its growth over time. Here’s what it looks like for a run of StoryMachine, which is currently a simple workflow.

For context, the initial prompt provides product requirements and a tech spec, then it reads the repository to get a relevant summary, and then breaks down the work into user stories in stages with some human feedback. There’s more details in a section below.

You can see a list of components, and how they change and grow over time. Recency matters in context, so just looking at space won’t do, it has to be across time and space.

It visualises growth of components as counts and percentages of total token count.

And it also lets you filter, sort, search through messages in a conversation however you’d like. Not all messages have the same prominence even at the same token count. User messages that direct the conversation carry more weight.

Here’s the Github link again, if you want to try it out: nilenso/context-viewer

How it works

It’s a fairly simple process with a few steps.

1. Parse the conversation: This is currently one of the open-ai formats of chat-completions / responses / conversations API logs. It’s easy enough to parse another format. You can also drop multiple conversations to process them in parallel. Interface has basic search, filtering and sorting too.
2. Count the tokens: It uses dqbd/tiktoken’s WASM bindings to count tokens per message in parallel.
3. Segment large messages: Messages over a certain threshold of tokens (which is 500, somewhat arbitrarily) get broken down into smaller pieces using AI. If there are multiple parts in a single prompt / message, the parts get broken down here. This is basically a dead-simple-single-prompt version of semantic chunking, lot of room to improve this.
4. Find components: Given the entire conversation, an AI call identifies the components, and another call assigns components to individual messages. You can also tweak the prompts right in the UI, and iterate on the components until you get to a classification you’re happy with.
5. Visualise: The components view follows anthropic’s visualisation of context. There’s a time-slider that lets one traverse through how context fills up, and there’s also a simple stacked bar graph that shows growth over time.
6. Synthesize: Given the generated data of component growth over time, and a summary of the conversation, an AI can do a pretty great job of telling us about growth patterns good and bad, redundancy, and relevance. It might then also provide reasonable solutions since there’s sufficient light on the problem now.

The prompts are very short descriptions of the goals, targeted at what the simplest model is very good at. While I have found these default prompts to be good, they’re meant to be iterated on to assist us in a process of finding the right segment sizes and the right dimension to split components with. Some might want much more granular components, some purposes need more domain specific categories, and some might prefer hierarchical categorisation with the ability to zoom in/out of them. Being able to slice and dice the data in a way that fits the needs is essential in observability.

Analysing a ~13k tokens conversation for all the above steps takes ~15s. A ~35k tokens conversation takes ~40s. I’d say it scales linearly. I didn’t care too much about performance or cost with this tool. But if I find that this is worth building out into something more full fledged and practical for large scale usage, then I’ll probably redesign parts of it. Suggestions welcome!

Why I built this, and how it has helped

I ran into this need pretty quickly when working on StoryMachine. StoryMachine is a fairly simple workflow currently. It takes a PRD, a tech spec, reads the codebase, and generates stories with some user feedback. Still, it’s a fair number of turns in a long conversation. And more importantly, the conversation involves iterating over artefacts of stories, and I knew that every iteration of a story would be adding the whole story to the context again, leading to redundancy.

I wanted to see this happening. So that I could verify that this is indeed a problem, and a big enough problem warranting an engineering solution. I also wanted this feedback loop for myself so I could prioritise my attention to truly important issues.

A few ways in which I immediately saw benefits:

• I could “see” the conversation happen. A summary of what happened in the conversation, how many turns, tool calls etc, helped me quickly get a grip on what I’m analysing.
• The components it detected were “project_specifications”, and “product_specifications”. I had made a typo! And the LLM could have been looking at these as two different docs when they were actually the same doc. And I guess more importantly, this was duplicated context occupying about 13% of context window.
• The questions used to generate the relevant repository context are not relevant after that activity, but they remain in context unnecessarily. The AI generated analysis told me this, and I could see it visually too. This was about 2% in terms of size. But then again, even a small amount of poison could be lethal depending on the type of poison; size isn’t everything.
• And by the end of iterating through 2 stories with user feedback, because the full stories were being regenerated at every turn, it added up to 41%. This validated my hypothesis, and allowed me take it on as a real problem to solve.

About using AI to analyse

This is certainly a debatable design choice, so here are my thoughts around it:

1. It’s a bit annoying to use another AI, burning more tokens and money to come up with something that could be a first class concept in an AI SDK. If we segmented our prompts correctly, split apart using XML tags with attributes for identifying components, and also tag all tool calls and tool responses, then we should have a lot of the data already organised for observability. Similar to emitting rich product events from applications. I think this is a good idea worth exploring. DSPy’s signatures is a layer of abstraction that’s well suited for this sort of annotation at the request side. However, the responses from LLMs that also go into context do need to be broken down, and we don’t really control that.
2. The model I’ve been using so far is gpt-4o-mini, which is 2 generations old, but I use it because it is fast, and produces good enough results. I bet one could use a gpt-oss model, or some other local tiny model to similar results. That way this analysis could be fast, and fully private too.
3. Semantic chunking, and identifying components can happen across any arbitrary dimension. It is essential to have a way to modify the process to be better suited to each conversation / product / workflow’s needs. Using simple AI prompts here is a nice method that permits this. context-viewer lets you change the prompts and re-run the analysis so you can iteratively get the observability you need.
4. At scale, some of this would not be feasible anymore, especially inside a browser tab. If one wanted to analyse 1000 conversations of a kind, running through 10s of AI calls per conversation isn’t sustainable or even meaningful. We’d have to engineer around it. Perhaps one can use this kind of a tool to explore and figure out how they want to analyse their conversations, and then engineer a custom data pipeline that scales, perhaps without AI. Semantic chunking methods that are not LLM based could be one way to go.

On existing observability tools

The LLM observability space is large with lots of players in the field like Braintrust, Helicone, Langfuse, Arize, WhyLabs, Fiddler, Evidently, etc. Their focus seems to be largely on:

1. Logging and tracing: see input, output, spans, traces, latencies, flame graphs, timelines for agents, etc.
2. Token Usage & Cost Monitoring: observability for optimising unit economics, cost per request, performance, efficiency.
3. Latency, Throughput & Error Rate Monitoring: more traditional engineering systems observability features to help keep systems reliable.
4. Anomaly and outlier detection: To improve the effectiveness of above metrics
5. ML Observability: Data and concept drift detection in input, output, or features
6. Output Quality Evaluation: Broad relevance, quality, success scores

Not that these aren’t important, but they don’t give the kind of feedback I’m looking for on my context. I find that almost all of these tools treat the context as a singular entity, and don’t really break them down for analysis.

Further, the kind of analysis I’m referring to here feels more domain, or product specific. Something I’d expect to find in a Mixpanel / Statsig dashboard, not in a datadog dashboard. The analysis of events is however still not similar to the way product metrics are usually analysed.

LangChain’s insights agent that came out last week comes closest to what I’m looking for.

Try it out

• Go to nilenso/context-viewer, and clone it
• Add AI API keys in environment variables as README.md says, build and run it.

• Export your context as a JSON file. Honestly, this isn’t as straightforward as I would like for it to be. Currently I’ve built support for Open AI formats:

  • For responses format you can use the input-items API
  • For conversations format you can use list-items API
  • For chat-completions, I used an internal dashboard API: https://api.openai.com/v1/dashboard/chat/completions/chatcmpl-id
  • For other formats, adding another parser is just a prompt away.
• Drag-drop it into context-viewer on the browser

I had a lot of fun building this, and I plan to write about that experience. Meanwhile, here’s an end-to-end demo at 2x:

I’d love to hear what you think. Join the discussion on Hacker News! Contributions welcome through PRs. You can also tweet @nilenso, or email us at hello@nilenso.com to reach us.

Everyone is talking about Richard Sutton’s Bitter Lesson once again¹.

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

Rich Sutton, The Bitter Lesson

I highly recommend taking a detour to read Richard’s essay if you haven’t yet, before coming back to this page. It’s very short.

Here’s my observation: The bitter lesson applies to developers building and working with AI applications as well—and many have not yet digested the bitter lesson.

How not to code with AI

I’ve observed a type of AI-maximalist programmer often found at vibe coding events, workshops and demos. Their setup often has a folder full of text files that describe “rules”, “modes”, “roles”, prompts, or subagents. It often looks like a dump of all possible individual actions a developer can take—PRD analyser, planner, user story writer, code reviewer, UAT tester, etc. These files are full of long instructions, with lots of “pleading” language (or threats), capitalisation and even step-by-step logic telling an LLM how it should think and act.

The fundamental error in the above methods is that they bake in assumptions of what a workflow should look like, and how the agent should operate. They meddle with the model’s behaviour. It is what Sutton would describe as a “human knowledge based” method.

Some of these tricks were necessary when the models were weaker and less agentic. Today, they can reason well and learn from feedback in the environment. Force-fitting a complex web of workflows and roles is potentially fighting against the model weights.

The engineer that has digested the bitter lesson will instead set up an environment that can provide feedback loops to the agent. This setup is simpler and better accommodates frontier reasoning models that are scaled with reinforcement learning by getting out of their way.

How not to build LLM wrappers

I have observed engineers directly jump to complex workflows, indiscriminate application of prompting tricks and multiple agents with fixed roles when designing an LLM-integrated application. These add unnecessary complexity and should not be the default starting point. To better illustrate why, we can look at how coding agents have evolved over time.

The first generation of AI coding tools (Cursor, Sourcegraph Cody, Codeium², Copilot) heavily relied on chunk-and-embed paradigm, ie, use a separate vector embeddings storage layer that prefills retrieved chunks into the LLM’s context window. ³

Newer AI tools (Cline, Windsurf, Amp, Claude Code, Codex, OpenHands) eschew pre-filled retrievals in favour of agentic search—ie, tell the AI how to invoke a search, and let it figure it out from there. How the search is performed is an implementation detail. This is a much simpler fundamental architecture. ⁴

The latter approach better embodies the bitter-lesson. Do not bake in your human knowledge assumptions by prefilling items into the agent’s context window.

Reinforcement learning produces goal-seeking agents. Anyone who has digested the bitter lesson knows that more compute is being poured into these LLMs to make goal-seekers (they get a reward signal when they achieve their goal). Leverage this fact. As models get better at goal-seeking in general, they will get better inside applications that mirror this action → feedback loop.

We can generalise this for most LLM-enabled applications.

Let’s contrast some human-knowledge driven “artisanal” architectures against more “bitter-lessoned” architectures which could represent two ends of a spectrum.

Signals affirming the bitter lesson influencing application design

• GPT-5 Codex resulted in a new system prompt in Codex CLI. It shrunk from ~300 lines to ~100 lines.
• The Claude Code prompt also shrank. The multi-edit tool call was removed, further simplifying the program.
• Claude Code creators actively avoided prefilled context and embed-and-chunk, directly citing the bitter lesson.
• Cloudflare introduced code mode, where it rewrites MCP tool interfaces into typescript interfaces, because the LLM is much more competent at writing and composing code. While providing tools are more bitter lessoned than prefilled context windows, I thought this goes a step further.

When to use artisanal architectures

This is not to say that artisanal architectures are bad. It’s that artisanal architectures must account for the bitter lesson.

When the model isn’t good at your task yet, but may get there eventually under the current scaling regime—design an artisanal architecture to build what is needed today, but do so with the knowledge that some day you may have to throw this functionality away—make the artisanal parts especially easy to remove when the bitter lesson inevitably strikes. ⁵

A more permanently artisanal architecture also makes sense when your task does not require a repeated sequence of actions and deep thinking, for example, a classification task in a pipeline or a task to link similar address records.

Make a note of what is not scaling with compute

With current scaling methods verifiable tasks with clear goals will continue to improve: coding, searching, mathematics. Leave the methods of achieving the goal to the agent.

Current training methods have also not scaled context window sizes as reliably—so you might want to hold on to subagents and context-compaction tricks.

Training methods will also not solve for important parts that are gluing things together, like retries and reliable execution, or good interface design.

Summary

• Make a note of what aspects of models will improve with the current scaling regime.
• Account for the scaling in your AI application—avoid human-informed workflows in lieu of action-infer-feedback loops.
• Artisanal architectures and methods may be necessary for many applications—but do so with the knowledge that you might have to throw it away when the next frontier model drops.
• Some things won’t scale with compute—most of your artisanal ingenuity may be required here.

Similar articles

• Learning the Bitter Lesson
• The Model is the Product

Thanks to Srihari and Ravi Chandra Padmala for reviewing drafts of this.

They measure something narrower than what their names suggest. In general, they are a lot less messy than how we write software. This is why Claude scoring 80% on SWE-bench does not translate to Claude one-shotting 80% of the things I throw at it.

Let’s look at what these benchmarks are actually measuring¹.

Table of Contents

• SWE-bench Verified and SWE-bench Pro
  • What it measures
  • The specifics
  • Verdict
• Aider Polyglot
  • What it measures
  • The specifics
  • Verdict
• LiveCodeBench
  • What it measures
  • The specifics
  • Verdict
• Other benchmarks
• Benchmarking is hard and this makes me bullish on coding agents
• References and further reading

SWE-bench Verified and SWE-bench Pro

What it measures

How well a coding agent can submit a patch for a real-world GitHub issue that passes the unit tests for that issue.

The specifics

There are many variants: Full, Verified, Lite, Bash-only, Multimodal. Most labs in their chart report on SWE-bench Verified, which is a cleaned and human-reviewed subset.

Notes and quirks of SWE-bench Verified:

• It has 500 problems, all in Python. Over 40% are issues from the Django source repository; the rest are libraries. Web applications are entirely missing. The repositories that the agents have to operate are real, hefty open source projects.
• Solutions to these issues are small—think surgical edits or small function additions. The mean lines of code per solution are 11, and median lines of code are 4. Amazon found that over 77.6% of the solutions touch only one function.
• All the issues are from 2023 and earlier. This data was almost certainly in the training sets. Thus it’s hard to tell how much of the improvements are due to memorisation.

Recently, Scale AI published an improved version called SWE-bench Pro that tries to address some quirks of Verified. Here are my notes:

• 1865 problems, from 41 repositories. It’s a mix of copyleft/GPL public repositories and some private repositories. I’m skeptical that choosing copyleft would meaningfully combat contamination, given AI labs are known to resort to piracy to train their models. But this is still an improvement.
• The repositories are Python, Go, JS and TS—we don’t know the distribution, except that they are “not uniform”. But they also ensure every repository has only 50-100 problem instances.
• They claim to sample repositories “from a diverse range of topics, including consumer applications with complex UI logic, B2B platforms with intricate business rules, and developer tools with sophisticated APIs”—much better than SWE-Bench Verified. That said, I could not find what this distribution looks like.
• The solutions have a mean of 107 lines of code, and a median of 55 lines of code, and usually span an average of 4 files. Also good.
• They actually got humans to rewrite problems based on issues, commits and PRs to ensure there’s no missing information. They also added “requirements [that] are grounded on the unit tests that are used for validation”. They may also add interface code for some problems.
• They also have dockerized environments set up with all the dependencies installed and configured, so the benchmark explicitly does not test if your agent can setup your repository.

Verdict

Overall, I think SWE-bench is a good, if still very flawed benchmark (most other benchmarks are a lot worse). I also think SWE-bench Pro addresses some severe problems with Verified (which at this point should just be ignored in any frontier model report). I’ll note that there’s significant drift from what this measures and how I actually work with AI coding agents.

SWE-Bench is measuring how well AI performs on well-defined units of work. So when we say that an agent scores 25% in SWE-bench Pro, we are saying: “In a problem set of well-defined issues with pointed requirements and (the occasional) specification of code interfaces, 25% of the solutions from the agent get the respective problem’s unit test cases to pass”.

This is a useful measurement of progress. But this is not SWE as I understand it—most of the high-leverage parts are in working with product owners to come up with a good specification, translate them into useful interfaces, and then writing secure, maintainable code. With this benchmark we do not have any idea if the code is maintainable, secure, provably correct, or well-crafted—we just know that the unit test cases for it will pass².

Aider Polyglot

What it measures

If the coding agent (specifically, Aider) can solve hard-level Exercism problems and apply file edits that pass unit tests after at most one round of feedback.

The specifics

Exercism is a learning platform with “kata-style” programming exercises. It’s not as algorithmic as LeetCode, but still pretty contained.

class BankAccount {
    void open() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }


    void close() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized int getBalance() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized void deposit(int amount) throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized void withdraw(int amount) throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }
}

It seems to have far more language diversity than most other popular benchmarks: C++, Java, Go, Python, JavaScript and Rust are covered. The more functional programming languages are still unrepresented.

Glancing at individual examples, it seems to me that most solutions are in the range of 30-200 lines of code, and spanning at most 2 files.

Like SWE-bench, the evaluation is based on how many unit test cases pass. Everything runs on the Aider harness and prompts.

Verdict

This is a benchmark that will tell you how good a model is at solving small, tight and well-defined problems. It’s a good measure to check how well a model will perform on Aider across a range of languages. But this is nowhere near a benchmark for SWE (nor does it claim to be). Like SWE-bench, it also only checks unit test case pass rate, which does not account for many aspects of correctness and software quality.

LiveCodeBench

What it measures

Python competitive-programming skills under hidden test suites with a rolling, “fresh” problem set. Think LeetCode.

The specifics

• It consists of the following tasks:

  • Generate solutions to competitive coding problems from scratch.
  • Fix incorrect solutions to coding problems.
  • Predict the output of a function given the code (weird but why not!)
  • Given only the problem statement and test input, predict the output (okay?)
• Everything is Python.
• There’s a fairly even balance between Easy, Medium and Hard problems.
• The evaluations are just like LeetCode: run hidden test cases.

• We know there’s little contamination—only problems released after each model’s cutoff date are evaluated.

  • That said, because it’s LeetCode style, lots of problems will look quite similar to each other.

Verdict

This isn’t a SWE benchmark. This will tell you how good a model is at for solving LeetCode style Python problems, along with a mix of some slightly unusual skills like “mental execution” of code and test case output prediction.

Other benchmarks

TerminalBench: This is interesting because it exclusively focuses on terminal use. So SWE-Bench paired with TerminalBench will give a broader picture of SWE-like capabilities.

SWE-Lancer: OpenAI released this earlier in the year, and I thought it was neat because it directly maps the work to economic value by getting the agents to work on Expensify/Upwork tasks. The validation comes from E2E tests rather than unit tests. Unfortunately, their reporting of this benchmark has been quite lowkey since. And nobody is running this benchmark on non-OpenAI models anymore (which I’m curious to see especially since Claude Sonnet 3.5 outshone the o1 model back in the day). There are other flaws with this benchmark. But this framing seems to be in the right direction.

METR’s Long Horizon Benchmark: This was an interesting framing, as it considered the time horizons of LLMs working autonomously. They also have a detailed rubric for the “messiness” of a task. I have talked about this benchmark in my article about managing units of work for AI agents.

Multi-SWE-Bench: ByteDance made a polyglot benchmark that works similarly to SWE-Bench, that spans seven languages: Java, TypeScript, JavaScript, Go, Rust, C, and C++.

SWE-Bench Multilingual: Another polyglot benchmark that spans nine languages, compatible with the SWE-Bench harness. Has data from popular C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust repositories.

HumanEval (and its variants): An old coding benchmark that should be totally ignored today. The tasks seem to require implementing extremely easy Python toy functions.

Benchmarking is hard and this makes me bullish on coding agents

A large lesson I took away from studying the specifics of popular benchmarks is that designing a good benchmark is highly labour-intensive. Without human review and annotations, it’s nearly impossible to make a good benchmark. The more sophisticated the benchmark gets, the more it seems to require human intervention to ensure that the tasks are high-quality and not nonsensical or impossible.

And then there are the actual evaluation methods. The way to scale up evaluations is to have automated verification across all tasks. It’s not surprising that most evaluations boil down to “make the unit tests pass” due to this reason. But this will always fall short when it comes to actually benchmarking what I consider the core work of an SWE—which is to translate a problem into structured, verifiable solutions. There is some subjectivity and fuzzy judgement involved around satisfying business needs and timelines, making the right architectural tradeoffs and ensuring the solution is good over a long time horizon.

Considering how state-of-the-art benchmarks fall woefully short of capturing the nuance and messiness of SWE work, the coding agents we have are fantastic. One could imagine how much better they would get once we have better benchmarks (and RL environments) that do a better job than what we have today. I’m not sure about how we’d solve for the more subjective parts, but until recently we didn’t even have good polyglot benchmarks. There’s still a lot of low-hanging fruit. This suggests to me that we are yet to hit any kind of wall for coding abilities in the near future.

On a very short notice, I can already think of a bunch of ideas for what could be improved in benchmarks:

• Validate using generative testing methods, such as PBT or fuzz testing instead of unit tests.
• Use formal methods to check for correctness, where possible.
• Validate answers against automated User Acceptance Criteria checks, where possible.
• Start with product-level documents as the input for the benchmark, such as business context, PRDs and technical specifications. Validate against automated UATs and end-to-end tests.
• Create a benchmark that accounts for the information acquisition and clarification that real SWEs have to do—I could imagine having a benchmark that intentionally does not give all the necessary information up front, and it’s up to the agent to present the required clarifications or search for the necessary context.
• Use well-calibrated human judges to score on the fuzzier criteria of quality (this seems quite hard to do right, and “well-calibrated” is doing a lot of heavy lifting in that sentence).

References and further reading

SWE-bench / Verified

• Dataset
• Leaderboard
• AWS SWE-bench analysis
• Original paper

SWE-bench Pro

• Scale AI blog
• Paper
• Leaderboard

Aider Polyglot

• Aider blog post

LiveCodeBench

• Project page

TerminalBench

• Official site
• Docs

SWE-Lancer

• OpenAI blog
• Paper

METR Long-Horizon

• Blog update
• Paper

Polyglot SWE-bench variants

• Multi-SWE-bench (ByteDance)
• ArXiv paper
• Hugging Face dataset
• SWE-bench Multilingual

HumanEval

• Paper
• GitHub

The unit of work would be the fundamental abstraction. We’d build state machines and workflows around it, carrying it from specification to deployment through activities performed by product managers, engineers, designers, and others. The process could be customised to each team’s needs, with all the bells and whistles. But fundamentally, its effectiveness and adaptability depends on how good this central abstraction is.

Get this abstraction wrong, and complexity scales exponentially. All the processes built around it inherit the dysfunction. Planning becomes chaotic, progress becomes opaque, and coordination becomes an expensive mess.

We deal with leaky abstractions by periodically refactoring them, so why not do the same thing with the unit of work? What makes a good unit of work? Let’s walk through these familiar activities and observe the properties that emerge.

Breaking it down

We typically start with product or feature requirements. We don’t usually take on a full feature in one shot, it’s “too big”. Especially if it’s complex enough to need some technical design and specification written along with it. We break it down into small parts that are more approachable for solving, and also give us a steady sense of progress.

Now the product requirement is actually a hypothesis for creating business value, and we need to validate the hypothesis as early as possible. So, the small parts need to be valuable to the customer.

In other words, we need the unit of work to be a slice of the cake, not a layer.

Of course, bug fixes and refactors aren’t providing value in the same way, and that’s okay. Sometimes there are technical tasks that are best left independent. That’s okay too. No need to be dogmatic as long as the broad needs of value and sense of progress are being met.

Planning

Before starting work, we want to prioritise, because it saves a lot of time. We want to ship the most valuable slices first, and perhaps discard some low priority slices. But we can’t prioritise without weighing the business value against the implementation effort. All slices aren’t the same size, so we estimate the implementation effort first.

Then, some large slices can have low product value, so we would want to break them into even smaller slices to prioritise parts we care about most. Some other large slices can’t be sliced further meaningfully, and that’s okay. Some smaller slices can’t be engineered independently, so we build the larger slice anyway. The unit needs to be negotiable.

And since we’re doing this as a team, we’ll want to ensure that the slices are as independent as possible, so that we can each do our part without waiting, and we don’t step on each other’s toes.

Gathering context

A unit can be specified today, picked up for execution next month, blocked by another task, and then deprioritised into the backlog. Over its life, it gathers context about various things:

• What value it provides, how to verify it
• How it needs to be implemented
• Missing pieces of context that came together after conversations
• Unknowns that were resolved or unresolved
• Who worked on it, what issues they ran into
• What bugs came up in testing, and QA before release

Keeping these pieces of context collected in a single place helps in picking it up from where it was left off. When discussing, implementing, or tracking, it’s useful to have the same artifact in front of us.

Solving

Knowing exactly what we’re solving for is very helpful, so we can build just enough software™️. No more, no less. So we need to define the acceptance criteria that we can all agree on.

Then, solve until we meet them.

It’s good to automate checking whether they meet the acceptance criteria, because we’re going to be doing that an awful lot while solving.

Verifying

Confidence usually doesn’t require checking every possible case, only the key ones that capture most of the impact. Yes, we checked this slice at every step of the way, but it is useful to inspect it one last time before serving.

When is a unit considered done? When the slice has been served. When it’s in the hands of the user, in production, potentially behind a feature flag.

And that’s it. To manage the life cycle of software development, we manage the unit of work. Some would say we need to INVEST in good units of work. And some of you might rightly recognise that it looks like a User Story. But as long as the described properties and affordances for its users exist, it should make for a decent unit of work regardless of what we call it.

Does your unit of work need refactoring?

We’re fairly aware of the penalties of leaky abstractions in software. The incidental complexity of getting our primary real world abstractions wrong, grows exponentially with each layer of software built over it, until the whole system is slow, sludgy slop that’s difficult to work with. We can hack it here and there, and celebrate minor wins, but the big wins were lost in the ignored opportunities to refactor that central abstraction.

If we apply the same thought process to software development, we’ll see that our core abstraction, the unit of work, might need refactoring.

Big gains in developer productivity in this economic weather are important. Organisations that use DORA measure deploy or commit frequencies might find them valuable in some dimensions, but they’re not a measure of productivity in terms of outcomes for the customer. I love these last lines in Kent Beck’s writing about measuring developer productivity:

Be suspicious of anyone claiming to measure developer productivity. Ask who is asking & why. Ask them what unit they are measuring & how those units are connected to profit.

I am 100% pro-accountability. Weekly delivery of customer-appreciated value is the best accountability, the most aligned, the least distorting.

And I think a unit of work as defined above could be used to measure productivity holistically. Prioritising by value, eliminating unnecessary work, and validating quickly then become obvious, and measurable ways to increase productivity.

Productivity gains through use of AI assistants is also popularly reported and benchmarked in terms of % of code generated, but that’s not a very valuable dimension for measurement. If the benchmarks for AI productivity revolved around units of work valuable to the customer, then we’d be talking true productivity gains. AI assistants also need small, well specified slices of work, and hence, will also benefit from a well defined unit of work. My colleague Atharva has written a wonderful blog post about that in detail.

Yeah, this article is mostly about rehashing a two-decade-old pitch for some common sense agile. But I hope it has been worth your time.

Annexes

• In reality, the workflow isn’t as linear, and there is much back and forth between the steps. I’ve kept it simple to focus on the properties.
• Yes, I’m aware the classic definition of user stories doesn’t have implementation details.
• Slicing can happen across many dimensions, and breaking down a hard problem effectively, can actually be a very hard problem.

• If you want to read the OG Agile material, you can read:

  • Kent Beck introducing story cards in XPX (Chapter 15 on planning)
  • Bill Wake’s writing, and the INVEST criteria are condensed, quick reads
  • The C2 page on User Stories for opinions and some discussions
  • Ron Jefferies on Card, Conversation and Confirmation.
  • Mike Cohn’s User Stories Applied is a deep dive.
• I like Gergely Orosz and Kent Beck’s response to McKinsey on measuring developer productivity. Gergely’s writing about DORA, and SPACE is interesting, but I wonder if metrics can be more granular, around this unit of work, and its affordances. That would shift-left the feedback on productivity, to where it matters.

When I was new to this emerging craft of AI-assisted coding, I was getting lousy results, despite the models being rather intelligent. Turns out the major bottleneck is not intelligence, but rather providing the correct context.

Andrej Karpathy, while referencing my earlier article on this topic, described the work of AI-assisted engineering as “putting AI on a tight leash”. What does a tight leash look like for a process where AI agents are operating on your code more independently than ever? He dropped a hint: work on small chunks of a single concrete thing.

The right sized unit of work respects the context

I like the term context engineering, because it has opened up the vocabulary to better describe why managing units of work is perhaps the most important technique to get better results out of AI tools. It centers our discussion around the “canvas” against which our AI is generating code.

I like Anthropic’s visualisation from their docs:

The generated output of the LLM is a sample of the next token probability. Every time we generate a token, what has already been generated in the previous iteration is appended to the context window. What this context window looks like has a huge influence on the quality of your generated output.

Drew Breunig wrote an excellent article about all kinds of things that can go wrong with your context and proposed various techniques to fix them.

The best AI-assisted craftsmen are often thinking about the design and arrangement of their context to get the AI to one-shot a solution. This is tricky and effortful, contrary to what the AI coding hype suggests.

If you don’t provide the necessary information in the context to do a good job, your AI will hallucinate or generate code that is not congruent with the practices of your codebase. It is especially brittle at integration points of your software system.

On the other hand, if you fill up the context with too much information, and the quality of your output degrades, because of a lack of focused attention.

Breaking down your task into “right-sized” units of work, which describe just the right amount of detail is perhaps the most powerful lever to improve your context window, and thus the correctness and quality of the generated code.

The right sized unit of work controls the propagation of errors

Time for some napkin maths.

Let’s say your AI agent has a 5% chance of making a mistake. I’m not just referring to hallucinations—it could be a subtle mistake because it forgot to look up some documentation or you missed a detail in your specification.

In an agentic multi-turn workflow, which is what all coding workflows are converging to, this error compounds. If your task takes 10 turns to implement, you will have a (1 – 0.95)¹⁰ = 59.9% chance of success. Not very high.

Utkarsh Kanwat in his blog post has made the same argument. His conclusion was that any AI agent would need some kind of pause-and-verify gating mechanism at each step for a long-horizon task.

What does the state of the art for multi-turn error rates look like? METR recently published a popular chart describing how AI models are getting better at long-horizon tasks. Currently GPT-5 is at the top of the leaderboard, where it can perform ~2-hour long tasks at around a 70% success rate. Working backwards (let’s say a 2 hour task is 50+ turns) this would amount to a sub-1% error rate per action.

Doesn’t a <1% error rate per action seem suspicious to you? As a regular user of agentic coding tools (my current one is Codex CLI), I’ll eat my shoe if GPT-5 starts nailing my tasks 99.9% of the time.

My intuition derived from experience tells me that even the best AI right now isn’t even 95% likely to be correct. So where is the difference coming from? It needs a closer look at the actual paper:

Our tasks typically use environments that do not significantly change unless directly acted upon by the agent. In contrast, real tasks often occur in the context of a changing environment.

[…]

Similarly, very few of our tasks are punishing of single mistakes. This is in part to reduce the expected cost of collecting human baselines.

This is not at all like the tasks I am doing.

METR acknowledges the messiness of the real world. They have come up with a “messiness rating” for their tasks, and the “mean messiness” of their tasks is 3.2/16.

By METR’s definitions, the kind of software engineering work that I’m mostly exposed to would score at least around 7-8, given that software engineering projects are path-dependent, dynamic and without clear counterfactuals. I have worked on problems that get to around 13/16 levels of messiness.

An increase in task messiness by 1 point reduces mean success rates by roughly 8.1%

Extrapolating from METR’s measured effect of messiness, GPT-5 would go from 70% to around 40% success rate for 2-hour tasks. This maps to my experienced reality.

I am not certain that pure intelligence can solve for messiness. Robustness to environmental chaos and the fuzzy nature of reality is fundamentally about managing context well. Until we find the magic sauce that solves this, it is clear that we need a workflow that can break down our problem into units of work, with verifiable checkpoints to manage the compounding of errors.

These verifiable checkpoints need to be legible to humans.

So, what is the “right sized” unit of work?

The right sized unit of work needs to be small and describe the desired outcome concisely.

The desired outcome on completion of a unit of work needs to be human-legible. I argue that it needs to provide legible business value. Ultimately, the users of software are going to be humans (or systems that model human constructs). Therefore, an elegant way to break down a project is to model it as small units of work that provide legible business value at each checkpoint. This will serve the purpose of respecting the context window of the LLM and help manage the propagation of errors.

Software engineers have already defined a unit of work that provides business value and serve as the placeholder for all the context and negotiation of scope—User Stories. I think they are a good starting point to help us break down a large problem into smaller problems that an LLM can one-shot, while providing a concrete result. They center user outcomes, which unlike “tasks”, are robust to the messy dynamic environment of software development. Srihari has elegantly written about the value of user stories in the software development process, and I recommend reading his post to better understand the properties that make them suitable for the messy work of building software.

Deliverable business value is also what all stakeholders can understand and work with. Software is not built in a vacuum by developers—it needs the coordination of teams, product owners, business people and users. The fact that AI agents work in their own context environment separate from the other stakeholders hurts effectiveness and transfer of its benefits. I think this is an important gap that needs to be bridged.

Most AI agents today have well-functioning “planning” modes. These are good at keeping the agent on rails, but they mostly provide technical value, and not necessarily a legible business outcome. I believe planning is complementary to our idea of breaking down a project into small units of business value. My proposed unit of work can be planned with existing planning tools. And I believe this is superior to planning over a large unit of work due to the context rot issues described earlier.

Of course, plain old User Stories as described in the Agile canon is not sufficient. It needs to be accompanied by “something more” that can nudge the agents to gather the right context that serves the business value outcome of the stories. What that “something more” could look like is something we hope to answer in the coming months.

The StoryMachine experiment

To test whether user stories with “something more” can indeed serve as optimal units of work that that have the properties I described above, we are running an experiment called StoryMachine. Currently StoryMachine does not do much—it reads your PRD and Tech Specs and produces story cards. It is still early days. But we will set up an evaluation system that will help us iterate to a unit of work description that helps us build useful software effortlessly. I hope to share updates on what we find in the coming months.

I want the craft of AI-assisted development to be less effortful and less like a slot-machine. And our best lever to get there is managing the unit of work.

In this post, I propose a structure for this operational exercise (complementing a business review) that lasts 2-4 hours, every month or quarter. I see quality as solving for the Pareto front with the tangible dimensions of reliability, performance, cost, delivery and security, and the more intangible dimensions of simplicity and social structures. For each dimension, go through the list of questions below and try to answer them together. The questions are:

• Intentionally informal to provoke honest discussion.
• Intuitive proxies for metrics. Numbers matter, and we should look at dashboards during discussions, but we need to go beyond the numbers and talk about the problems they represent.
• Intentionally instinctive, and emotional. They work by poking at symptoms, and leveraging the learned senses of trusted engineers, rather than breaking everything down to the raw facts. How they feel is an important signal, and it is a leader’s job to listen to them.
• Per system, where a system refers to a software service, or a group of them that you want to treat as a whole. I would suggest different review meets for systems, or teams you want to treat independently.
• Meant to be answered by people who actively work on the software every day
• About effectiveness at the Pareto front, not necessarily efficiency.
• Not novel. They’re what I hope most experienced developers would consider common sense.

Simplicity

This is the most important dimension to reflect on quality. But it is best treated as an intangible dimension, and is hard to measure objectively. A simple system is performant, does one thing well, is cheap, and reliable. And a good engineer knows this intuitively.

1. What would a new engineer experience?
   • Can we explain the system’s responsibility in plain english, within 5 minutes?
   • Can they form a correct mental model in under one hour using only docs and diagrams?
   • Do we find ourselves apologising when explaining how the system works?
   • How long does it take for a new engineer to be onboarded? Is the time to first PR acceptable?
2. Is the domain simple? And Is the domain modelling simple?
   • What are its core domain entities (aggregates), and would you say there are many?
   • Is the current design made of small and composable components?
   • Do simple modifications you expect in hours, take many days? And is that surprising?
   • Are there Architecture Decision Records (ADRs) that you keep referring to for key decisions?
   • Does a small feature need modifications in many places, or in multiple modules?
   • Do users mostly figure out how to use, or build-on the system through its readme, and interfaces, or do we have to often answer questions around the usage?
3. Is it simple to observe, debug, and diagnose?
   • Do you need to look at values from the database or cache to diagnose most issues?
   • Can you reproduce bugs simply by making the same request again?
   • If the system is stateful, how is state represented? Is it simple to reconstruct the state at a given point in time?
4. Is the domain inherently complex, or is the software incidentally complex?
   • Are we afraid of making changes in this system because we might break things we don’t understand?
   • How often do we need to confer with the one person in the team who remembers why things are the way they are?
   • If we squash a bug, do two others take its place?
   • Given a chance to rewrite, what exactly would we change and why?

Delivery

That is, delivery of business value, not code. It’s faster to deliver newer and smaller software. As it gets older and larger, how do we maintain the speed? It’s mostly about the flywheel. How quickly do we get feedback, and how much of it do we get before going to production?

1. Does it feel like we are moving slowly?
   • If so, what do you think makes us slow?
   • Are we happy with our deploy frequency, and rollback rate?
   • How much time does it take, on average, for a user-story (business value), to go from in-progress to done (in-production)? Has this been improving or deteriorating?
   • How confident are we about our estimates? Is our predictability improving? What’s our track record of sticking to estimated timelines?
2. How much confidence do we have in our tests, builds and deployments?
   • Do green builds on main automatically deploy to production?
   • Will we deploy during peak hours, or during Friday nights?
   • Can we refactor anything as long tests pass? What are we afraid of refactoring?
   • Do tests pass consistently on CI, and everyone’s machines?
   • What % of the codebase is abandoned, irrelevant, or barely used in production?
3. How good is our local setup?
   • Does everyone in the team run their service locally on their machine?
   • Do the service dependencies like databases, queues, caches etc also run locally? Are they in-memory so they’re faster?
   • How much time does it take to run tests locally?
   • How much time does it take to install dependencies and build a service locally?
   • Is the local setup a single command, and does it “just work”?
4. What’s the % split of feature / bug / chore (tech-task)?
   • Do we understand reasons for current split?
   • Does the split reflect our strategic focus?

Reliability

This dimension is fairly well quantified, usually. So it’s possible to be objective, and look at the numbers for SLOs, uptime, etc and make meaningful judgements from it. However, I’ll still write out questions for the subjective / qualitative aspects that should be evaluated in addition to the objective ones (that I am not writing about).

1. Is our incident management healthy?
   • Do we ignore alerts because they’re noisy?
   • How many alerts do we receive or every week? Does it feel like we’re constantly fire fighting?
   • How many of those alerts were for previously diagnosed issues?
   • How often are incidents detected by users rather than alerts?
   • Which stage needs most attention? Detection, Triage, Diagnosis, Recovery, or Review?
   • Does a newcomer know what to do when they receive an alert?
2. Are our reliability expectations reasonable, and clear?
   • What are the critical user journeys supported by this system?
   • Do we have product-level SLOs for user journeys end-to-end that are well monitored?
   • Are the reliability issues due to essential or incidental complexity?
   • How can we get away with lesser reliability requirements?
   • Do we need to be transactionally, or eventually consistent? If eventually consistent, in how much time? Can that be relaxed for better reliability?
   • Do we feel like we’re re-inventing the wheel? Would using existing, or managed solutions be more reliable?
3. How much control do we have on our reliability?
   • How much are our systems dependent on external systems? Are they optional or required?
   • Are we happy with our timeouts, fallbacks, and defaults?
4. Fault isolation, graceful degradation, and automatic recovery.
   • When one system fails, do others fail? And is that failure cascade reasonable as per their domain responsibilities?
   • At loads over capacity, does the system continue to work at capacity?
   • What parts of recovery are manual?

Performance

We just need to be performant enough to enable the next growth curve of the business. This section cares about that kind of performance. Effective, not efficient.

1. Are the performance expectations clear, and reasonable?
   • Does the workload or throughput actually match business metrics, or is it inflated incidentally? Reducing scale is the best way to deal with scale.
   • Are the performance expectations for the system clear? Are SLOs set beforehand? Are they reasonable? Begin with the end in mind.
   • Is performance of the journey user-bound, or system-bound? If user-think-time dominates, performance of the system isn’t the most important concern.
2. Is the team truly aware of the current state?
   • Do people know (by memory) the approximate p99s of critical operations?
   • Does the team know the normal throughput patterns (morning and evening peaks for example) to know that something is amiss by just looking at the shape of throughput through time?
   • What are the normal and peak resource utilisations of the system? Do we have leading indicators of trouble, or just lagging ones?
   • Which components will require redesign before horizontal scaling is viable?
3. Do we know what direction to improve performance in?
   • Which resource is the bottleneck? Compute, memory, I/O, network, or something else?
   • Is the synchronous request path tight, with everything non-critical kicked to async tasks?
   • What kind of throughput disrupts performance? Unpredictable spikes and bursts, or more predictable plateaus and sawtooths?
   • What accuracy, or consistency requirements constrain performance?
   • Has optimisation created rigidity? Are we boxed into micro-optimizations that block larger design moves?

Organisation

Software architecture is socio-technical. Organisation design, system design, and process design are deeply connected. How people are organised, and how they communicate, is reflected in the software architecture, and vice-versa. Yet, these aspects are often seen as independent, or unrelated.

In order to enable high ownership and agency, we should be willing to restructure teams or rescope responsibilities, just as much as we’re willing to change software architecture.

1. Is the team’s responsibility and ownership clear?
   • How dissonant is the product <> team <> system responsibility overlap?
   • If this system is responsible for a journey, does the team own the journey and funnel too?
   • Does the system fit into the boundaries of the team’s business domain?
2. Are we fighting against organisational structure with software structure?
   • What are the downstreams and upstreams of the system, and are they owned by teams that are close to this team in the org-chart?
   • What is the average number of people required to own a system, and what are the outliers (on both ends) to that?
   • Is the system responsible for many things?
   • Is the work-life balance for people different based on the systems they work on? Is that warranted?
3. Do our processes complement or contrast our architecture?
   • Do incentives reward short term velocity over long term manoeuvrability?
   • How many meetings could have been avoided if we had clear contracts?
   • Do users build on the system by composing and configuring, or do they need to collaborate and coordinate with us to get work done?
   • Which people have “meetings all day”, and is their time best spent that way? Does it compensating for poor system or org structure?
   • Which processes would we drop if we trusted everyone like we trust ourselves?
   • Is there a CONTRIBUTING.md defined?
   • What is the average number of PRs open at any point in time?

Cost, Security

These are important dimensions, and are very much part of the Pareto frontier. Unfortunately, I haven’t built a lot of intuitions with these dimensions. I understand them enough to work on related problems, but not enough to be writing a health check questionnaire on. I hope someone does this. It would be useful.

Write in if you would like to collaborate on these sections with me!

The questions will make conversations happen, but it is up to you to truly listen, understand, and make the most of it. I would suggest using the meeting to focus on the problems. You can prioritise and solve for them later on.

Also, I’m also assuming you’re doing the work of ensuring the work is focused on the right problem to begin with. Climbing a ladder fast isn’t useful if it’s against the wrong wall.

Thanks to Atharva for his thoughtful review of this post.