What an Early AI Product Taught Me About Breaking Up Model Work

When a product idea is ambitious, it is tempting to ask the model to do the whole job in one pass. For an early AI-enabled decarbonization-planning feature, that was a reasonable starting point. We were learning what the experience should be and what the model could contribute. A broad request helped us explore the shape of an answer quickly.

It also taught me that a compelling first answer and a dependable product workflow are different things.

I do not view that early design as a mistake. It gave us something concrete to evaluate. But once a feature moves toward production, the questions change. Which parts of the output can be checked? Which steps can be retried? What should the user be allowed to correct? Where does the application need durable state? Those questions made me more interested in breaking model work into purposeful stages.

One call can conceal several different jobs

A single prompt might ask a model to interpret input, infer missing context, choose an approach, produce structured records, and write an explanation. The response may look coherent while the intermediate decisions remain hard to inspect. If something is wrong, you may not know which part to revise.

The remedy is not “use more calls” as a rule. More calls add latency, cost, and new places for failure. The useful move is to identify tasks with different inputs, checks, or retry behavior, then give them clear boundaries.

For example, understanding an incoming document is not the same task as matching a derived record to a known reference set. Generating a candidate explanation is not the same task as accepting a user’s correction and updating stored state. If the application treats those as separate steps, it has more opportunities to validate, clarify, or recover without repeating everything.

That separation can also make the product easier to explain. The user can see what the system understood, correct a specific interpretation, and continue. A model becomes part of a workflow rather than an opaque answer machine.

Design for the handoffs

Later, while working on AI-assisted ingestion of customer-provided spreadsheets, I helped build a more explicit flow. The system needed to take inconsistent source material and turn it into structured records. It used a persisted state machine to track progress, async workers for longer-running work, OpenSearch to match records against relevant reference factors, and a user-facing AI chat for refinement.

The point of that architecture was not to make an impressive diagram. It was to give each stage a job and make the transitions durable. If a step was delayed or failed, the product could understand where it was. If a user needed to clarify data, the conversation could connect to the workflow state rather than floating beside it. If the input source changed later, downstream record generation could remain useful.

That last point mattered when we extended the system beyond spreadsheets. A third-party utility-data integration could provide structured bill data, so it did not need the same AI parsing step. We normalized its events into the same downstream record-generation shape and continued through the matching pipeline. The original architecture was not a universal abstraction we predicted in advance; it was flexible enough to admit a new source without pretending that every source needed identical processing.

Parallel work needs a reason

Some model tasks can run in parallel. Others depend on the output of a preceding step. I try to distinguish those cases instead of choosing an orchestration pattern because it sounds sophisticated.

Parallel calls are useful when independent inputs can be examined separately and later combined. Chaining is useful when the second step genuinely needs the first step’s result. Smaller targeted calls are useful when they create a checkable intermediate result. None of these choices is free. They affect cost, latency, consistency, and the number of states the application must handle.

My rule of thumb is to make a stage explicit when doing so gives the product a meaningful control point: a validation, a retry, a user decision, or a boundary between uncertain model interpretation and deterministic application behavior. If splitting the work only adds coordination without improving those things, the extra architecture may not be worth it.

Keep the human in the loop where it counts

“Human in the loop” can be a vague slogan. In a product, I want to know what the person can actually inspect and change. Can they see the system’s interpretation before it becomes a record? Can they correct the data that matters? Can they recover from an error without restarting an entire process? Does the interface tell them whether work is pending, complete, or in need of attention?

Those questions influence backend design as much as UI design. The model may propose; the application has to preserve the state of the work, enforce constraints, and offer a comprehensible path forward. Trust comes less from claiming that the model is smart and more from making its contribution legible and correctable.

The lesson I took from that early feature was not that big prompts are bad. They are often a good way to explore a product idea. The lesson was to revisit the shape of the work as we learn more. As the product matures, I want model calls that have clear responsibilities, handoffs the system can observe, and user interactions that make the result useful even when the first answer is imperfect.