AI cost is an architecture problem.

An AI feature becomes expensive in places that a model pricing table cannot explain: repeated retrieval, growing context, tool loops, failed attempts, and the human work needed to correct a plausible answer.

I treat cost as a property of the workflow. The useful question is how much effort the system consumes to produce an acceptable outcome, under the latency and reliability constraints of the product.

The examples below are illustrative design scenarios, not descriptions of an employer’s systems or measured results.

Define the outcome before optimizing the call

Consider a document review workflow. It retrieves evidence, identifies relevant clauses, interprets ambiguity, and produces a recommendation. A cheaper recommendation that omits critical evidence is a regression. A more expensive first attempt may be preferable if it avoids repeated review.

I would track two related measures:

• Automated cost per accepted outcome: model, retrieval, tool, and infrastructure spend across all attempts, divided by outcomes that meet an explicit acceptance rule.
• Operational cost per accepted outcome: that spend plus the cost of human review and correction, where those costs can be measured responsibly.

Report the acceptance rate beside either number. Otherwise, a workflow could appear efficient by rejecting difficult work or quietly narrowing what counts as success. Keep the task mix stable when comparing versions, and inspect expensive or unsuccessful cases separately from the average.

An accounting helper should include failed attempts in spend, while counting only accepted outcomes in the denominator. Keep the acceptance decision external to the helper:

type Attempt = {
  cost: number; // Same currency, including all metered workflow components.
  outcomeId: string;
};
 
function costPerAcceptedOutcome(
  attempts: Attempt[],
  acceptedOutcomeIds: Set<string>,
): number | null {
  if (acceptedOutcomeIds.size === 0) return null;
  const totalSpend = attempts.reduce((sum, attempt) => sum + attempt.cost, 0);
  return totalSpend / acceptedOutcomeIds.size;
}

Both inputs must cover the same measurement window and workload. Returning null when nothing is accepted prevents an undefined ratio from appearing as zero cost. Production accounting also needs a consistent policy for outcomes completed after the reporting window.

Make the decision boundary explicit

Separate evidence acquisition, interpretation, policy checks, and execution. These responsibilities change for different reasons and need different evidence of correctness.

A model’s ability to call a tool does not make it the right place to enforce authorization or validate a transaction. Keep those checks at the service boundary, where every caller is subject to them.

For the document example, exact metadata extraction might use structured fields. Interpretation of an unusual clause may require a model. The final action still passes through explicit policy checks. This division also makes it possible to change the interpretation layer without rewriting the execution path.

Control the amount of work, not only its unit price

A workflow with a predictable sequence is easier to budget than one that decides its next step indefinitely. Start with a fixed path when the task structure is known. Introduce dynamic planning where its flexibility solves a demonstrated limitation.

Every additional reasoning step should have a purpose. Before allowing another retrieval, ask what missing evidence it can obtain. Before allowing another model call, ask what decision it can change.

Use separate limits for elapsed time, model calls, tool calls, retrieved bytes, and estimated spend. A token limit alone does not constrain a slow external dependency. A tool-call limit alone does not constrain a single very large response.

Reserve capacity for producing a useful terminal response. A system that exhausts its budget halfway through should return a clear incomplete state with the evidence it has, rather than presenting partial analysis as a final answer. For work that cannot safely degrade, route to review or stop explicitly.

Treat context as a designed interface

Passing the entire conversation and every retrieved document to each step is simple, but it couples cost to history length and makes relevance harder to inspect.

Prefer a compact, typed handoff: the question being answered, the evidence identifiers, relevant excerpts, unresolved issues, and the permitted next action. Preserve source references so that later steps can distinguish retrieved facts from earlier model interpretations.

Compression has a failure mode: it can remove the exception that determines the answer. Evaluate summaries against downstream decisions, not only readability. When exact wording matters, retain the original passage alongside the summary.

Caching needs the same discipline. A cache key should reflect the inputs that make an answer valid, including tenant or access scope, source version, and relevant policy or prompt version. A cache hit that serves stale or unauthorized evidence is not a successful optimization. Define invalidation before relying on reuse.

Route by evidence, then validate the routing policy

A smaller model or deterministic path can handle a subset of tasks well. The hard part is identifying that subset reliably.

Do not use a model’s stated confidence as the only escalation signal. Useful signals may include missing evidence, contradictory sources, failed schema checks, unfamiliar input patterns, and disagreement with explicit constraints. Each signal needs validation against representative cases.

A routing policy should be evaluated as part of the whole system. If the first stage misses difficult cases, the stronger fallback never sees them. Measure incorrect non-escalations as carefully as successful escalations. Retain an abstain path for cases where the system lacks enough evidence to decide.

Compare quality on the same workload

Build a versioned evaluation set that includes routine inputs, ambiguous inputs, incomplete evidence, long documents, and cases where no answer is justified. Keep a held-out portion so that repeated prompt tuning does not become tuning to the entire benchmark.

For extraction or classification, precision and recall can expose different mistakes. For recommendations, use a rubric that checks evidence support, required considerations, and appropriate uncertainty. Human labels need documented criteria and a way to resolve disagreement. A model-based grader can help triage cases, but should be calibrated against reviewed examples rather than treated as an independent source of truth.

Compare cost, latency, acceptance, and failure categories together. A lower average can conceal a degraded tail or a particularly important class of errors. Segment by task complexity and inspect the cases whose outcomes changed.

Make retries an explicit design choice

Retrying a transient timeout may help. Repeating a request with the same invalid input usually adds cost without changing the cause. Distinguish dependency failures, invalid requests, insufficient evidence, and poor generated output before choosing a retry policy.

State-changing tools require additional care. If a response is lost after an action succeeds, repeating the call can duplicate the action. Use service-side idempotency where appropriate and reconcile uncertain outcomes before attempting another write.

Instrument attempts under a shared workflow identifier. Record the selected path, model and configuration versions, usage, tool timings, termination reason, and evaluation outcome. Keep sensitive payloads out of routine telemetry; retain only the evidence needed for debugging under appropriate access and retention rules.

Roll out the architecture as a hypothesis

Change one major variable at a time when possible: routing, context assembly, model choice, or retry policy. Offline evaluation helps reject obvious regressions; a limited rollout reveals production behavior that the benchmark did not capture.

Where practical, compare a candidate in shadow mode before allowing it to affect user-visible decisions. Shadow execution has its own cost and data-handling requirements, so scope it deliberately. Define rollback conditions in advance around quality, latency, and failure rates as well as spend.

The goal is a workflow whose cost can be explained by the work it performs. Once that relationship is visible, model selection becomes one engineering decision among several, supported by evidence instead of a price comparison alone.

Further reading

Anthropic’s Building effective agents offers a useful distinction between predefined workflows and dynamically directed agents. The budgeting, acceptance measures, and rollout approach above are design recommendations for the illustrative workflow.