Reliable automation needs more than generation.

A generated test can compile, execute, and pass while protecting almost nothing. It might click the wrong control, assert only that a page exists, or reproduce a defect as the expected result.

The difficult engineering problem is preserving intent through an automated feedback loop. Generation proposes an implementation. Validation establishes what is mechanically acceptable. Execution supplies observations. Evaluation asks whether those observations support the requirement.

I would design those as separate responsibilities with explicit handoffs. The examples below are illustrative, rather than a description of a particular production framework.

Begin with a testable contract

A manual scenario such as “verify that checkout works” leaves important decisions unspecified. What state does the user start in? Which payment path is in scope? What observable result establishes success? Which failures must remain visible?

Before generating code, convert the scenario into a reviewable contract:

This contract is the basis for reviewing both the initial candidate and every repair. If a requirement is ambiguous, surface the ambiguity before generating increasingly elaborate code around an assumption.

Keep fixture setup distinct from the behavior under test. An API can be a good way to prepare an account, yet an inappropriate shortcut for the UI interaction the test is supposed to exercise.

Separate the evidence each stage can provide

These stages should exchange structured artifacts: a contract version, candidate revision, diagnostics, execution evidence, and an explicit disposition. A free-form message saying “the test is fixed” is not enough to reconstruct why it was accepted.

Generate within a narrow repository interface

Give the generator the conventions it must follow: approved fixtures, authentication helpers, page abstractions, test-data factories, and assertion patterns. A smaller interface reduces the number of plausible but incompatible implementations.

Require a proposed file path and a mapping from each requirement to its assertion. Check that output against a schema before writing files. Then enforce repository boundaries: reject unexpected dependencies, changes outside allowed paths, disabled tests, and modifications to the application merely to make the test pass.

Treat generated code as untrusted input. Execute it in a disposable environment with restricted credentials, network access, and filesystem permissions. A test harness often has powerful access; that access should not become an accidental capability of arbitrary generated code.

Design assertions around observable behavior

Consider a save operation. Asserting that the Save button disappeared says little about persistence. Asserting only a success toast may still miss a backend failure if the UI updates optimistically.

The assertion should match the contract. If persistence matters, verify the saved value after a reload or fresh navigation. If a permission boundary matters, test the forbidden action and its observable rejection. Avoid reaching into implementation state when a user-visible result provides the required evidence.

For example, this simplified Playwright fragment checks more than completion of a click:

await page.getByLabel("Display name").fill("Ada");
await page.getByRole("button", { name: "Save", exact: true }).click();
await expect(page.getByRole("status")).toContainText("Saved");
await page.reload();
await expect(page.getByLabel("Display name")).toHaveValue("Ada");

The application still determines whether this is a sufficient contract. A profile page that reads stale local state after reload would require a stronger persistence observation. The code is evidence for a specific behavior, not a universal template.

Repair the cause without weakening the contract

A repair loop needs a classification of failure before it needs another prompt.

• Invalid candidate: syntax, type, or unsupported API errors. Return precise diagnostics to the generator.
• Environment failure: unavailable dependency, expired test credentials, or broken fixture setup. Investigate the harness rather than rewriting assertions.
• Interaction mismatch: a changed locator or timing assumption. Inspect the actual UI and verify that the intended action is still the same.
• Behavioral failure: the application did not satisfy the requirement. Preserve the evidence and investigate; do not automatically redefine success.
• Ambiguous requirement: the expected result cannot be established. Return for clarification instead of guessing.

Changing an assertion can sometimes correct a mistaken test. That is a semantic change, so it needs explicit review against the contract. It should not be an ordinary repair action justified solely by a passing rerun.

Bound the loop by attempts, elapsed time, and resource usage. Stop if the same diagnostic repeats or the candidate oscillates between failures. Retain each revision and the evidence that motivated it. An unresolved result is a valid terminal state.

Prevent the evaluator from inheriting the same blind spot

Asking the generator whether its own test is good creates a weak feedback loop. A separate evaluator can still share the same mistaken assumptions if it receives only the generated code and a success log.

Supply the evaluator with the original contract, the requirement-to-assertion mapping, and execution evidence. Use deterministic checks for rules that can be stated precisely. Use model-assisted review for questions that require interpretation, and calibrate those judgments against human-reviewed examples.

A useful challenge is counterfactual: which plausible defect would make this test fail? In the save example, deliberately prevent the persisted update in a controlled test environment and confirm that the persistence assertion detects it. Mutation testing and seeded defects can provide evidence of sensitivity, although neither proves complete coverage. Keep such fault injection isolated from production.

Make reproducibility a property of the harness

Use isolated test state, deterministic fixtures, and explicit cleanup. Prefer locators tied to user-facing semantics or stable contracts over incidental DOM structure. Use assertions that wait for the expected condition instead of arbitrary delays. Playwright’s guidance on test isolation, resilient locators, and retrying assertions is a useful foundation for these mechanics.

A rerun that passes after a failure should remain visible as a flaky outcome. Otherwise, retries convert instability into an apparently healthy dashboard. Capture enough diagnostic evidence to distinguish application defects from fixture collisions and environmental failures, with sensitive data redacted.

Track flakiness by test and environment. Quarantine can protect delivery while an issue is investigated, but it needs an owner and an exit condition. Permanently ignored tests create the appearance of coverage without its protection.

Measure the useful artifact, not the generated volume

The number of tests produced rewards output, including redundant or fragile output. More useful measures include reviewed contract coverage, defect detection on controlled cases, repeated-run stability, repair attempts, review effort, and maintenance burden after application changes.

Compare the workflow against a baseline on the same scenarios. Include straightforward cases, ambiguous requirements, permission boundaries, and deliberately broken behavior. Keep a held-out set to detect whether improvements generalize beyond the examples used to tune the generator.

Introduce automation in stages: generate reviewable proposals first, allow bounded mechanical repairs next, and expand autonomy only when the evidence supports it. Keep promotion into the trusted suite explicit. The architecture should make it easy to explain why a test belongs there and what would cause that trust to be reconsidered.

Further reading

Playwright’s best practices and test isolation documentation explain the underlying browser-testing mechanics. The contract, repair boundaries, and promotion process above are recommendations for building an automation workflow around those mechanics.