There are two different subjects that get called “AI testing,” and confusing them wastes a lot of time. One is testing an AI product — how to assert on non-deterministic model output, covered in end-to-end testing for AI products . The other, this guide, is using AI to test an ordinary application: a web app, a GUI, a form-heavy internal tool with no model in it at all.

The second is where most teams stand to gain, because UI test suites fail for boringly mechanical reasons — a selector changed, a dialog appeared, a test was never written — and those are tractable problems.

What AI actually changes

Three capabilities are real and shipping. It is worth being precise about each, because the marketing around them is not.

1. Test generation from an application, not from a spec

The useful form is not “describe your test in English and get code.” It is an agent that opens the running application, explores it, and writes tests against what it observes, verifying each selector as it goes.

Playwright ships this in the framework itself. It provides three agents out of the box — planner, generator, and healer [1]:

AgentWhat it does
PlannerExplores the app from a seed test and produces a Markdown test plan of flows and scenarios
GeneratorTurns that plan into executable Playwright tests, verifying selectors and assertions against the live page
HealerReplays failing steps, inspects the UI, and applies corrections — or reports that the feature is genuinely broken

They are set up with npx playwright init-agents, targeting whichever AI loop you use, and Playwright’s documentation notes the definitions “should be regenerated whenever Playwright is updated to pick up new tools and instructions” [1].

The important design decision here is the seed test. The planner needs an environment that is already logged in and seeded with data. Most of the value you get from generated tests is decided by how good that fixture is, not by the model.

2. Browser agents driven by the accessibility tree

The mechanism underneath is worth understanding, because it explains both the capability and its limits. Agents driving a browser through the Model Context Protocol — microsoft/playwright-mcp being the reference implementation [2] — operate primarily on the page’s accessibility tree rather than on screenshots.

This matters. The accessibility tree is a structured, textual representation of the page: roles, names, states. Acting on it is cheaper, faster, and far more reliable than asking a vision model to interpret a picture of a page. It also means the agent addresses elements the way a screen reader does — by role and accessible name — which is exactly how a durable Playwright locator is written anyway.

A practical side effect: applications with poor accessibility are hard for these agents to test. If your buttons are unlabelled <div>s, the agent has nothing to grip. Improving accessibility improves testability, and the reverse.

3. Self-healing locators

When a UI change breaks a locator, the healer inspects the current DOM and proposes a replacement. This addresses the single largest maintenance cost in UI suites.

It also carries the sharpest risk in this guide, and it deserves stating plainly: a test that heals itself can heal itself into passing when the application is actually broken. If a “Submit” button is renamed to “Send,” healing is correct. If it disappeared because of a regression and the agent binds to a different button that happens to be nearby, the suite goes green over a real defect.

The mitigation is process, not technology:

  • Treat every heal as a proposed diff that a human reviews, never an automatic in-place rewrite on CI.
  • Run healing in a separate maintenance job, not in the pipeline that gates deployment.
  • Require the healed test to fail against the pre-fix build. A heal that passes on both the broken and the fixed version is testing nothing.

Where this earns its place

  • Legacy applications with no test coverage. Exploratory generation against a running system is dramatically faster than writing a suite from scratch, and the marginal value of the first hundred smoke tests is enormous.
  • Regression suites over stable, form-heavy UIs. High test count, mechanical maintenance, low ambiguity.
  • Locator churn during redesigns. Healing genuinely helps when markup changes but behaviour does not.
  • Coverage discovery. Planner agents are good at noticing flows nobody wrote a test for.

Where it does not

  • As a correctness oracle. An agent exploring your app learns what it does, not what it should do. Generated assertions encode current behaviour, including current bugs. Requirements still come from humans.
  • Anywhere the assertion is subjective. Visual polish, brand compliance, “does this feel right” — vision models are inconsistent judges and expensive at scale.
  • In the deployment gate, if generation runs at test time. Calling a model during a CI run makes the run non-deterministic, slow, and billable. See below.
  • Highly dynamic canvas or WebGL interfaces. No meaningful accessibility tree, so the agent is reduced to vision and becomes unreliable.

Keeping the suite deterministic

The critical architectural rule: use AI at authoring time, not at execution time.

The output of an agent should be committed Playwright code — ordinary, readable, reviewable tests in your repository. CI then runs plain Playwright, with no model call, no network dependency on a provider, no token cost, and no run-to-run variance. This preserves everything a test suite is for.

The anti-pattern is a suite that calls a model on every run to decide what to click. It is slow, costs money per execution, fails when the provider has an incident, and — worst — is not reproducible, so a failure cannot be bisected. If you cannot re-run a red build and get the same red, you do not have a test suite. The general problem is covered in testing non-deterministic systems .

A workable pipeline:

  1. Author — agents explore, plan, and generate tests against a seeded environment.
  2. Review — a human reads the generated tests like any other pull request. Delete the ones asserting nothing.
  3. Execute — CI runs deterministic Playwright. No models involved.
  4. Maintain — a scheduled healing job proposes fixes for failures as diffs, for human review.

What does not change

Everything that made UI suites work before still applies, and AI does not substitute for any of it:

  • Stable, semantic locators — role and accessible name over CSS paths. Agents produce better tests against accessible markup, so this compounds.
  • Test isolation and data seeding. Tests that depend on each other’s state fail unpredictably regardless of who wrote them.
  • The test pyramid. Generated UI tests are cheap to create, which makes it tempting to build a suite of hundreds of slow end-to-end tests. Resist it: the economics of running them are unchanged. See test pyramid for AI .
  • Deleting tests. A generated suite accumulates redundant coverage quickly. Pruning is maintenance work that agents will not do for you.

The honest summary: AI has substantially reduced the cost of writing and repairing UI tests, and changed nothing about the cost of running them or the judgement required to know what is worth asserting.

Further reading

Sources

  1. Playwright. “Test Agents” (planner, generator, healer). https://playwright.dev/docs/test-agents
  2. Microsoft. “Playwright MCP — Playwright Model Context Protocol server.” https://github.com/microsoft/playwright-mcp
  3. Playwright. “Locators — locating elements by role and accessible name.” https://playwright.dev/docs/locators
  4. W3C. “Core Accessibility API Mappings — the accessibility tree.” https://www.w3.org/TR/core-aam-1.2/
  5. Model Context Protocol. “Specification.” https://modelcontextprotocol.io/