Skip to main content

​
Write targets that survive a redesign

In rough order of durability:
  1. A test id: [data-testid="checkout-submit"]. Nothing in a redesign moves it.
  2. An accessible name: the button’s visible label, the field’s <label>. Durable and readable.
  3. A row-scoped control: “Delete in the Blue Top row”. Survives reordering.
  4. An ordinal: “the first product”. Fine for listings, fragile if the order is not stable.
  5. Loose prose: resolved by the grounding model. It works, and the warning tells you what it picked, but it is the one most affected by a redesign.

​
Set the auth mode

The single most common cause of a test that passes alone and fails in a batch. Sign-in, sign-up and sign-out tests need fresh_logged_out; dashboard workflows need reuse_login_state. See Reusable flows.

​
Flake tracking

The project overview carries a flakiness section: pass rate per saved test over 7, 30 or 90 days, sorted worst-first by a composite score built from three signals. Fix the top of that list before adding tests.

​
Retry on fail

Use it for genuine flake such as an animation or a lazy-loaded element, and not as a band-aid for a real bug. A step that passed only on retry is marked, and those marks are a flake signal. press steps are never retried, because re-sending a key after the page has already acted on it produces a false pass.

​
Visual baselines

Set a baseline per saved test, step and viewport. Later runs diff against it with a pixel comparison and show baseline, actual and difference side by side. Best on pages whose layout is meant to be stable: a pricing table, a receipt, an empty state. Worst on anything with live data in it.

​
Performance budgets

Per saved test, cap LCP, FCP and TTFB with a tolerance percentage. Metrics are collected per page load and checked after the run; a breach marks the run’s performance status degraded and alerts like a failure. Set the caps from what the site does today, not from an aspiration. A budget that is always red is a budget nobody reads.

​
Adversarial fuzzing

For a saved test with type steps, fuzzing generates a payload per field across six categories (empty, boundary, XSS, SQL injection, unicode, special characters) and fires them as a batch of variants alongside the parent run. A confirmation shows the variant count and estimated time before anything runs, and the whole batch can be cancelled from the active-runs toast.

​
Role boundary probe

Give it a list of URLs and it walks each one as a privileged session and as a low-privilege session. A URL both can reach is reported as a possible IDOR. A redirect to a login page counts as blocked; only a 2xx counts as reachable.

​
Run hygiene

  • Give slow flows a real time limit instead of letting them hit the default.
  • Cancel runs you do not need. It frees a browser immediately and gives the allowance back.
  • Keep pull-request test selections small. A slow PR comment gets ignored.
  • Split long tests. Three signals beat one, and a 40-step plan is a ceiling you do not want to be near.