Skip to main content

​
Write targets that survive a redesign

In rough order of durability:
  1. A test id: [data-testid="checkout-submit"]. Nothing in a redesign moves it.
  2. An accessible name: the button’s visible label, the field’s <label>. Durable and readable.
  3. A row-scoped control: “Delete in the Blue Top row”. Survives reordering.
  4. An ordinal: “the first product”. Fine for listings, fragile if the order is not stable.
  5. Loose prose: resolved by the grounding model. It works, and the warning tells you what it picked, but it is the one most affected by a redesign.

​
Set the auth mode

The single most common cause of a test that passes alone and fails in a batch. Sign-in, sign-up and sign-out tests need “Fresh logged-out each leg”; dashboard workflows need “Reuse saved login state”. See Reusable flows.

​
Flake tracking

The project overview carries a flakiness section once there are 30 days of runs: pass rate per procedure over 7, 30 or 90 days, sorted worst-first by a composite score built from three signals. Runs that need review or were cancelled are left out of pass rates. Fix the top of that list before adding tests.

​
Retry on fail

Use it for genuine flake such as an animation or a lazy-loaded element, and not as a band-aid for a real bug. A step that passed only on retry is marked, and those marks are a flake signal. press steps are never retried, because re-sending a key after the page has already acted on it produces a false pass.

​
Visual baselines

Set a baseline per procedure, step and viewport from a replay’s run page. Every step of a replay records a lossless screenshot, and only that can become a baseline; a failure screenshot cannot. Later runs diff against it with a pixel comparison and show baseline, actual and difference side by side. Best on pages whose layout is meant to be stable: a pricing table, a receipt, an empty state. Worst on anything with live data in it.

​
Performance budgets

Per procedure, cap LCP, FCP and TTFB with a tolerance percentage, in project settings → Performance budgets. Metrics are collected per page load and checked after the run; a breach marks the run’s performance status degraded. It does not send an alert by itself. A run whose browser shared the test machine with other runs is not judged: its timings measure our load, not your site. Set the caps from what the site does today, not from an aspiration. A budget that is always red is a budget nobody reads.

​
Adversarial fuzzing

For a procedure with type steps, Fuzz on its row generates a payload per field across six categories (empty, boundary, XSS, SQL injection, unicode, special characters), up to 20 variants. It fires them as a batch with a control run of the unchanged procedure. Up to three browsers run at once. A confirmation shows the variant count and estimated time before anything runs, and the whole batch can be cancelled from the active-runs pill. Each variant checks only its own steps with the payload typed in. Nothing detects a server error, script execution or a reflected payload on its own, and the variant’s run page says so. Variants keep no recording.

​
Role boundary probe

In project settings → Role probe, give it a list of URLs and it walks each one as a privileged session and as a low-privilege session. A URL both can reach is reported as a possible IDOR. A redirect to a login page counts as blocked; only a 2xx counts as reachable.

​
Run hygiene

  • Give slow flows a real time limit instead of letting them hit the default.
  • Stop runs you do not need. It frees a browser immediately.
  • Keep pull-request test selections small. A slow PR comment gets ignored.
  • Split long tests. Three signals beat one, and a 40-step plan is a ceiling you do not want to be near.