Verdicts
A finished run carries one of three verdicts. They appear on every run badge, in the report’s Result line, in alerts and PR comments, and asoverallStatus in the API.
A run reads Needs review when no step failed and any of these holds:
- a check you asked for was never performed (a coverage gap no step’s reading settled);
- a check passed, but on something that was already true before the action it was about, so it proved nothing;
- the report found that the result does not match the request, or only partly does;
- Testorim refused one of its own planned steps as written, for example an absence check it could not certify on that page. The site did nothing wrong and neither did you.
Run states
The lifecycle state, separate from the verdict.
A cancelled run has no verdict. It shows as Cancelled everywhere, is
never counted as a failure, and its report says why it stopped and how many
steps ran: “The run was stopped before it finished. 3 of 8 steps ran (3
passed, 0 failed); the rest did not run.” A run stopped before its first
step says so. A stopped run keeps the steps that ran, but no
recording, trace or final screenshot, because stopping closes the browser at
once.
Step states
A step can also pass with a warning: the action worked, but something is
worth knowing. The app moved the page mid-step. An Enter key was ignored and a
click stood in for it. A click changed nothing on the page. A typed value was
no longer in any visible field by the end of the run. A resolution had to
guess and is telling you what it picked.
Warnings do not change the verdict. They are the difference between a green
test you trust and one you should look at. Alerts can also fire on them (see
Integrations).
How the verdict is decided
- If any executed step failed, the run failed.
- Otherwise, if a coverage gap is open, a check proved nothing, or Testorim refused its own planned step, the run needs review.
- Otherwise, if the report judged the result failed or partial, the run needs review.
- Otherwise it passed.
Failure causes
Every failed step is labelled with whose problem it is. The label is set where the failure was raised, not guessed from the wording afterwards.
A text or value mismatch on an element that was found is marked
app, but
unconfirmed: the message ends by saying it cannot tell whether the page or
the expected text is wrong. Check which before filing it.
A coverage gap step also says why no step covered the check: unsupported
(nothing in a browser can observe it), capacity (the plan reached its
40-step ceiling), or omission (the plan had room and missed it).
An internal failure is deliberately excluded from three things: the silent
retry budget, another lap of a long-polling assertion, and a “rejected as
expected” pass on a negative test. Only the application refusing an action can
satisfy a negative test.