Python package and pytest

The testorim package runs Testorim tests from Python code and adds a pytest plugin, so a real-browser check can sit next to your other tests and fail the build the same way. It has no dependencies and needs Python 3.9 or newer.

pip install testorim

Create an API key in Testorim under Settings → API keys and put it in TESTORIM_API_KEY. See Authentication for what a key may change.

From code

from testorim import Testorim

client = Testorim()  # reads TESTORIM_API_KEY

run = client.run_test(
    "Shop",  # a project's name, address or id
    "Add the Blue Top to the cart, open the cart and check that it shows 1 item.",
)
print(run.summary())
run.assert_passed()  # raises TestorimRunFailed unless the verdict is passed

run_test starts the run and waits for the verdict (up to 10 minutes by default). Other calls:

client.projects()                                    # the workspace's projects
client.tests("Shop")                                 # a project's saved tests
client.run_saved("Checkout", project="Shop")         # replay a saved test by name or id
client.run_test("Shop", "...", base_url="https://pr-42.preview.example.com")
client.run_test("Shop", "Sign in with a wrong password and check that an error is shown.", expect_failure=True)
run = client.run_test("Shop", "...", wait=False)     # start without waiting
client.wait_for(run.id)                              # then wait
client.cancel(run.id)

A run has a verdict (passed, failed or needs_review; None while it runs or when cancelled), the step counts, the written report, failed_steps (each with the reason and blame: app, test, unsupported or internal), the run’s url, and links to the video, Playwright trace and final screenshot, which expire an hour after they were read. summary() leaves those links out because it often lands in CI logs.

In pytest

Installing the package adds a testorim fixture and a testorim marker:

import pytest

@pytest.mark.testorim
def test_checkout(testorim):
    testorim.run_saved("Checkout", project="Shop").assert_passed()

Run only these tests with pytest -m testorim, or leave them out with pytest -m "not testorim".
--testorim-base-url URL points every run made through the fixture at another address, such as a pull request’s preview.
Without an API key the test fails with a message saying how to create one; set TESTORIM_SKIP_WITHOUT_KEY=1 to skip instead (useful for pull requests from forks, which get no secrets).

A failed run fails the test with its summary:

TestorimRunFailed: Verdict: FAILED (1 passed, 1 failed, 1 skipped, 41s)
Run: https://app.testorim.com/runs/...
Failed step 2: click "Create account": The page showed Something went wrong [the app did not behave as described]

In GitHub Actions

name: Browser tests
on: pull_request

jobs:
  testorim:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install testorim pytest
      - name: Run the Testorim tests against the preview
        env:
          TESTORIM_API_KEY: ${{ secrets.TESTORIM_API_KEY }}
          TESTORIM_SKIP_WITHOUT_KEY: "1"
        run: pytest -m testorim --testorim-base-url "https://pr-${{ github.event.number }}.preview.example.com"

Put your preview’s real address in --testorim-base-url, and make sure the preview is deployed before the job runs. A saved test that types a saved password replays only on its project’s own host or one of the project’s environments.

Errors

Every exception derives from TestorimError. The ones you will meet:

Exception	When
`AuthenticationError`	401: the key is missing, wrong or revoked.
`RefusedError`	402, 429 or 503: Testorim will not start the run now. `code` and `retryable` say why and whether waiting helps; see API errors.
`NotFoundError`	No project or saved test matches, or it is in another workspace.
`TestorimUnreachable`	The API could not be reached.
`TestorimRunFailed`	`assert_passed()` on a run that did not pass.

The client never retries a trigger by itself, because a retried trigger is a second run. If your workspace lives on another host, set TESTORIM_API_URL. The full reference is on the package’s PyPI page.

Limits

The site must be reachable from the internet: Testorim’s browsers refuse localhost and private addresses.
Every run counts against your plan, whichever tool started it.
For coding agents, use the MCP server instead; for JavaScript or TypeScript test suites, the library; for shell scripts and other CI, the CLI.

​From code

​In pytest

​In GitHub Actions

​Errors

​Limits

From code

In pytest

In GitHub Actions

Errors

Limits