qa.pro from CI.

Create an API key in Settings → API keys (Team plan and up) and store it as a secret. The workflow below tests every successful preview deployment. qa.pro takes preview URLs under a domain you verified (a verified apex covers preview-123.shop.example); any other host waits until its domain is verified, and the run's waiting field says so.

# .github/workflows/qapro.yml
name: qa.pro
on: [deployment_status]
jobs:
  check:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Test the preview URL
        env:
          QAPRO_URL: https://qa.pro
          QAPRO_KEY: ${{ secrets.QAPRO_KEY }}
          TARGET: ${{ github.event.deployment_status.target_url }}
        run: |
          set -u
          run=$(curl -sS --fail-with-body -X POST "$QAPRO_URL/api/v1/runs" \
                -H "Authorization: Bearer $QAPRO_KEY" \
                -H "Content-Type: application/json" \
                -d "{\"urls\": [\"$TARGET\"]}") \
            || { echo "::error::could not start the run: $run"; exit 1; }
          # a run may be split in two when some addresses wait for their domain's proof:
          # poll every id the API returned
          ids=$(echo "$run" | python3 -c '
          import json, sys
          d = json.load(sys.stdin)
          print(" ".join(str(i) for i in [d["id"]] + d.get("also_queued", [])))')
          fail=0
          for id in $ids; do
            echo "run $QAPRO_URL/runs/$id"
            status=queued; st=""
            for i in $(seq 1 90); do
              sleep 20
              # a transient 502 must not fail the job
              st=$(curl -sS --fail-with-body "$QAPRO_URL/api/v1/runs/$id" \
                   -H "Authorization: Bearer $QAPRO_KEY") \
                || { echo "poll: $st"; sleep 20; continue; }
              status=$(echo "$st" |
                python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')
              if [ "$status" = done ] || [ "$status" = failed ] ||
                 [ "$status" = cancelled ]; then break; fi
            done
            # nothing finished it in 30 minutes: say why
            if [ "$status" = queued ] || [ "$status" = running ]; then
              waiting=$(echo "$st" | python3 -c '
          import json, sys
          print(json.load(sys.stdin).get("waiting") or "no browser was free")')
              echo "::error::run $id: $waiting"; fail=1; continue
            fi
            echo "$st" | python3 -c '
          import json, sys
          d = json.load(sys.stdin); s = d["summary"]; r = s["failing_sites"]
          print("run %d: %d passed · %d failed · %d not run; %d of %d browsers reported"
                % (d["id"], s["pass"], s.get("fail", 0), s.get("not_checked", 0),
                   s["results"], s["expected"]))
          # needs a look: the check stopped and nothing on the page says whose fault;
          # watch its film on the report
          if s.get("look"):
              print("run %d: %d %s a look; that is not a failure: watch what happened on %s"
                    % (d["id"], s["look"], "needs" if s["look"] == 1 else "need",
                       d["url"]))
          # a look is not a failure
          ok = (d["status"] == "done" and r == 0 and s["ended"] + s.get("look", 0) > 0
                and s["results"] >= s["expected"])
          sys.exit(0 if ok else 1)' || fail=1
          done
          exit $fail

A check that needs a look (it stopped before it could tell, and nothing on the page says whose fault) is not a failure: the job prints it on its own line with the report's link, and the exit code does not change. The run's summary counts it as look, beside pass, fail and not_checked.

Ad-hoc URLs use the team's default checks; pass "checks": ["add_to_cart", "search_product"] to choose.

Raw file: ci.md.