Running tests and results
Running a test opens a browser, replays every step and tells you whether the test passed. You can run one test, the tests with a tag, or the whole project, and read the results on the Dashboard.
Run one test
Open the test on the Tests tab and press Run, or ⌘R (Ctrl+R). The result shows below the steps. Last run shows the full output of the last run of that test.
A test you have not saved yet cannot run: the app asks you to save it first. Press Stop & save, then Run.
A project on a ruby_raider older than 4.0 cannot run until you update it; see Projects.
If the test fails, the app shows the step that failed and offers fixes; see Fixing a failed test.
Run tagged tests
On the Tests tab, pick a tag in the Tag list, open More and choose Run tagged tests. To tag a test, see The Tests tab.
Run the whole project
On the Files tab, click the run button on the project's row at the top of the file tree. It runs every test the Run setting on the toolbar selects:
- All tests runs everything.
- Smoke runs only the tests tagged smoke.
- Regression runs only the tests tagged regression.
- Custom tag… runs only the tests with the tag you type.
Next to it:
- Re-run failed runs only the tests that failed last time. These re-runs go one test at a time. The button is unavailable when nothing failed.
- Run in parallel runs several tests at once, which is faster on big suites. Set how many at the same time with Workers, under More options.
The run panel
While a run is going, the run panel lists each test as Queued, Pending, Running, Passed, Failed, Quarantined or Skipped, with a count such as "3 of 10". Before the first test, it may say "Installing gems…" or "Starting Ruby…".
- The filters All, Failed, Running and Queued narrow the list.
- Cancel stops the run. Run again runs the same tests again. Re-run failed runs only the ones that failed.
- Technical details shows the raw output.
If you leave the page, an indicator keeps showing the progress; click Show the run to come back. When the run finishes, your computer shows a notification with how many tests passed and failed.
Only one run can go at a time. If a run is already going, in this project or another, the app tells you.
To run tests on a schedule, even when the app is closed, see Scheduling tests.
Dashboard
The Dashboard tab shows the results of the last run.
- Overall summary: the number of tests that passed, failed, were skipped and were quarantined, with a chart.
- Result cards: one per test. Click a failed card to see where it failed, the error, and a screenshot from the moment it failed (View screenshot).
- Open report opens the full HTML report in your browser.
- View trace shows every step of a failed test in plain words, for example 'Clicked the "Login" button', with how it found the element, how long it waited, a screenshot, the page as it was, and on Chrome, Edge and Firefox the network requests and console messages. Back to results returns to the list. You can also open a saved trace file (
raider-results/attachments/<n>-trace.zip) with File › Open trace….
Quarantined tests failed, but were quarantined in Raider, so they did not fail the build. To treat them like any other failure, turn on Count quarantined failures as failures under More options on the Files tab.
Failure video
Raider can record a video of a failed test. To watch it, click a failed result card on the Dashboard, then expand Failure video.
Recording a video needs ffmpeg. Until a release ships with its own ffmpeg, set the RAIDER_FFMPEG_PATH environment variable to an ffmpeg program built with libvpx before you start Raider Desktop.
Accessibility results
In a project with Accessibility testing (axe), a failed accessibility check lists each problem it found on the page, with how serious it is and a suggestion for fixing it.
Screenshot comparison
When a test compares the page with a saved picture, the result shows the Expected, Actual and Diff pictures and how many pixels differ. If the change is intended, click Accept as baseline to make the new picture the one to compare with.