Scheduling tests
Raider can run your tests on a schedule, for example every weekday at 08:00, even when Raider Desktop is closed. A scheduled run is an ordinary run of your tests, and its results show on the Dashboard. You choose what runs, when, and whether you see the browser.
Scheduling needs a project on ruby_raider 4.2 or later. On an older project, the Schedules panel says "Scheduling needs ruby_raider 4.2 or later." To update, run bundle update ruby_raider in the project folder.
Choose what to run
A schedule runs one of these:
- All tests in the project.
- Tests tagged one tag, such as smoke. To tag a test, see The Tests tab.
- Ticked tests: the tests you tick in the Tests list.
To tick tests, press Select at the top of the Tests list on the Tests tab. Each test gets a checkbox, and clicking a test ticks it instead of opening it. Shift-click ticks every test in between. The bottom of the list shows how many are ticked, such as "2 ticked", with Run to run them now, Schedule… to schedule them and Clear to untick them all. Press Select again, or Esc, to go back to opening tests.
A schedule of ticked tests remembers their files. Renaming a test keeps it in the schedule; deleting its file does not (see When a scheduled run fails).
Schedule tests
On the Tests tab, open the Schedules panel and press Schedule…, or press Schedule… under the ticked tests. Then fill in:
- Name: lowercase letters, digits and _, up to 60. Raider suggests one, such as
all_testsortag_smoke. A name that is taken replaces that schedule. - Run: All tests, Tests tagged (pick one of the project's tags or type one), or Ticked tests with how many are ticked.
- Repeat: Every hour, Every day, Every weekday (Monday to Friday) or Every week. Every hour asks for the Minute past the hour; the others ask for a Time on the 24-hour clock, such as 08:00, and every week also for a Day. Times are your computer's local time.
- Show the browser: see Show the browser. Off at first.
- Retry failed tests: 0 to 3. A test that fails is run again, up to that many times, inside the same run, with a fresh browser each time.
- Notify me when a scheduled run fails: on at first; see When a scheduled run fails.
Press Schedule, and the schedule shows in the panel with when it runs, such as "Every weekday at 08:00", and its next run. With Notify me when a scheduled run fails on, the row also says "notify on failure".
Your computer's own scheduler runs the tests, so nothing needs to stay open and nothing asks for an admin password. Schedules belong to this computer: they are kept out of git, and a fresh copy of the project has none.
Show the browser
Scheduled runs hide the browser unless you turn on Show the browser for that schedule. It works whatever the project's Headless setting on the Files tab says, and it never changes that setting: the run itself tells the browser to show.
- It needs a project made with Ruby Raider 4.0 or later.
- It is not available on Linux, where a scheduled run has no screen to show a browser on, and the option is hidden there.
- On macOS a shown browser opens while you are logged in. On Windows the run happens in your session while you are logged on, but a shown browser has not been tested there yet.
Next run, last result and reports
Each schedule in the Schedules panel shows its name, what it runs (All tests, a tag, or how many tests), when it runs, "browser shown" when Show the browser is on, the next run, and the last result:
- Passed with how many tests passed.
- Failed with how many failed of how many ran.
- Could not start: the run could not start or did not finish normally, for example it was stopped or found no tests.
- Skipped, another run was going: see below.
A new schedule shows "Not run yet".
The buttons on each schedule:
- View report opens the report of that schedule's last run in your browser.
- Run now runs the schedule's tests now, as a normal run in the app, with its Show the browser setting. It does not count as a scheduled run.
- ⋯ › Edit opens the schedule with its settings filled in; press Save to replace it.
- ⋯ › Stop schedule removes the schedule.
A scheduled run is an ordinary run of your tests, so it replaces the last run on the Dashboard, as a run from the app does; see Dashboard. The report on the schedule stays until that schedule runs again.
Only one run goes at a time in a project. While a scheduled run is going, the app does not start another run and says "A scheduled run (nightly) is going; try again when it finishes." A scheduled run that is due while another run is going is skipped and leaves the Dashboard alone.
When a scheduled run fails
With Notify me when a scheduled run fails on, your computer shows a notification when a scheduled run fails or could not start, with Raider Desktop open or closed. Its title names the schedule, such as "Ruby Raider: nightly failed", and it says how many tests failed, such as "3 of 40 tests failed", or why the run could not start. Passed and skipped runs never notify.
With it off, you get a notification, "Scheduled run failed", only while Raider Desktop has the project open. Clicking it opens the Tests tab with the Schedules panel.
If a schedule did not run, or its row shows a warning:
- The computer was off, asleep or logged out. See On each computer for what happens to a missed run.
- Test file not found: a ticked test's file was deleted, or renamed or moved outside the Tests tab. Press Edit: the missing files are left out, so press Save to schedule the rest. If none of its files are left, Raider asks you to choose what the schedule runs.
- Schedule problem: the computer's scheduler lost the schedule (
missing from launchd, or Task Scheduler, systemd or cron), or still runs one that Raider has no record of (missing from schedules.yml). Press Re-schedule to save it again, or Stop schedule to remove it. - Schedule problem: Ruby not found: re-schedule: you upgraded or moved Ruby since you scheduled it. Press Re-schedule and save the schedule again.
- On macOS and Linux, the scheduler's own output is in
.raider/schedules/<name>/scheduler.log. Windows keeps no such log.
On each computer
Your computer's own scheduler runs the schedules:
- macOS (launchd): runs only while you are logged in. A run missed while the Mac slept happens once when it wakes.
- Windows (Task Scheduler): runs only while you are logged in. A missed run happens as soon as it can.
- Linux with systemd: a missed run happens once at your next boot or login. Schedules stop when you log out unless you run
loginctl enable-linger $USERonce. - Linux without systemd (cron): missed runs are skipped.
On Linux the failure notification often can't show under cron, or in a systemd session without your desktop's session bus.
A schedule runs your tests with the Ruby that was active when you made it. After you upgrade or move Ruby, schedule them again.
Each schedule is tied to the project's folder. Stop its schedules before you move or rename the folder, then schedule them again in the new place.
From the command line
Everything in the Schedules panel is also a Ruby Raider command. Run it in the project folder, or add --project PATH at the end. raider sc is short for raider schedule.
| Command | What it does |
|---|---|
raider schedule add NAME --every day --at 02:00 |
Schedules all tests, or replaces the schedule NAME. |
raider schedule add NAME --every day --at 02:00 --tag smoke |
Schedules the tests tagged smoke. |
raider schedule add NAME spec/a_spec.rb spec/b_spec.rb --every day --at 02:00 |
Schedules those test files. |
raider schedule list |
Lists the schedules with their next run, last result and state. |
raider schedule run NAME |
Runs the schedule now, the way the scheduler does. It counts as the schedule's last result and notifies like a scheduled run. |
raider schedule remove NAME |
Removes the schedule. |
add takes --every hour, day, weekday or week, --at 08:00 (or --at :15 for every hour), --on mon to sun for every week, --show-browser, --retries N (0 to 3) and --notify. For example:
raider schedule add nightly --every day --at 02:00 --notify
raider schedule add smoke --every hour --at :15 --tag smoke --retries 1
add checks the project first. A project locked to an older ruby_raider is refused with "Scheduling needs ruby_raider 4.2.0 or later in this project: bundle update ruby_raider".
raider schedule run ends with one of these exit codes:
- 0: the tests passed.
- 1: a test failed.
- 2: the run could not start, or the command was refused.
- 3: another run was going in the project, so this one was skipped.
The other commands end with 0, or with 2 and one line saying why. A mistyped command, such as a missing NAME, ends with 1 and the usage text. The Ruby Raider README has every option.
Upgrading from 4.1
Schedules made with Ruby Raider 4.1 can't run in 4.2. They are removed automatically the first time any raider schedule command runs in the project, or the next time one of them is due, and one line says which were removed. Add them again with raider schedule add.
Apart from the old list of schedules, nothing is deleted: the folders that 4.1 schedules used are left alone; the Ruby Raider README says which ones you can delete and when, and how to remove the schedules of a project you moved.