Testing should not only be used to prove that a feature works, but also to find out as soon as possible that a change has caused a feature to stop working.
The advantages of testing more often is that you will typically have more time before the next big milestone to find and fix the problem that causes the tests to fail, and because the time between test executions is short, there is only a limited number of changes that could have introduced the problem.
Currently, I am also working on an embedded product and we have organized our testing activities like this:
- Development is done according to the Git Flow model: Changes are made on feature branches and are only integrated into the main software after a (successful) review. In parallel with the review, unittests (on a build machine) and a limited integration test (on the real target hardware) are executed. This limited integration test takes about 20 minutes to execute.
- Every night, if there are changes to the main software branch, a regression run of nearly all integration tests is executed. This test run takes about 5 to 6 hours to execute on the target hardware and excludes only the test cases that take an excessive amount of time.
- Every weekend, the tests that require a lot of time (typically between 10 and 24 hours per testcase) are executed.
A story/feature is not considered to be completed until the code has been integrated into the main software branch and the nightly test run shows no problems (that can be attributed to the changes made for the story/feature).
Besides these internal integration tests, there are also other teams that perform integration tests on a system level to verify how well our product works in the wider ecosystem.
The advantage of this way of working is that you get faster feedback when your change broke something that you didn't intend, but it does require a way of working where the work-in-progress is not mixed with the work that was already completed.
