7

In my team we have been working on a firmware for a product. The product was released some years ago but we still implement new features in the firmware and regularly provide a new firmware for our customers. Recently we have developed a new product which use the same firmware as the first one. Of course, we had to change the firmware to allow for some new capabilities (while still keeping functionality for existing product intact).

We have a release test specification which contans mostly manual tests which we carry out before we release any new software. We have adapted this for the new product since it has additional capabilities. But most of it is common.

I know that new products are in the pipeline and in my simplistic reasoning we need to repeat our release test for each product every time we release a firmware. So i am looking for a new approach which is less time consuming. One of the things we are already working on is automation, but due to the nature of the product many tests have to be manual.

Manoj R
  • 4,076
  • 22
  • 30
Vandhunden
  • 231
  • 1
  • 4
  • If many of the tests are common amongst products then, why is more automated testing not possible? If your test plan can be written into a specification then it should be possible to write automated tests. The only thing I see holding you back may be technology or platform limitations. – maple_shaft Jan 02 '13 at 12:06
  • I can give an example - one of our tests involve making a phone call to a satellite operator in order to verify functionality of our equipment. When we release a new firmware we need to ensure that the system works with the new firmware (making it essentially a system test). – Vandhunden Jan 02 '13 at 12:14
  • I see, then yes this is a high level test meant to be a manual process, however is it possible to emulate the hardware so that more automated tests can be run against the firmware? – maple_shaft Jan 02 '13 at 12:38
  • Yes, and this is something we are also doing where possible. The most time consuming tests are test where we have to make some change to the physical environment, e.g. temperature tests or removing cables (to test that we can detect broken cables), etc. In both cases, yes, we can emulate hardware but this still this does not test the system as a whole. But you are right in that it would take us some of the way and perhaps be sufficient. – Vandhunden Jan 02 '13 at 12:50
  • It sounds like your situation is rather unique. If I were the client, and I was purchasing hardware that interacts with satellite systems, then I would demand manual system tests as well no matter how labor intensive. Room for error when dealing with satellites I understand is unacceptable. – maple_shaft Jan 02 '13 at 12:55
  • Certainly is unacceptable - I was just wondering whether I was missing some obvious way to avoid all the somewhat-duplicate testing. Thanks for your inputs though. – Vandhunden Jan 02 '13 at 13:15
  • 1
    One thing I have learned the last years is: there is almost always room for more test automation, often it is only a matter of having the right idea. For example, your phone call scenario may be not be easily automated right now. But assumed your satellite operator does nothing but reading some data from a computer screen to validate the correctness, is it impossible to think of some kind of web service which provides exactly that data directly to you, in a way which could be queried automatically? – Doc Brown Jan 02 '13 at 13:28

2 Answers2

4

The broader scope of software testing involves analyzing all of your products, supported platforms, and then identifying what needs to be tested to provide a particular degree of assurance about the results.

The most important aspect is to document what you decide and why you decided that. You'll want to provide that documentation to your clients so they can be assured of your testing process.

For your case:

  1. Since you're expanding, you should look for the commonality between products and firmware releases. Aspects that are truly common don't necessarily have to be tested on every single target platform. A representative sample should be sufficient.

  2. Provide some discipline around your release process so you can understand what has changed between releases. If you can demonstrate that functional area XYZ won't be affected by any of this release's changes, then you can either eliminate or reduce the amount of testing of that area. Your product domain will dictate the balance point between eliminate or reduce testing for that cycle.

  3. It can be expensive, but consider adding robotic automation around your test processes. Pulling a cable; pressing buttons; dropping the device; etc... can all be performed with a robotics test harness. This approach allows you to test around the clock, which is the other solution for time consuming tasks. In other words, you found more time instead of decreasing the amount of time required.

2

I would split your tests in three categories:

  1. (Unit)tests on a (simulated) reference platform. These tests should cover all the software that does not directly interface with hardware components and should include both unit-tests for the various software components and functional tests for all the features. The (simulated) reference platform should be able to support all features (even if there is no single product that has them all) and should be able to simulate hardware failures.
  2. Tests of the hardware interfaces. These tests should give you confidence in the interfacing between hardware and software. By necessity, these tests must be executed on the hardware itself and repeated for each distinct hardware design. It might be good if these tests are specified/designed by a combined team of hardware and software people.
  3. System tests. These tests should give confidence that the final assembled system works as designed. They should focus on the most-used features (and perhaps some easy-to-test error scenarios) and be repeated for each product that has a distinct hardware design or non-overlapping feature-set (if two products are technically identical, but one has a reduced feature set, there is no need to test that second one as extensively).
Bart van Ingen Schenau
  • 71,712
  • 20
  • 110
  • 179