Introduction to testing

Question

I am very new to the concept of testing and I had not yet have any experience with this.

At my company, no one does write tests as of now, neither is any test framework set up or anything. Also, we don't have testers. You bet how testing works: send new version to one customer, then wait for his call... if he hasn't called in for 24 hours to report bugs, call him and ask, and if the program seems to work, send it to the other customers. And then, wait for their calls...

Now, you can guess how much I like this. In fact, it's down to two options by now: either I can change my company, or I can change to another company. For both, I could use some background knowledge on testing, which I am collecting right now. But where to start?

A friend told me the basics on how his company does automated tests each night - a status I want to establish in our company as well. I asked him to find out which frameworks they use for unit tests and web tests, and he will find out for me, but I would also like to ask you which ones are recommended or even de-facto standard.

What we deliver are ExtJS applications which do CRUD operations on data via C# backends into MSSQL databases.

So I guess I would have to look for

• a webtest framework, which "clicks" through the browser/webapp
• an extjs/javascript unittest framework,
• a C# unittest framework (I will have a closer look at VisualStudio UnitTests on Monday)
• perhaps a framework which tests whether web calls give correct return values, and how web calls react to "garbage in"?
• perhaps even an sql unittest framework for stored procedures?

What exactly would I really need; and what would be best/most useful to start with? (from the accepted answer here I read "webtest" it is, but then, testing web calls could be easier to start with!?)

Are recommendable low-cost/no-cost frameworks available, or are there frameworks which provide a limited no-cost version, so I can try whether it suits my/our needs before I go to my boss for funding?

Where can I start finding information, are there good books on testing?

Answer 1

Here are a few suggestions to at least get you started. Everything listed here is free.

• a webtest framework, which "clicks" through the browser/webapp - Selenium

• an extjs/javascript unittest framework - Jasmine or Mocha

• a C# unittest framework - NUnit

• perhaps a framework which tests whether web calls give correct return values, and how web calls react to "garbage in"? - NUnit, implemented as tests calling the web services and verifying expected responses

• perhaps even an sql unittest framework for stored procedures - NUnit, implemented as tests calling the stored procs and verifying the expected results (either from the stored proc output, or with follow-up queries to confirm expected data changes)

By the way, I worked at a company that was nearly clueless about automated/programmatic testing (testing was done manually or not at all). But I threw together an automated UI regression test using Selenium, and it blew them away. Sometimes people just need a good demonstration.

Answer 2

This is all general, not specific to your needs.

Random stuff I learned from the process of adding unit tests to my C++ project (not all from personal experience):

• integration tests are not like unit tests. There are a lot of harnesses that only support unit tests, not integration tests. The same script that calls your test harness should also run a code formatter and error out if it makes a diff.
• A properly-written make test will be able to only rerun tests if the relevant code has been changed. Hint: each test binary outputs a log file (more useful than a stamp file).
• a good test framework supports process-level isolation and splits the responsibility of choosing tests and recording their results from actually running them. TAP, which originated from perl, is a good example of doing this and is usable for any language.
• Use a freaking CI service with a matrix. This is useful even without any testsuite, to catch compiler errors that only occur with one compiler version.
• If your matrix is too big, or if your CI bot is taking to long for a single matrix entry, maybe you have too many bundled libraries. Even if it's a library specifically written for this project, if it is isolated enough to be a separate repo with its own CI builds (which indirectly generate binaries for use in your main project's CI builds), it should be. If it is not isolated enough, make it so.
• it's impossible to unit test in the presence of global variables. Kill them with fire. If killing them is too hard, at least isolate them into a single Globals globals object per library/binary. But libraries should never expose an API that relies on globals anyway.
• realistically, you won't add tests to all your old code. Rather, add tests both when writing new code and when bug reports come in.
• when writing new code, first quickly write some code so you get an idea of what you really need, then delete it all and write unit tests, then build an implementation as necessary to pass the unit tests. This ends up being a lot easier.
• your tests can and will rely on implicit assumptions about the class/function they're testing, but the tested class/function will be changed by an idiot someday (probably you).

My general advice is: do not tie yourself to one particular technology. You have special needs, write a little code that works with your code instead of twisting around to fit in a framework. TAP is particular good at letting you write your own glue instead of forcing you to use legos, and even letting you call out to other test harnesses.