31

I'm working in a software development team as software developer. I've been working on the same project for three years now. The software is a 32-bit desktop based C# application in .NET 4. Our target platform in Windows 7 (we had to support Windows XP till last year). The software communicates with various custom hardware for which custom drivers are written. The hardware manufacturing and driver software is written by our client. There is different driver for 32-bit and 64-bit Windows of course.

During our system testing phase, we execute all/most test cases in both 32-bit and 64-bit Windows 7. I can't recall if we got any bug in our software that exist in only one flavor of Windows. Having this experience I've started to wonder, do we really need to test 32-bit software on 64-bit Windows?

What's the industry standard?

gnat
  • 21,442
  • 29
  • 112
  • 288
Donotalo
  • 1,219
  • 1
  • 12
  • 19
  • 1
    Does your .NET application have any dependencies on native DLLs? I've been bitten a few times only testing on one platform mainly because I forgot to package the x86 native DLLs with my software along with the x64 DLLs. If you start to use a new third party library, that library might also try to load native DLLs behind the scenes, and you won't notice until it crashes on an x86 PC. I've also had to write code that picks which DLLs to use based on whether my .NET app is running in 64-bit mode or not, and that code needs to be tested as well. – Phil Oct 07 '14 at 23:26
  • @Phil: Point noted. The DLL does use a lot of external libraries. I believe all of those DLLs are compiled for x86. The application itself doesn't have any dependency on native DLLs, but it calls native win32 API. – Donotalo Oct 13 '14 at 06:32

6 Answers6

31

Most of the bugs we encountered with running 32-bit software on 64-bit windows had to do with the location of the software (Program Files (x86) instead of Program Files), locations of registry keys (some were found in Wow6432Node). We had these problems mostly because we needed to communicate with other software (also 32-bit), and so we needed to test the software on both 32-bit and 64-bit...

When you didn't have these problems, I believe it is quite safe not testing on both platforms when you explicitly compile in 32-bit mode. When compiled in 32-bit, the .NET runtime will run everything in 32-bit mode, and it should work the same as the 32-bit mode on 32-bit platforms.

According to 64-bit Applications (MSDN), 32-bit applications are run in Wow64 mode and Running 32-bit Applications (MSDN) explains this mode in more detail.

Peter Mortensen
  • 1,050
  • 2
  • 12
  • 14
David Perfors
  • 740
  • 4
  • 8
  • Are you saying that a 32 bit software if run on 64 bit OS, the .NET on that OS will run the software in 32 bit mode - which is same as running the software in 32 bit OS in .NET? Is there any documentation? – Donotalo Oct 03 '14 at 07:56
  • 4
    @Donotalo: you should inform yourself about the basic switch in your Visual Studio configuration manager (or each project's compilation settings) named "platform", with the options "x86", "x64" and "Any CPU". When you found that switch, F1 is probably your friend. – Doc Brown Oct 03 '14 at 08:02
  • Documentation is added to my answer – David Perfors Oct 03 '14 at 08:04
  • 1
    The biggest issue we had was running with other 32 bit libraries, .NET would simply fail to load them when run on a 64-bit machine. – gbjbaanb Oct 03 '14 at 09:20
  • 1
    @gbjbaanb: you surely meant, when you forgot to use "x86" as platform. – Doc Brown Oct 03 '14 at 09:38
  • err. yes, sorry - when a "all CPU" platform was used, the resulting binary worked fine on a 32-bit dev box but the exact same binary failed horribly when put on the 64-bit server. You can spend a lot of time checking the application's configuration data in such cases. – gbjbaanb Oct 03 '14 at 12:02
  • @gbjbaanb that's correct - anything compiled to run on 32-bit won't be able to use 64-bit binaries and vice-versa. Even if you believe your software shouldn't have any problems, nothing beats thorough testing. – Thebluefish Oct 03 '14 at 17:40
  • @gbjbaanb: The issue isn't just with DLLs. If a software package has some applications that are 32-bit only and some that aren't, x64 apps won't be able to see registry keys written by the 32-bit apps and vice versa (any idea why MS deliberately broke x86/x64 interop?). – supercat Oct 03 '14 at 18:26
23

The hardware manufacturing and driver software is written by our client. There is different driver for 32 bit and 64 bit Windows of course.

So on 32 bit Windows, your software talks to one driver, and on 64 bit Windows, it talks to a different one? Let us assume there are new versions of these drivers from time-to-time. So when you only test your software on 32 bit Windows you cannot be sure there won't be some differences in the 64 bit driver which will make the combination of your software + 64 bit driver fail. And from the viewpoint of your users, it does not matter who is to blame (you or the author of the driver), all they see is a non-working system. So even if your code is free of bugs, a test might reveal a bug in the 64 bit driver, and finding such a bug might help you take the right measures (like sending a bug report to the author of the driver).

Of course, when you have used those two drivers for years, and you are very confident that the behaviour is exactly the same, you might skip the tests for one platform, following the arguments in @DavidPerfors's answer. As a compromise, you could run tests on 64 bit Windows only whenever a new driver version is available. Actually, this depends on the complexity of the drivers, your experience with and confidence in them.

Some additional things to consider:

  • what OS type is your user base using most? 32 bit or 64 bit Windows? If you decide to test only on one platform, choose the one your users use most often.
  • how severe is it when a new release of the software will not work on a less frequently used platform? For example, can your customers immediately step back and install the previous working release? Do they have only some inconvenience or real financial loss by this? If it is the former, testing on only one platform may be fine, if it is the latter, obviously not.
Doc Brown
  • 199,015
  • 33
  • 367
  • 565
16

The default assumption in enlightened QA circles is "If you didn't test it, then it doesn't work".

As a practical matter that's usually an unreachable goal to strive for in much the same way application engineers might like to have unit-tests for everything; but they don't believe they'll ever reach it and release on schedule.

However your question can only be answered by, or in conjunction with, sales and marketing. You provide them with a cost to test and they provide an analysis of the market benefit. If estimations on both sides were sufficiently accurate, the answer would be simple 

if B > C:
    test_32bit_version()

In my experience, everyone's cost estimates are inaccurate. As far as the other side of the equation, Dilbert once parodied decision making there with "I just asked my cat, Mittens". To do much better, they'd need training in anthropological field methods.

msw
  • 1,857
  • 10
  • 16
  • *The default assumption in enlightened QA circles is "If you didn't test it, then it doesn't work".* -- and the default assumption in operations circles is "If you didn't test it, then be prepared to be hunted down like a rabid dog by angry sysadmins". It's hard to test all scenarios but it's a good idea to try hard to test all reasonable ones. It's very rare that a project post mortem ends up deciding that time spent testing a system works properly was time wasted. – Rob Moir Oct 04 '14 at 15:04
6

Seeing as 99% of all Windows installations of Windows 7 and up, and a good portion of Vista as well, are 64 bit, why the hell would you even consider not testing for that platform?
It's just a no-brainer, unless you're making it specifically for a very limited group of users you KNOW are using 32 bit Windows and will continue to do so for the life of your product.

So yeah, test for 64 bit problems. In fact develop on 64 bit platforms, and probably supply a 64 bit version as standard with a 32 bit compiled version as an option for those few customers who've not upgraded to a new computer and OS in the last 6-8 years or so.

jwenting
  • 9,783
  • 3
  • 28
  • 45
  • 1
    I concur mostly, but supplying both 32 and 64 bit versions adds complexity and probably shouldn't be done without a good rationale based on performance or functionality. –  Oct 03 '14 at 08:10
  • @dan1111 given that he's apparently still needing to support customers on old 32 bit platforms (though for how much longer is impossible to tell), he has a valid business case to supply 32 bit binaries at least for now. – jwenting Oct 03 '14 at 08:21
  • 4
    I understand the need to supply 32 bit binaries. I just don't know if he should bother to supply native 64 bit ones without a compelling reason. –  Oct 03 '14 at 08:24
  • 1
    If I were releasing desktop software here in 2014, I would probably only release 64 bit binaries. Remember in the 90s when people were transitioning from DOS to Windows 95? We had a similar argument then -- and clearly DOS got left in the dust, just like 32 bit is now (at least on the desktop, not embedded or mobile). –  Oct 03 '14 at 15:13
  • 4
    I must ask @jwenting. Where did you get that it's 99%? Could you improve on that and post the source? – Malavos Oct 03 '14 at 15:52
  • Problems can arise when releasing both 32-bit and 64-bit binaries. Especially with LAA available, there's very little reason to support 64-bit unless you really need the extra memory. The extra time and effort into testing and supporting 64-bit binaries generally isn't worth it, especially for smaller developers. This gets even more complex when you have many 3rd-party dependencies, and I'll bet that many developers of 3rd-party libraries don't explicitly test 64-bit as well. – Thebluefish Oct 03 '14 at 17:47
  • @Thebluefish: Given Microsoft's bizarre decision to have 32-bit applications effectively use a different registry from 64-bit ones, any package which uses a mixture of 32-bit and 64-bit programs that need to interact with each other is likely to have problems (and needs to be tested for them) on 64-bit machines. – supercat Oct 03 '14 at 18:24
  • 1
    Yeah, I don't believe the 99% at all. The business world still has a _lot_ of 32 bit Windows installations out there, between old machines not updated (running Win7 from the early days) or out of fear of incompatibilities. "The majority" are probably 64 bit, but I'd be very unlikely to believe any very high percentage without very good evidence. – Joe Oct 03 '14 at 18:55
  • @supercat I was referring to a program that gets compiled for both target platforms, sorry for the confusion. – Thebluefish Oct 03 '14 at 19:01
  • @Thebluefish: Right. My point related to some experience with an application that contained two programs, one of which is .NET and the other of which is written using an x86-only development system, if the .NET one wasn't compiled for x86 only, it wouldn't see the registry keys set by the other. – supercat Oct 03 '14 at 19:07
2

I would test any installer on as many different Windows setups as possible as from my experience the installers are most likely to fail on different systems.

Otherwise, as you know from your experience with the given software, bugs are most unlikely to just show up on 32-bit or 64-bit, and you can take some calculated risk.

Firstly, you should be having many test cycles with very little code changing between the later cycles as you get close to shipping. Any time you can save, you can use to create more test cases, and/or allow more (and hence smaller) cycles, so giving quicker feedback. (The risk of spending time testing X may be more than the risk of not testing Y, because you are testing X too much.)

Therefore

  • Try to test on the “other bitness” to what you used for the proceeding test cycle.
  • If you know the “bitness” your developer use, start with testing on other one.
  • Divide the test cases up between the “bitness” to give come coverage on each
  • But swap them between the “binnes” on each test cycle.
Peter Mortensen
  • 1,050
  • 2
  • 12
  • 14
Ian
  • 4,594
  • 18
  • 28
2

Nope. Likewise, when the FDA is done testing new medicines on mice and rats, they skip over testing on monkeys and just sell it for human consumption.

< /sarcasm>

Yes, yes yes yes yes. There is nothing but sadness in store for your software if you do not test every platform you possibly can. Things are always different, and the assumptions in the designer/coder's head during the project usually only come passably close to modelling real life. So please, test your software. Please.

kmort
  • 131
  • 3