8

Having read a lot about this topic --- such as on this Software Testing Fundamentals site on verification and validation and Software Testing and Quality Assurance: Theory and Practice by Naik and Tripathy --- I still do not get it. Verification should prove that you are building the product right, while validation proves that you built the right product. But only static techniques (code reviews, requirements checks...) are mentioned as being verification methods. How can you say if its implemented correctly if you do not test it? It is said that verification ensures that the product meet specified requirements. Again, if the function is specified to work somehow, only by testing I can say that it does.

Could anyone explain this to me please?

Thomas Owens
  • 79,623
  • 18
  • 192
  • 283
John V
  • 4,898
  • 10
  • 47
  • 73
  • 1
    Where are you reading this? Usually, verification and validation is treated as a singular activity that includes everything that goes into product quality. Both static and dynamic techniques are part of V&V. – Thomas Owens Sep 10 '12 at 14:35
  • Please let use know where you are reading this. It is just plain wrong. See, for example, [this link.](http://www.asc-tech.com/vv.htm) – Peter K. Sep 10 '12 at 14:49
  • For example here: http://softwaretestingfundamentals.com/verification-vs-validation/ Also a book I have says the same, that verification is only done by static analysis, while validation by dynamic (actual testing) – John V Sep 10 '12 at 14:56
  • 1
    @user970696 the _book you have_ - would you mind telling its title and author? also, you mention "Wiki" - what's this? – gnat Sep 10 '12 at 15:56
  • Thanks for the link... but it's wrong. [Check out CMMI definition of Verification, specifically SP3.1](http://www.software-quality-assurance.org/cmmi-verification.html#sp31). – Peter K. Sep 10 '12 at 16:56
  • @PeterK. Thanks, yet now there is a conflicting answer :) I really dont know anymore what is correct.. – John V Sep 10 '12 at 17:24
  • Verification is checking that the system meets its specifications; Validation is checking that the customer's needs are met (in the customer's environment). I cannot see how you can check that the system meets its specifications without _some_ testing; static methods just don't cut it. – Peter K. Sep 10 '12 at 17:31
  • I'll try to answer this when I get home, but what book are you reading? Can you provide a title and author or link to it on Amazon? – Thomas Owens Sep 10 '12 at 17:41
  • Software testing and QA from Naik&Tripathy (Wiley,2008) – John V Sep 10 '12 at 17:53
  • 1
    @user970696 I don't have the book, but I have their slides from [here](https://ece.uwaterloo.ca/~snaik//mybook1.html). They don't appear to make the distinction you're talking about. For example, Ch3 of the slides talks about "Dynamic Unit Testing" which is certainly verification, not validation (because the whole system is not available). – Peter K. Sep 11 '12 at 01:17

2 Answers2

14

As this question created some controversy, let me start off this answer with my background: Apart from being exposed to V&V in daily project work, I worked for several years in the software engineering department of my alma mater and am a lecturer for software engineering. While this does not guarantee that anything I say is correct, I hope it at least gives me the benefit of the doubt that there might be some truth in this answer.

Let me assure you, that you are not as confused as you may believe you are. What you have stated in your question is as much true as it is misleading. Let me first point out, what you have correctly stated:

  • Verification = build the product right vs validation = build the right product
  • Static techniques are part of verification - mainly because they take your program and some formal input derived from the requirements and evaluate them against each other.
  • Verification ensures correct implementation of the requirements (i.e. that you have built it the right way)

Now let me clean up the confusion about testing. First, as many comments have stated before, dynamic testing of code via automated tests (unit, integration, ...) is indeed part of verification. What causes most of the confusion, however, is that people in validation will talk about testing as well, yet mean something different: in validation, testing usually involves a person using the application for its intended purpose. In the optimal case, this is the customer him-/herself.

However, the "errors" [1] found by testing in verification and in validation differ fundamentally:

  • verification testing errors: these are errors that violate your requirements in one way or another.
  • validation testing errors: these are errors with the very product you have built, not its functionality, hence, they reveal errors within the requirements.

For most people, it helps to look at concrete examples of different V&V cases. The following are extreme examples of errors:

  • You have a low-level requirement that states "f(x) should return x+1" and your implementation of "f" always returns the constant 2. You may find this error by several different verification approaches, but your customer probably won't find it during validation, because you are building an e-shopping site and he neither knows nor cares about "f".

  • You have a requirement that states "The system should be able to handle 1000 requests per second at a CPU load maximum of 80%". Again, validation will have a hard time, just as much as most static techniques. In fact, the simplest way to verify this is to dynamically test your application by hammering it with requests and monitoring your CPU load during that time.

  • Consider the above requirement for "f" once more, this time with a "correct" implementation. All your reviews, static analysis and dynamic tests will report a success, but then your customer tests your software and tells you that he misses the shopping cart icon on the webpage. No amount of verification will be able to find this error, as you have made it during the requirements phase.

As you can see, "testing" - if not defined more precisely - is part of both verification and validation, and in fact, "testing" should be performed for both.

[1] "error" is used colloquially here, so as to avoid the distinction between error, failure, mistake, fault, ...

Frank
  • 14,407
  • 3
  • 41
  • 66
  • That was really great! Leaving to cinema, I would really clarify: also both black box or test case based testing are part of verification proces, apart from UAT testing by users. Am I right here? – John V Sep 10 '12 at 18:46
  • But I must add, that ISO 12207:2008 does state only static techniques for verification :/ – John V Sep 10 '12 at 18:50
  • Unfortunately many (most) developers do not work to requirements, so they get confused about the difference between Verification and Validation, hence the pervasive though V&V are the same and even worse, Testing == QA. – mattnz Sep 10 '12 at 21:12
  • @Frank: Well, in the ISO I mentioned, it specifically mentiones that testing is conducted as a part of validation (7.2.5.3.2.1 and ..2). How come there is so much of dispute :( – John V Sep 10 '12 at 22:06
  • 1
    @user970696: Sure, those sections explicitly mention testing *against user needs*. The two sections I mentioned (implicitly: 7.2.4.3.2.3 and 7.2.4.3.2.4) state `The code implements proper event sequence, consistent interfaces, correct data and control flow, completeness, appropriate allocation timing and sizing budgets, and error definition, isolation, and recovery.` and `The software components and units of each software item have been completely and correctly integrated into the software item` --- how would you do either of those things *without* testing? – Peter K. Sep 10 '12 at 23:26
  • @PeterK. Thanks, that makes sense. But what is then the validation testing mentioned under that 7.2.5.3.2.1, it even mentiones border values for input testing, which implies typical functional test. I mean, nobody has still stated where functional testing belong (verification, validation?) – John V Sep 10 '12 at 23:29
  • @user970696: It does, but not at the unit test or integration test level --- it's at the "system boundary" (i.e. what the user can input). – Peter K. Sep 10 '12 at 23:31
  • @PeterK. Right, so when testers do the testing, what is that then? They to functional tests in the SIT. According to one of the first links, it would be verification. From that ISO, I would say validation. – John V Sep 10 '12 at 23:33
  • @user970696: SIT testing starts looking more like validation than verification. There is sometimes a "fuzzy line" between where one starts and the other stops. [Check out this "design V" diagram](http://en.wikipedia.org/wiki/V-Model) : Verification is closer to code (and goes against the detailed design); Validation is higher up. Verification needs to be done "inside" the system (mostly); validation needs to be done "outside" the system (again, mostly). – Peter K. Sep 10 '12 at 23:43
  • @PeterK. Alright, that was my point as it does not fit either of these two perfectly. Also, Unit testing and Integration is verification, while functional testing (or non functional) on the actual product is validation, correct? But of course, now its disagree with the link in the comments under the original question, where black box and white box is stated to be verification. – John V Sep 10 '12 at 23:47
0

Indeed: verification proves that you are building the product right, while validation you build the right product.

Therefore

  • verification is proving of the process... that you have followed the required standards and procedures, usually checked by static analysis and code review.
  • validation is proving of the resultant product... that the code does what is required, usually checked by dynamic analysis and testing, to show that specified inputs result in specified outputs

I don't usually cite Wikipedia, but in this case, it's helpful...

A more detailed explaination of the two processes can be found in ISO 12207 Software Life Cycle Processes (one of the other answers posts a link to an uncontrolled copy):

7.2.4.3.2 Verification Tasks

  • Requirements Verification
  • Design Verification
  • Code Verification
  • Integration Verification
  • Documentation Verification

7.2.5.3.2 Validation Tasks

  • Prepare Test Requirements, Cases and Specifications
  • Conduct the Tests

Now, the opening question asked why verification does not include testing. And despite some of the other answers, by P.SE members of high reputation, I put it to you that that is not the point of the verification activity, but of the validation activity.

Some answers suggest that you have to test to verify code or integration. No - that activity is to prove the modularity and the interfaces, etc, not the execution.

Utimately, what this discussion shows is that there is a lot of confusion as to what is verification and what is validation, and this is not helped by the boundary being a grey area, and defined slightly differently in different standards.

What is important is that both parts of V&V should be covered, and as long as that is the case then semantically, it doesn't really matter whether it's V or V... just V and V.

Andrew
  • 2,018
  • 2
  • 16
  • 27
  • Thanks but now I am really confused. The comments to the question shows that verification is the testing phase, you say the opposite. – John V Sep 10 '12 at 17:21
  • I see I've now got two down-votes... I'd be curious to know why people think my answer is wrong? – Andrew Sep 10 '12 at 17:37
  • Well IMHO as stated in the comments, verification is the actual testing, while valiadion is more about user acceptance of the final product. – John V Sep 10 '12 at 17:39
  • I *absolutely* disagree with that summary. In fact, that is plain wrong. Verification is *nothing* to do with the testing – Andrew Sep 10 '12 at 17:40
  • Well, look at the links provided in the comments. Looks like its comomonly accepted the way I said, yet I too read different opinions. But reading university texts, there is eg.: Verification approaches attempt to idenfity product faults or errors.." – John V Sep 10 '12 at 17:51
  • ISO 8402:1994 > ...Software testing is one of many verification activities.." Another one: testing specific requirements is a verification activity. – John V Sep 10 '12 at 17:57
  • The great thing about Standards, is that there are so many to chose from. I refer you to ISO 12207:2008 (Software Development Life Cycle) s7.2.4 and 7.2.5 - these definitions tie in with your original reference... – Andrew Sep 10 '12 at 18:07
  • Thank you. But one thing - if you look at the waterfall model, there is a Implementation phase followed by Verification, which is testing. What do you think? – John V Sep 11 '12 at 09:41
  • That depends on which Waterfall model you refer to... citing Wikipedia (http://en.wikipedia.org/wiki/Waterfall_model) Royce's original model had "Testing and debugging (AKA validation)" although the neighbouring image shows "Verification". Which all just adds to the confusion! – Andrew Sep 11 '12 at 09:52
  • Yeah, to be honest, I am still confused as there is so much of contradiction. E.g. look here, it even mentiones testing types as verification activities: http://www.ece.cmu.edu/~koopman/des_s99/verification/ BTW in your opinion, is unit and integration testing verification? – John V Sep 11 '12 at 10:08
  • Maybe we should move this discussion to Chat? But my opinion (and others clearly disagree) is that all static activity is verification, and dynamic testing (unit, module, integration) is a validation activity. – Andrew Sep 11 '12 at 10:13
  • One more thing please :) According to that ISO, verification should confirm that each software work product and/or service of a process or project properly reflects the specified requirements. But how can answer that from static analysis? If the requirement is "User is able to display screen with..." or "the controls will be placed in the blue box". I do not get this.. – John V Sep 11 '12 at 11:39
  • "Reflects" the specified requirements is the key... this is achieved by the design reviews (etc) and *traceability/flowdown* as well as the static analysis. Validation then *confirms* the requirements... – Andrew Sep 11 '12 at 11:51
  • thanks, language barrier I guess. Not sure how to translate "to reflect" in this context. If something is reflecting requirements, doesnt it mean that it is fullfilling? – John V Sep 11 '12 at 15:10