16

Every textbook I've seen makes a big deal of the fact that testing and verification are two different concepts. Yet none of them provides a clear (or clear enough to me, at last) distinction.

To provide some context, I'm interested in the verification of digital hardware designs using hardware design languages (HDLs).

I've seen some explanations that resort to a "physical" or "tangible" difference: if it is about a manufactured device, then it's testing. Is this the whole story? If so, why does the word "test" come up so often in verification (especially in functional verification, we talk about testcases, testbenches, DUT (device under test), directed tests, random tests, etc...)

VHDL Addict
  • 161
  • 1
  • 1
  • 5

8 Answers8

24

Was an ASIC Design Verification Engineer at Qualcomm. In the most simple way I can explain it:

Testing: Making sure a product works, after you've created the product (think QA).

Verification: Making sure a product works BEFORE you've created it.

They're both testing, just that verification is more complicated because you have to figure out a way to test the product before it exists and you have to be able to make sure it works as designed and to spec when it actually comes out.

For example, Intel is designing their next processor, they have the specs, they have the schematics and the simulations. They spend $1 Billion USD to go through fabrication and manufacturing. Then the chip comes back and they test it and find out that it doesn't work. They just threw a lot of money out the window.

Throw verification in. Verification engineers create models that simulate the behaviour of the chip, they create the testbench that will test those particular models. They get the results of these models and then they compare it with the RTL (model of the circuit writting in a hardware design language) results. If they match, things are (usually) OK.

There are a number of different methodologies for the verification process, a popular one is Universal Verification Methodology (UVM).

There is a lot of depth in the field and people can spend their entire career in it.

Another random tidbit of information: Usually you need 3 verification engineers for 1 design engineer. That's what everyone in the field says anyway.

EDIT: A lot of people think of verification as a testing role, but it's not; it's a design role in itself because you have to understand all the intricacies of your IC like a designer does, and then you have to know how to design models, testbenches, and all the test cases that will cover all the feature functionality of your IC, as well as trying to hit every single line of RTL code for all possible bit combinations. Remember that a processor nowadays has billions of transistors due to the fabrication process allowing smaller and smaller (now 14nm).

Also, in large corporations like Intel, AMD, Qualcomm, etc, designers don't actually design the chip. Usually the architect will define all the specs, layout the types of pieces that need to go together to get a particular function with a specific requirement (i.e. speed, resolution, etc.), and then the designer will code that into RTL. It's by no means an easy job, it's just not as much designing as a lot of engineers coming out of school think it is. What everyone wants to be is an architect, but it takes a lot of education and experience to get to that point. A lot of architects have PhD's, and like 15-20 years of experience in the field as a designer. These are brilliant people (and sometimes crazy) who deserve to be doing what they're doing, and they're good at it. The architect on the very first chip I worked on was a bit awkward and didn't really follow some social norms, but he could solve anything you're stuck with regarding the chip, and sometimes he would solve it in his head and tell you to look at one signal and you'd be like, "how the hell did he do that?". Then you ask him to explain and he does and it goes way over your head. Actually inspired me to read textbooks even though I've graduated already.

PGT
  • 675
  • 5
  • 17
  • +1 Thanks for the last remark, helps us see that the field is indeed important (though RTL and design engineering sounds more attractive to most engineers, I think) – VHDL Addict Nov 06 '14 at 23:05
  • For completeness, would you mind adding what a testcase is? – VHDL Addict Nov 06 '14 at 23:07
  • I added a tidbit about what verification actually is because of your first comment about the design role being more attractive; they're both good roles, just depends on what you like. As for a test case, a SoC like the Snapdragon could have tens to hundreds of thousands of test cases, and with random testing, in the millions. To put it simply: you're applying a set of input bits and that gets changed going through many modules, then you get some output bits as a result that you compare with your model's results. Something as simple as testing a picture showing up on your phone would have... – PGT Nov 07 '14 at 03:31
  • a large number of testcases. Say you want to show a single pixel on your mobile screen. What someone outside the field will think is apply 1 bit for white, and 0 for black. In the actual mobile world, that pixel can differ by size, intensity, rotation, color format (YUV###,RGB###, etc.). And you're probably testing 1 bit within a set of bits that gets applied to the input together. The other bits may be 0 because it's black, or may be 1 because it's handling other information like how to handle transmission mode, CLK, enables/disables, triggers, fancy stuff like that. – PGT Nov 07 '14 at 03:42
6

In my book Verification is ensuring that what you have designed "does the job" - i.e., you have a set of things the "device" needs to perform, and verification ticks those off on the list.

Testing, though, is making sure that the things the "device" does are done right. You have a set of functions, and you test each function making sure that function performs properly.

In a nutshell, Verification is checking the design, and Testing is checking the product.

Majenko
  • 55,955
  • 9
  • 105
  • 187
  • I think I'm starting to understand... Could you please provide some examples of each? – VHDL Addict Nov 06 '14 at 21:56
  • How does that fit with a *verification plan*, which specifies what should be implemented and also how to know that the functionality is correct? It would be of little use to implement or tick-off a function if it does not work. – VHDL Addict Nov 06 '14 at 21:58
  • @ Majenko - So you have written a book on Verification? Would you share some more details about it? – Michael Karas Nov 07 '14 at 02:53
4

Coming from a an ASIC (hardware) design background, there are three important terms: validation, verification, and test. The earlier answers generally talk about one or two of these terms, but don't clearly contrast all three in the way I would. Here's the way I understand them:

  • Validation: does the specification (often a C model) meet market or customer requirements
  • Verification: does the implementation (RTL, netlist, or GDS2) match the specification
  • Test: does the manufactured device match the implementation
Winston Smith
  • 163
  • 3
  • Can the netlist and GDS2 simulations give different results? – Ciro Santilli OurBigBook.com Jul 15 '16 at 18:03
  • 1
    @CiroSantilli巴拿馬文件六四事件法轮功, I suppose you're asking about behavior of gates vs. transistors. For normal digital voltages and waveforms, I'd say they will give same results. But there could be "analog" effects not considered by idealized gates, such as power/ground variations or signal charge sharing. If those effects are present, then ideal digital behavior may not hold true. – Winston Smith Jul 30 '16 at 23:31
  • 1
    @CiroSantilli新疆改造中心法轮功六四事件 Yes, they can give significantly different results. Been there, made that mistake. – Elliot Alderson Jan 10 '20 at 12:59
2

ISO9000 talks about verification and validation. In the context of ISO9000 verification means testing a prototype design to prove it meets functional and performance expectations. Validation means testing the first production run also meets design expectations. Verify first, validate later is my little way of remembeeibg the order of things.

Several software standards reverse the order of verify and validate and this could really cause confusion so be aware of this.

Bottom line is... Why are you testing something - is it a prototype design - if so then the quality standards tend to call this verification. If you are testing a production run for the first time then hardware guys call this validation.

Just my personal experiences.

Andy aka
  • 434,556
  • 28
  • 351
  • 777
1

A test is designed to see if a specification is met. Verification is to see if the device meets the design inputs -- i.e., all the specs. I suppose there are many more interpretations, but this is what I've seen in FIA guidance docs.

Scott Seidman
  • 29,274
  • 4
  • 44
  • 109
  • I see, I've changed the wording a little (from *test* to *testing*) to make it clearer that both are processes. I agree with you that for individual tests the word *test* is appropriate (sometimes I feel like I'm restating the obvious with these terminology questions...) :) – VHDL Addict Nov 06 '14 at 21:22
1

We make a distinction between verification testing and validation testing. Let's say you are designing a fan that cools off some equipment. Verification testing is done to make sure the fan meets all the design requirements. So you might test airflow, thermal cycling, vibration, etc.

Validation testing makes sure the design requirements were the right one. Did the design inputs we had for the fan actually give us the fan we wanted? For example you'd make sure the fan actually cools the equipment as intened.

Eric
  • 3,010
  • 15
  • 21
  • This is how the terms are understood by the books on Software Engineering I have read. Validation = make sure the requirements are right (check them with the customer, regulations, etc.); Verification = make sure the product is right (test it against the specification) – Wouter van Ooijen Nov 07 '14 at 07:40
1

From reading these replies I am now realizing there is no set definition of how "testing" differs from "verification" in the industry.

When working with HW design ("real" HW design, like stuff on PCB, not VHDL programming) we go through a verification&validation phase, and a production testing phase (actually just designing the production tests themselves and delivering them to production site). - Verification - (1)verify that prototype/mass-produced-item fulfills the HW requirements (2)validate HW requirements against SYS requirements. - Production Testing - smoke test and simplified & quick verification test-cases adapted to screen out mass production defects without having to go through entire verification process for each of the 500000 units produced a year.

So in that particular multinational company "testing" refers to production tests and nothing more.

user3038015
  • 11
  • 1
1

Going by the book definition, "Verification is the process of ensuring the functional correctness of the design according to the specification document"whereas "testing is the process of checking whether the physical chip works as intended after manufacturing". Verification is about checking if the design is logically correct while testing is about whether the hardware works.

Girish M
  • 11
  • 1