2

Suppose that you have a design that has had an error after running 14, 18, and 20 hours. You're not exactly sure what is causing the error, but you make a change to the design that MIGHT fix it.

How long do you have to run without seeing an error until you can be 99% sure that you've fixed the problem?

Note that the system is reset and restarted after each error, so the times given are for the first error after a reset/restart.

Also, assume that undoing the change makes the error come back at the original rate.

crj11
  • 5,480
  • 1
  • 13
  • 32
  • Not a complete answer, but personally I'd say you keep checking through your design until you are certain about the cause of the error, and then fix it, rather than guess fixing it and hoping. (p.s. this is not intended to be a snarky comment) – Tom Carpenter Sep 18 '18 at 22:49
  • 1
    @TomCarpenter: While it is theoretically possible to track down the ultimate root cause for every issue, anyone who has done this kind of work realizes that it doesn't always make economic sense. Therefore, you restart the test (while optionally continuing to search for root cause -- or more likely, starting to work on a different problem) and if the systems in the meantime run for long enough, you declare success and move on. – Dave Tweed Sep 18 '18 at 23:46
  • Is the error random or is it certain to happen at those intervals, or is it random, but always at those intervals? @DaveTweed, you've stumbled upon an issue that I've had a lot of cause to think about over the years as a less trained individual often trying to learn from people like engineers. To a person with incomplete knowledge like me, an engineer's often-internal rules of thumb are precious information, but engineers are precise people and these rules can be dangerous when misinterpreted, and much of this information remains internal. – K H Sep 19 '18 at 01:57
  • @KH I don't have enough incidents for any real statistics, just the three errors. So far after the change, it has been running for 30 hours without an error. – crj11 Sep 19 '18 at 02:06
  • @KH The syetem was restarted from scratch each time. The delay was until the initial error. I have edited the question to reflect this. – crj11 Sep 19 '18 at 02:54
  • I've had a student recompile the code in a debug mode to fix something and then forget to recompile in release settings - the settings slowed the timings from the design and _only_ after about 3 hours to 5 hours the system would become unhappy. Took ages to fix. – D Duck Sep 19 '18 at 08:00

2 Answers2

2

Part of the way I did DVT's different from everyone else was to verify the all the design specs including HALT/HASS with margin measurements, not just pass-fail.

The objective is not unique to raise stress levels to a non-destructive fault level but rather find a way to find how to measure the margin to error or functional failure.

This can be accomplished by injecting radiated and conducted & radiated noise of all types; RF, ESD, arc noise, thermal and vibration, and injecting false data on the interface etc.

If you can measure margin to failure at all the Environmental stress levels expected (Climatic, mechanical, electrical, human error, etc( then find the weakest links).

Hopefully, you have some way to automate the measurements with built-in self-test.

Proving that a new product means verify the design, process and components are all defects free with adequate margin. There are statistical ways to measure this before acceptance such as Dpk based on std dev and min-max, fault detection and correction coverage, etc.

Your criteria for acceptance ( parameters, methods and values) must be defined by detailed design specs in order to have a reliable product.

My favorite final process test for new SMD MOBO'S was random vibration with heat on a fixture or a vibrator on backplane cards. But for high-speed synchronous links prone to errors such as magnetic disk drives, I used SERDES "window margin", a digital form of eye pattern on worst-case data patterns.

For each project a DVT might be 30 pages with 1 page per test such as climatic. I spent 8 years as Test Eng. Mgr responsible for conducting DVT's on new products and all factory production with over 20 mainframes testing diskdrives and 100's of special test gear with a dozen ATE for board test, and Burn-in heated chambers for every system final test in production at 40'C

Tony Stewart EE75
  • 1
  • 3
  • 54
  • 182
  • I once worked on a datalink concentrator that rarely but halted with a "timeout". It would happen once in a day or not. So I put a 64 bit logic analyzer on all the busses and chips needed to analyze the traffic and after a week of no fault, I gave up. No trouble found. It didn't matter that much because this advanced 2 way wireless automated meter reading system using the ISM band with 1 million potential users sharing color diversified TDM in a 6kHz BW was sold to the biggest competitor, Itron and we got our last paycheque. I expected if there was local EMI, probes might make it worse. – Tony Stewart EE75 Sep 19 '18 at 19:39
1

Back in the 1980s, when I was designing motherboards for engineering workstations, we did two kinds of testing:

  • DVT - Design Verification Testing - run a small number of systems at the limits of their ranges for voltage, frequency and temperature (8 corners), looking for any problems with the design margins.

  • DMT - Design Maturity Testing - run a large number of systems for as long as possible under nominal conditions.

Our rule of thumb for either kind of testing was to run for 3× the average before-fix failure time (MTBF) before we considered a problem "fixed".

Dave Tweed
  • 168,369
  • 17
  • 228
  • 393
  • In the past we have used a similar rule of thumb of three times the longest error delay. There is probably some math that supports such rules of thumb, but I personally don't know it. Given the rules of thumb, assuming no more errors, I guess I can declare the problem fixed on Thursday morning. – crj11 Sep 19 '18 at 02:11
  • FYI, it failed again after 38 hours, so back to the drawing board :-( . – crj11 Sep 19 '18 at 13:04