NPIC6C596 resets itself

Question

This is a board with an ESP32, 16 relays, and other IO.

I'm using two power shift registers NPIC6C596 to drive the relays...

...and here's one relay.

NPIC6C596 normally runs on 5V, I used 3V3 by mistake in the schematic, but that's fixed on the board thanks to a bit of wire.

Without any load connected to the relays, everything works fine.

However, when I connect a load to one of the relays, NPIC6C596 sometimes resets itself when switching. It's quite reliable in failing, it happens about 10-20% of the time, depending on load.

I suspected the reset line, so I removed the power-on reset RC circuit and just connected the active low reset to VCC with a wire. No luck, it still resets itself.

Here's the layout:

+24V for the relays comes from a Recom PCB mount switching brick, there's a canned buck converter to make +5V for the rest of the circuitry.

I've done some investigation:

230VAC LED lightbulb as a load.

This has capacitance, so the relay will spark at turn-on but not at turn-off. Since these are SPDT relays, there are two possibilities: bulb on NO, and bulb on NC.

Bulb on NC: reset occurs at turn-on only.

Bulb on NO: no reset occurs

This hints that, when the NPIC drives the relay and shorts its output to ground, the low impedance sinks whatever noise spike would reset it into ground. But when the bulb is on the NO contact, the spark occurs when it does not drive the relay, and it looks like it is more sensitive then.

Inductive load: 230VAC motor driven ball valve

This isn't a solenoid, rather a ball valve driven by a small motor that draws a few watts.

Reset occurs both at turn-on and turn-off, no matter what relay contact is used.

This means the whole thing is unusable, because when NPIC6C596 resets itself, it turns off all the 8 outputs, which means all the relays that were on then turn off. When the micro tries to turn them on again to open the ball valves... the chip resets again. So it keeps blinking.

It happens with all the relays. It doesn't seem to propagate through the reset line to the other NPIC6C596 though.

Everything else on the board works fine, the micro doesn't crash, etc. It's just these NPIC6C596 chips.

Probing with a scope isn't very useful: no matter where or what I probe, or just the ground clip looped to the tip of the probe, I get the usual spikes that happen when a relay sparks in the vicinity.

How are your relay ground coils routed w/r/t the shift register grounds? (They look connected together?) You want to decouple as much as possible, dumping the coil ground current without shaking the shift register, I.e. separate grounds. I'm surprised the esp32 is alright: I've had them completely lock up from relay emi. — 2e0byo, Dec 09 '21 at 23:45
Looks like an EMI generator. YOu need to use STP wire and perhaps RC snubbers on the loads with ferrite beads. Stray capacitance is more sensitive to the arc rise time on either open or closed. Perhaps an ICL on LEDs and TVS on the motor loads will help greatly too with low ESR caps on the coil power. Minimize all current arc loop area with STP pairs with PE on shields at source. — Tony Stewart EE75, Dec 09 '21 at 23:58
@2e0byo positive pin of relay coil to +24V, negative to shift register. It's a power shift register with integrated MOSFETs, it's supposed to be able to switch inductive loads, solenoids, etc. But then, of course... it has only one ground pin. I'm going to try the TI version (TPIC6C596) to avoid redoing the board... — bobflux, Dec 10 '21 at 00:11
Bob that won't fix anything to swap drivers. Your loop area impedance from 24V cap to driver to return must be a low inductance to avoid impulse ground shift on common currents induced by loads. V= LdI/dt and L increases about 10nH per cm — Tony Stewart EE75, Dec 10 '21 at 02:01
The logic ground must never share any coil current and be directly back to supply pwr gnd. — Tony Stewart EE75, Dec 10 '21 at 02:22
Reading the specs of the npic6C595 say the logic high in 0.85 times vcc. Not going to work with 3V3 levels from the esp32 at least not without level translation. — Kartman, Dec 10 '21 at 02:29
@TonyStewartEE75 I'm going to test adding some decoupling caps on +24V. Now I would agree with the comment on grounding, but this chip has only one ground pin, which is both logic and power, so it cant' have a separate ground return for the coils... Well, it looked like a convenient one-chip solution on paper lol. — bobflux, Dec 10 '21 at 08:29
I missed the shared ground, sorry That seems to throw one if the usual solutions under the bus... Still surprised the esp32 is alright, but count your blessings I guess :) — 2e0byo, Dec 10 '21 at 08:35
@TonyStewartEE75 Added 2x 1µF MLCCs as decoupling on +24V next to relay and next to chip: no luck. With the motorized ball valve as a load, sometimes, the NPIC6C596 on the *other side* of the board also resets. This ball valve has a limit switch to stop the motor, and when it trips, NPIC6C596 also sometimes resets even though it is not actually switching. Now... I fired this chip and replaced it with a 2N3904 driven by a GPIO. So far, it looks like it works. I'll have it switch the ball valve a few thousand times in a loop, see if the micro crashes... — bobflux, Dec 10 '21 at 10:17
@2e0byo (followup to previous comment) -- With the transistor instead of NPIC6C596, no problem at all with the LED lightbulb. Without the resetting NPIC6C596 I could test it for much longer, and reach the point where the motorized ball valve crashes the micro, lol. I opened the valve motor, and it has about zero EMI mitigation components inside, just a limit switch. — bobflux, Dec 10 '21 at 10:47
I've used a triac I had lying around. It works a lot better. https://electronics.stackexchange.com/questions/598793/triac-driving-motor-with-limit-switch-needs-protection — bobflux, Dec 10 '21 at 12:05
looks the answer here, sadly, is 'don't use a power shift register with a single ground'. Other than driving your relays with transistors/triacs a long way from the mcu (i.e. keeping the mcu ground pristine) I would seriously consider a *hardware* WDT in any application where an MCU is switching external loads. I had an esp32 controlling a fridge/freezer for a year: despite good grounding, isolation and decoupling emi would sometimes cause it to lock up. With a *hardware* WDT in the loop you *know* things are going to be alright. Ironically a bug later killed the fridge, but that's my fault ;) — 2e0byo, Dec 10 '21 at 12:19
@2e0byo yes I agree, this chip looked so convenient but in the end, it was a really bad idea... Did you use the ESP32 hardware WDT or a separate circuit? — bobflux, Dec 10 '21 at 14:14
@bobflux I meant a completely separate wdt. The esp32 'hardware' wdt depends on the main clock IIRC and can lock up. I actually used *both* the esp32 onboard wdt *and* a hardware wdt (there are decent chips--I breadboarded with a 555 and pnp + diode + cap on the reset line for a pulse). That said there are multiple onboard wdts, and I simply used whatever uPython exposes. A hardware wdt is still a very cheap investment, and if you have dangerous loads... — 2e0byo, Dec 10 '21 at 14:26

bobflux · Answer 1 · 2021-12-13T10:47:34.450

Self-answer:

These motorized valves turned out to be rather nasty loads due to the absence of any EMI mitigation and bouncy limit switches. With snubbers added (10nF+47R), they became civilized and there were no longer any problems with crashing/rebooting ESP32. I tested other loads this board was supposed to drive like circulator pumps, without problems on the ESP32 side either.

However, the NPIC6C596 kept resetting all the time.

So, the conclusion is this chip is way too sensitive to drive relays. Since I'm not going to bodge 16 transistors to drive the relays, I'll have to redo the board, so I'll switch to opto-triacs, which I should have done to begin with.

--

I've done more testing.

Test 1: I suspected the long trace for shift register /RESET acting like an antenna.

I had already connected each /RESET pin on each shift register to its VCC pin directly with a short wire. So I also cut the /RESET trace, leaving the short wire only. Then I tested with the snubberized ball valve.

-> No change, and when the relay on one side of the board switches, both shift registers reset, including the one on the other side of the board. ESP32, which is in the middle, doesn't care.

The fact they both reset is quite weird. It means whatever is resetting them is either going through the air, or through a trace that connects them together (data, /OE, two clocks). I'm not suspecting +24V and +5V yet, because there are several decoupling capacitors between the two chips.

Test 2: Add more decoupling on +24V next to relay.

No change.

Test 3: Add 10nF ceramic capacitor between the driven (negative) side of relay coil and GND.

It works! It's been switching reliably every second for about 10 minutes now, whereas before it would reset about every 3 relay clicks.

So, that was the answer... The chip's internal MOSFETs have body diodes, and the datasheet says they will avalanche over 33V, so they should be protected and the capacitor should not be necessary. But... the chip has only one ground pin.

Glad you found it! I would *still* recommend thinking of an external WDT on the esp32's reset. The kind of loads you drive with motorised ball valves deserve a proper fail safe. (It goes without saying that your code needs to fail safe: it should reset the system to a safe state on reboot and take things from there. You might also want some kind of system with a timestamp written to flash to count reboot loops and fail to dormant if e.g. the system reboots x times in y minutes. Ball valves have less water hammer than solenoids, but depending on how quickly they close pounding might be bad...) — 2e0byo, Dec 13 '21 at 12:38
They take about 2 min to close, and it's for heating, so there's no risk of hammer. However the motor has been "cost optimized" to open and close infrequently: it gets very hot when repeatedly opening and closing. I will definitely add a watchdog as you say! — bobflux, Dec 13 '21 at 12:42

NPIC6C596 resets itself

1 Answers1

Linked