Help! Nixie's Gone Wild!!!

[FlaMike]

When I got up a few hours ago, I noticed the pool water level was low--below that of the skimmer weir opening. Last night before I crashed, everything was normal. So, I decided I needed to investigate what was happening. There were no weather issues overnight (storms, precipitation of any kind, observable power fluctuations, etc.) It seemed like just another night in paradise. But Nixie was going wild!

The pump was running, freeze protect was on (the air temperature was 75-80 at the time. Nixie reported air & water temperatures of zero, the emergency drain was open (it has an egg timer of 10 minutes so the pool does not empty) and the overflow sensor--not part of Nixie--never tripped. I assume the drain line emptied the pool until the skimmers ran dry. I don't know if there was damage to the pump yet.

I did a hard reboot of Nixie & REM. When they started up, boom! Right where they left off--most/all of the relays went on, temperatures were zero, the pump started, and the drain line re-opened. Ugh!

I did a shutdown command, turned off all breakers, replaced the SD card with a backup from a couple of days ago, & flipped the breakers back on. Nixie/REM had temperature readings of zero, then then normalized, then went back to zero. The relays went back on, etc. The Relay HAT graphic in REM showed several/all of the relays toggling on & off repeatedly. REM temperature readings fluctuated between "real" temperatures & zero--probably causing the freeze protect cycle to go nuts & mess with the pump.
I did another shutdown & killed all of the breakers.

That's where things stand at the moment--everything is off & I am refilling the pool. I don't know if the Building Automation HAT just crapped out, if I need to reload Sequent software & reconfigure the world, or what. I don't want to just power things up again because it appears Nixie/REM are hurting pups & will just go nuts again.

It looks like things fell apart around 4:00 AM. Below is some history from various items under Nixie/REM control.

My question is, where do I go from here? Do I need to get all new Sequent stuff? The relay board appears to be OK, but ??? What could have caused things to go nuts & how do I recover & prevent it from happening again?

All thoughts/ suggestions/sympathies are welcome!! Thank you for your help!

I apologize for the images that were posted. They were all supposed to be different ones & the image names were. I should have previewed the posting but didn't. My bad. Hopefully, these are more helpful...

The blue line is water temperature in the heater graph.

The graph above is the valve actuator for the drain line, not the sensor that never signaled an overflow situation.

The pool filter pump normally runs from 8:00 AM to 8:00 PM.

[tagyoureit]

You have my sympathies. It sucks when shit breaks for no apparent reason.

I'd start with any of the Sequent command line tools and see if they respond correctly to checking status, toggling relays, reading inputs, etc. Then move only to REM. See what happens with the comms to the Sequent cards. Is the temp coming in correctly? Can you toggle relays? etc. These steps should highlight any hardware issues. njsPC only responds to the data that's coming in so it should be the last part that you bring back into the equation.

[FlaMike]

I corrected the graphs above. Don't know why they had appropriate file names listed but all turned out to be the same graph. I think there's a dark cloud in my life today.

Working on your command line suggestions. Also checking all wiring in case something magically became loose. Earlier just before my previous post, it appeared that REM & the Sequent boards were communicating as expected, but I'll verify as you suggest. Hoping to find a smoking gun!

[FlaMike]

OK, I am completely flummoxed. I checked all wiring--everything was secure, tight, connecting what it should, & looked good. I added water to the pool so there was plenty to nourish the skimmers. I turned on the breaker for Nixie & REM but nothing else. I turned on/off all relays via REM & all appeared to be OK. I added water to the strainer basket on the pump to help the beast prime. I powered up the rest of the breakers & turned on the pump.

The pump primed, Nixie was getting good data once the flow sensor had been up for a few minutes (I think it's a 2 minute delay for the salt cell to check flow & salinity). And Nixie is humming along as if nothing ever happened.

It's only been about 15-20 minutes, but Nixie is shrugging her shoulders & asking why I'm so concerned & confused. She's right, I'm both!

Hoping this situation never occurs again, but holy smokes, what a lousy way to start your day!

If the poop hits the fan again, I'll let y'all know.