Feature request, read all modbus meters and also log voltage, cos phi etc #132
Comments
|
I would love that! I have tried to do readouts with two masters, but failed. |
|
I am thinking about a sort of configurable gateway from modbus to REST API.... |
|
Or an implementation of Modbus TCP, so all Modbus RTU devices can also be read/controlled over Modbus TCP. But I think it would be enough to go full read-out with SmartEVSE v3 and then using the REST API to send Modbus meter data to Home Assistant or other HA systems. |
|
In this version your MainsMeter, EVMeter and PVMeter present themselves at port 502 of your SmartEVSE, with the modbus-tcp protocol; so this makes it possible for you to use whatever modbus function you want on whatever modbus register you want. Please test, have fun! |
|
Looks like it's working can it interfere with the SmartEVSE RTU modbus polling? I mean can the TCP requests from HA happen on the same time as the SmartEVSE is polling this data from the RTU modbus? Because sometimes I see this: Home Assistant yaml config |
|
Thats why I ask you to test it thoroughly: it uses the same client (=modbus server) as the rest of the firmware, so I would expect no collisions. But I enabled modbus lowlevel error routine, so if you have telnet log running you should see no timeout or other modbus errors... |
|
Without disconnecting the second ESP32 module with ESPEasy firmware, CRC errors 99% sure coming from the second modbusmaster trying to fetch data: After disconnecting the second ESP32 modbus device:
|
|
Ok going to try second setup... stay tuned! |
|
modbus-tcp-bridge2.zip |
|
No, E0 Time out remains. |
|
I did add another, separate client for the TCP bridge. Now are you polling the extra registers you requested or are you also polling the registers that are already available through the REST API? Because that would definitely mess up traffic on the bus.... |
|
When I try to copy your setup, I get this error in HomeAssistant: And also, every 30 seconds I get this: When I comment out the device_class: power of this sensor, both messages are gone. I still get an occasional timeout in telnet but not as many as you do .... would this change anything in your setup? EDIT: Testing with the FIRST firmware file modbus-tcp-bridge.zip since this makes more sense. |
|
I cannot get the timeouts reproduced in my setup; I get an occasional timeout, but this is not often enough to do some decent testing. I can see the following test scenarios:
This represents your setup, if you use the commented parameters this might improve the behaviour of HomeAssistant. Let me know which way you want to go.... |
|
For test scenario 1 you can use since it has no modbus TCP bridge, but it does have the extended log messages for modbus errors. |
|
OK great, will look into this. Possible that the current 1.6.2 release has Telnet disabled? |
|
You've got to rename firmware.debug.bin to firmware.bin |
|
This is interesting. Could it be possible to have modbus TCP addressing for the meters ? I have tested the Elfin EW11 devices to convert modbus RTU to modbus TCP. |
|
It would not solve the problem that your MainsMeter is fixed as slave at address 1, and SmartEVSE is fixed as master at address 1. |
|
Finally had some time to flash the debug version (from the now pulled 1.6.2) , stupid me, not checking the whole folder contents... Second ESP32 modbus module is disconnected. So don't really get why I get the timeouts. My meters have 101, 102, 103 and 104 as modbus address. |
|
Got it, it's because Load balancing is set to master and no slaves are connected. In the past we had to this to get the Circuit menu option. In newer releases this is not needed anymore, disabling this fixed the timeouts :). |
|
Ok thx for discovering this. So does this also solve your timeout problems for modbus-tcp-bridge.zip? |
|
Yes exactly. I used your first tcp build.
These time-outs are just 7 messages to send to all 7 slaves. And since there are none, these don’t get received/acked. With verbose on my “euro” fell…
This weekend I’ll add more meters and see what happens. Good stuff!
|
|
I have a nice weekend present for you, this version has the problems in 1.6.2 fixed, and also has added detailed logging to the timeout/other modbus error messages; thanks to your trick to put LoadBl = 1 I could test this now! In normal logging the timeout-messages to the slaves are suppressed, in verbose logging they are still shown... Also a broadcast message that times out will show up if you have loadbalancing enabled without extra EVSE's connected, this one is not suppressed since it is not that annoying... Let me know what happens if you add more meters! |
|
|
|
Currently only traffic for the MainsMeter address, the EVmeter address and the PVmeter address is forwarded. The forwarding has to be configured per address at the RTU modbus, so if you want more addresses to be read a config routine has to be built... |
|
Yes, would be nice if we could add addresses to the list. I also think PV_Meter / PVmeter is not used anymore. I'm not able to find the menu option for it, I can remember with a v2 or v3 with original firmware this menu item was somewhere. Only MainsM and EVM show up. |
|
I think it depends on the setting MainsMeterMeasure... |
|
Indeed, you can enable it by changing in the menu what the mains meter measures. Would you need to add menu's like "Extra kWh meter 1" "Extra kWh meter 2" etc? So you can also put more modbus devices in. |
|
Looks like I can't use the mainsmeter setting with PV :). Purple is sensorbox data to my second SmartEVSE (for now a separate modbus network) and blue the data from the first SmartEVSE with several SDM230's. Yesterday was with the old working settings, then the data is very similar. Can I see anywhere what kind of code you added to make the TCP Modbus server? |
|
I pushed the code to master, Im thinking of detecting TCP requests that are not yet forwarded and opening a forward for them dynamically; not sure if it can be done simply in the emodbus library... |
|
I think beginning of July I have some more time to test this version. At the moment I run the latest build with the Modem features. |
|
tcpbridge_with_modem.zip |
|
Hi sorry, I haven't got time yet to test it. |
|
I also recently 'migrated' my EV meter to modbus because of the addition of the TCP Modbus bridge (before it was pushing in the relevant EV meter details through MQTT on SmartEVSE from HA, which worked very well). Now I see a lot of bogus values appearing. E.g. big jumps in amperage values for the EV meter, big jumps in 'ChargePower' aka the power in watts being transferred into the EV, while nothing is flowing through.. It seems like some sort of corruption of the values or the checksum not being checked properly. Is this a known issue?
PS: I am using the Sensorbox as well, with my own ESPHome fork running on it. Its hardcoded on slave address 10, EV Meter is on 11. The Sensorbox values are coming in correctly; no weird stuff happening there. PS2: Apparently only the values that get processed inside of EVSE get scrambled. It seems like the values bridged over by Modbus TCP bridge into HA are actually valid and do not contain the same jumps. Whut? |
|
Thanks for testing, this is new info since the others in this thread didnt have time to test it. If I understand correctly : |
|
I must not forget to mention that this is with the Modbus Bridge logic incorporated in the current master branch, not the custom build versions that were shared in this pull request thread.
The Sensorbox is not read out through the modbus bridge (only by SmartEVSE) but directly connected to HA through ESPHome. Since the whole reason I moved the EV meter from HA to SmartEVSE was the TCP Bridge functionality, being able to still read out all values in HA as well while keeping the EVSE stuff functioning independent from my WiFI/HA etc. So I am actually not sure whether this issue also occurs without using the Modbus bridge functionality.. I may have to do some testing soon to confirm that. I do see some Modbus communication CRC/timeout errors appearing from time to time, in the debug output on telnet. I do not use load balancing btw. |
|
Small update; somehow it fixed itself; I'm not getting weird values anymore through MQTT.. I'll keep an eye on it and report back if the situation changes again. |
|
Holiday season started, so again a bit more time for other projects :) I tested the code, it works but it is lacking the feature to get the current status. I have 3 concurrent meters and works. |
|
Ok thx for testing! |
Sadly it wasn't fixed; my HA installation just wasn't querying the modbus bridge TCP server. After I 'fixed' this and started requesting all SDM630 values again, the same issue came back. It seems to directly garble the information it gets from the SDM630, as if requests for the same register are colliding or something, resulting in weird measurements. I guess I can try removing the 'double register' reads from HA and see if that fixes the issue (trust the bridge will then forward all incoming data from the SmartEVSE's own requests to HA as well). |
|
I would appreciate it if you could test disabling the "double register reads"! |
This didn't work sadly. For now I've resorted again to disabling the readings from HA. It's been stable ever since again. |
|
This is strange, since the HA readings (both MQTT or REST API) dont force any extra readings from the modbus. It looks like some overload (memory and/or cpu) that is not happening when reading directly through modbus tcp... |
|
I also noticed quite some errors (timeouts, CRC mismatches) inside of the telnet debug output. It indeed looked to be some kind of overload happening. The errors occurred at both the SDM630 as EV Meter and the Sensorbox v2 as MainsMeter (even though it runs my custom ESPHome FW). I also noticed the Sensorbox being slow and noticing the DSMR component calls were stalling the loop with 0.2s every time it was updating it's values. Knowing this, I did a couple of things;
Overall, all these things seem to result in a much more stable, happy modbus climate for the EVSE. Even after re-enabling the modbus TCP bridge and having HA query the SDM630 through the EVSE, the readings do not contain the weird value spikes anymore. So I'd call this success. Now, since this touches some internals (e.g. extra task created, modbus value request loop slowed down, etc) I am a bit unsure if I should publish it as a branch yet until I've done some more testings. So I'll report back next week whether my issues came back or if it's definitively solved now. I do not have slave EVSE's nor use the load balancing feature, so maybe someone who does could test the 'improvements' in their set-up before we actually consider merging it. |
|
Sounds great, but your third point worries me: how would these changes impact other users with "standard" sensorbox v2 firmware? Also: the stalling of 0.2s occurs within the sensorbox with custom firmware, or within the SmartEVSE loop? If its within the sensorbox, who cares, because this is the primary and only task of the sensorbox; and if you would care, is it possible to solve this delay (it shouldnt be memory or cpu intense to get that dsmr value). So when the dsmr freq is reduced from 2s to 5s, now the SmartEVSE reads that value every 200ms, so reading the same value 25 times? Since the problem of "strange values" arised after I places the MQTT publish-calls in the updatecurrentdata routine, and diminished when I placed a mutex, I am looking at replacing the MQTT library with a threadsafe version, because I think that will solve the root cause of the problem. I cannot test anything until december, and will be reluctant to merge any major changes I couldnt test myself... |
It shouldn't. Since SmartEVSE is the master on the modbus, it will query the Sensorbox when it sees fit. The standard Sensorbox firmware has a wildly different way to get/parse the specific values from the energy meter (custom build, just a couple of values). It'll just respond if asked.
This has been changed in my custom ESPHome firmware, so on the Sensorbox and not on the EVSE. Changing the frequency of polling the actual data doesn't affect SmartEVSE in any way. The ESPHome version of DSMR polls and processes all of the values from the Smart Meter (not just 5), so it takes some time to complete it's job which is why it stalls the loop for 0.2s. This normally wouldn't be a problem if you use multi-threading on an ESP, but the developers of ESPHome are convinced one core is all you will ever need with ESPHome and thus give no option to actually utilise the second thread for anything (except internal stuff such as WiFi etc). So this is more an ESPHome thingy than anything else.
That isn't the case since the SmartEVSE uses some fancy loop that queues up multiple readings and kind of iterates over them. In the original 100ms timer it states it will query the meters every 2 seconds. So using a 200ms timer would logically result in a request every 4 seconds. I suppose I could also try to have the ESPHome Sensorbox update it's values every 4 seconds, see how that fares. I think it mostly was struggling with the Modbus readouts/calls every 2 seconds while also doing a DSMR readout every 2 seconds. Sometimes coinciding and causing delays due to the stalls the DSMR component was causing when reading out it's values. They're both using Serial IO (modbus + readout) which can be blocking, so that makes sense. But above probably doesn't do much for the strange value issue, as that seems to be happening inside of the SmartEVSE, not the Sensorbox. It may well explain the timeouts I was seeing at requests for Sensorbox values (but that'd be very specific to my own use-case).
I see. I didn't even think of this yet, sounds like the proper fix indeed.
Totally get that! No worries. I will be using this custom version for myself as it seems to fix my issues. I agree with you that we shouldn't merge anything that can't properly be tested. The mentioned changes are found here btw; arpiecodes@0205b73. Not saying this is the proper solution for now or that we should merge it as such, but just to give an indication of what I've done here. |
|
@arpiecodes I found this MQTT library that claims to be thread-safe: I don't think it'll take too much effort to move the library to this one and give it a test, after removing the mutex's I placed in the code. |
|
Nice, curious of the outcome. I did "discover" that the bogus values are sometimes kWh meter reading or voltage etc. So it's maybe a timing error? That it receives a register value from before or after the one it thinks it's requesting? |
|
@fluppie That is very interesting, it supports the theory that the non-threadsafeness of the library causes the problem... |
|
Sad to report my code changes didn't completely alleviate the 'weird values' issue. They did help for the timeout/CRC modbus errors. One thing I do not quite understand; why would the addition of the MQTT client be the issue? Since it is not really writing anything incase of EVMeter/MainsMeter being received through modbus, just reading them out (and then sending out to the MQTT broker). UpdateCurrentData is not called there if there was no received messages/commands over MQTT. Correct me if I'm wrong. I do see the issue when you have multiple threads triggering the UpdateCurrent data, but then still it's weird it's only happening to modbus received values. Wouldn't logically the mutex be placed on the modbus calls/logic instead? To me it'd make perfect sense if the collision would happen in the case the BridgeModbus connected client requests a value from the meter while SmartEVSE also requests it at the same time (or any situation where multiple requests are done for whatever reason). I'd imagine that to be a very weird situation for the meter to handle and it would probably cause issues. This gets supported by the observation that it only seems to happen (for me at least) while having the ModbusBridge in use reading out the same meter. I suppose because I lowered the interval for the SmartEVSE readouts from 2s to 4s, it made the issue less frequent, but didn't fully solve it. |
|
The modbus logic (and protocol) is supposed to be thread safe; in the protocol the master sends data requests (through a single routine that "rotates" all the different meters/functions), and the (single) receiving routine sorts them out. So the only collisions would be when the "rotating sending routine" is called before the last call was finished. You took care of that possibility by putting that in a 200ms loop. Also, AFAIK the problem only arises with the MQTT interface, not with the REST API. So in my view the strange values have to do with the MQTT library. |
I see how it's thread safe for sending/receiving parts, and the part of SmartEVSE requesting those values by itself. But; what does the ModbusBridge have to bring to the table here? It's not part of that routine rotation method inside of the SmarEVSE, but requests values on it's own behalf (as requested by its clients). The single routine inside of the calls made by SmartEVSE itself won't help there.
How did you test/verify this? MQTT is obviously very easy to log values and see the spikes as they occur. Through API it's probably a lot harder to catch them (but I still saw them happen there as well - just didn't try with MQTT logic disabled/removed). |
|
True! (as to the bridge). REST API: I stand corrected, I didnt know strange values were detected there too! |
|
Here you can see voltage being logged as current. Doesn't happen often, only a few times per week. But I'm polling "only" every 15s instead of the SmartEVSE every two seconds or so.
|
|
Interesting to see in my case it's always the value 2387W, which seems to (in this case) match the total KWh charged since reset;
EDIT: Yep, I also confirmed the earlier spikes are showing 2386W with contactors closed and KWh since reset being 2386.x. |
|
I found a threadsafe MQTT library, that should/would solve the strange outlier values on the API. It doesn't use the PubSubClient underlying library, but another, ESP-IDF library that looks a lot more reliable, although it is pretty expensive in flash ram usage: @arpiecodes @fluppie Could you test whether this solves /diminishes the problem? |
|
Ok Scatman_II on tweakers reports still problems, this is a version on the old library where the current-publishments are synchronized with the other publishments: |
Currently I have a ESP32 with modbus shield from enri.nl to read all parameters / all modbusmeters. This gives 2 modbus masters which is far from ideal. I don't know if it's a lot of work to read out more registers than current and power as well as kWh import / export.
Screenshot of the ESP32 with ESPEasy software:
The Nan values are because of modbus collisions. I think the SmartEVSE is requesting every 2 seconds data on the modbus. My mainsmeter is at 15s intervals and the PV and EV meter are at 60s intervals. To limit the number of collisions.
List with all available registers:
https://github.com/reaper7/SDM_Energy_Meter/blob/master/SDM.h
Just a question, if not I keep using it like this. On the other hand I can save a ESP32, a 5V power supply and modbus collisions :).
The text was updated successfully, but these errors were encountered: