Error setting up integration #8

screenagerbe · 2023-07-14T06:27:18Z

The defaut DSMR integration works just fine, but the moment I install this custom integration the P1 port seems to be unreachable (and it isn't).

HA version: 2023.6.3, running as docker container
P1 is connected via ser2net module (https://www.zuidwijk.com/product/p1-reader-ethernet/)
Once I configure the HACS integration the connection is lost and cannot be restored for some reason, at the same time connecting via PuTTY is working fine though.

There are also some rather weird errors in the logs:

Logger: dsmr_parser.parsers
Source: runner.py:179
First occurred: 13 July 2023 at 21:30:46 (1231 occurrences)
Last logged: 08:06:34

Unexpected <class 'ValueError'>: unconverted data remains: 525

Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:137
First occurred: 13 July 2023 at 21:30:46 (1231 occurrences)
Last logged: 08:06:34

Error doing job: Fatal error: protocol.data_received() call failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1009, in _read_ready__data_received
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 128, in data_received
    self.handle_telegram(telegram)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 154, in handle_telegram
    parsed_telegram = self.telegram_parser.parse(telegram)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 95, in parse
    dsmr_object = parser.parse(match)
                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 272, in parse
    timestamp_occurred = ValueParser(timestamp).parse(values[i * 3 + 1])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 393, in parse
    value = self.coerce_type(value) if value is not None else value
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py", line 7, in timestamp
    naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 525

Logger: dsmr_parser.clients.protocol
Source: /usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py:144
First occurred: 13 July 2023 at 21:30:46 (1231 occurrences)
Last logged: 08:06:34

disconnected due to exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1009, in _read_ready__data_received
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 128, in data_received
    self.handle_telegram(telegram)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 154, in handle_telegram
    parsed_telegram = self.telegram_parser.parse(telegram)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 95, in parse
    dsmr_object = parser.parse(match)
                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 272, in parse
    timestamp_occurred = ValueParser(timestamp).parse(values[i * 3 + 1])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 393, in parse
    value = self.coerce_type(value) if value is not None else value
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py", line 7, in timestamp
    naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 525

The telegram message:

/FLU5\253770234_A

0-0:96.1.4(50217)
0-0:96.1.1(315[...])
0-0:1.0.0(230713201636S)
1-0:1.8.1(000001.177*kWh)
1-0:1.8.2(000003.947*kWh)
1-0:2.8.1(000175.145*kWh)
1-0:2.8.2(000083.569*kWh)
0-0:96.14.0(0001)
1-0:1.4.0(00.001*kW)
1-0:1.6.0(230705121500S)(00.086*kW)
0-0:98.1.0(2)(1-0:1.6.0)(1-0:1.6.0)(230501000000S)(632525252525W)(00.000*kW)(230701000000S)(230623050000S)(01.316*kW)
1-0:1.7.0(00.000*kW)
1-0:2.7.0(00.000*kW)
1-0:21.7.0(00.000*kW)
1-0:22.7.0(00.000*kW)
1-0:32.7.0(236.7*V)
1-0:31.7.0(001.28*A)
0-0:96.3.10(1)
0-0:17.0.0(999.9*kW)
1-0:31.4.0(999*A)
0-0:96.13.0()
0-1:24.1.0(007)
0-1:96.1.1(385[...])
0-1:24.2.1(230713201606S)(00006.338*m3)
!F4C9

Mind you:
There is only an electricity meter and a water meter installed, there is no gas meter connected

The text was updated successfully, but these errors were encountered:

dupondje · 2023-07-14T06:55:01Z

Seems like the disconnect is caused by: disconnected due to exception

And it seems like it fails to parse because:
0-0:98.1.0(2)(1-0:1.6.0)(1-0:1.6.0)(230501000000S)(632525252525W)(00.000kW)(230701000000S)(230623050000S)(01.316kW)

The '632525252525W' value is not a valid date it expects.
BUT

This should already be fixed in dsmr_parser that this is handled, see ndokter/dsmr_parser@edf535f

Can you check "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py" if it contains that commit?

screenagerbe · 2023-07-14T07:11:26Z

No, it does not. 🤔

def timestamp(value):
    naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')

    # TODO comment on this exception
    if len(value) == 13:
        is_dst = value[12] == 'S'  # assume format 160322150000W
    else:
        is_dst = False

    local_tz = pytz.timezone('Europe/Amsterdam')
    localized_datetime = local_tz.localize(naive_datetime, is_dst=is_dst)

    return localized_datetime.astimezone(pytz.utc)

dupondje · 2023-07-14T07:39:08Z

That's not correct! Are you using the latest version of this integration? Cause this HACS integration depends on dsmr-parser=1.2.3: https://github.com/dupondje/homeassistant-dsmr/blob/main/custom_components/dsmr/manifest.json#L10

screenagerbe · 2023-07-14T08:47:32Z

yeah running the latest version:

screenagerbe · 2023-07-18T07:13:34Z

So I checked the current version of dsmr-parser, and it isn't 1.2.3

Do I need to update the version myself or should it be taken care of because of the defined requirement in the custom integration?

dupondje · 2023-07-18T09:01:12Z

Home-Assistant should update the version to the one required in the dsmr integration.
It's weird it didn't do that. Do you see anything in the startup logs?

screenagerbe · 2023-07-18T16:27:25Z

Nope but i fiddled a bit with the access rights of the custom_components folder in the docker container.
and suddenly:

so the newest version of dsmr-parser is installed, but unfortunately I still get the Failed to communicate error

errors in log (looks like these are the same errors):

Logger: dsmr_parser.parsers
Source: runner.py:179
First occurred: 18:20:06 (1 occurrences)
Last logged: 18:20:06

Unexpected <class 'ValueError'>: unconverted data remains: 525

Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:137
First occurred: 18:20:06 (1 occurrences)
Last logged: 18:20:06

Error doing job: Fatal error: protocol.data_received() call failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1009, in _read_ready__data_received
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 128, in data_received
    self.handle_telegram(telegram)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 154, in handle_telegram
    parsed_telegram = self.telegram_parser.parse(telegram)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 95, in parse
    dsmr_object = parser.parse(match)
                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 272, in parse
    timestamp_occurred = ValueParser(timestamp).parse(values[i * 3 + 1])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 393, in parse
    value = self.coerce_type(value) if value is not None else value
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py", line 7, in timestamp
    naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 525

Logger: dsmr_parser.clients.protocol
Source: /usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py:144
First occurred: 18:20:06 (1 occurrences)
Last logged: 18:20:06

disconnected due to exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1009, in _read_ready__data_received
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 128, in data_received
    self.handle_telegram(telegram)
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 154, in handle_telegram
    parsed_telegram = self.telegram_parser.parse(telegram)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 95, in parse
    dsmr_object = parser.parse(match)
                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 272, in parse
    timestamp_occurred = ValueParser(timestamp).parse(values[i * 3 + 1])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/parsers.py", line 393, in parse
    value = self.coerce_type(value) if value is not None else value
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py", line 7, in timestamp
    naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 525

dupondje · 2023-07-19T13:27:16Z

But the file still doesn't seem to contain the correct code.
/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py

File "/usr/local/lib/python3.11/site-packages/dsmr_parser/value_types.py", line 7, in timestamp
naive_datetime = datetime.datetime.strptime(value[:-1], '%y%m%d%H%M%S')

While the strptime call should be on line 8 on 1.2.3
https://github.com/ndokter/dsmr_parser/blob/v1.2.3/dsmr_parser/value_types.py#L8

screenagerbe · 2023-07-20T07:20:05Z

Yeah, you're correct. The files don't have the correct code in them...
I even setup an Home Assistant Operating System up on a spare RPI and it throws exactly the same error.

I'll leave it for what it is and just wait for the new parser to be included in the container.

screenagerbe · 2023-07-20T20:34:08Z

Couldn't leave it alone 🤷‍♂️

But now it is working in HAOS.
Apparently there were two files which were not updated via the version=1.2.3 requirements.
Those files were also not updated via pip install --no-cache-dir --upgrade --force-reinstall dsmr-parser namely:

parsers.py
value_types.py

I've been playing around with them (updating the content, renaming them) and after a system reboot of HAOS it just worked, restarting Home Assistant wasn't enough though.

Now I need to see if I get it also working in the HA container.

screenagerbe · 2023-07-20T21:16:22Z

Couldn't leave it alone 🤷‍♂️

But now it is working in HAOS. Apparently there were two files which were not updated via the version=1.2.3 requirements. Those files were also not updated via pip install --no-cache-dir --upgrade --force-reinstall dsmr-parser namely:
* parsers.py

* value_types.py
I've been playing around with them (updating the content, renaming them) and after a system reboot of HAOS it just worked, restarting Home Assistant wasn't enough though.

Now I need to see if I get it also working in the HA container.

same for docker containers.
I had to exec into it and replace the two files mentioned above.
stop/start the container and then it worked...

screenagerbe mentioned this issue Aug 3, 2023

Missing water entities in dsmr home-assistant/core#95895

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error setting up integration #8

Error setting up integration #8

screenagerbe commented Jul 14, 2023

dupondje commented Jul 14, 2023

screenagerbe commented Jul 14, 2023

dupondje commented Jul 14, 2023

screenagerbe commented Jul 14, 2023

screenagerbe commented Jul 18, 2023

dupondje commented Jul 18, 2023

screenagerbe commented Jul 18, 2023

dupondje commented Jul 19, 2023

screenagerbe commented Jul 20, 2023

screenagerbe commented Jul 20, 2023

screenagerbe commented Jul 20, 2023

Error setting up integration #8

Error setting up integration #8

Comments

screenagerbe commented Jul 14, 2023

dupondje commented Jul 14, 2023

screenagerbe commented Jul 14, 2023

dupondje commented Jul 14, 2023

screenagerbe commented Jul 14, 2023

screenagerbe commented Jul 18, 2023

dupondje commented Jul 18, 2023

screenagerbe commented Jul 18, 2023

dupondje commented Jul 19, 2023

screenagerbe commented Jul 20, 2023

screenagerbe commented Jul 20, 2023

screenagerbe commented Jul 20, 2023