Update Apache IoTDB version

[slawr]

The current CDSP docker compose deployment uses IoTDB version 1.2.2 and was used to develop IoTDB features such as the VISSR and RemotiveLab integrations.

Several months have passed and the upstream IoTDB project is now in the 1.3.x series which brings useful enhancements such as improved sync between databases, a feature we see as useful in the CDSP Information Layer. We should therefore upgrade.

High level task outline:

• Decide IoTDB version and create summary of likely code impact (required changes)
• Update version used in docker
• Check IoTDB connection in RemotiveLabs bridge
• Check IoTDB connection in VISSR
• Update CDSP documentation where necessary

[slawr]

IoTDB version

1.3.2 was in focus as the current latest version, but 1.3.3 has just been released with useful looking improvements to existing sync and subscription features, whilst adding a new one AINode that integrates LLM AI models into timeseries SQL https://dlcdn.apache.org/iotdb/1.3.3/RELEASE_NOTES.md. So it's natural to consider jumping to that, with possible downside to lower amount of wide testing

Update: 1.3.3 was released Nov 20 '24 which gives sufficient time for major issues to become clear. It also brings various new features so decision taken to adopt it.

[slawr]

IoTDB changes 1.2.x..1.3.3

Upstream IoTDB list of changes here: https://github.com/apache/iotdb/releases
Github compare: apache/iotdb@v1.2.2...v1.3.3

Whilst there are of course new features that we may usefully adopt here is a list of changes from above that may effect existing implementations:

• v1.3.3
  • Storage Engine: Added new data types String, Blob, Date, and Timestamp.
  • System Management: Optimized configuration files, with the original three configuration files merged into one, reducing user operational costs.
  • Scripts and Tools: The import-tsfile script is expanded to support running the script on a different server from the IoTDB server.
• v1.3.2
  • Scripts and Tools: New export-data/import-data scripts added, supporting data export in CSV, TsFile formats or SQL statements
  • [IOTDB-6302] Enhance the support of ISO_LOCAL_DATE_TIME timestamp format by @HTHou in [IOTDB-6302] Enhance the support of ISO_LOCAL_DATE_TIME timestamp format apache/iotdb#12086
• v1.3.0
  • [IOTDB-6194] Rename target_config_node_list to seed_config_node
  • [IOTDB-6218] Rename storage_query_schema_consensus_free_memory_proportion to datanode_memory_proportion

As a starting point the renaming of the two config variables should be checked.

Implementation notes:

• Adding UDF library to the image
  • The UDF library binary (jar) release internal directory structure has changed. For v1.2.2 there was a directory structure and now in v1.3.3 the binaries are in a flat structure. This affects extracting the release archive in the CDSP IoTDB Dockerfile and modifying the registration script.
• Enabling REST API used by Grafana connector
  • The three config files in 1.2.x were merged into by commit 9b2efea9a2fcd0e941a4 into the single file iotdb-system.properties. So need to edit (sed) that file instead. Also the file appears to no longer contain all properties, including the REST property, so it must now be added instead of modified.
• IoTDB online documentation
• Whilst some links are still valid, others such as the SQL reference manual have moved in the tree and therefore require updates.

[slawr]

CDSP Docker updates

Completed CDSP Docker updates so upstream IoTDB v1.3.3 pre-built image is used and successfully starts.

Check IoTDB connection in VISSR

Confirmed VISSR can still connect and get/set data. Pre-populated some DB data and used VISSR HTML client to connect and get/set an actuator and get Vehicle.Speed. No changes necessary.

Check IoTDB connection in RemotiveLabs bridge

Used existing bridge/feeder to stream Nightdrive data into the DB using the bridge readme example. No changes necessary.

Grafana connections

Used the Vehicle.Speed down-sample example Grafana section as a sanity test. This exposed that the IoTDB config behaviour has changed requiring a different method to enable the REST API. Previously all config entries were present, now they are not. So a 'sed' to change false to true fails. Now the entry to enable it must be added. Asked upstream what the correct way to do this is, but for now simply adding the entry to make it work in the DockerFile. With that Grafana can connect to IoTDB and perform queries.

[slawr]

Completed related pull request. Waiting for review comments.