Update to Apache IoTDB v1.3.3 + Docker improvements

docker/README.md

@@ -50,13 +50,6 @@ There is a current issue with the upstream VISSR VISS Server Dockerfile in which
 
 If your project requires Access Grant support please discuss enabling it with the VISSR community.
 
-### Generate `vss_vissv2.binary`
-The VISSR server component requires a file called `vss_vissv2.binary` to understand the VSS tree it must work with. Unfortunately, VISSR provides no default file and you must therefore generate it yourself.
-
-Instructions for doing that can be found in the VISSR documentation site [here](https://covesa.github.io/vissr/server/#vss-tree-configuration)
-
-Tip: The playground maintainers have found that the method involving running `make binary` in a git clone of the VSS source tree to generate the file is straight forward. Note: check the VSS readme for the python requirements for the tooling.
-
 ### Tip: Corporate CA security (download error "tls: failed to verify certificate:")
 If you are working behind a corporate security system that places a _man-in-the-middle_ between your host and the internet you may see security errors when artifacts are downloaded as part of the build process.
 
@@ -130,10 +123,10 @@ $ sudo docker compose -f docker-compose-cdsp.yml down
 Listing should show three running containers as shown below:
 ```
 $ sudo docker ps
-NAME            IMAGE                           COMMAND                  SERVICE         CREATED          STATUS          PORTS
-app_redis       redis                           "docker-entrypoint.s…"   redis           10 minutes ago   Up 10 minutes   6379/tcp
-iotdb-service   apache/iotdb:1.2.2-standalone   "/usr/bin/dumb-init …"   iotdb-service   10 minutes ago   Up 10 minutes   0.0.0.0:6667->6667/tcp, :::6667->6667/tcp
-vissv2server    cdsp-vissv2server               "/app/vissv2server -…"   vissv2server    10 minutes ago   Up 4 seconds    0.0.0.0:8081->8081/tcp, 0.0.0.0:8600->8600/tcp, 0.0.0.0:8887->8887/tcp, 127.0.0.1:8888->8888/tcp
+CONTAINER ID   IMAGE                COMMAND                  CREATED              STATUS              PORTS                                                                                              NAMES
+f43ed0c6ba0a   cdsp-iotdb-service   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute   0.0.0.0:6667->6667/tcp, :::6667->6667/tcp                                                          iotdb-service
+7829813bdcb8   cdsp-vissv2server    "/app/vissv2server -…"   5 days ago           Up 8 seconds        0.0.0.0:8081->8081/tcp, 0.0.0.0:8600->8600/tcp, 0.0.0.0:8887->8887/tcp, 127.0.0.1:8888->8888/tcp   vissv2server
+8e21a556e398   redis                "docker-entrypoint.s…"   5 days ago           Up 5 days           6379/tcp                                                                                           app_redis
 ```
 #### Apache IoTDB
 You can confirm the Apache IoTDB server is running by connecting to it with the IoTDB CLI client (_quit_ to exit the client):
@@ -172,5 +165,3 @@ The following example uses the javascript HTML client from `vissr/client/client-
   Server: readyState=3, status=200
   Server: {"data":{"dp":{"ts":"2024-01-10T14:56:48Z","value":"Data-not-found"},"path":"Vehicle.Speed"},"ts":"2024-01-10T14:56:48Z"}
 ```
-
-You can query what VSS nodes the server understands by asking for the VSS path list using the URL `http://localhost:8081/vsspathlist`. Entering that URL in your web browser will typically give you a graphical rendering of the JSON data returned.

docker/docker-compose-cdsp.yml

@@ -9,29 +9,26 @@ services:
     build:
       context: iotdb  # CDSP IoTDB image
       args:
-        iotdb_version: 1.2.2
-        service_hostname: iotdb-service
+        IOTDB_VERSION: 1.3.3
+        SERVICE_HOSTNAME: iotdb-service
     restart: on-failure:3
     ports:
       - "6667:6667"
     environment:
       - cn_internal_address=iotdb-service
       - cn_internal_port=10710
       - cn_consensus_port=10720
-#      - cn_seed_config_node=iotdb-service:10710
-      - cn_target_config_node_list=iotdb-service:10710
+      - cn_seed_config_node=iotdb-service:10710
       - dn_rpc_address=iotdb-service
       - dn_internal_address=iotdb-service
       - dn_rpc_port=6667
       - dn_mpp_data_exchange_port=10740
       - dn_schema_region_consensus_port=10750
       - dn_data_region_consensus_port=10760
       - dn_seed_config_node=iotdb-service:10710
-#      - dn_seed_config_node=iotdb-service:10710
-      - dn_target_config_node_list=iotdb-service:10710
     volumes:
-        - ./data:/iotdb/data
-        - ./logs:/iotdb/logs
+        - ./iotdb-data:/iotdb/data
+        - ./iotdb-logs:/iotdb/logs
 
 # VISSR acting as VISS Data Server
 
@@ -73,6 +70,6 @@ services:
       - "0.0.0.0:8887:8887"
       - "0.0.0.0:8600:8600"
     volumes:
-      - ./logs:/app/logs
+      - ./vissr-logs:/app/logs
     volumes_from:
       - tmp

docker/iotdb/Dockerfile

@@ -4,7 +4,7 @@
 # Dockerfile for the CDSP build of Apache IoTDB.
 # ---------------------------------------------------------------------------
 
-ARG IOTDB_VERSION=1.2.2
+ARG IOTDB_VERSION=1.3.3
 ARG IOTDB_IMAGE=apache/iotdb:${IOTDB_VERSION}-standalone
 
 FROM ${IOTDB_IMAGE} AS cdsp-extras
@@ -15,9 +15,9 @@ ARG IOTDB_IMAGE
 # UDF Data Quality library
 ARG UDF_ARCHIVE_NAME=apache-iotdb-${IOTDB_VERSION}-library-udf-bin.zip
 ARG UDF_ARCHIVE_URL=https://archive.apache.org/dist/iotdb/${IOTDB_VERSION}/${UDF_ARCHIVE_NAME}
-ARG UDF_LIB_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/ext/udf/library-udf.jar
+ARG UDF_LIB_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/library-udf.jar
 ARG UDF_REG_SCRIPT_NAME=register-UDF.sh
-ARG UDF_REG_SCRIPT_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/tools/${UDF_REG_SCRIPT_NAME}
+ARG UDF_REG_SCRIPT_PATH=apache-iotdb-${IOTDB_VERSION}-library-udf-bin/${UDF_REG_SCRIPT_NAME}
 ARG SERVICE_HOSTNAME=iotdb-service
 
 # General configuration
@@ -35,7 +35,7 @@ RUN unzip -j -o ${UDF_ARCHIVE_NAME} ${UDF_LIB_PATH} ${UDF_REG_SCRIPT_PATH} \
 
 WORKDIR ${IOTDB_HOME}/conf
 # Enable IoTDB REST API as it is used by the IoTDB Grafana Connector for queries
-RUN sed -i 's/^# enable_rest_service=false/enable_rest_service=true/g' iotdb-common.properties
+RUN echo "enable_rest_service=true" >> iotdb-system.properties
 
 # Set the entry point path to the IoTDB sbin directory where the IoTDB admin scripts are.
 # This mimics the upstream image.

docs/docs-gen/content/examples/_index.md

@@ -35,25 +35,32 @@ Topic examples: Virtual signal platforms, VISSR etc.
 |------|-------------|
 | [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example bridge that streams vehicle data from the RemotiveLabs cloud platform into the IoTDB data store |
 
-
 ## COVESA Touchpoints
 Topic examples: Low level vehicle abstraction, Mobile devices, Car2Cloud / Cloud etc. 
 
 ## COVESA Technologies
 Topic examples: vsome/ip (SOME/IP), uServices, Vehicle API, VISS etc.
 
+| Name | Relationship to the category |
+|------|-------------|
+| [VISSR/VISS hello-world](https://covesa.github.io/vissr/build-system/hello-world/) | hello-world tutorial for making VISS requests using VISSR in the upstream VISSR project |
+| [VISS transport examples](https://raw.githack.com/COVESA/vehicle-information-service-specification/main/spec/VISSv3.0_TransportExamples.html) | Examples in the VISS Specification for making VISS requests and the responses for various transport protocols |
+
 ## Databases
 Topic examples: Apache IoTDB, MongoDB Realm, Redis/SQLite/memcache etc.
 
- Name | Relationship to the category |
+| Name | Relationship to the category |
 |------|-------------|
 | [vehicle-speed-downsample-iotdb](https://github.com/COVESA/cdsp/tree/main/examples/vehicle-speed-downsample-iotdb) | Using the IoTDB [Data Quality Library ]({{< ref "apache-iotdb#data-processing-functions" >}} "IoTDB Data Quality Library") for advanced (VSS) timeseries data processing|
-| [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example of streaming (writing) southbound (VSS) timeseries data into IoTDB|
-
+| [RemotiveLabs feeder](https://github.com/COVESA/cdsp/tree/main/examples/remotivelabs-feeder) | Example of streaming (writing) southbound (VSS) timeseries data into IoTDB |
 
 ## Frameworks / Protocols
 Topic examples: vsome/ip (SOME/IP), VISS, uServices/uProtocol/Capabilities, Vehicle API, MQTT, Kafka, Apache Zeppelin etc.
 
+| Name | Relationship to the category |
+|------|-------------|
+| VISS | Find examples in the COVESA Technologies section and or search for VISS/VISSR in this page |
+
 ## Big data
 Topic examples: Hadoop, Flink, Spark, Cloud DB,  Nifi etc.
 

docs/docs-gen/content/manuals/apache-iotdb.md

@@ -14,11 +14,11 @@ The IoTDB project [website](https://iotdb.apache.org/) has extensive documentati
 Info: The intention is to add more information such as a guide for feeder integration as the project progresses.
 
 ## Integrating VSS data into the IoTDB data model
-The ["Basic Concept"](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Data-Model-and-Terminology.html) section of the IoTDB documentation introduces the IoTDB data model, data types, encoding and compression.
+The ["Basic Functions"](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Data-Model-and-Terminology.html) section of the IoTDB documentation introduces the IoTDB data model, data types, encoding and compression.
 
 In IoTDB terminology *measurement* is the key in a key/value pair. In VSS terms the leaf node name. The timeseries is the record of the measurement on the time axis. A timeseries is a series of time/value data points.
 
-The IoTDB data model supports hierarchical partitioning and like VSS uses a dot notation to separate the levels. This means if we simply appended a VSS leaf node name like `Vehicle.CurrentLocation.Longitude` as the measurement (key) name to the end of a IoTDB path such as `root.test2.dev1` the `Vehicle.CurrentLocation.` IoTDB would treat it as part of the IoTDB data model partitioning which could cause unwanted issues when scaling over millions of vehicles.
+The IoTDB data model supports hierarchical partitioning and like VSS uses a dot notation to separate the levels. This means if we simply appended a VSS leaf node name like `Vehicle.CurrentLocation.Longitude` as the measurement (key) name to the end of a IoTDB path such as `root.test2.dev1` IoTDB would treat the VSS path prefix `Vehicle.CurrentLocation.` as part of the IoTDB data model partitioning which could cause unwanted issues when scaling over millions of vehicles.
 
 We have separated those two concepts by quoting the VSS leaf node name using backticks when processing the name in IoTDB. As shown below:
 ```
@@ -127,19 +127,20 @@ IoTDB has an extensive collection of built-in data processing functions covering
 - Sampling Functions, such as `M4`.
 - Change Points Function, such as `CHANGE_POINTS`.
 
-A full function list with examples can be found in the upstream [IoTDB Function reference manual](https://iotdb.apache.org/UserGuide/latest/Reference/Function-and-Expression.html).
+A full function list with examples can be found in the upstream [IoTDB Function reference manual](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/Function-and-Expression.html).
 
 ### Data Quality Library functions
 The IoTDB project also maintains a Data Quality Library which provides an additional collection of functions covering:
-- Data Quality, such as `Accuracy`.
+- Data Quality, such as `Completeness`.
 - Data Profiling, such as `Sample`.
 - Anomaly Detection, such as `Outlier`.
 - Frequency Domain Analysis, such as `HighPass`.
+- Data Matching, such as `Cov`.
 - Data Repair, such as `TimestampRepair`.
 - Series Discovery, such as `ConsecutiveSequences`.
 - Machine Learning, such as `AR`.
 
-A full function list with examples can be found in the upstream [IoTDB Data Quality Library reference manual](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html).
+A full function list with examples can be found in the upstream [IoTDB UDF Data Quality Library reference manual](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html).
 
 #### Setup
 In the upstream IoTDB project the library is an optional install.
@@ -156,7 +157,7 @@ Steps:
 
 
 ### User Defined functions
-IoTDB also allows you intergrate your own functions as User Defined Functions (UDF). The [UDF section](https://iotdb.apache.org/UserGuide/latest/User-Manual/Database-Programming.html#user-defined-function-udf) of the IoTDB documentation explains how to develop and register your own.
+IoTDB also allows you to integrate your own functions as User Defined Functions (UDF). The [UDF development section](https://iotdb.apache.org/UserGuide/latest/User-Manual/UDF-development.html) of the IoTDB documentation explains how to develop and register your own.
 
 ## VISSR (VISS) integration
 As part of the initial development of the playground the team extended VISSR to support connections to Apache IoTDB as a VISSR data store backend and upstreamed the support.

examples/cdsp-hello-world/README.md

@@ -1,9 +1,14 @@
 # Central Data Service Playground Hello World
+
+> [!NOTE]
+> Apologies. As you can see the team has been busy adding features and not got around to finishing this example yet. In the meantime, please take a look at the deployment section of the [Docker README.md](https://github.com/COVESA/cdsp/blob/main/docker/README.md#deploy-with-docker-compose) which gives guidance on starting the Playground and sanity testing it is running correctly.
+
 This is a simple "hello-world" example for the Central Data Service Playground (CDSP).
 
 Goals:
 1. Act as a sanity test that the Playground is running correctly
 2. Provide a simplistic example of VSS and VISS use in the Playground.
+
 ## Logical/Implementation Concept
 This example does not expand the concept of the Playground. Therefore please see the Playground documentation for a description of the Logical and Implementation concepts behind it.
 

examples/vehicle-speed-downsample-iotdb/README.md

@@ -35,10 +35,10 @@ The dataset contains just over 13500 values recorded over approximately 5 minute
 ### Load
 In the example we will import the dataset into a VSS `Vehicle.Speed` timeseries within IoTDB.
 
-IoTDB provides [import/export tools](https://iotdb.apache.org/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.html) for its native TsFile file format and CSV and which are included in the image. We will use the [`import-csv.sh`](https://iotdb.apache.org/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.html#usage-of-import-csv-sh) tool to perform the import.
+IoTDB provides import/export tools for its native TsFile file format, SQL and CSV and which are included in the runtime image. We will use the [`import-data.sh`](https://iotdb.apache.org/UserGuide/latest/Tools-System/Data-Import-Tool.html) tool to perform the import.
 
 ### Transform
-IoTDB has a library of Data Quality functions which includes the function [`Sample`](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html#sample) for sampling. Sample has three sampling methods: `Reservoir`, `Isometric` and `Triangle`. 
+IoTDB has a library of Data Quality functions which includes the function [`Sample`](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html#sample) for sampling. Sample has three sampling methods: `Reservoir`, `Isometric` and `Triangle`. 
 
 Sample function:
 
@@ -64,7 +64,7 @@ The following screenshot shows the results graphed in Grafana. The green graph i
 *Figure 1: Grafana visualisation of the results. Key: Green=input data, yellow=down-sampled result*
 
 Tips:
-+ Documentation for each Sample method and the other functions can be found in the [IoTDB library documentation](https://iotdb.apache.org/UserGuide/latest/Reference/UDF-Libraries.html).
++ Documentation for each Sample method and the other functions can be found in the [IoTDB library documentation](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/UDF-Libraries_apache.html).
 
 + Triangle uses a *Largest-Triangle-Three-Buckets (LTTB)* algorithm to calculate the output timeseries. Details of which can be found in the originating academic research: [Downsampling Time Series for Visual Representation, Sveinn Steinarsson, 2013](https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf)
 
@@ -106,6 +106,8 @@ $ sudo docker exec -ti iotdb-service /iotdb/sbin/register-UDF.sh
 
 Info: See the online documentation site for [details](https://covesa.github.io/cdsp/manuals/apache-iotdb/#setup)
 
+Tip: If for some reason you are failing to register the Data Quality Library, substitute one of the [built-in sample functions](https://iotdb.apache.org/UserGuide/latest/SQL-Manual/Operator-and-Expression.html#sample-functions) rather than `Sample()` in the SQL below.
+
 ### Import the dataset
 
 1. Create the database in IoTDB into which we will import
@@ -120,19 +122,19 @@ Info: See the online documentation site for [details](https://covesa.github.io/c
 
     2.1. Copy the dataset file to a volume visible within the IoTDB image:
     ~~~
-    $ cp vehicle_speed_rl_dataset.csv ../../docker/data
+    $ sudo cp vehicle_speed_rl_dataset.csv ../../docker/iotdb-data
     ~~~
 
     2.2. Import the dataset
 
-    To import the dataset into IoTDB we need to execute the `import-csv.sh` tool in the IoTDB image. That can be done from the host terminal using the docker `exec` command.
+    To import the dataset into IoTDB we need to execute the `import-data.sh` tool in the IoTDB image. That can be done from the host terminal using the docker `exec` command.
     ~~~
-    $ docker exec -ti iotdb-service /iotdb/tools/import-csv.sh -h iotdb-service -p 6667 -u root -pw root -f /iotdb/data/vehicle_speed_rl_dataset.csv
+    $ sudo docker exec -ti iotdb-service /iotdb/tools/import-data.sh -h iotdb-service -p 6667 -u root -pw root -s /iotdb/data/vehicle_speed_rl_dataset.csv
     ~~~
 
     Example successful execution:
     ~~~
-    $ docker exec -ti iotdb-service /iotdb/tools/import-csv.sh -h iotdb-service -p 6667 -u root -pw root  -f /iotdb/data/vehicle_speed_rl_dataset.csv
+    $ sudo docker exec -ti iotdb-service /iotdb/tools/import-data.sh -h iotdb-service -p 6667 -u root -pw root -s /iotdb/data/vehicle_speed_rl_dataset.csv
     ------------------------------------------
     Starting IoTDB Client Import Script
     ------------------------------------------
@@ -325,7 +327,7 @@ It costs 0.028s
 
 ### Suggested next steps
 + Repeat the query with the same input data, but with different `k` parameter values to see what affects the number of samples has on the accuracy of the trace compared to the input data.
-+ Amend the query with other [SQL clauses](https://iotdb.apache.org/UserGuide/latest/User-Manual/Query-Data.html) to shape what you are interested in, e.g. use `WHERE` to define a time filter.
++ Amend the query with other [SQL clauses](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Query-Data.html) to shape what you are interested in, e.g. use `WHERE` to define a time filter.
 + A great way to explore these sampling queries is by using Grafana as discussed below.
 + Try the other [IoTDB data processing functions](https://covesa.github.io/cdsp/manuals/apache-iotdb/#data-processing-functions)