add const UID attribute to allow constant IDs (#352)

* add const UID attribute to allow constant IDs --------- Signed-off-by: Niclas Wesemann <[email protected]>
COVESA · May 17, 2024 · a30bb0a · a30bb0a
1 parent e96ed5d
commit a30bb0a
Show file tree

Hide file tree

Showing 8 changed files with 266 additions and 69 deletions.
diff --git a/docs/vspec2id.md b/docs/vspec2id.md
@@ -1,14 +1,15 @@
 # vspec2id - vspec static UID generator and validator
 
-The vspec2id.py script is used to generate and validate static UIDs for all nodes in the tree.
-They will be used as unique identifiers to transmit data between nodes. The static UIDs are
-implemented to replace long strings like `Vehicle.Body.Lights.DirectionIndicator.Right.IsSignaling`
-with a 4-byte identifier.
+The `vspec2id.py` script is used to generate and validate static UIDs for all
+nodes in the tree. These static UIDs serve as unique identifiers for
+transmitting data between nodes. They are designed to replace long strings like
+`Vehicle.Body.Lights.DirectionIndicator.Right.IsSignaling` with compact 4-byte
+identifiers.
 
 ## General usage
 
 ```bash
-usage: vspec2id.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style] [--format format] [--uuid] [--no-expand] [-o overlays] [-u unit_file]
+usage: ./vspec2id.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style] [--format format] [--uuid] [--no-expand] [-o overlays] [-u unit_file]
                    [-q quantity_file] [-vt vspec_types_file] [-ot <types_output_file>] [--yaml-all-extended-attributes] [-v version] [--all-idl-features]
                    [--validate-static-uid VALIDATE_STATIC_UID] [--only-validate-no-export] [--strict-mode]
                    <vspec_file> <output_file>
@@ -33,59 +34,124 @@ IDGEN arguments:
 ## Example
 
 To initially run this you will need a vehicle signal specification, e.g.
-[COVESA Vehicle Signal Specification](https://github.com/COVESA/vehicle_signal_specification). If you are just starting
-to use static UIDs the first run is simple. You will only use the static UID generator by running the command below.
+[COVESA Vehicle Signal Specification](https://github.com/COVESA/vehicle_signal_specification).
+If you are just starting to use static UIDs the first run is simple. You will
+only use the static UID generator by running the command below. Please note that
+if you want to use any overlays, now is the time to do so:
 
 ```bash
 cd path/to/your/vss-tools
 ./vspec2id.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec ../output_id_v1.vspec
+# or if you are using an overlay e.g. called overlay.vspec
+./vspec2id.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec ../output_id_v1.vspec -o overlay.vspec
 ```
 
-Great, you generated your first overlay that will also be used as your validation file as soon as you update your
-vehicle signal specification file.
+Great, you generated your first vspec including static IDs that will also be
+used as your validation file as soon as you update your vehicle signal
+specification or your overlay.
 
-If needed you can make the static UID generation case-sensitive using the command line argument `--strict-mode`. It
-will default to false.
+If needed you can make the static UID generation case-sensitive using the
+command line argument `--strict-mode`. It will default to false.
 
 ### Generate e.g. yaml file with static UIDs
 
-Now if you just want to generate a new e.g. yaml file including your static UIDs, please use the overlay function of
-vspec2x by running the following command:
+Now if you just want to generate a new e.g. yaml file including your static
+UIDs, please use the corresponding exporter (here we will use
+`./vspec2yaml.py`). Please note that if you are outside of the spec folder in
+the vehicle specification you will have to specify the path to the units.yaml
+using `-u`.
 
 ```bash
 cd path/to/your/vss-tools
-./vspec2yaml.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec -o ../output_id_v1.vspec -e staticUID vehicle_specification_with_uids.yaml
+./vspec2yaml.py output_id_v1.vspec vehicle_specification_with_uids.yaml -e staticUID -u ../vehicle_signal_specification/spec/units.yaml
 ```
 
+### Using constant UIDs for specific attributes
+
+In case you want to use a specific predefined 4-byte hex ID for an attribute in
+your tree, you can define a `constUID` in your overlay which will be used
+instead of the generated static UID. Let's say you want to use a constant ID
+`0x00112233` for the signal if the driver is wearing a seatbelt which looks like
+this
+
+```yaml
+Seat:
+  type: branch
+  instances:
+    - Row[1,2]
+    - ["DriverSide","Middle","PassengerSide"]
+  description: All seats.
+#include SingleSeat.vspec Seat
+
+```
+
+In the overlay define the one you would like to give a constant ID to and define
+a new attribute called `constUID`:
+
+```yaml
+Vehicle.Cabin.Seat.Row1.DriverSide.IsBelted:
+  datatype: boolean
+  constUID: '0x00112233'
+  type: sensor
+```
+
+Let's say the snippet above is a file called `const_id_overlay.vspec`, you could
+run the vspec2id.py like this:
+
+```bash
+./vspec2id.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec const_test.vspec -o const_overlay.vspec
+```
+
+which will give you the following `INFO` msg and write the defined constant ID
+as the static ID in your generated vspec.
+
+```bash
+...
+INFO     Calling exporter...
+INFO     Generating YAML output...
+INFO     Using const ID for Vehicle.Cabin.Seat.Row1.DriverSide.IsBelted. If you didn't mean to do that you can remove it in your vspec / overlay.
+INFO     All done.
+```
+
+This works for all types of signal even enums (as shown above). Please note that
+if you use a constant UID, you will not be able to do proper validation on the
+signal. Validation will be further explained in the next section.
+
 ### Validation
 
-In this case you want to validate changes of your vehicle specification. If you are doing a dry run try temporarily
-renaming a node or changing the node's datatype, unit, description, or other. You will get warnings depending on your
+In this case you want to validate changes of your vehicle specification. If you
+are doing a dry run try temporarily renaming a node or changing the node's
+datatype, unit, description, or other. You will get warnings depending on your
 changes in the vehicle signal specification.
 
-The validation step compares your current changes of the vehicle signal specification to a previously generated file,
-here we named it `../output_id_v1.vspec`. There are two types of changes `BREAKING CHANGES` and `NON-BREAKING CHANGES`.
-A `BREAKING CHANGE` will generate a new hash for a node. A `NON-BREAKING CHANGE` will throw a warning, but the static
-ID will remain the same. A `BREAKING CHANGE` is triggered when you change name/path, unit, type, datatype, enum values
-(allowed), or minimum/maximum. These attributes are part of the hash so they a `BREAKING CHANGE` automatically
-generates a new hash for a static UID.
-In case you want to keep the same ID but rename a node, this we call a `SEMANTIC CHANGE`, you can add an attribute
-called `fka` in the vspec (which is a list of strings) and add a list of names to it as shown below for `A.B.NewName`.
-The same holds for path changes, if you move a node between layers you can add the `fka` attribute containing the
-full path as shown below.
+The validation step compares your current changes of the vehicle signal
+specification to a previously generated file, here we named it
+`../output_id_v1.vspec`. There are two types of changes `BREAKING CHANGES` and
+`NON-BREAKING CHANGES`. A `BREAKING CHANGE` will generate a new hash for a node.
+A `NON-BREAKING CHANGE` will throw a warning, but the static ID will remain the
+same. A `BREAKING CHANGE` is triggered when you change name/path, unit, type,
+datatype, enum values (allowed), or minimum/maximum. These attributes are part
+of the hash so they a `BREAKING CHANGE` automatically generates a new hash for a
+static UID. In case you want to keep the same ID but rename a node, this we call
+a `SEMANTIC CHANGE`, you can add an attribute called `fka` in the vspec (which
+is a list of strings) and add a list of names to it as shown below for
+`A.B.NewName`. The same holds for path changes, if you move a node between
+layers you can add the `fka` attribute containing the full path as shown below.
 
 Before renaming `A.B.NewName` its name was `A.B.OldName`.
 
-```
+```yaml
 A.B.NewName:
   datatype: string
   type: actuator
   allowed: ["YES", "NO"]
   description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'.
   fka: ['A.B.OlderName', 'A.B.OldName']
 ```
+
 or
-```
+
+```yaml
 A.B.NewName:
   datatype: string
   type: actuator
@@ -94,40 +160,45 @@ A.B.NewName:
   fka: A.B.OlderName
 ```
 
-In order to add fka attribute, one can add fka directly into the vspec file or use overlay feature of vss-tools.
+In order to add fka attribute, one can add fka directly into the vspec file or
+use overlay feature of vss-tools.
 
 Example mycustom-overlay-fka.vspec
-```
+
+```yaml
 A.B.NewName:
   datatype: string
   type: actuator
   fka: A.B.OlderName
 ```
-As stated if you want to rename the node `A.B.NewName` to `A.NewName` you can also write the Formerly Known As `fka` attribute stating its legacy path. For hashing function in previous case `A.B.OlderName` will be used.
 
-To summarize these are the `BREAKING CHANGES` that affect the hash and `NON-BREAKING CHANGES` that throw
-warnings only:
+As stated if you want to rename the node `A.B.NewName` to `A.NewName` you can
+also write the Formerly Known As `fka` attribute stating its legacy path. For
+hashing function in previous case `A.B.OlderName` will be used.
+
+To summarize these are the `BREAKING CHANGES` that affect the hash and
+`NON-BREAKING CHANGES` that throw warnings only:
 
-| BREAKING CHANGES      |     | NON-BREAKING CHANGES |
-|-----------------------|-----|----------------------|
-| Qualified name        |     | Added attribute      |
-| Data type             |     | Deprecation          |
-| Type (i.e. node type) |     | Deleted Attribute    |
-| Unit                  |     | Change description   |
-| Enum values (allowed) |     | Qualified name (fka) |
-| Minimum               |     |                      |
-| Maximum               |     |                      |
+| BREAKING CHANGES | | NON-BREAKING CHANGES |
+|-----------------------|-----|----------------------| | Qualified name | |
+Added attribute | | Data type | | Deprecation | | Type (i.e. node type) | |
+Deleted Attribute | | Unit | | Change description | | Enum values (allowed) | |
+Qualified name (fka) | | Minimum | | | | Maximum | | |
 
-Now you should know about all possible changes. To run the validation step, please do:
+Now you should know about all possible changes. To run the validation step,
+please do:
 
 ```bash
 ./vspec2id.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec ../output_id_v2.vspec --validate-static-uid ../output_id_v1.vspec
 ```
 
-Depending on what you changed in the vehicle signal specification the corresponding errors will be triggered.
+Depending on what you changed in the vehicle signal specification the
+corresponding errors will be triggered.
 
-Now, if the warning logs correspond to what you have changed since the last validation, you can continue to generate
-e.g. a yaml file with your validated changes as described in the `Generate e.g. yaml file with static UIDs` step above.
+Now, if the warning logs correspond to what you have changed since the last
+validation, you can continue to generate e.g. a yaml file with your validated
+changes as described in the `Generate e.g. yaml file with static UIDs` step
+above.
 
 ### Tests
 
@@ -139,5 +210,5 @@ export PYTHONPATH=${PWD}
 pytest tests/vspec/test_static_uids
 ```
 
-Depending on how you are using the implementation you might have to activate your virtual environment as described on
-the top README.
+Depending on how you are using the implementation you might have to activate
+your virtual environment as described on the top README.
diff --git a/tests/vspec/test_static_uids/test_static_uids.py b/tests/vspec/test_static_uids/test_static_uids.py
@@ -27,14 +27,25 @@
 # HELPERS
 
 
-def get_cla_test(test_file: str):
-    return (
-        "../../../vspec2id.py "
-        + test_file
-        + " ./out.vspec --validate-static-uid "
-        + "./validation_vspecs/validation.vspec "
-        + "--only-validate-no-export"
-    )
+def get_cla_test(test_file: str, overlay: str | None = None):
+    if overlay:
+        return (
+            "../../../vspec2id.py "
+            + test_file
+            + " -o "
+            + overlay
+            + " ./out.vspec --validate-static-uid "
+            + "./validation_vspecs/validation.vspec "
+            + "-u ./test_vspecs/units.yaml"
+        )
+    else:
+        return (
+            "../../../vspec2id.py "
+            + test_file
+            + " ./out.vspec --validate-static-uid "
+            + "./validation_vspecs/validation.vspec "
+            + "-u ./test_vspecs/units.yaml"
+        )
 
 
 def get_cla_validation(validation_file: str):
@@ -77,6 +88,12 @@ def change_test_dir(request, monkeypatch):
     monkeypatch.chdir(request.fspath.dirname)
 
 
+@pytest.fixture(scope="function", autouse=True)
+def delete_files(change_test_dir):
+    yield None
+    os.system("rm -f out.vspec")
+
+
 # UNIT TESTS
 
 
@@ -329,6 +346,10 @@ def test_added_attribute(caplog: pytest.LogCaptureFixture):
     for record in caplog.records:
         assert "ADDED ATTRIBUTE" in record.msg
 
+    result = yaml.load(open("./out.vspec"), Loader=yaml.FullLoader)
+    keys: list = [key for key, _ in get_all_keys_values(result)]
+    assert "A.B.NewNode" in keys
+
 
 @pytest.mark.usefixtures("change_test_dir")
 def test_deleted_attribute(caplog: pytest.LogCaptureFixture):
@@ -341,3 +362,58 @@ def test_deleted_attribute(caplog: pytest.LogCaptureFixture):
     )
     for record in caplog.records:
         assert "DELETED ATTRIBUTE" in record.msg
+
+    result = yaml.load(open("./out.vspec"), Loader=yaml.FullLoader)
+    keys: list = [key for key, _ in get_all_keys_values(result)]
+    assert "A.B.Max" not in keys
+
+
+@pytest.mark.usefixtures("change_test_dir")
+def test_overlay(caplog: pytest.LogCaptureFixture):
+    test_file: str = "./test_vspecs/test.vspec"
+    overlay: str = "./test_vspecs/test_overlay.vspec"
+    clas = shlex.split(get_cla_test(test_file, overlay))
+    vspec2id.main(clas[1:])
+
+    assert len(caplog.records) == 1 and all(
+        log.levelname == "WARNING" for log in caplog.records
+    )
+
+    for record in caplog.records:
+        assert "ADDED ATTRIBUTE" in record.msg
+
+    result = yaml.load(open("./out.vspec"), Loader=yaml.FullLoader)
+    keys: list = [key for key, _ in get_all_keys_values(result)]
+    assert "A.B.OverlayNode" in keys
+
+
+@pytest.mark.usefixtures("change_test_dir")
+def test_const_id(caplog: pytest.LogCaptureFixture):
+    test_file: str = "./test_vspecs/test.vspec"
+    overlay: str = "./test_vspecs/test_const_id.vspec"
+    clas = shlex.split(get_cla_test(test_file, overlay))
+    vspec2id.main(clas[1:])
+
+    result = yaml.load(open("./out.vspec"), Loader=yaml.FullLoader)
+    for key, value in get_all_keys_values(result):
+        if key == "A.B.Int32":
+            assert value["staticUID"] == "0x00112233"
+
+
+@pytest.mark.usefixtures("change_test_dir")
+def test_iterated_file(caplog: pytest.LogCaptureFixture):
+    test_file: str = "./test_vspecs/test.vspec"
+    clas = shlex.split(get_cla_test(test_file))
+    vspec2id.main(clas[1:])
+
+    with open("./out.vspec", "r") as f:
+        result = yaml.load(f, Loader=yaml.FullLoader)
+
+    # run again on out.vspec to check if it all hashed attributes were exported correctly
+    clas = shlex.split(get_cla_test("./out.vspec"))
+    vspec2id.main(clas[1:])
+
+    with open("./out.vspec", "r") as f:
+        result_iteration = yaml.load(f, Loader=yaml.FullLoader)
+
+    assert result == result_iteration
diff --git a/tests/vspec/test_static_uids/test_vspecs/test_added_attribute.vspec b/tests/vspec/test_static_uids/test_vspecs/test_added_attribute.vspec
@@ -5,6 +5,7 @@
 # https://www.mozilla.org/en-US/MPL/2.0/
 #
 # SPDX-License-Identifier: MPL-2.0
+
 A:
   type: branch
   description: A is a test node

diff --git a/tests/vspec/test_static_uids/test_vspecs/test_const_id.vspec b/tests/vspec/test_static_uids/test_vspecs/test_const_id.vspec
@@ -0,0 +1,12 @@
+# Copyright (c) 2024 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+A.B.Int32:
+  datatype: int32
+  type: sensor
+  constUID: '0x00112233'
diff --git a/tests/vspec/test_static_uids/test_vspecs/test_overlay.vspec b/tests/vspec/test_static_uids/test_vspecs/test_overlay.vspec
@@ -0,0 +1,12 @@
+# Copyright (c) 2024 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+A.B.OverlayNode:
+  datatype: int32
+  type: sensor
+  description: This is the description of the overlay node.
diff --git a/tests/vspec/test_static_uids/validation.yaml b/tests/vspec/test_static_uids/validation.yaml
@@ -23,6 +23,7 @@ A.B.IsLeaf:
 A.B.Max:
   datatype: uint8
   description: A leaf that uses a maximum value.
+  min: 0
   max: 100
   staticUID: '0xCF708DCC'
   type: sensor