diff --git a/README.md b/README.md index 98eff1de..67d74be2 100644 --- a/README.md +++ b/README.md @@ -23,20 +23,21 @@ Tools in the Obsolete Tools category may be deleted after 12 months. Examples on tool usage can be found in the [VSS Makefile](https://github.com/COVESA/vehicle_signal_specification/blob/master/Makefile) and in tool-specific documentation, if existing. - Tool | Description | Tool Category | Documentation | -| ------------------ | ----------- | -------------------- |-------------------- | -| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl` | Community Supported | Try `./vspec2x --help` or check [vspec2x documentation](docs/vspec2x.md) | -[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | -[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) | -[vspec2json.py](vspec2json.py) | Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | -[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | -[vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported | [vspec2binary Documentation](binary/README.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) | -[vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | -[vspec2c.py](obsolete/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Obsolete (2022-11-01) | [Documentation](obsolete/vspec2c/README.md) | -[vspec2ocf.py](obsolete/ocf/vspec2ocf.py) | Parses and expands a VSS and generates a OCF specification | Obsolete (2022-11-01) | - | -[vspec2protobuf.py](vspec2protobuf.py) | Parses and expands a VSS and generates a Protobuf specification | Contrib | - | -[vspec2ttl.py](contrib/vspec2ttl/vspec2ttl.py) | Parses and expands a VSS and generates a TTL specification | Contrib | - | -[vspec2graphql.py](vspec2graphql.py) | Parses and expands a VSS and generates a GraphQL specification | Community Supported | [Documentation](docs/VSS2GRAPHQL.md) | + Tool | Description | Tool Category | Documentation | +| ------------------ | ----------- |-----------------------|-----------------------------------------------------------------------------------------------------------------------| +| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl` | Community Supported | Try `./vspec2x --help` or check [vspec2x documentation](docs/vspec2x.md) | +[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | +[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) | +[vspec2json.py](vspec2json.py) | Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | +[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | +[vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported | [vspec2binary Documentation](binary/README.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) | +[vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) | +[vspec2c.py](obsolete/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Obsolete (2022-11-01) | [Documentation](obsolete/vspec2c/README.md) | +[vspec2ocf.py](obsolete/ocf/vspec2ocf.py) | Parses and expands a VSS and generates a OCF specification | Obsolete (2022-11-01) | - | +[vspec2protobuf.py](vspec2protobuf.py) | Parses and expands a VSS and generates a Protobuf specification | Contrib | - | +[vspec2ttl.py](contrib/vspec2ttl/vspec2ttl.py) | Parses and expands a VSS and generates a TTL specification | Contrib | - | +[vspec2graphql.py](vspec2graphql.py) | Parses and expands a VSS and generates a GraphQL specification | Community Supported | [Documentation](docs/VSS2GRAPHQL.md) | +[vspec2id.py](vspec2id.py) | Generates and validates static UIDs for a VSS | WIP | [vspec2id Documentation](./docs/vspec2id.md) | ## Tool Architecture diff --git a/docs/vspec2id.md b/docs/vspec2id.md new file mode 100644 index 00000000..0de0592d --- /dev/null +++ b/docs/vspec2id.md @@ -0,0 +1,122 @@ +# vspec2id - vspec static UID generator and validator + +The vspecID.py script is used to generate and validate static UIDs for all nodes in the tree. +They will be used as unique identifiers to transmit data between nodes. The static UIDs are +implemented to replace long strings like `Vehicle.Body.Lights.DirectionIndicator.Right.IsSignaling` +with a 4-byte identifier. + +## General usage + +```bash +usage: vspec2id.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style] [--format format] [--uuid] [--no-expand] [-o overlays] [-u unit_file] + [-vt vspec_types_file] [-ot ] [--json-all-extended-attributes] [--json-pretty] [--yaml-all-extended-attributes] [-v version] [--all-idl-features] + [--gqlfield GQLFIELD GQLFIELD] [--validate-static-uid VALIDATE_STATIC_UID] [--only-validate-no-export] + + +Convert vspec to other formats. + +positional arguments: + The vehicle specification file to convert. + The file to write output to. + +... + +IDGEN arguments: + + --validate-static-uid VALIDATE_STATIC_UID + Path to validation file. + --only-validate-no-export + For pytests and pipelines you can skip the export of the +``` + +## Example + +To initially run this you will need a vehicle signal specification, e.g. +[COVESA Vehicle Signal Specification](https://github.com/COVESA/vehicle_signal_specification). If you are just starting +to use static UIDs the first run is simple. You will only use the static UID generator by running the command below. + +```bash +cd path/to/your/vss-tools +./vspec2id.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec ../output_id_v1.vspec +``` + +Great, you generated your first overlay that will also be used as your validation file as soon as you update your +vehicle signal specification file. + +### Generate e.g. yaml file with static UIDs + +Now if you just want to generate a new e.g. yaml file including your static UIDs, please use the overlay function of +vspec2x by running the following command: + +```bash +cd path/to/your/vss-tools +./vspec2yaml.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec -o ../output_id_v1.vspec -e staticUID vehicle_specification_with_uids.yaml +``` + +### Validation + +In this case you want to validate changes of your vehicle specification. If you are doing a dry run try temporarily +renaming a node or changing the node's datatype, unit, description, or other. You will get warnings depending on your +changes in the vehicle signal specification. + +The validation step compares your current changes of the vehicle signal specification to a previously generated file, +here we named it `../output_id_v1.vspec`. There are two types of changes `BREAKING CHANGES` and `NON-BREAKING CHANGES`. +A `BREAKING CHANGE` will generate a new hash for a node. A `NON-BREAKING CHANGE` will throw a warning, but the static +ID will remain the same. A `BREAKING CHANGE` is triggered when you change name/path, unit, type, datatype, enum values +(allowed), or minimum/maximum. These attributes are part of the hash so they a `BREAKING CHANGE` automatically +generates a new hash for a static UID. +In case you want to keep the same ID but rename a node, this we call a `SEMANTIC CHANGE`, you can add an attribute +called `fka` in the vspec (which is a list of strings) and add a list of names to it as shown below for `A.B.NewName`. +The same holds for path changes, if you move a node between layers you can add the `fka` attribute containing the +full path as shown below. + +Before renaming `A.B.NewName` its name was `A.B.OldName`. + +``` +A.B.NewName: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +``` + +As stated if you want to rename the node `A.B.NewName` to `A.NewName` you can also write the `fka` attribute +stating its legacy path. + +To summarize these are the `BREAKING CHANGES` that affect the hash and `NON-BREAKING CHANGES` that throw +warnings only: + +| BREAKING CHANGES | | NON-BREAKING CHANGES | +|-----------------------|-----|----------------------| +| Qualified name | | Added attribute | +| Data type | | Deprecation | +| Type (i.e. node type) | | Deleted Attribute | +| Unit | | Change description | +| Enum values (allowed) | | | +| Minimum | | | +| Maximum | | | + +Now you should know about all possible changes. To run the validation step, please do: + +```bash +./vspecID.py ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec ../output_id_v2.vspec --validate-static-uid ../output_id_v1.vspec +``` + +Depending on what you changed in the vehicle signal specification the corresponding errors will be triggered. + +Now, if the warning logs correspond to what you have changed since the last validation, you can continue to generate +e.g. a yaml file with your validated changes as described in the `Generate e.g. yaml file with static UIDs` step above. + +### Tests + +If you want to run the tests for the vspec2id implementation, please do + +```bash +cd path/to/vss-tools +export PYTHONPATH=${PWD} +pytest tests/vspec/test_static_uids +``` + +Depending on how you are using the implementation you might have to activate your virtual environment as described on +the top README. diff --git a/tests/vspec/.gitignore b/tests/vspec/.gitignore index 2edf7d26..07e76579 100644 --- a/tests/vspec/.gitignore +++ b/tests/vspec/.gitignore @@ -1,3 +1,4 @@ # Ignore files produced during testing, if any **/out.json **/out.txt +**/out.vspec diff --git a/tests/vspec/test_static_uids/test_static_uids.py b/tests/vspec/test_static_uids/test_static_uids.py new file mode 100644 index 00000000..24dfd9c4 --- /dev/null +++ b/tests/vspec/test_static_uids/test_static_uids.py @@ -0,0 +1,295 @@ +#!/usr/bin/env python3 + +import os +import pytest +import shlex +import vspec +import vspec.vssexporters.vss2id as vss2id +import vspec2x +import yaml + +from typing import Dict +from vspec.model.constants import VSSTreeType, VSSDataType, VSSUnit +from vspec.model.vsstree import VSSNode +from vspec.utils.idgen_utils import get_all_keys_values + + +# HELPERS + + +def get_cla_test(test_file: str): + return ( + "../../../vspec2id.py " + + test_file + + " ./out.vspec --validate-static-uid " + + "./validation_vspecs/validation.vspec " + + "--only-validate-no-export" + ) + + +def get_cla_validation(validation_file: str): + return ( + "../../../vspec2id.py ./test_vspecs/test.vspec ./out.vspec " + "--validate-static-uid " + validation_file + ) + + +def get_test_node( + node_name: str, + unit: str, + datatype: VSSDataType, + allowed: str, + minimum: str, + maximum: str, +) -> VSSNode: + source = { + "description": "some desc", + "type": "branch", + "$file_name$": "testfile", + } + node = VSSNode(node_name, source, VSSTreeType.SIGNAL_TREE.available_types()) + node.unit = VSSUnit(unit) + node.data_type_str = datatype.value + node.validate_and_set_datatype() + node.allowed = allowed + node.min = minimum + node.max = maximum + + return node + + +# FIXTURES + + +@pytest.fixture +def change_test_dir(request, monkeypatch): + # To make sure we run from test directory + monkeypatch.chdir(request.fspath.dirname) + + +# UNIT TESTS + + +@pytest.mark.parametrize( + "node_name, unit, datatype, allowed, minimum, maximum, result_static_uid", + [ + ("TestNode", "m", VSSDataType.UINT16, "", 0, 10000, "156365B2"), + ("TestNode", "mm", VSSDataType.UINT32, "", "", "", "0931A8FA"), + ("TestUnit", "degrees/s", VSSDataType.FLOAT, "", "", "", "C733138C"), + ("TestMinMax", "percent", VSSDataType.UINT8, "", 0, 100, "72A24EF1"), + ("TestEnum", "m", VSSDataType.STRING, ["YES, NO"], "", "", "DEA50110"), + ], +) +def test_generate_id( + node_name: str, + unit: str, + datatype: VSSDataType, + allowed: str, + minimum: str, + maximum: str, + result_static_uid: str, +): + node = get_test_node(node_name, unit, datatype, allowed, minimum, maximum) + result, _ = vss2id.generate_split_id(node, id_counter=0) + + assert result == result_static_uid + + +@pytest.mark.parametrize( + "test_file, validation_file", + [("test_vspecs/test.vspec", "./validation.yaml")], +) +def test_export_node( + request: pytest.FixtureRequest, + test_file: str, + validation_file: str, +): + dir_path = os.path.dirname(request.path) + vspec_file = os.path.join(dir_path, test_file) + + vspec.load_units(vspec_file, [os.path.join(dir_path, "test_vspecs/units.yaml")]) + tree = vspec.load_tree( + vspec_file, include_paths=["."], tree_type=VSSTreeType.SIGNAL_TREE + ) + yaml_dict: Dict[str, str] = {} + vss2id.export_node(yaml_dict, tree, id_counter=0) + + result_dict: Dict[str, str] + with open(os.path.join(dir_path, validation_file), "r") as f: + result_dict = yaml.load(f, Loader=yaml.FullLoader) + + assert result_dict.keys() == yaml_dict.keys() + assert result_dict == yaml_dict + + +@pytest.mark.parametrize( + "children_names", + [ + ["ChildNode", "ChildNode"], + ["ChildNode", "ChildNodeDifferentName"], + ], +) +def test_duplicate_hash(caplog: pytest.LogCaptureFixture, children_names: list): + tree = get_test_node("TestNode", "m", VSSDataType.UINT32, "", "", "") + child_node = get_test_node(children_names[0], "m", VSSDataType.UINT32, "", "", "") + child_node2 = get_test_node(children_names[1], "m", VSSDataType.UINT32, "", "", "") + tree.children = [child_node, child_node2] + + yaml_dict: Dict[str, dict] = {} + if children_names[0] == children_names[1]: + # assert system exit and log + with pytest.raises(SystemExit) as pytest_wrapped_e: + vss2id.export_node( + yaml_dict, + tree, + id_counter=0, + ) + assert pytest_wrapped_e.type == SystemExit + assert pytest_wrapped_e.value.code == -1 + assert len(caplog.records) == 1 and all( + log.levelname == "CRITICAL" for log in caplog.records + ) + else: + # assert all IDs different + vss2id.export_node( + yaml_dict, + tree, + id_counter=0, + ) + assigned_ids: list = [] + for key, value in get_all_keys_values(yaml_dict): + if not isinstance(value, dict) and key == "staticUID": + assigned_ids.append(value) + assert len(assigned_ids) == len(set(assigned_ids)) + + +# INTEGRATION TESTS + + +@pytest.mark.usefixtures("change_test_dir") +def test_full_script(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 0 + + +@pytest.mark.usefixtures("change_test_dir") +def test_semantic(caplog: pytest.LogCaptureFixture): + validation_file: str = "./validation_vspecs/validation_semantic_change.vspec" + clas = shlex.split(get_cla_validation(validation_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "SEMANTIC NAME CHANGE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_vss_path(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_vss_path.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "PATH CHANGE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_unit(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_unit.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 2 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "BREAKING CHANGE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_datatype(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_datatype.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "BREAKING CHANGE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_name_datatype(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_name_datatype.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 2 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for i, record in enumerate(caplog.records): + if i % 2 == 0: + assert "ADDED ATTRIBUTE" in record.msg + else: + assert "DELETED ATTRIBUTE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_deprecation(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_deprecation.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "DEPRECATION MSG CHANGE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_description(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_description.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "DESCRIPTION MISMATCH" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_added_attribute(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_added_attribute.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "ADDED ATTRIBUTE" in record.msg + + +@pytest.mark.usefixtures("change_test_dir") +def test_deleted_attribute(caplog: pytest.LogCaptureFixture): + test_file: str = "./test_vspecs/test_deleted_attribute.vspec" + clas = shlex.split(get_cla_test(test_file)) + vspec2x.main(["--format", "idgen"] + clas[1:]) + + assert len(caplog.records) == 1 and all( + log.levelname == "WARNING" for log in caplog.records + ) + for record in caplog.records: + assert "DELETED ATTRIBUTE" in record.msg diff --git a/tests/vspec/test_static_uids/test_vspecs/temp_test.vspec b/tests/vspec/test_static_uids/test_vspecs/temp_test.vspec new file mode 100644 index 00000000..75ac0cee --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/temp_test.vspec @@ -0,0 +1,51 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is the legacy for test deprecation. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.OldName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 0 + unit: percent + description: A leaf that uses only a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test.vspec b/tests/vspec/test_static_uids/test_vspecs/test.vspec new file mode 100644 index 00000000..688c9b3d --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_added_attribute.vspec b/tests/vspec/test_static_uids/test_vspecs/test_added_attribute.vspec new file mode 100644 index 00000000..df2093af --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_added_attribute.vspec @@ -0,0 +1,61 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. +A.B.NewNode: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A newly added node. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_datatype.vspec b/tests/vspec/test_static_uids/test_vspecs/test_datatype.vspec new file mode 100644 index 00000000..ab12cc91 --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_datatype.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int32 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_deleted_attribute.vspec b/tests/vspec/test_static_uids/test_vspecs/test_deleted_attribute.vspec new file mode 100644 index 00000000..76eaa9df --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_deleted_attribute.vspec @@ -0,0 +1,47 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_deprecation.vspec b/tests/vspec/test_static_uids/test_vspecs/test_deprecation.vspec new file mode 100644 index 00000000..ee05b1ec --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_deprecation.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: Here we want to trigger a deprecation msg change. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_description.vspec b/tests/vspec/test_static_uids/test_vspecs/test_description.vspec new file mode 100644 index 00000000..5d917791 --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_description.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype int +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_name_datatype.vspec b/tests/vspec/test_static_uids/test_vspecs/test_name_datatype.vspec new file mode 100644 index 00000000..47919f60 --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_name_datatype.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int8: + datatype: int8 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_unit.vspec b/tests/vspec/test_static_uids/test_vspecs/test_unit.vspec new file mode 100644 index 00000000..45ea1398 --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_unit.vspec @@ -0,0 +1,54 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: m + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: degrees/s + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.B.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/test_vss_path.vspec b/tests/vspec/test_static_uids/test_vspecs/test_vss_path.vspec new file mode 100644 index 00000000..b4f8f23c --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/test_vss_path.vspec @@ -0,0 +1,55 @@ +A: + type: branch + description: A is a test node +A.Float: + datatype: float + type: actuator + unit: mm + description: A.Float is a leaf of A of datatype float +A.Int16: + datatype: int16 + type: sensor + unit: rpm + description: A.Int16 is a leaf of A of datatype int16 +A.String: + datatype: string + type: sensor + description: A.String is a leaf of A of datatype string + deprecation: This is test deprecation, let's say it used to be called Str instead String. +A.StringArray: + datatype: string[] + type: sensor + description: A.StringArray is a leaf of A of datatype string array +A.B: + type: branch + description: B is a branch of A +A.Int32: + datatype: int32 + type: sensor + unit: rpm + description: A.B.Int32 is a leaf of A.B of datatype int32 + fka: ["A.B.Int32"] +A.B.NewName: + datatype: uint32 + type: sensor + unit: mm + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: ['A.B.OldName', 'A.B.OlderName'] +A.B.IsLeaf: + datatype: string + type: actuator + allowed: ["YES", "NO"] + description: This node is a leaf of the tree and it has allowed values (aka an enum). +A.B.Min: + datatype: uint8 + type: sensor + min: 10 + unit: percent + description: A leaf that uses a minimum value. +A.B.Max: + datatype: uint8 + type: sensor + unit: percent + min: 0 + max: 100 + description: A leaf that uses a maximum value. diff --git a/tests/vspec/test_static_uids/test_vspecs/units.yaml b/tests/vspec/test_static_uids/test_vspecs/units.yaml new file mode 100644 index 00000000..e725c8f0 --- /dev/null +++ b/tests/vspec/test_static_uids/test_vspecs/units.yaml @@ -0,0 +1,22 @@ +# Taken from the units.yaml of VSS catalog +units: + mm: + label: millimeter + description: Distance measured in millimeters + domain: distance + m: + label: meter + description: Distance measured in meters + domain: distance + degrees/s: + label: degree per second + description: Angular speed measured in degrees per second + domain: angular speed + rpm: + label: revolutions per minute + description: Rotational speed measured in revolutions per minute + domain: rotational speed + percent: + label: percent + description: Relation measured in percent + domain: relation diff --git a/tests/vspec/test_static_uids/validation.yaml b/tests/vspec/test_static_uids/validation.yaml new file mode 100644 index 00000000..f41d1de2 --- /dev/null +++ b/tests/vspec/test_static_uids/validation.yaml @@ -0,0 +1,68 @@ +A: + description: A is a test node + staticUID: '0x6865518C' + type: branch +A.B: + description: B is a branch of A + staticUID: '0x5EE0702C' + type: branch +A.B.Int32: + datatype: int32 + description: A.B.Int32 is a leaf of A.B of datatype int32 + staticUID: '0x4322A0BF' + type: sensor + unit: rpm +A.B.IsLeaf: + allowed: + - 'YES' + - 'NO' + datatype: string + description: This node is a leaf of the tree and it has allowed values (aka an enum). + staticUID: '0x51887178' + type: actuator +A.B.Max: + datatype: uint8 + description: A leaf that uses a maximum value. + max: 100 + staticUID: '0xACFF72AC' + type: sensor + unit: percent +A.B.Min: + datatype: uint8 + description: A leaf that uses a minimum value. + min: 10 + staticUID: '0xF66F948E' + type: sensor + unit: percent +A.B.NewName: + datatype: uint32 + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: + - A.B.OldName + - A.B.OlderName + staticUID: '0x63C22EF6' + type: sensor + unit: mm +A.Float: + datatype: float + description: A.Float is a leaf of A of datatype float + staticUID: '0xC83DDDBF' + type: actuator + unit: mm +A.Int16: + datatype: int16 + description: A.Int16 is a leaf of A of datatype int16 + staticUID: '0xC50253A3' + type: sensor + unit: rpm +A.String: + datatype: string + deprecation: This is test deprecation, let's say it used to be called Str instead String. + description: A.String is a leaf of A of datatype string + staticUID: '0x215470FC' + type: sensor +A.StringArray: + datatype: string[] + description: A.StringArray is a leaf of A of datatype string array + staticUID: '0x2115794B' + type: sensor diff --git a/tests/vspec/test_static_uids/validation_vspecs/validation.vspec b/tests/vspec/test_static_uids/validation_vspecs/validation.vspec new file mode 100644 index 00000000..f41d1de2 --- /dev/null +++ b/tests/vspec/test_static_uids/validation_vspecs/validation.vspec @@ -0,0 +1,68 @@ +A: + description: A is a test node + staticUID: '0x6865518C' + type: branch +A.B: + description: B is a branch of A + staticUID: '0x5EE0702C' + type: branch +A.B.Int32: + datatype: int32 + description: A.B.Int32 is a leaf of A.B of datatype int32 + staticUID: '0x4322A0BF' + type: sensor + unit: rpm +A.B.IsLeaf: + allowed: + - 'YES' + - 'NO' + datatype: string + description: This node is a leaf of the tree and it has allowed values (aka an enum). + staticUID: '0x51887178' + type: actuator +A.B.Max: + datatype: uint8 + description: A leaf that uses a maximum value. + max: 100 + staticUID: '0xACFF72AC' + type: sensor + unit: percent +A.B.Min: + datatype: uint8 + description: A leaf that uses a minimum value. + min: 10 + staticUID: '0xF66F948E' + type: sensor + unit: percent +A.B.NewName: + datatype: uint32 + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: + - A.B.OldName + - A.B.OlderName + staticUID: '0x63C22EF6' + type: sensor + unit: mm +A.Float: + datatype: float + description: A.Float is a leaf of A of datatype float + staticUID: '0xC83DDDBF' + type: actuator + unit: mm +A.Int16: + datatype: int16 + description: A.Int16 is a leaf of A of datatype int16 + staticUID: '0xC50253A3' + type: sensor + unit: rpm +A.String: + datatype: string + deprecation: This is test deprecation, let's say it used to be called Str instead String. + description: A.String is a leaf of A of datatype string + staticUID: '0x215470FC' + type: sensor +A.StringArray: + datatype: string[] + description: A.StringArray is a leaf of A of datatype string array + staticUID: '0x2115794B' + type: sensor diff --git a/tests/vspec/test_static_uids/validation_vspecs/validation_semantic_change.vspec b/tests/vspec/test_static_uids/validation_vspecs/validation_semantic_change.vspec new file mode 100644 index 00000000..fc72af8f --- /dev/null +++ b/tests/vspec/test_static_uids/validation_vspecs/validation_semantic_change.vspec @@ -0,0 +1,66 @@ +A: + description: A is a test node + staticUID: '0x6865518C' + type: branch +A.B: + description: B is a branch of A + staticUID: '0x5EE0702C' + type: branch +A.B.Int32: + datatype: int32 + description: A.B.Int32 is a leaf of A.B of datatype int32 + staticUID: '0x4322A0BF' + type: sensor + unit: rpm +A.B.IsLeaf: + allowed: + - 'YES' + - 'NO' + datatype: string + description: This node is a leaf of the tree and it has allowed values (aka an enum). + staticUID: '0x51887178' + type: actuator +A.B.Max: + datatype: uint8 + description: A leaf that uses a maximum value. + max: 100 + staticUID: '0xACFF72AC' + type: sensor + unit: percent +A.B.Min: + datatype: uint8 + description: A leaf that uses a minimum value. + min: 10 + staticUID: '0xF66F948E' + type: sensor + unit: percent +A.B.OldName: + datatype: uint32 + description: A.B.NewName's old name is 'OldName'. And its even older name is 'OlderName'. + fka: + - A.B.OlderName + staticUID: '0xD4DFF5FD' + type: sensor + unit: mm +A.Float: + datatype: float + description: A.Float is a leaf of A of datatype float + staticUID: '0xC83DDDBF' + type: actuator + unit: mm +A.Int16: + datatype: int16 + description: A.Int16 is a leaf of A of datatype int16 + staticUID: '0xC50253A3' + type: sensor + unit: rpm +A.String: + datatype: string + description: A.String is a leaf of A of datatype string + staticUID: '0x215470FC' + type: sensor +A.StringArray: + datatype: string[] + description: A.StringArray is a leaf of A of datatype string array + staticUID: '0x2115794B' + type: sensor diff --git a/vspec/model/vsstree.py b/vspec/model/vsstree.py index 3a62f8e4..a339e9ad 100644 --- a/vspec/model/vsstree.py +++ b/vspec/model/vsstree.py @@ -40,13 +40,13 @@ class VSSNode(Node): core_attributes = ["type", "children", "datatype", "description", "unit", "uuid", "min", "max", "allowed", "instantiate", "aggregate", "default", "instances", "deprecation", "arraysize", - "comment", "$file_name$"] + "comment", "$file_name$", "fka"] # List of accepted extended attributes. In strict terminate if an attribute is # neither in core or extended, whitelisted_extended_attributes: List[str] = [] - unit: Optional[VSSUnit] + unit: Optional[VSSUnit] = None min = "" max = "" @@ -60,6 +60,7 @@ class VSSNode(Node): instances = None expanded = False deprecation = "" + fka = "" def __deepcopy__(self, memo): # Deep copy of source_dict and children needed as overlay or programmatic changes diff --git a/vspec/utils/__init__.py b/vspec/utils/__init__.py index 5d20e1e5..eca75d9d 100644 --- a/vspec/utils/__init__.py +++ b/vspec/utils/__init__.py @@ -1,3 +1,4 @@ + # Copyright (c) 2023 Contributors to COVESA # # This program and the accompanying materials are made available under the @@ -5,3 +6,19 @@ # https://www.mozilla.org/en-US/MPL/2.0/ # # SPDX-License-Identifier: MPL-2.0 + +import argparse + + +def remove_options_argparse(parser: argparse.ArgumentParser, options) -> None: + # ToDo: this allows removing arguments from argparse but involves accessing protected members + # also when removing '--format' it removes the positional argument vspec_file + for option in options: + for action in parser._actions: + if ( + vars(action)["option_strings"] + and vars(action)["option_strings"][0] == option + ): + parser._handle_conflict_resolve(action, [(option, action)]) + break + diff --git a/vspec/utils/idgen_utils.py b/vspec/utils/idgen_utils.py new file mode 100644 index 00000000..f4e2939d --- /dev/null +++ b/vspec/utils/idgen_utils.py @@ -0,0 +1,74 @@ +def get_node_identifier_bytes( + qualified_name: str, + data_type: str, + node_type: str, + unit: str, + allowed: str, + minimum: str, + maximum: str, +) -> bytes: + """Get a node identifier as bytes. Used as an input for hashing + + @param qualified_name: full path to the node + @param data_type: its datatype + @param node_type: its node type + @param unit: the unit if it uses one + @param allowed: the enum for allowed values + @param minimum: min value for the data if exists + @param maximum: max value for the data if exists + @return: a bytes representation of the node + """ + + return ( + f"{qualified_name}: " + f"unit: {unit}, " + f"datatype: {data_type}, " + f"type: {node_type}" + f"allowed: {allowed}" + f"min: {minimum}" + f"max: {maximum}" + ).encode("utf-8") + + +def fnv1_32_hash(identifier: bytes) -> int: + """32-bit hash of a node according to Fowler–Noll–Vo + + @param identifier: a bytes representation of a node + @return: hashed value for the node as int + """ + id_hash = 2166136261 + for byte in identifier: + id_hash = (id_hash * 16777619) & 0xFFFFFFFF + id_hash ^= byte + + return id_hash + + +def fnv1_32_wrapper(name: str, source: dict): + """A wrapper for the 32-bit hashing function if the input node + is represented as dict instead of VSSNode + + @param name: full name aka qualified name + @param source: + @return: + """ + allowed: str = source["allowed"] if "allowed" in source.keys() else "" + minimum: str = source["min"] if "min" in source.keys() else "" + maximum: str = source["max"] if "max" in source.keys() else "" + identifier = get_node_identifier_bytes( + name, + source["datatype"], + source["type"], + source["unit"], + allowed, + minimum, + maximum, + ) + return format(fnv1_32_hash(identifier), "08X") + + +def get_all_keys_values(d: dict): + for key, value in d.items(): + yield key, value + if isinstance(value, dict): + yield from get_all_keys_values(value) diff --git a/vspec/utils/vss2id_val.py b/vspec/utils/vss2id_val.py new file mode 100644 index 00000000..302c2c43 --- /dev/null +++ b/vspec/utils/vss2id_val.py @@ -0,0 +1,154 @@ +from anytree import PreOrderIter # type: ignore +import argparse +import logging +import sys +from typing import Optional +from vspec.model.vsstree import VSSNode +from vspec.utils.idgen_utils import fnv1_32_wrapper + + +def validate_static_uids( + signals_dict: dict, validation_tree: VSSNode, config: argparse.Namespace +): + """Check if static UIDs have changed or if new ones need to be added + + @param signals_dict: to be exported dict of all signals containing static UID + @param validation_tree: tree loaded from validation file + @param config: the command line arguments the script was run with + @return: None + """ + + def check_description(k: str, v: dict, match_tuple: tuple): + validation_node: VSSNode = validation_tree_nodes[match_tuple[1]] + + try: + assert v["description"] == validation_node.description + + except AssertionError: + logging.warning( + "[Validation] " + f"DESCRIPTION MISMATCH: The description of {k} has changed from " + f"\n\t Validation: '{validation_node.description}' to \n\t Current " + f"vspec: '{v['description']}'" + ) + + def check_semantics(k: str, v: dict) -> Optional[int]: + """Checks if the change was a semantic or path change. This can be triggered by + manually adding a fka (formerly known as) attribute to the vspec. The result + is that the old hash can be matched such that a node keeps the same UID. + + @param k: the current key + @param v: the current value (dict) + @return: boolean if it was a semantic or path change + """ + if "fka" in v.keys(): + semantic_match: Optional[int] = None + for fka_val in v["fka"]: + old_static_uid = "0x" + fnv1_32_wrapper(fka_val, v) + for i, validation_node in enumerate(validation_tree_nodes): + if ( + old_static_uid + == validation_node.extended_attributes["staticUID"] + ): + logging.warning( + f"[Validation] SEMANTIC NAME CHANGE or PATH CHANGE for '{k}', " + f"it used to be '{validation_node.qualified_name()}'." + ) + semantic_match = i + return semantic_match + else: + return None + + def check_deprecation(k: str, v: dict, match_tuple: tuple): + if ( + "deprecation" in v.keys() + and validation_tree_nodes[match_tuple[1]].deprecation + ): + if v["deprecation"] != validation_tree_nodes[match_tuple[1]].deprecation: + logging.warning( + f"[Validation] DEPRECATION MSG CHANGE: Deprecation message " + f"for '{k}' was " + f"'{validation_tree_nodes[match_tuple[1]].deprecation}' " + f"in validation but now is '{v['deprecation']}'." + ) + + def hashed_pipeline(): + """This pipeline uses FNV-1 hash for static UIDs. + + If no UID was matched we check if the user has written an old path or old name, + so we can tell if it was a semantic or path change. If semantic or path change + we trigger a corresponding warning. + If not semantic or path change we try to match the key: + If key was matched we know that it was a unit, datatype, type, enum, min/max change + which triggers `BREAKING CHANGE` warning. If key cannot be matched the node must have + been added since the last validation, so it triggers an `ADDED ATTRIBUTE` warning. + If there was a UID match we will continue to check for non-breaking changes and throw + corresponding logs. + In the end the remaining nodes correspond to deleted nodes, so we throw a + `DELETED ATTRIBUTE` warning. + """ + nonlocal validation_tree_nodes + + for key, value in signals_dict.items(): + matched_uids = [ + (key, id_validation_tree) + for id_validation_tree, other_node in enumerate(validation_tree_nodes) + if value["staticUID"] == other_node.extended_attributes["staticUID"] + ] + + # if not matched via UID check semantics or path change + if len(matched_uids) == 0: + semantic_match = check_semantics(key, value) + if semantic_match is None: + key_found: bool = False + for i, node in enumerate(validation_tree_nodes): + if key == node.qualified_name(): + key_found = True + validation_tree_nodes.pop(i) + break + + if key_found: + logging.warning( + f"[Validation] BREAKING CHANGE: " + f"There was a breaking change for '{key}' which " + f"means its name, unit, datatype, type, enum or " + f"min/max has changed." + ) + else: + logging.warning( + f"[Validation] ADDED ATTRIBUTE: " + f"The node '{key}' was added since the last validation." + ) + else: + validation_tree_nodes.pop(semantic_match) + + elif len(matched_uids) == 1: + check_deprecation(key, value, matched_uids[0]) + check_description(key, value, matched_uids[0]) + + validation_tree_nodes.pop(matched_uids[0][1]) + + else: + logging.error( + "[Validation] Multiple matches do not make sense " + "for the way we load the data. Please check your " + "input vspec!" + ) + sys.exit(-1) + + for node in validation_tree_nodes: + logging.warning( + "[Validation] DELETED ATTRIBUTE: " + f"'{node.qualified_name()}' was not matched so it must have " + f"been deleted." + ) + + if validation_tree.parent: + while validation_tree.parent: + validation_tree = validation_tree.parent + + validation_tree_nodes: list = [] + for val_node in PreOrderIter(validation_tree): + validation_tree_nodes.append(val_node) + + hashed_pipeline() diff --git a/vspec/vssexporters/vss2id.py b/vspec/vssexporters/vss2id.py new file mode 100644 index 00000000..93f87d57 --- /dev/null +++ b/vspec/vssexporters/vss2id.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2023 Contributors to COVESA +# +# This program and the accompanying materials are made available under the +# terms of the Mozilla Public License 2.0 which is available at +# https://www.mozilla.org/en-US/MPL/2.0/ +# +# SPDX-License-Identifier: MPL-2.0 +# +# Generate IDs of 4bytes size, 3 bytes incremental value + 1 byte for layer id. + +import argparse +import logging +import os +import sys +from typing import Dict, Tuple +from vspec import load_tree +from vspec.model.constants import VSSTreeType +from vspec.loggingconfig import initLogging +from vspec.model.vsstree import VSSNode +from vspec.utils import vss2id_val +from vspec.utils.idgen_utils import ( + get_node_identifier_bytes, + fnv1_32_hash, + get_all_keys_values, +) +import yaml + + +def add_arguments(parser: argparse.ArgumentParser) -> None: + """Adds command line arguments to a pre-existing argument parser + + @param parser: the pre-existing argument parser + """ + parser.add_argument( + "--validate-static-uid", type=str, default="", help="Path to validation file." + ) + parser.add_argument( + "--only-validate-no-export", + action="store_true", + default=False, + help="For pytests and pipelines you can skip the export of the vspec file.", + ) + + +def generate_split_id(node: VSSNode, id_counter: int) -> Tuple[str, int]: + """Generates static UIDs using 4-byte FNV-1 hash. + + @param node: VSSNode that we want to generate a static UID for + @param id_counter: consecutive numbers counter for amount of nodes + @return: tuple of hashed string and id counter + """ + + identifier = get_node_identifier_bytes( + node.qualified_name(), + node.data_type_str, + node.type.value, + node.get_unit(), + node.allowed, + node.min, + node.max, + ) + hashed_str = format(fnv1_32_hash(identifier), "08X") + + return hashed_str, id_counter + 1 + + +def export_node(yaml_dict, node, id_counter) -> Tuple[int, int]: + """Recursive function to export the full tree to a dict + + @param yaml_dict: the to be exported dict + @param node: parent node of the tree + @param id_counter: counter for amount of ids + @return: id_counter, id_counter + """ + node_id, id_counter = generate_split_id(node, id_counter) + + # check for hash duplicates + for key, value in get_all_keys_values(yaml_dict): + if not isinstance(value, dict) and key == "staticUID": + if node_id == value[2:]: + logging.fatal( + f"There is a small chance that the result of FNV-1 " + f"hashes are the same in this case the hash of node " + f"'{node.qualified_name()}' is the same as another hash." + f"Can you please update it." + ) + # We could add handling of duplicates here + sys.exit(-1) + + node_path = node.qualified_name() + + yaml_dict[node_path] = {"staticUID": f"0x{node_id}"} + yaml_dict[node_path]["description"] = node.description + yaml_dict[node_path]["type"] = str(node.type.value) + if node.unit: + yaml_dict[node_path]["unit"] = str(node.unit.value) + if node.is_signal() or node.is_property(): + yaml_dict[node_path]["datatype"] = node.data_type_str + if node.allowed: + yaml_dict[node_path]["allowed"] = node.allowed + if node.min: + yaml_dict[node_path]["min"] = node.min + if node.max: + yaml_dict[node_path]["max"] = node.max + + if node.fka: + yaml_dict[node_path]["fka"] = node.fka + elif "fka" in node.extended_attributes.keys(): + yaml_dict[node_path]["fka"] = node.extended_attributes["fka"] + + if node.deprecation: + yaml_dict[node_path]["deprecation"] = node.deprecation + + for child in node.children: + id_counter, id_counter = export_node( + yaml_dict, + child, + id_counter, + ) + + return id_counter, id_counter + + +def export(config: argparse.Namespace, signal_root: VSSNode, print_uuid): + """Main export function used to generate the output id vspec. + + @param config: Command line arguments it was run with + @param signal_root: root of the signal tree + @param print_uuid: Not used here but needed by main script + """ + logging.info("Generating YAML output...") + + id_counter: int = 0 + signals_yaml_dict: Dict[str, str] = {} # Use str for ID values + id_counter, _ = export_node(signals_yaml_dict, signal_root, id_counter) + + if config.validate_static_uid: + logging.info( + f"Now validating nodes, static UIDs, types, units and description with " + f"file '{config.validate_static_uid}'" + ) + if os.path.isabs(config.validate_static_uid): + other_path = config.validate_static_uid + else: + other_path = os.path.join(os.getcwd(), config.validate_static_uid) + + validation_tree = load_tree( + other_path, ["."], tree_type=VSSTreeType.SIGNAL_TREE + ) + vss2id_val.validate_static_uids(signals_yaml_dict, validation_tree, config) + + if not config.only_validate_no_export: + with open(config.output_file, "w") as f: + yaml.dump(signals_yaml_dict, f) + + +if __name__ == "__main__": + initLogging() diff --git a/vspec2id.py b/vspec2id.py new file mode 100755 index 00000000..399468fd --- /dev/null +++ b/vspec2id.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2023 Contributors to COVESA +# +# This program and the accompanying materials are made available under the +# terms of the Mozilla Public License 2.0 which is available at +# https://www.mozilla.org/en-US/MPL/2.0/ +# +# SPDX-License-Identifier: MPL-2.0 +# +# Initialize static IDs. +# + +import sys +import vspec2x + +if __name__ == "__main__": + vspec2x.main(["--format", "idgen"] + sys.argv[1:]) diff --git a/vspec2x.py b/vspec2x.py index 07e19a85..76d2a4c1 100755 --- a/vspec2x.py +++ b/vspec2x.py @@ -23,7 +23,7 @@ from vspec.vssexporters import vss2json, vss2csv, vss2yaml, \ - vss2binary, vss2franca, vss2ddsidl, vss2graphql, vss2protobuf, vss2jsonschema + vss2binary, vss2franca, vss2ddsidl, vss2graphql, vss2protobuf, vss2jsonschema, vss2id SUPPORTED_STRUCT_EXPORT_FORMATS = set(["json", "yaml", "csv", "protobuf", "jsonschema", "idl"]) @@ -44,7 +44,9 @@ def export(config: argparse.Namespace, root: VSSNode): idl = vss2ddsidl graphql = vss2graphql protobuf = vss2protobuf + jsonschema = vss2jsonschema + idgen = vss2id def __str__(self): return self.name @@ -57,10 +59,9 @@ def from_string(s): raise ValueError() -parser = argparse.ArgumentParser(description="Convert vspec to other formats.") - - def main(arguments): + parser = argparse.ArgumentParser(description="Convert vspec to other formats.") + initLogging() parser.add_argument('-I', '--include-dir', action='append', metavar='dir', type=str, default=[],